Jared Spool is Not a Fan of NPS

Metrics, metrics, metrics. I've been working in the call center space for a number of years. If there's one thing that call centers do a lot of, it's measure stuff. Agents get measured on talk time. Call centers get measured on queue times. And then of course we move into the more generalized realm of customer experience. In this area there are three metrics that are by far and away the most common ones: CSAT (customer satisfaction), NPS (net promoter score) and CEI (customer effort index). Of the three, the first two are far and way the most common, and NPS is one that you will see all over the industry.

To catch you up to speed: NPS is based on one question: "How likely are you to recommend XYZ to a friend or colleague?" The participant is asked to respond using an 11 point scale. The results are then segmented into Detractors (scores 0 to 6) , Passive respondents (7-8), and Supporters (9-10). It's a super-common metric and is used all over a wide variety of organizations, built into CX platforms, and so on.

And Jared Spool is not a fan. Here's an article from him, "Net Promoter Score Considered Harmful (and What UX Professionals Can Do About It)".

So. When I read the headline of that article, I was a bit shocked. For those of you who don't know Jared Spool, he's pretty much the OG of user experience. Or one of them, anyway. He's been around for years, and his company, UIE (User Interface Engineering) is a very significant player in that space. And of course, NPS has also been around for years and is a pretty significant player in the marketing research place. So this article is very much along the lines of Godzilla vs. Mothra. Or something similar.

In a nutshell, here are his points:

• The way that NPS is calculated, your participants could all answer 0s to your question, or, they could all answer 6s to your question, but after you run your results through the NPS calculator, both of these look the same. You don't get any credit on your incremental improvements.

My thoughts: I haven't done enough reading on the "guts" of how NPS is calculated to be able to speak intelligently on that. However, he is right about one thing: an organization is built on incremental improvements, especially as you grow larger, and your customer experiences become more fine-tuned. Failing to give yourselves credit for moving the needle even a little bit is failing to recognize this basic reality of what it's like to work in the UX field: your wins, most often, come one drop at a time.

Next point:

• An 11 point scale is too noisy and difficult for respondents to use.

My thoughts: It's quite true that different participants will have different ideas about what constitutes a 7 experience vs. a 5 experience. However, I do feel like if you get a large enough dataset, this kind of thing does tend to shake out over the big picture. The problem arises, once again, in the scoring:

In NPS, a dataset full only of sixes scores -100 and a dataset full only of sevens scores 0.

So that's a very big difference for something which is not going to be immediately apparent to participants. Point goes to Spool on this one.

Final point:

• The NPS question itself is flawed. The question, remember, is: "How likely are you to recommend XYZ to a friend or colleague?" Spool points out that:

The best research questions are about past behavior, not future behavior. Asking a study participant Will you try to live a healthy lifestyle? or Are you going to give up sugar? or Will you purchase this product? requires they predict their future behavior. We are more interested in what they’ve done than what they’ll do. We’re interested in actual behavior, not a prediction of behavior.

If there's one thing that Big Data has taught us, it's this: Human beings really suck at predicting what they are going to do. Our predictions about our future actions are aspirational, not based in reality. Look at how Netflix moved away from making recommendations based on your queue, to making recommendations based on your actual viewing patterns:

A lot of people tell us they often watch foreign movies or documentaries. But in practice, that doesn’t happen very much.

That means that the core question itself is not a reliable measurement. Spool wins this one by a landslide. Game, set, match.

There's a lot more at the link, so for anyone who is interested in customer experience and metrics, I'd recommend reading it. Here's Spool's ultimate takeaway:

We can't reduce user experience to a single number.

That's really the bottom line here: Metrics are great, but they never tell the whole story.