How The Bangor Daily News uses Net Promoter Score to learn more about its subscribers

How likely is it that would you recommend your local newspaper to a friend or colleague?

This question is at the heart of Net Promoter Score surveys, a methodology that aims to determine the loyalty of a company or organization’s customers.

NPS has been frequently used in the corporate world for years, but as news organizations have become more reliant on direct reader revenue through memberships and subscriptions, some outlets have begun using NPS to survey their readers.

One of these outlets is the Bangor Daily News, a local paper in Maine. And this week in Solution Set we’re going to dig into the Daily News’ approach to NPS and how it combines quantitative and qualitative data from the surveys.

Solution Set is a weekly report from The Lenfest Institute for Journalism. Every Thursday, we take an in-depth look at one rad thing in journalism, share lessons, and point you toward other useful resources.

We’re also partnering with GroundSource so you can now get Solution Set delivered each week via text message. You can sign up by clicking here or by texting SOLUTION to (215) 544–3524.

Before we dive in, some disclosures. The Lenfest Institute has supported the Daily News through a couple of programs. The paper has received funding through the Community Listening and Engagement Fund and it’s also one of the first publications participating in Newspack, a project the Institute is co-funding to build a WordPress-based CMS for small to medium-sized publishers.

Anyway, here’s the TLDR:

TLDR

• The Challenge: After moving to a digital subscription model, The Bangor Daily News wanted to learn more about its audience.

• The Strategy: Last year, it began Net Promoter Score surveys, which are based around asking whether readers would recommend the product to a friend or colleague on a 0-10 scale.

• The Numbers: The Daily News wouldn’t share specific figures, but it said the qualitative data it receives from the survey is more valuable at this point.

• The Lessons: It’s not enough to just conduct surveys, news organizations must then take that information and then make changes based on the results.

• The Future: The Daily News plans to continue conducting NPS surveys on a quarterly basis moving forward.

• Want to know More?: Scroll down for more on how you can start using NPS in your organization.

The Challenge

Like every newspaper, Maine’s Bangor Daily News has had to adapt to a changing business model.

In fall 2017, the paper introduced a digital pay meter on its website. The Daily News spent months researching how to implement the meter, and working out how to best communicate the change to readers.

“We spent about half a year studying what other publishers have done, what mistakes they have made, what they have learned and planning how we would handle digital subscription here,” Bangor Daily News COO Todd Benoit wrote in an introductory article. “One of our biggest lessons has been to communicate often, clearly, and openly and to focus on customer service.”

The Daily News has tried to spread that ethos throughout the organization by putting readers and subscribers at the center of its operations.

But as it has built out its paying digital audience, the Daily News has wanted to better understand what its subscribers think and how it can better serve them.

The Strategy

To get inside subscribers’ heads, the Daily News began using Net Promoter Score surveys to try and gauge what its subscribers think about coverage and their experience with the paper.

The Daily News had previously used NPS, but it began using the survey methodology again in earnest early last year. It sent out email surveys to its subscribers, and it segmented the audience based on print-only subscribers, digital-only subscribers, print plus digital subscribers, and readers who have registered with an email address.

Here’s how Net Promoter Score surveys work, and what the Daily News asked its readers as part of the survey:

In an NPS survey, readers are asked to respond to the question: How likely is it that would you recommend The Bangor Daily News (Ed note: or any other product) to a friend or colleague? Respondents then answer on a 0-10 scale.

Respondents are then grouped into three categories based on their response:

Promoters: Respondents who choose 9 or 10. Promoters are your biggest and most loyal fans.
Passives: Respondents who choose 7 or 8. Passives are typically satisfied, but unenthusiastic customers. Typically, they’re most likely to churn to competitors.
Detractors: Respondents who choose 6 or below. Detractors are unhappy customers.

Once you’ve completed the survey, you calculate the Net Promoter Score by subtracting the percentage of Detractors from the percentage of Promoters. If every single one of your respondents is a Detractor, you can get the lowest NPS score of -100, and conversely, if everyone is a Promoter, you’ll have a score of 100.

Typically, if you’re running an NPS, survey, you then ask a follow-up question to ask respondents why they chose their answer. The Daily News asked different follow-ups based on these responses:

8 or below got asked: What changes would The Bangor Daily News have to make for you to give it a higher rating?
9 and 10 got asked: What does the Bangor Daily News do well?

When it gets the responses, it hand codes them into buckets so they can ascertain what issues subscribers care about,”

“We only just implemented digital subscriptions since shortly before our first NPS,” Daily News Director of Product Marketing Conrad Lumm told me. “It’s been interesting to track people’s perceptions of the products that they’re paying for. It’s really instructive and useful for us. Those qualitative responses, to me as a marketer, are the most interesting piece of the information, although everything else is important too.”

“And they’re the most actionable,” Director of Audience Development Joellen Easton added. “It gives us an early signal.”

For example, the Daily News recently started printing its paper at a new facility, affected delivery times in certain parts of the state. As a result, it had to move up its deadlines, which meant some sports scores no longer make it into the paper.

The Daily News heard about it in its most recent NPS survey.

“In this most recent round, we started hearing some complaints about the slowness of our sports score reporting,” Easton said.

“It was a slight uptick, but it was noticeable,” Lumm added. “If you’re looking at comments that you get on Facebook or elsewhere, it can be kind of scattershot, and you don’t know how seriously to take everything, and as an organization, like all newspapers, we have to prioritize pretty carefully. We aren’t swimming in resources. We have to pick our battles. The qualitative data we get back from our NPS questions are incredibly helpful in helping us to prioritize what we do next to reduce our churn and to make sure people still like us.”

As a result, it’s planning to create a sports-focused morning email newsletter, that it hopes will satiate the subscribers that complained about the change.

Similarly, another issue that stood out was complaints about bias. Most of the comments, Easton said, criticized the paper for having a liberal bias, but it also received some comments about perceived conservative bias.

“There was a consistent high-level complaint about bias,” Easton said. “In the past, we had seen that, but only among our print readers. We hadn’t noticed it clustered amongst the detractors, it had been more broadly felt. That was a real warning. That’s due to a number of external factors as well as things, I’m sure, about how we conduct ourselves and what the audience expects of us.”

“It was also good timing because there’s a lot more discussion about bias and trust in the industry now,” she continued. “But it has given a lot of momentum for a project that is pretty wide-reaching across the organization to try and affect that perception both through changing how we interact with our readers in the comments, the kinds of events we hold, how we signal the difference between editorial and news content, and how we talk about our story choices. We’re planning on trying a bunch of things this year.”

(I’ll talk more about the Daily News’ response to this in The Lessons, which will highlight the cross-departmental nature of the work.)

Every time it’s conducted an NPS survey, the Daily News has added a third question to the survey as well.

“The third one is where we customize,” Lumm said. “We think about where we’re at today, what information do we really need that’s a little bit of a one-off. We’re less likely to have a representative sample by the third question, and we’re aware of that. We just want to be nudged in a direction.”

In one survey, the paper asked respondents to choose what Maine news and information providers they find essential to their lives. It found significant differences between audiences. While all segments said the Daily News was the most essential to their lives, print-only subscribers said local TV news was the next most essential. Digital plus print and digital-only subscribers, however, said public radio was the next most essential.

Additionally, the gap between the Daily News and the next most essential news organization was widest for the digital plus print subscribers, Easton said, adding that they’d like to do more research to better understand it.

“I would theorize that the bundle folks see us as much essential because they’re being served more thoroughly on more platforms by us, and our newsroom is a digital-first newsroom so they’re getting the fast [pace] on digital and the warmth…of print. There are a lot of questions we’d like to ask, to understand this.”

It’s also considering adding a membership-like approach to its subscription offerings, so last year it also asked subscribers what they found valuable about other membership groups they’re involved in.

“We were starting to think with digital subscriptions, whether moving toward a membership model was something that’s viable or would be interesting to our audience,” Lumm said. “We were also thinking about our events: Why do people want to convene in a place? Is it to hang out with like-minded folks? Is it niche interest? There were a lot of different reasons we put down.”

The Numbers

The Daily News would not share any specific figures with me. In particular, I wanted to know the paper’s NPS scores across its segments.

(Easton and Lumm said the paper’s leadership would not allow them to share numbers as soon as I reached out, but I still thought it was worthwhile to talk with them and learn from their broad experience using NPS. Let me know what you think though!)

The average benchmark NPS score for news organizations is -7, Lumm said. “We do better than that. I can’t go into details in terms of numbers, but we are certainly doing better than -7,” he said.

(In a follow-up email, I asked the source for that industry average, and Lumm replied: “We’re not really comfortable discussing numbers. I’m sorry that I can’t help you with a source. This is closely held information for everyone who gathers it.”)

Easton and Lumm repeatedly said though that, at least for now, they find the qualitative responses they get from the surveys more useful than the quantitative Net Promoter Score even though they have seen some minor change in the scores.

“We have seen some change, but when we look at the sample size, that change is for the most part not statistically significant, and if it is, it’s only barely,” Easton said. “I wouldn’t want to say: Oh we changed something, and then anybody who understands statistics would look at it and say, you haven’t changed anything.”

The paper began this current round of NPS surveys last year. So far, it has conducted three surveys.

The Daily News uses Survey Monkey to conduct its NPS surveys. You need to create a paid account to utilize branching questions that change depending on respondents’ answers. Lumm said it can cost as little as $300 to use Survey Monkey.

There are other free tools, but Easton recommended a paid tool that saves time and makes it easier to code and sort the responses.

The Lessons

• NPS has its limitations: Many people have critiqued NPS, and the Bangor Daily News recognizes that there are limitations to the number itself.

In the Daily News’ case, it’s only run three surveys, so it doesn’t want to extrapolate too much from its Net Promoter Score. The paper is planning to continue sending NPS surveys quarterly, so it will be able to better track trends over time.

“Doing it once is of limited value,” Lumm said. “You don’t go in and play one hand of blackjack. If you’re a good blackjack player you’re going to want to play more than one hand. We want to get our reader feedback many times over a long period of time so we can really chart what the impact of any changes we make are.”

That’s also why the Daily News pairs the quantitative question with qualitative feedback. It’s not enough to know just that subscribers are satisfied or unsatisfied. They need qualitative follow-ups to better understand why respondents actually feel the way they do.

The Daily News also said that while it can be a challenge to ensure the respondents are representative of its readership, it’s hopeful that its subscribers are responding in good faith.

“We don’t have a good way of making sure that the sample is representative,” Lumm said. “We send this to everybody. Our numbers are high enough, that we don’t think we see a lot of false positives. That’s just because our readership…the people who are putting cash on the barrelhead are more engaged than other folks and more likely to give us this feedback.”

Easton interjected: “And they’re the folks we’re particularly interested in hearing from. There is some self-selection, but any time you ask people to voluntarily share information about themselves there is some self-selection, but that’s one of the reasons we’re doing it regularly and over time so we can observe change. Let’s say you heard from 100 people that something was an issue, and we thought that rises to the level of our interest, so we run an experiment. The next time we do the NPS we ask the same question, and we’re segmenting, bucketing and tagging the same way, so we can observe the effect of what we’re trying to do to move the needle.”

• Actionable insights across the org: Any type of survey or reader research is useless unless you actually use the feedback and information to improve your work. And the paper has worked across departments to try and iterate based on the findings.

For example, the Daily News got feedback that some subscribers thought that the paper was biased and that reporters often put their opinion into stories.

“We need to explain our story choices, we need to explain our process,” Easton said. “There are so many things to try and affect that and change that. There is also training for how does something that seems totally normal to you as a reporter based on conversations you had in the newsroom, actually seem different to a reader who might come from a different perspective.”

To change this perception, the Daily News has undertaken two initial projects in conjunction with both newsroom and editorial page leadership. The first is to rethink its approach to comments.

Some commenters are miffed when they’re contributions are removed or edited, so the Daily News plans to more actively engage in the comments and explain the policies more clearly.

“We’ve been having a very public process in changing our moderation policy, changing our moderation consistency, engaging more often in the comments, and not allowing people to beat up on each other,” Easton said. They are constantly reiterating to readers that their comments are being more tightly moderated because “it’s how you’re saying it not what you’re saying,” she said.

The Daily News is also going to migrate to Talk, the Coral Project’s commenting platform. (This was as a result of a Lenfest Institute-backed CLEF grant.)

The newsroom leadership is also thinking more seriously about how it can better differentiate between news stories and opinion stories, so it’s clear to readers what section of the paper they’re reading.

“We’re looking at what other news organizations are doing but also thinking about, given our resources, what we can pull off. We’re basically trying to make a change without requiring development resources that can be consistent and also doesn’t appear in print, because this delineation is more clear already in print than it is in digital.”

She continued: “A piece of this is that whatever the delineation is, is that it carries to the object level, because folks form opinions before they ever read anything. The headline just flitted past them on Facebook and they have an opinion on that. To have that information that it’s an opinion piece in the headline itself, that matters. But then we have to have a conversation about whether that will have a chilling effect on pageviews, and then do we care? What’s the most important value here?”

The Daily News is thinking about how it can spread the data and lessons more broadly throughout the organization.

It’s going to work with Trusting News to further develop, measure, and learn from these reader-focused editorial strategies. It’s also in the process of rethinking how it handles and moderates comments.

But the Daily News has been able to begin to implement these plans because it knew from the NPS surveys that this was an issue that its subscribers cared about. Traditionally, newsrooms may have been hesitant to use data that emerged from the marketing department to help inform their editorial strategies, but by taking a cross-functional approach to the work, the Daily News is better able to serve its most loyal readers.

• Be intentional with your questions: The Daily News purposely keeps its surveys short, so it doesn’t overwhelm respondents. It wants to make it as easy as possible for them to participate, while still ensuring that it gets good data.

Every NPS survey it has sent is three questions long, and it thinks about the first two questions — the NPS question, and the question asking why they chose that number on the scale — as ways to “prime the pump,” Easton said.

“Sometimes the first question you ask isn’t actually the question you’re looking for the answer to. It’s a gateway drug. It limbers you up to answer questions. Sometimes you would ask a seven-question survey of folks, when you’re really looking for is the answer to that last question: What’s on your mind? What do you want to talk to us about? But you ask a bunch of other questions earlier in the survey to prime the pump and get them accustomed to thinking about something,” she said.

“Maybe you might ask them to think about something that changed and how they reacted,” she continued. “That’s more interesting than what their initial status was and what their later status was. What you’re trying to get at is how they think about change. With NPS, the 0-10 scale, that’s a super easy thing. They hit a button. And then we’re asking them, ‘Why did you say that?’ If we just asked them what are we doing well and what we’re doing wrong, without asking that initial question we may not get as thoughtful responses or we may get more of a knee jerk reaction instead of a more considered one.”

The Future

The Bangor Daily News plans to send out its next NPS survey later this month. In the added third question, it plans to ask about what other publications they subscribe to.

“We want to understand where there is overlap in terms of who is stacking up subscriptions between us and other entities,” Easton said. “Do we see ourselves more in the mix with The New York Times and The Wall Street Journal? Or with our other competitor in the state? These are things we want to find out.”

And moving forward, the Daily News plans to conduct NPS surveys on an ongoing quarterly basis over the next few years.

Its hope is that the regular surveys will enable it to better track and measure the change over time.

“Our intention is for us to continue doing this on a quarterly basis, and over the course of a couple of years start to see actual change at the overall NPS output number for each of those segments,” Easton said. “[We want] to be able to say, through our responses to the qualitative feedback that we’re getting and discerning through a short-term basis, we’re able to affect the overall direction of this organization and the overall relationship that it has with its audience.”

Want to know more?

• Net Promoter Score was created by a strategist named Fred Reichheld. He wrote a 2003 Harvard Business Review article that introduced the broader world to the concept and it’s a good foundation for better understanding how to use NPS.

• Here’s another HBS story on how Philips, the technology conglomerate, uses NPS.

• NPS isn’t for everyone, and writer Jared Spool put together this thoughtful critique of the methodology.

• Finally, whether or not you decide to use NPS, it’s a good idea to survey your readership. Sarah Schmalbach, my colleague who leads the Institute’s Lenfest Lab, wrote a thoughtful post this week about what the Lab learned from its first experiment building a location-aware news app. One of the lessons highlighted the need to get more survey-based qualitative feedback. You can check out the whole post here.