Guide

Statewide News Collective: A guide to measuring community impact

A guide to measuring community impact that is harder to see, based on a 2024 research project from the Statewide News Collective.

By Danny Hayes

October 15, 2024

News organizations want to make an impact, but it’s not always straightforward to identify all the downstream effects of an outlet’s journalism.

Some types of impact are obvious — a news story produces a law enforcement investigation, an elected official resigns, or a legislator launches hearings into an issue or problem. But news organizations also produce more fundamental impacts that can be harder to observe. This is especially true at the state and local level. Local journalism provides communities with a trusted source of news, keeps residents informed, and mobilizes audiences to take political action. But because this kind of community impact takes place over the long term, at the individual level, and out of the sight of newsrooms, it is not always easy to see. Identifying and measuring community impact is crucial to demonstrating how local news organizations contribute to the democratic process.

This guide documents one approach to measuring the type of community impact that is harder to see. It draws on insights from a 2024 research project that demonstrated the impact of three members of the Lenfest Institute for Journalism’s Statewide News Collective — New Hampshire Public Radio, Bridge Michigan, and Montana Free Press.

The Community Impact of Independent Journalism

The guide describes two strategies. One focuses on how to measure community impact through the use of a public opinion survey, the approach that we used in the SNC research project. A second strategy explains how to use focus groups to gain similar insights into community impact, an approach that may be useful for organizations with more limited resources or staff time.

Jump to:

Surveys
Focus groups

The key principle in both tracks is to conduct research with a wide swath of constituents, including residents who get news from your outlet and those who don’t.

There are two reasons for this. First, gathering information about both consumers and non-consumers creates points of reference that make it easier to interpret findings from a survey or focus group. For instance, in the SNC community impact research, we found that between 29% and 43% of SNC outlet consumers said they had “a lot” of trust in the information they got from the outlet. On their own, the figures are difficult to evaluate – do those numbers indicate high or low levels of trust?

But it is easier to evaluate those results by comparing them to the levels of trust that other consumers have in the news outlets they use. As shown in the graph below, respondents in New Hampshire, Michigan, and Montana expressed lower levels of trust in other news sources than in the SNC organizations. Consequently, SNC outlets actually have above average levels of trust, a positive indicator of community impact.

Second, comparing the views and behaviors of consumers and non-consumers makes it easier to draw inferences about the impact of a news outlet’s coverage. In the SNC impact research, we asked respondents how well informed they felt about issues facing their state. When we compared respondents who regularly got news from an SNC outlet to those who didn’t, we found large gaps. In each case, as the graph below shows, SNC outlet consumers were significantly more likely to report feeling informed, suggesting that exposure to those organizations’ journalism helped residents keep up with major developments.

Seeking to measure community impact this way differs from audience surveys that many news organizations already conduct. Audience surveys are valuable for gauging users’ preferences for content, learning how they view an organization’s journalism, and so forth. But because they often lack a point of comparison — especially with non-consumers — their ability to provide systematic evidence of impact is limited. The approaches described in this guide are designed to address that.

Measuring community impact can have several benefits. First, it can give you a longer-term perspective on the contributions your organization’s journalism makes. For example, staff at Bridge Michigan said that the SNC impact research project helped them “view Bridge Michigan’s work in the totality of the year as opposed to the daily work immediately in view.”

Second, it can help identify strategies to expand your audience and reach new consumers. As one example, the impact research in Montana showed that many residents of the state get their news from local radio and television. Consequently, MTFP leaders are “planning on leveraging our relationships with both television and radio broadcasters to produce regular content featuring MTFP reporters in order to reach news-consuming Montanans where they’re at.” Likewise, the research in New Hampshire suggested that NHPR might have a stronger audience among young people than other outlets in the state.

Third, data on community impact can help with fundraising. Donors and grantmakers often want to see tangible effects. One way to do that is to show that your journalism is improving the civic life of your community. That kind of evidence may persuade donors that their investment would make a substantial impact. “Being able to show the tangible impact of our reporting will strengthen our case for continued and increased support,” according to MTFP staff.

Using surveys to measure community impact

Surveys can be a valuable tool for measuring the impact of news organizations because they provide direct insights from the community. By surveying a representative sample of the local population, surveys can reveal how your outlet is informing citizens about important local issues and developments. Surveys can also allow you to track changes in community perspectives over time, offering a clear picture of how effectively your organization is shaping public discourse and keeping residents civically engaged.

Surveys are particularly useful because of their scale. With the emergence of online survey platforms, it is possible to distribute surveys to a large number of community members and gather responses quickly. Large samples can also give you the ability to examine responses among different demographic groups, such as age or income, providing a deeper analysis of your organization’s impact.

Here, we provide an overview of the process of recruiting a survey sample, designing a questionnaire, and analyzing the survey data.

Recruiting a sample

For organizations interested in conducting a survey to measure community impact, the first step is to identify the relevant geographic area you serve. For statewide news outlets, the area will likely be an entire state. Other local outlets might serve a city or a particular region. A sample of respondents needs to be drawn from the relevant geographic area. For outlets that serve small audiences (e.g., a small town or neighborhood), surveys may not be suitable, and studying impact with focus groups may be a more effective strategy.

To draw the sample, you will likely want to contract with a company that specializes in public opinion or market research and is familiar with your coverage area. The firm would be responsible for recruiting a sample from the relevant geographic area and administering a questionnaire. The SNC project partnered with the firm Qualtrics to recruit a sample in the three states where research was conducted, but numerous national companies, including Survey Monkey and Cint, provide similar services. We mention these companies as examples, not endorsements. Ultimately, the cost of a survey can vary significantly depending on the geographic location, the scope of the research, and how the organization plans to analyze the data.

One consideration in recruiting a sample will be its size. How many respondents should the survey include? There is no “right” answer to this question; it will depend on a variety of factors, including the size of your news organization’s audience. News outlets with smaller audiences will need to ensure that any sample contains enough of its consumers to draw inferences about its impact. The appropriate sample size can be determined through consultation with a professional survey research firm. In the SNC research, our sample sizes varied in part because the population of the states we studied were different. Consequently, the sample sizes in New Hampshire (655) and Montana (650) were smaller than the one in Michigan (2,020).

A second consideration will be the composition of a sample. You will want to ensure that the sample is as similar to the relevant geographic area’s population as possible. For instance, if 35% of a state’s population has a college degree, then it is ideal for a sample of that state to have the same share of college-educated respondents. This helps ensure that results from the survey are as close to the “real world” as possible. Of course, no sample will be a perfect replica of the population. But maximizing the representativeness of the sample can be achieved through the sampling and weighting process in consultation with a professional survey research firm. For background, the Pew Research Center has helpful explainers on sampling and weighting.

Designing a questionnaire

A well-designed survey will collect several different pieces of information. Examples of questions used in the SNC community impact surveys are available here.

Survey questions

A survey should include questions to gather demographic information about respondents — age, race, education levels, and so forth. These will allow a news organization to learn about how its impact may vary among different groups — young people, residents of different socioeconomic status, and so forth.

To measure community impact, the survey will also need to collect information on several different attitudes and behaviors. One approach, which we used for the SNC research, is to focus on three levels of potential impact: residents’ awareness of the news outlet and its journalism; attitudes about the news outlets they use; and their levels of information and civic engagement.

Here’s how you should think about survey questions in each of those areas:

Awareness

Survey questions in this section can be focused on gauging whether respondents are regular consumers of your organization’s journalism, how they consume it (e.g., via newsletters, web site, etc.), and how often. This portion of the survey can also ask respondents if they are familiar with major investigations/stories/projects that your outlet has devoted significant attention to.

This survey should also ask respondents about their use of other relevant news outlets. This is one way to provide a point of comparison that can give more context to the survey responses about your own organization.

Attitudes

The items in this section of the survey should tap respondents’ feelings about your news organization and journalism in general. These include measures of trust and credibility (of your outlet and other competing news organizations); respondents’ views of the journalism your organization is producing; and how well that journalism addresses the concerns of the community.

Once again, this is a place where asking respondents the same questions about other news outlets they use is important. This provides important context that can help interpret the findings about your own organization.

Information and civic engagement

A third level is information and civic engagement. By this, we mean how news organizations inform their users and encourage their participation in civic life.

Questions can focus on respondents’ civic engagement and actions they have taken in the community. In particular, the survey can ask about various forms of civic participation, including voting, attending community meetings, volunteering in the community, contacting elected officials, and running for political office.

This section may also ask people about how well informed they feel, as well as their familiarity with key issues and figures in the community. These can provide both subjective and objective measures of civic knowledge.

Finally, the survey can also include measures of what are frequently called internal efficacy (how well people feel like they understand important debates and issues) and external efficacy (whether they feel like they can have an impact on their community). One goal here is to eventually allow for comparisons as to whether consumers and non-consumers of the news organizations have different levels of knowledge or civic participation.

Qualitative measures

Although the quantitative measures can provide a systematic assessment of impact, it can also be useful to collect qualitative evidence. As a result, it is a good idea to include open-ended questions that allow respondents to describe how a news outlet has made an impact on them.

One approach is to ask respondents to explain how their news habits may affect their understanding of or engagement in the community. Another idea is to ask them to recall a specific story that made an impact and to explain how that affected them in some way. For users of your news outlet, you can ask specifically about stories from the outlet.

Analyzing the survey

The survey will provide a lot of information that can be used to characterize the impact of your journalism.

You can determine the share of respondents who regularly get news from your outlet; how familiar they are with major stories; how often they share the information with others; and how much trust and confidence they have in your organization’s journalism. The SNC research report provides examples of this kind of analysis. If the survey is large enough, it is also possible to examine how these findings differ across groups (e.g., by age, gender, income level, etc.).

A central part of the analysis can be to compare the levels of information and civic engagement between consumers and non-consumers of the outlet. For instance, if your organization’s consumers report higher rates of voter turnout or participation in community events than non-consumers, this may suggest that your journalism is contributing to civic engagement. That was one key finding from the SNC report — respondents who regularly got news from one of the SNC outlets in our study reported higher levels of civic participation across a range of activities, as shown in the figure below.

Of course, caution is required, since a survey like this cannot “prove” that an organization’s journalism caused residents to engage in some civic action. But correlations like this are evidence that is consistent with community impact.

These quantitative measures can also be paired with the qualitative findings about how respondents describe the impact of an outlet’s journalism, as we did in the SNC report. We explained how respondents told us in their own words that stories they encountered had increased their knowledge of important issues, heightened their emotional engagement, and gave them new routes to participate in the community. The combination can allow for a rich portrayal of the many ways that a news outlet may have a meaningful impact on the community.

Using focus groups to measure community impact

Focus groups are a valuable way to study the impact of your news organization because they allow for in-depth, qualitative discussions that can reveal deep insights into audience perceptions and reactions to news content. In a focus group setting, participants can express their thoughts in their own words, providing nuanced feedback that might be missed in the structured format of surveys. The group setting can also encourage discussions among participants that may yield insight into community sentiments about your outlet’s journalism and its impact.

One advantage is that organizing a focus group can require fewer resources than conducting large-scale surveys, especially when the target audience is localized or difficult to reach through traditional survey methods. Additionally, focus groups allow moderators to probe deeper into issues that arise, offering more flexibility than a fixed survey format. This adaptability can make focus groups an effective way to explore issues like media trust or how specific news stories resonate with an audience.

Here, we provide an overview of the process of recruiting participants, conducting a focus group, and analyzing the findings.

Recruiting participants

For organizations interested in conducting focus groups to measure community impact, the first step is to identify the relevant geographic area that they serve. For statewide news outlets, the area will likely be an entire state. Other local outlets might serve a city or a particular region. The key is to draw a sample of respondents from the relevant geographic area.

As with a survey, a key component of the focus group is to recruit participants who are users of your outlet as well as those who are not. This will allow for a comparison of the views and behaviors of consumers and non-consumers. Consumers can be recruited through existing email lists, social media announcements, or other channels of communication that you may have established with your audiences. Residents who do not get news from your outlet might be recruited through online/social media ads, local message boards, or other public forums that draw a wide swath of the community.

One limitation of focus groups is that the people interested in participating are often different — typically more civically engaged — than those who are not. Consequently, you should aim to recruit as diverse a sample as possible in order to increase the chance that the results of the study are broadly generalizable to the community. One way to do this is to offer a small payment to encourage participation by a more representative set of residents.

In general, focus groups should be no larger than 10-12 individuals, which is small enough to facilitate discussion. News organizations should aim to conduct 3-4 focus group sessions to gather a large enough set of responses to facilitate analysis.

Conducting a focus group

Within the focus groups, one initial activity is to administer a short survey that seeks to measure some of the indicators that are suggested for the survey. Although the sample size for this abridged survey would be small, it can still provide a general sense of how the perspectives of your outlet’s consumers may differ from those who get their news elsewhere. (If you were to take this approach, you could utilize some of the same sample questions we included in the survey section.)

The conversation in the focus groups should center on how consumers get information, how they feel about information from the outlets they use (e.g., levels of trust), how well informed they feel, and their involvement in the community. In these discussions, you should encourage participants to describe any connections they see between their news habits and their civic engagement.

It may be useful to think about questions that get at the three different levels of community impact — awareness, attitudes, and information and civic engagement.

Focus group questions

Awareness

In an initial discussion, you may want to have participants describe where they get news about the community, how they consume the news (e.g., via newsletters, web site, etc.), and how often. You may also want to ask respondents if they are familiar with major investigations/stories/projects that your outlet has devoted significant attention and resources to.

Attitudes

In this portion of the discussion, you can learn about participants’ feelings about the outlets where they get news. For instance, how much trust do they have in the information they get from various outlets? Do the outlets where they get news address the important issues facing the community? This will offer you a chance to learn about how your own consumers view your journalism, as well as how that compares to the way users of other outlets view their own preferred sources.

Information and civic engagement

A third level is people’s information levels and civic engagement. By this, we mean how news organizations inform their users and encourage their participation in civic life.

This part of the discussion would focus on respondents’ civic engagement and actions they have taken in the community. In particular, it can be useful to ask participants various forms of civic participation, including voting, attending community meetings, volunteering in the community, contacting elected officials, and running for political office. Questions can also focus on how well-informed people feel, as well as their familiarity with key issues and figures in the community. You can also ask participants to discuss how well they feel like they understand important debates and issues and whether they feel like they can have an impact on their community. One goal here is to be able to say whether there are differences on these dimensions between the people who get news from your outlet and those who don’t.

It can also be a good idea to ask participants to describe how a news outlet has made an impact on them. In particular, questions could ask them to recall a specific story that made an impact and to explain how that affected them in some way.

Analyzing the focus groups

The analysis can focus on two aspects of the findings from the focus groups: (1) how consumers say your outlet and its journalism has affected them, shaped their engagement in the community, and other relevant outcomes; and (2) how this compares with what you learned from non-users.

Some key questions may be: Are there differences in the trust that consumers and non-consumers have in the information they get about what’s going on in their community? Are there differences in the way that consumers and non-consumers talk about their engagement in the community? Do your consumers describe their news habits playing a different role in shaping their civic engagement than do non-consumers?

The answers to these questions will provide a sense of the way that your outlet’s journalism shapes the lives of community members.