The methods we’ve used to guide our thinking and tips on when to use them.

Research lays the foundation for product development because it’s important to get to know your audience before making something for them. It’s also important to have potential audience members provide feedback on product features before you finalize them if you want to increase your chances of success. We’ve launched a handful of experiments this year in the Lab and used a wide range of light-touch user experience research techniques to guide our thinking. We’re sharing our “toolkit” as an easy guide for teams who want to deepen their UX research practice.

An introduction to our “toolkit”

There are many user research methods that can be used for a project, but depending on its scope, timing, and budget, some methods may be better than the others. In our lab, for example, we are satisfied with interviewing 30 people to explore a concept or running usability tests with 10 to 15 users before launching an experiment because the risk is low and our timeframes are tight. Even on a small scale though, we get enough information from potential users to continue with our experiments.

Meanwhile, there are a few research methods that we haven’t used given the time and effort they need to be done well. For example, we haven’t set up any diary studies, a method that asks users to document their activities, interactions, and emotions over time. This method isn’t ideal for our lab since the process is long, sometimes taking up to a month or longer to finish while requiring researchers to establish a well-organized timeframe to collect and analyze data submitted by users. Diary studies also require specific tools for users to record data.

Even though the methods are different, the overall research process stays the same. At the beginning of a project, your research should be exploratory and generative, and the focus should be on collecting as much information as possible about the product or the audience. Mid-way through a project, your research should start focusing more on evaluating ideas rather than generating information. It’s very important to understand which methods are more suitable and why to use them at the time.

Here is an overview of our “toolkit”, explaining when and why we take on each phase of research, the goal, the methods we use and what stage of design we’re in each step along the way. Extended details about each research method, along with practical tips and real-life examples, are below.

The tools in our “toolkit”

• In-person user interviews
• Competitive analysis
• Online surveys
• Card sorting exercise
• A/B testing
• Usability testing

---

Explore

In-person user interviews about concepts

Great for: Gathering qualitative feedback, which is an observational and conversational rather than binary or data-driven.

What to do: Meet with users, typically one at a time, to discuss any user-related topics.

End goal: Understand users’ feelings, motivations, objectives, habits, routines, and pain points.

Tips

• Prepare a script of between 5–20 questions.
• Ask open-ended questions.
• Have casual conversations and ask follow up questions.
• Have neutral reactions and don’t ask leading questions.
• Record all the answers.
• Conduct an analysis right away while memories are fresh.

Examples: Interview script and research analysis for our Philly Eats app experiment.

Competitive analysis of existing products

Great for: Gathering general information about competitors’ products; Helping identify the unique value your product provides.

What to do: Find similar existing products in the marketplace, then collect and compare data (i.e. features, reviews, and content) about those products.

End goal: Make better product strategy decisions by looking into similar existing products and examining what they are doing well and what they are not.

Tips

• Remember to keep your definition of a competing product broad at first; a competing product could share just one or many features with your product.
• Update the analysis if there are any changes among the competitors or if new ones enter the market.
• Look into the features of each existing product and see what they have in common and what’s different.
• Look into the customer reviews, both positive and negative, about each existing product.

Example: Competitive analysis for the Philly Eats app experiment.

---

Evaluate

Online surveys about products

Great for: Reaching out to more users in a short period of time; Gathering data-driven quantitative feedback.

What to do: Post survey links online or hand out printed survey forms to people.

End goal: Collect a larger amount of responses to understand trends in users’ motivations, objectives, habits, routines, and pain points.

Tips

• Summarize the goal of the survey at the beginning.
• Less is more. Use a small number of questions. Keep the questions short and easy to answer (multiple choice vs. written responses).
• Use online tools to create, analyze, and group user responses. Google Forms does that automatically.

Survey tools you can use

Google Forms (free, easy to use; our preferred tool)

Survey Monkey (free version offers unlimited surveys up to 10 questions; paid version offers more templates, question types, collaboration, and embedding features as well as branching and conditional logic options)

Typeform (good for designing beautiful and visually sophisticated surveys; free and paid versions available)

Example: Our survey for the Philly Eats app, regarding feature prioritization.

---

Iterate

Card Sorting for product elements prioritization

Great for: Validating information architecture schemes, which are outlines of the way information appears in your product; Gathering quantitative feedback.

What to do: Write down the individual product elements you want people to consider, and have them put them into groups. One example is ranking the elements (i.e. 1–4).

End goal: Understand people’s perceptions about your product’s information hierarchy based on their own preferences, and how intuitive a product’s flow is to people.

Tips

• Make sure the words on the cards are legible and make sense on their own.
• Make the process fun for people.
• Be open-minded and willing to find out how users see the information and the product.
• Encourage people to think out loud while they are grouping things.
• Record the results.

Example: Results from a card sorting exercise for the Philly Eats app.

A/B Testing on design

Great for: Comparing two different versions of a design; Gathering quantitative feedback.

What to do: Set two different designs side by side and ask people specific questions about things like color, text or image placement, graphics, etc.

End goal: Determine which variations of the design have the most positive responses from users.

Tips

• It is best to show two versions at the same time if possible.
• Make sure to ask specific questions regarding the things that need to be tested.
• Record the results.

Example: Notes and results from the A/B testing session for the sorting functionality in our Philly Eats app.

Usability Testing on features

Great for: Gathering both qualitative and quantitative feedback.

What to do: Meet with users, typically one at a time, to discuss any user-related topics.

End goal: Understand how people react to a product’s look, feel, and usability by observing:

1. How easily people can complete a task?
2. How quickly people can complete a task?
3. Do people remember how to complete a similar task?
4. How many errors do people make?
5. How satisfied people are when completing a task?

Tips

• Prepare a script and a list of 5–15 tasks and up to 5 follow-up questions.
• Ask open-ended questions before and after each task.
• Pick a comfortable environment, have casual conversations and make people feel relaxed.
• Have neutral reactions to their answers and don’t lead people in any way.
• Ask people to think out loud and record the answers. Remember to also observe people’s body language.
• Conduct an analysis right away while memories are fresh.

Usability testing tools you can use

• Paper prototypes, digital wireframe testing (i.e. an InVision prototype or a Sketch file), interactive prototype testing (i.e. beta version of an app), Usertesting.com moderated testing (you can ask testers questions in real-time) or Usertesting.com unmoderated testing (testers complete the tasks on their own)

Examples: Usability testing script for the Philly Eats app; Notes and results from the usability testing on Philly Eats notifications.

After looking at the UX research methods we’ve shared, I hope it’s clearer how essential they are to creating something valuable for users. Research allows teams to put themselves in other people’s shoes to understand what they need and how they feel about the product. If you have questions about these methods please reach out to us, or if you have a different method that you would like to share, please leave a comment below.

---

The Lenfest Local Lab is a multidisciplinary product and user experience innovation team located in Philadelphia supported by The Lenfest Institute for Journalism.

The Lenfest Institute for Journalism is a non-profit organization whose mission is to develop and support sustainable business models for great local journalism. The Institute was founded in 2016 by entrepreneur H.F. (Gerry) Lenfest with the goal of helping transform the news industry in the digital age to ensure high-quality local journalism remains a cornerstone of democracy.