Everything Looks Like a Nail: Selecting the Right Data at the Right Time
We all agree that we should use data to make decisions. Sometimes, though, we don’t put a lot of time into thinking about the types of data we should use. I want you to pause a second to think about that. When you say “data”, what do you mean? Are you treating every question like a 6D nail and trying to bash it into the wall?
For a lot of people, “data” means whatever comes out of their analytics package. That’s the “hammer” in the product management tool belt. Don’t get me wrong, that’s very important data, and you should be collecting it. But it’s not the only type of data you can gather and use. Just ask Amplitude — they make one of the most sophisticated behavioral analytics products on the market, and their PMs still spend 50% of their time talking to customers. No matter how amazing your analytics stack, it’s not a magic bullet. In this post, I want to explore how you can hone in on the right kind of data to answer your most important questions.
There are three parts to making a decision using data:
- Understand what question you’re trying to answer;
- Understand what data you can gather;
- Understand how that data can answer your question.
In this post, I’m going to deal with #2 and #3. For more on #1, take a look at my post on the pitfalls of data.
Stay in the present
Remember, you’re trying to understand what data you can gather. Of course, this will vary depending on your company or product stage and the size of your business. Maybe you don’t have budget for a fancy behavioral analytics tool, or you don’t have enough users for statistically significant data. That’s ok. Focus on your current capabilities, not on all the things you wish you had.
I will add one caveat. You might not have enough data to make meaningful decisions based on your analytics package — yet. That doesn’t mean it’s too early to set up your stack. When you’re ready to use that data, you should already be collecting it. (If your company is very small, apply for one of Amplitude’s new startup scholarships!) Collect the data, but for now let’s put it to one side and see how else you might get the answers you need.
Identify your options
Sometimes you get stuck in your existing way of thinking. It happens to all of us, and it definitely happens when we’re thinking about our data.
Those of us who develop products have tools for overcoming this. When we’re thinking about solving problems for our customers, we’ll go broad in a “generative” phase of thinking. We can do the same thing when addressing our own problems with data collection. Here’s a quick three-step brainstorming process you can try.
Step 1: Frame a question
As always, you should start with a question. In this case, start with something pretty generic. Like, “How might we learn more about how people use our product?”, or “How might we find out if our product is successful?”. Be deliberately vague so you’re not trapping yourself in preconceived ideas.
Step 2: Gather some people
If you’re truly solo, skip this step. But I encourage you to think about who in your organization might have ideas about this. You could include people from:
- Marketing
- Sales
- Customer support
- Engineering
- Data Science
- Design
- Product
- IT
Anyone who might know about a channel for hearing from customers, or a source of data you might not have considered.
Step 3: Brainstorm
Write down all the sources of data you can think of. You can write individually and share, or go popcorn style and capture ideas as you go. Whatever works better for your team.
Don’t try to figure out whether your sources are relevant to a particular question you have; this list is a resource you can come back to.
Data sources: A cheat sheet
Here’s a sample list to get you started, especially if you’re on your own.
Qualitative
- Empathy interviews
- Survey responses (individual)
- Usability testing results
- Unsolicited feedback (eg from customer service)
Quantitative
- In-app analytics data
- Analytics data from other channels (eg social media ad buys)
- Survey responses (aggregated)
- Sales data
- Experiment results (eg A/B testing)
- Customer service call volume
- Logs
Mix & match
Now that you have a list, you can figure out which of these will help you answer your most pressing question. Note that you don’t have to identify what kinds of questions could be answered by every data type you identified. That’s not a good use of your time. Take your most pressing question, and find the data that can address it. Ignore the rest of the list; you can revisit it when you have a different question to answer.
Some folks would say you should prioritize statistically significant quantitative data. And they are not wrong. Unless you are unable to get such data in relation to your question. This may seem super obvious, but trust me, people are trying to put square data pegs into round holes all the time.
Examples
Let’s take a couple of real examples and see which data from our list would help us answer them.
Question: What areas of my product aren’t working?
This is a very broad question, and you will need to narrow it down. The data you gather will help you to do that.
If you have enough users, you can obviously dig into your analytics (because you already have your analytics set up, right?). If you don’t, you can get this information by talking to people. There’s an entire UX toolkit for this. Start with Erika Hall’s Just Enough Research; the second edition is out soon.
Question: Does my idea (for a product or feature) solve a real problem?
If you are figuring out whether your product solves a real problem and could attract users, youʼre not going to be gathering huge amounts of analytics data. You could, however, do empathy interviews and run some Facebook ads. Statistical significance is unlikely to factor into your decisions here; youʼre looking for signals.
Question: Which of these designs will increase conversions?
If youʼre working on an optimization project for an existing product where the goal is to increase conversions, empathy interviews might help with your design, but theyʼre not going to inform your decisions. And Facebook ads arenʼt relevant, unless Facebook is part of your funnel. Youʼll be running experiments like multivariate tests and looking at your analytics data. Statistical significance is both possible and pertinent here.
Answer the question
This may seem redundant; didn’t we already match up the question with the data?
We did, but only in terms of which data we could use to answer the question. There’s an important “how” in this as well.
The most important thing about how you use your data to answer a question is honesty. You need to be intellectually honest. And thatʼs difficult, because we are all biased. And this is the moment when your confirmation bias will kick in, hard. And you are smart. Youʼll be able to find a way to make the data say what you think it should.
One way that I like to do this is to set out to disprove my hypotheses. We talk a lot about “validation” in tech, but really, we should be trying to invalidate our ideas. Before you look at your data, express the question in the form of a hypothesis. (If you need a template, check out Lex Roman’s post on optimization experiments.) Now think about how you could show this hypothesis was incorrect. Look for the fastest, easiest way you could disprove your hypothesis. The faster you fail, the sooner you can go on to something that might work out. If we fail at invalidation, we might be on a path to success.
When we build products, we think carefully about the tools we’re going to use. When we’re measuring those products, we need to be just as thoughtful — and just as creative — about the data we use. Because when it comes to product success, what we build is only as good as what we know.
Thanks to Lex Roman for giving feedback on an early draft of this post.
