Who’s Driving This Thing? The Pitfalls of Being “Data-driven”

Are you a data-driven Product Manager? Of course you are! Data is at the heart of what we do in product. We should always use data in our decision-making.

Sometimes, though, we talk about being “data-driven” as though it’s a magical shield that will prevent bad decisions from sneaking by us. Not so. Here’s the awful truth.

You can make horrible decisions using data.

Don’t Let the Data Drive the Bus

First of all, let me semantically quibble with calling ourselves “data-driven”. We should make decisions informed by data. But should we be driven by data? Or by our strategy? Who put data in the driver’s seat?

There’s one sure way to keep yourself in the driver’s seat: Interrogate your data. You’re in charge. You get to define the questions your data is going to answer. When you don’t do that, you may fall into three common pitfalls:

• Making irrelevant decisions
• Asking the wrong questions
• Confirming assumptions

Let’s take a more detailed look at these pitfalls and how we can avoid them using clearly defined questions.

Irrelevant decisions

In user experience, thereʼs a classic piece of advice: listen to your users, but donʼt let them design the product. We should heed that advice when looking at our data. Listen to the data, but don’t slip into reactive decision-making based on what you learn.

Your data is full of signals; it “says” all kinds of things. Sometimes, you’ll see big effects in your data and it’s tempting to act on them right away. But beware — red herrings lurk in that ocean of insights!

These red herrings aren’t necessarily “bad” things. I once worked with a fast-growing company to launch a new product. We weren’t expecting (or chasing) large numbers of users, but one day we noticed a huge spike in traffic. We couldn’t figure out why; then it hit us that it was coming from a more mature product, which had begun displaying a component we had built. Now, this may look like an exciting opportunity. Could we capture those users? They were clearly into us! But in fact, they weren’t. The users of the mature product were explicitly not our users. We didn’t act on this spike in our data, but we did make sure to pass on our observation. These users were looking for something. Our red herring might be someone else’s catch of the day!

The reason we didn’t act on this red herring was that we knew who we wanted to attract to our product, and what we wanted them to do. We weren’t tempted to change our priorities to opportunistically attract a different set of users. We had a clear strategy, and we knew how we wanted to implement it.

Letting data drive can leave you staggering from one tactical decision to the next, none of them getting you closer to your destination. These decisions are irrelevant to your goals. Spending time on stuff that doesn’t matter is wasteful, whether your decisions are informed by data or not.

Instead, make sure your strategy is clear. If you’re not answering questions in service of your strategy, no amount of data will help you succeed.

Wrong questions

Speaking of questions, how do you know what questions your data ought to answer?

Let’s pause to make a distinction. Thereʼs nothing wrong with letting data inform your hypotheses. But seeing your data as an answer and then hunting for the question is going to get you into trouble.

For example, the data you have might answer a question at the wrong stage of the product development. Say you have an app youʼve prototyped, but you havenʼt validated your market yet. As you test it with users, youʼre getting a strong signal that there are aspects of the onboarding flow that are confusing. Should you make a decision to redesign the onboarding flow? Probably not right now. File that information away for later! Right now, you need to be asking questions about whether anyone is going to download this app in the first place. It could have the best onboarding flow in the world, but if it doesnʼt solve anybodyʼs problem, whatʼs the point?

Staying laser focused on the right questions is crucial, especially if you’re lucky enough to have a lot of data at your disposal. If you start chasing insights instead of answering questions, you could end up answering a bunch of questions that don’t matter at all. Worse still, you could fail to answer the existential questions that mean the difference between success and failure for your product.

Confirming assumptions

The third problem is perhaps the most pervasive. You’ve heard of this one, folks. It’s confirmation bias.

Confirmation bias is the tendency to cherry pick the evidence that supports what we already believe, ignoring evidence to the contrary. That is one reason to state your hypothesis clearly up front. If you have articulated what you expect to happen, and it doesn’t, it’s harder to justify a business decision predicated on that outcome.

If you don’t have a clear hypothesis or question, chances are that things will “jump out” at you from the data to support your preconceived ideas. Research published in 2018 shows that if we have a predetermined idea about an outcome, that idea acts as a “cue” for our attention; our selective attention causes us to see only the evidence that supports our initial idea. Digging into your data with a specific question you need to answer is one way to work against this bias.

Define, Articulate, Test

When working with data, make sure you’re staying in the driver’s seat.

• Define the goals of your product or feature up front
• Articulate the question you’re trying to answer with data
• Test a hypothesis based on that question

Once you’ve done that, you have data that’s really valuable. Now you’re ready to get out there and design a solution that delivers on your goal.

Thanks to Lex Roman for giving feedback on an early draft of this post.