From the school of interesting ways we’ve failed…
There is a vast chasm separating the ability to collect metrics and use metrics.
The lean startup canon pushes for being data driven, so you’ll find that every startup has a plethora of people using a plethora of tools to “be metrics driven.” Lots of data. A/B Testing. Multivariate testing. All of this lingo circles around as common knowledge.
So we “did” that. Dropped Google Analytics on every page. Kissmetrics? Sure, why not. Mixpanel was being delivered ~100 different types of data points. We set up A/B tests all over the place in Optimizely. We built at least a dozen different “dashboards” and specific reporting tools in our own application.
It didn’t work.
We suffered from information overload – we had so much data on our hands, we had no clue what was actually happening. We had no discipline to regularly look at and understand the data. A/B tests were so easy to set up, we set up a lot of them, yielding inaccurate results, which we would never check. Our designated time to review metrics would be a mess of clicking around the various tools, trying to just understand what we were seeing. KPIs assembled would be inconsistent from month to month, yielding mistrust in the data. Luckily thing were going well, but if they hadn’t, it would have been hard to figure out what wasn’t.
We tried instrumenting the tools to tell us what we thought we needed, but they never delivered on that.
It reached a crisis point where we were talking to interested investors and realized we didn’t know the current metrics off the top of our heads, nor, even after looking through the data we had, could we answer some of their deeper questions.
Here’s what we did, which might work for you.
- Decide what your important goals are for the company. These are usually pretty standard for whatever vertical you’re in. We’re a B2B startup, so these are standard things like MRR & churn.
- Decide the metrics that should be tracked. Come up with a set of metrics that will tell you how you’re performing on your top goals. We have ~15, of varying importance. These are divided among top level departments (product, sales, marketing, customer service).
- Put them in a Google Spreadsheet, one per line. Yeah, I know, it’s late 2013 and we’re still using spreadsheets. But there are a couple key advantages of using a spreadsheet. The primary benefit is you get to decide what you need to track – you’re not limited to the data that’s in a third party tool, or how they calculate & present it. We fought for too long trying to get our main dashboards to be part of other tools, rather than using them to get just the data we wanted.
- Instrument your tools to collect that data. Now you’re able to use those powerful tools like Mixpanel & Google Analytics to answer exactly what you need – and you’ll find that may be completely different than what’s readily available. So this is really hard, no way around it. Don’t believe me? Try getting an accurate MRR calculation from Stripe data, amortized properly for annual plans (it’s OK, I’ll wait).
- Collect weekly. This is another advantage of using a spreadsheet: each team leader, while putting together their metrics to report to the team, is now looking at each and every individual value.
- Discuss weekly. Every monday morning, these team leaders run through each individual metric, explain why it changed, and answer any questions. This process usually results in clear actions for the week, questions we need to resolve, and experiments to run. If there is any question about integrity of the data (like an underlying API changing, as happens) – those are top concerns.
- Review with the team. We start off every team meeting with talking about the top level metrics.
- Go deep. As needed, dig into individual tools to gain better insight. This is where tools like Mixpanel really shine.
This has made a substantial difference for us.
