There’s a problem with the way people report data that I’ve noticed in practically every context I have worked in. I’ve seen it in pitch decks from startup founders, in presentations from executives, in internal reporting documents from engineers, marketers, product managers, and even data scientists.
Today, the problem appeared yet again in the form of an investor update from a founder, with some data on the monthly churn rate. It looked something like this:
Aug: 8.24%Sept: 8.49%Nov: 8.36%Dec: 8.54%
Do you see the problem?
It’s significant figures.
This founder is reporting his churn rate to the hundredth of a percentage point.
It’s bad enough that this data is technically wrong: there’s no way his measurements are precise enough to warrant hundredth-of-a-percentage-point reporting. It’s worse that it’s unintentionally misleading. This founder is essentially baiting his investors to ask him why the churn rate is changing when, spoiler alert, it isn’t really changing at all.
Significant figures
The concept of significant figures is beaten into the heads of students in science classes from middle school through graduate school. The idea is this: When you take measurements, you can only be so precise.
Maybe your ruler only goes down to 1/10 of a centimeter. Maybe your scale only measures to the 1/1000 of a gram.
When you use these measurements to calculate a new quantity, that quantity is only as precise as your least precise measurement. If the limit of precision on your ruler is to 1/10 of a centimeter, you should not be reporting calculated quantities that go out to 1/100 of a centimeter. You don’t actually know the quantity to that degree of precision. You’re misleading your audience.
How you measure matters. (Photo: Patricia Serna)
Margins of error
When calculating quantities like churn rate, you will be subject less to issues of measurement precision than to issues of sample size. When you base your numbers off specific instances (for example, users who churned this month), you will always be subject to some margin of error.
The more users you’re working with, the smaller that margin will be. But no matter how large your pool of users, you should expect your numbers to be subject to normal statistical fluctuation.
The more ducks you collect, the more precisely you can measure their behavior. (Photo: Joshua Coleman)
More decimals != more accurate
If you had 231 users at the beginning of the month, and 19 of them churned, your calculator will return a churn rate of 19/231 = 8.225108225%. These extra decimals are an artifact of division, not an indicator that you actually know your churn rate to the ninth decimal place. Just because those decimals pop up in the calculation does not mean they make a meaningful contribution to the number’s accuracy.
The problem with reporting extra decimals is not just a matter of technical precision. Reporting too many decimals is problematic because it is misleading. People who read your results will think you know those quantities with the precision implied by the number of decimals you wrote down. They will take your mishmash of measured quantities and statistical fluctuations as the truth.
This is why scientific researchers have to care about these seemingly boring details. Scientists build on the conclusions of each others’ research. As a result, they need to be clear about what they know and what they don’t know — which means they have to be exacting with the degree of precision they report in their numbers.
It’s science! (Photo: chuttersnap)
Your data should tell a story
So why do so many people in business contexts assume that when it comes to reporting numbers, the more decimals, the better? In part, I think the practice stems from a desire to be transparent, and to give their audience as much information as possible. Heaven forbid someone get the wrong idea about where the business is headed because the person reporting the numbers masked the truth with some creative rounding.
But I think the bigger driver behind this behavior is that people are resistant to the idea that these numbers should tell a story.
Your data is not just a graph on a page. Your data communicates fundamental truths about how your business is operating — always. Don’t just copy-paste a number from your dashboard. Think before you report.
How to do better
As you prepare to report a number, ask yourself…
- Does the quantity in question look like what you expected, or is it larger/smaller than you would have guessed? Why? What does that mean for the business?
- Is the quantity increasing, decreasing, or staying flat? What are the business implications of each option?
- Is the quantity still increasing/decreasing, but at a slower/faster rate than last month? What would that mean for the business?
- Is the quantity changing in a cyclical way over the course of a week, month, quarter, or year — always dropping on the weekends or getting a boost around the holidays? How should this cyclical behavior influence your audience’s understanding of this data?
- Does the way the quantity is changing make sense? Is it what you expect, or not? Why?
- As needed, add context to your conclusions. Are you working with numbers based on a very small subset of users? Do you have evidence that the current trend might reverse? Do you have reason to believe you’re operating within a large margin of error, or that some logging issues might have messed up the numbers this week? Go ahead and say that. Transparency in numbers means giving the full context, not writing down every decimal point available.
(Photo: Emily Morter)
If, like the founder reporting churn rate in the example above, the quantity you want to report isn’t changing, make that obvious. Don’t write down the same number with some small fluctuations in the decimal values for month after month. If your churn rate is 8.24%, 8.49%, 8.36%, and 8.54%, then just say: The churn rate is consistently between 8–9% each month.
Go ahead and make your conclusion explicit. Don’t expect your audience to draw it for you.
Don’t abandon the narrative of your numbers
The point of reporting analytics is to tell a story about the health of your business, its challenges, and what you think the future will look like based on the evidence you have now. Analytics data is narrative. If you’re not using your numbers to craft that narrative, then you may as well not do any reporting at all.
You don’t need a degree in statistics to learn how to listen to what your data is telling you. You don’t even need to follow the principles of significant figures and margin-of-error calculations to the letter. But you do need to develop an intuition about your numbers, and think carefully about the story you intend to tell with every number you report.
Your data is narrative. Tell the story. (Photo: Aaron Burden)
Reporting excess decimals without thinking it through is lazy. It suggests that you expect your audience to do the work of interpretation for you. Don’t abandon the narrative of your numbers. Listen to what they’re telling you, and tell the story.