My job is to gather, study, and understand data and its implications, and then make recommendations to help the business improve – in short, to deliver business value from data.
One of the things you learn when you work in analytics is that there’s an endless depth to virtually any problem – you can keep digging deeper and deeper forever. One of the most valuable skills you can learn is deciphering what’s needed to solve the real problem – when has the bulk of the business value been delivered, and when are you doing things that are just intellectual interesting but not actually valuable?
I’ve found that I end up performing analyses in one of four different levels of detail:
- The quick ‘n dirty: These are short and simple – for example, a designer wants to know what the distribution of the number of posts on a project is because they’re designing a new screen, or David or Jason wants to know how our support ticket response time is trending. These are some mix of data retrieval and analysis, but the results don’t need a lot of explanation or interpretation. Most of the time, the results are communicated via IM or Campfire, and I end up spending between 30 seconds and 30 minutes.
- The basic look: The most common analysis I do is a moderate depth one – something like a look at conversion rates and retention by traffic source, or a basic overview of how people are using a specific feature in the new Basecamp compared to how they used a similar feature in Basecamp Classic. The results here are more involved and need some interpretation or “color commentary”, and may come with specific recommendations. This sort of analysis gets written up in a post on one of our Basecamp projects, and usually takes somewhere between a couple hours and a day.
- The deep dive: When it comes to understanding root causes and developing significant recommendations, a more in depth analysis is called for. For things like understanding the root causes of cancellation or support cases, the bulk of the work tends to be on analysis, interpretation, and then actionable recommendations to address those causes. Frequently, there’s some instrumentation or reporting project that spins off from this as well – I may add a report to our dashboard on the topic so we can more easily track it over time. These analyses usually get written up in a longer document with significantly more detail, and sometimes come with a live or recorded video explanation and discussion as well. This sort of analysis usually takes between 1 and 3 weeks.
- The boiled ocean: If you want to understand a substantive issue from every single possible angle, try every statistical technique in the book, and write a report with every possible visualization, then you’re probably looking at investing multiple months in a problem. We haven’t done anything like this in the 18 months I’ve been here at 37signals, and that’s by design: in most cases, this type of analysis ends up providing essentially the same business value as a deep dive that takes a fraction of the time.
Next time you’re faced with an analytical problem, ask yourself what the real underlying problem you’re trying to solve is, and figure out what depth of analysis is the required to deliver the bulk of the business value; after all, your job is probably really about improving the business.
Anonymous Coward
on 05 Jul 12“deep dive” . . . “boiled ocean” . . .agh!!!! investment banking analyst flash backs.
Michael
on 05 Jul 12Regarding #3: Is it right to say you can’t just give the answer; you have to educate other Signals and defend your work, and the video and explanation is your way of answering the questions that would inevitably come up?
NL
on 05 Jul 12@Michael: Absolutely. In some cases, there’s not a lot of education and persuasion that’s needed, but particularly for those analyses that take a while, there is.
In many if not most cases I spend more time communicating the results than I do coming up with them.
Anonymous Coward
on 05 Jul 12@NL
I hope your internal communication is more concise than this blog post.
Yes, I know this is rude to say.
But as a coaching point, it wasn’t until the last paragraph of this post which even explained why this post was even created at all.
That should have been addressed in the first sentence.
Forrest
on 05 Jul 12I think, for me, the question I’d struggle to answer isn’t so much “what are the options for how deep I can go,” but rather “how do I know how deep to analyse in the case of a given problem? How do I determine what depth of analysis is required to deliver the bulk of the business value?”
It’s helpful to have these as discrete answers, to sort of make a possible approach more concrete. But I’m still not sure in non-obvious cases how to know ahead of time which problems are worth analyzing deeply.
Ernest
on 05 Jul 12@Anonymous Coward
It was actually addressed in the title…
David Andersen
on 06 Jul 12And that’s why they call ‘em ‘Anonymous Cowards.’
Michael
on 06 Jul 12Noah, so, who gives you the hardest time about your conclusions (in a good way, of course?)
Amar Harolikar
on 06 Jul 12Nice one. Thanks.
GeeIWonder
on 06 Jul 12How do you test your analysis results?
Dmitry Nikolaev
on 06 Jul 12When I stuck with a problem, most of the time I don’t know how to name it (even after reading this article, seems I already forgot all those titles). Deep dive, basic look, whatever.
If 37signals have a separate man “to gather, study, and understand data” it’s ok. But I’m sure this is not very suitable for the little-sized companies (and for singletons, of cause).
So, what the case of the article ? To classify some problems by category ?
ADI
on 08 Jul 12The titles threw me off a little bit but they turned out to be genius. Great analysis.
Michael Schaarschmidt
on 09 Jul 12Nice analysis – i catch myself often by trying to boil the ocean … where a basic analysis would be enough.
Jimmy Moncrief
on 09 Jul 12Noah
Would love some tutorials on how use R and your other tools.
Love your blog posts!
Sharky
on 09 Jul 12Great Post. I felt it was concise, well presented and helpful. Thanks
Ben H
on 09 Jul 12+1 AC
I too was initially confused by this blog post.
Anonymous Coward
on 09 Jul 12Can we have an article on how 37 signals carries out code reviews?
Junjay
on 10 Jul 12Thanks for the post, Noah. I like your last paragraph best and think this is actually the most important part of any analysis task-defining what the real problem is. As you mention, most of the time we are so eager to start thinking about what level of analysis and the types of statistical and software tools we want to use that we forgot to verify whether the question that has been posed to us is even the right question at all! In fact, you might even consider this the first layer of analysis-analyzing the question/request itself!
Suddora.com
on 10 Jul 12Understand the problem is first and most difficult to phase to face when you are doing analysis using analytics. Now in advanced version, analytics has been categorized that made easy to point the problem then solve. You guess the solution by reading the guide.
Jon A
on 10 Jul 12Helpful analysis pointers…
daisy
on 11 Jul 12It’s really helpful to me. After reading “Get real”, I’m start to work my web app and want to know what’s tool are you using for mock-up or prototype? Thanks!
This discussion is closed.