An interesting thing happens when your customer base reaches a certain size: You cease having edge cases. I think we’ve probably been at that point for a good year now – maybe longer – but we’ve really felt it recently.
Mistakes, bugs, incompatibilities, and related issues that used to affect a handful now affect hundreds. 1% is a real number now.
This requires some organizational change. More caution, more testing, more contingency planning, more disaster planning. These are good things in one direction and frustrating things in another. Regardless, they’re real and here to stay.
It’s a healthy reminder that companies can change, policies can change, techniques can change, perspectives can change. This change can come quick or take many years, but it’s usually already happened before you really notice it. It’s your job to catch up with it. What once worked before may not work again just as what didn’t work before may work now.
Personally, I’m finding it invigorating. It’s a new challenge for us as we continue to grow—people, revenues, exposure, influence, and responsibility.
As we approach our 10th anniversary, I’m reminded of what we’ve always known to be true: simpler is better, clarity is king, complexity is often man-made, and doing the right thing is the right way to do things.
Dave Greiner
on 08 Jul 09This is the exact situation that’s been creeping up on us the last 6-12 months too Jason. We’re continually putting more focus on the QA stages before a deploy and getting more structure around our development process.
As things get more structured it’s a challenge to walk the line between having the right amount of process and getting bogged down by too much of it.
Congrats on your continued growth.
Jim Jeffers
on 08 Jul 09Time for you guys to start taking some notes out of Amazon’s playbook for rolling out new features/functionality eh?
rm
on 08 Jul 09I immediately thought of the #fixreplies fiasco on Twitter, which @ev insisted “only affected 3%!!!!” of their userbase. Which is at least 100,000 users (but probably a hell of a lot more).
MB
on 08 Jul 09Great insights Jason. Not something that is discussed much. Thanks.
andycamp
on 08 Jul 09Is that end of the edge case, birth of the enterprise…?
JA
on 08 Jul 09I’m not sure I understand why 1% is more important now than it was 5 years ago? It might be a bigger absolute number, but it’s still only 1%?
Dan
on 08 Jul 09@JA: Sure, it may be the same relative amount, but now it’s more people submitting bug reports, more people emailing you with support questions, more people blogging about your bugs and failures, etc.
Tathagata
on 08 Jul 09@JA I agree with you, I too didn’t get the 1% thing? For all you know it might be the other way round.
I haven’t had this experience of catching up though, because we develop large and complex(from a scientific viewpoint) software, which are really useful to a handful or around a hundred customers. We do have to play catchup, but it would seem in very different ways.
Derek
on 08 Jul 09I wonder if this is going to make it harder to “get real.”
Joe Casabona
on 08 Jul 09I agree with Derek- I’m curious to see how this plays out. I read Getting Real and agreed with it for the most part- however I’ve always thought that while an agile process is nice in theory, as things grow that’s something hard to maintain. Sometimes process can be just as important as product, to take a line from the agile manifesto.
Still, you guys are a great company that I really try to model and I am excited to see how you adjust.
Happy
on 08 Jul 09Don’t worry about issues affecting the 1%. It’s now more than a handful but still way less than the “20%” that seems to mark natural and systematic boundaries of diminishing returns. Heck, Tim Ferriss recommends getting rid of 20% of one’s customers, specifically those that represent high maintenance and low income. Following classic 80/20 rule: Cutting out, or at least ignoring the issues of a selected 20% of customers could result in 80% or more drop in support work without much impact to income, freeing up time to build new stuff.
ABasketOfPups
on 08 Jul 09Happy: and then you lose your customers, as that 1% or 20% isn’t a magical static number. It’s the folks having some sort of issue Right Now. Will some folks have trouble more often than others? Sure. But almost everyone will have some issue at some point, and how you respond helps (they tell everyone “when things rarely go wrong, they respond quickly and fix everything”) or really hurts (“they’ve completely ignored this issue, just like the last one. Let’s go google some replacement service…”)
Khürt Williams
on 08 Jul 09Just don’t lose your edge.
Man Made Complexity & Edgecases
on 08 Jul 09Here’s a great example from today’s paper of man-made complexity that resulted only in $5million wasted on a complicated solution to a non-existent problem. It’s also a real-life warning of the perils of wasting time and money on the edge cases.
RTD scraps real-time updates of Colorado light-rail arrivals.
A cheaper, simpler system will update train riders based on published schedules. “In the small number of cases where trains are delayed, RTD’s control center can make special announcements to passengers on platforms.”
http://www.denverpost.com/breakingnews/ci_12774365
John
on 08 Jul 09“We’re so popular, we don’t have edge cases any more!” seems like a particularly spin-worthy way of saying “We’re going to scale back on our release schedule due to internal reasons we don’t really want to talk about”
The fact is that with any product there will be vocal minorities. You still need to maintain a release schedule, stick to your focus, prioritize your requirements and care about your customers. “No More Edge Cases” sounds, IMO, like a whole lot of cow pie.
Dylan
on 08 Jul 09This is the case with all macroscopic systems when those systems are scaled. Giving rise to the saying “the devil is in the details.”
The solution I’m fond of most to address this right now is setting standards absurdly high and the threshold for what is tolerable extremely low.
Karmen Blake
on 08 Jul 09what @derek and @joe said
J
on 08 Jul 09“We’re so popular, we don’t have edge cases any more!” seems like a particularly spin-worthy way of saying “We’re going to scale back on our release schedule due to internal reasons we don’t really want to talk about”
John wins the award for Most Cynical Person of the Day.
Happy
on 08 Jul 09@J: I’ll pile onto the cynicism bandwagon with John. 37signals’ efforts and attention to PR and publicity are becoming more and more evident. John’s expressing what I think a lot of 37signals customers hope – that the attention to the needs of their publicist doesn’t detract from development of more cool and useful features and products.
JF
on 08 Jul 09Publicity has zero to do with our development timelines or the decision to work on this feature or that feature. Zip. It never has and it doesn’t now.
Our development is driven by customer request, internal vision, and the spirit of the products.
Happy
on 08 Jul 09JF: 37signals’ customers love to know that. Thanks for jumping in and making it crystal clear! Keep up the good work.
Seralat
on 09 Jul 09Ignoring edge cases is not just an unwise choice in situations involving large numbers of users, but also in situations involving mission critical applications. Any decision around the wisdom of ignoring edge cases needs to account for both of these dimensions, volume and criticality.
Banking applications are a particularly good example of this because they often face both scale and criticality issues.
When I was designing applications for online banking, transaction volumes for our personal banking applications (leaving aside ATM transactions where volumes were higher by a significant multiplier) were in the tens of millions per month, with an active user base of 6-20 million, depending upon functionality.
An error rate of .01% equals 10,000 errors per 10 million transactions. That’s a lot of errors in absolute terms.
But let’s assume that our transaction volumes were much lower, and that we were only dealing with 1 million transactions per month. In that case, an error rate of .01% results in only 1,000 transaction errors, affecting roughly the same number of people. Looked at just in terms of numbers, that seems like a manageable number of errors such that we could potentially ignore some edge cases likely to contribute to the error rate.
For most people, however, money and everything having to do with it is mission critical. Any error relating to a transaction is likely to generate an immediate customer service contact and a great deal of potentially adverse publicity for the company that causes the problem, not to mention whatever problems it causes for the affected customer.
You can kill your reputation, and therefore your business, faster than you can imagine if you tolerate sloppiness around edge cases when designing and building applications in a business like banking.
In design situations like this, every possible application outcome of every action the customer might take has to be accounted for and the outcome specified so that a positive, or at least non-negative situation results. These outcomes can be caused either by customer action (e.g. they start a transaction before the daily cut-off but complete it after the cut-off) or the system (e.g. the credit card gateway is down). This means spending uncounted hours with your technical teams and business experts to understand all of the possible permutations of how a request like a bill payment or account transfer can succeed - either partially or completely - or fail gracefully, and specifying exactly what needs to happen in each of those situations.
People need to understand that designing, building and releasing on the fly is a luxury available only in limited circumstances, not an absolute good.
Jemaleddin
on 09 Jul 09Not to jump into the fray with criticism, but the message above doesn’t really jibe with some of your decisions: dropping support for IE6 was a good idea from a number of reasons, but it almost certainly affected far more than 1% of your users, and reduced your potential user base even more considerably.
What’s the difference?
JF
on 09 Jul 09@Jemaleddin The IE 6 move (one of the best we’ve made in a long time) was a global, fundamental change. It wasn’t about a bug in a new feature or a specific software change.
I would make the IE 6 decision over and over if I could.
Anonymous Coward
on 10 Jul 09aa
This discussion is closed.