Over the past few months I’ve learned an immense amount from the all new Basecamp codebase. This year for RailsConf, I was able to show off some new (and old!) patterns in the codebase along with some neat libraries that we’ve put to serious use for our customers. The slides are over at SpeakerDeck if you’d like to see them all, and hopefully a video of the presentation will be up soon!

Here’s a few choice sections from the talk, and plenty of great links to check out in the meantime:

My dad is a firefighter and fire captain in Niagara Falls, NY. When I told him I had on-call duty for my new job, he was beyond excited. After relaying stories of waking in the middle of the night to head into the hall and getting overtime from his buddies that didn’t want to wake up for work were sick, I had to explain that it’s a different type of action for me. I face issues of a less life-threatening variety, but there’s still plenty of virtual fires to extinguish. Here’s a primer for what being an on-call programmer means at 37signals.

The routine

On-call programmers are rotated around every 2 weeks, and all 8 programmers currently get their fair share of customer support and interaction. Usually we have 2 to 3 programmers on-call on a given week. Support issues from email and Twitter are routed through Assistly, and handled by our awesome support team.

If there’s an issue they can’t figure out, we get a ping in our On Call room in Campfire to check out a certain issue:

These issues range in severity, and they might look like:

When developers first discover the wonders of test-driven development, it’s like gaining entrance to a new and better world with less stress and insecurity. It truly is a wonderful experience well worth celebrating. But internalizing the benefits of testing is only the first step to enlightenment. Knowing what not to test is the harder part of the lesson.

While as a beginner you shouldn’t worry much about what not to test on day one, you better start picking it up by day two. Humans are creatures of habit, and if you start forming bad habits of over-testing early on, it will be hard to shake later. And shake them you must.

“But what’s the harm in over-testing, Phil, don’t you want your code to be safe? If we catch just one bug from entering production, isn’t it worth it?”. Fuck no it ain’t, and don’t call me Phil. This line of argument is how we got the TSA, and how they squandered billions fondling balls and confiscating nail clippers.

Tests aren’t free (they cost a buck o’five)

Every line of code you write has a cost. It takes time to write it, it takes time to update it, and it takes time to read and understand it. Thus it follows that the benefit derived must be greater than the cost to make it. In the case of over-testing, that’s by definition not the case.

Think of it like this: What’s the cost to prevent a bug? If it takes you 1,000 lines of validation testing to catch the one time Bob accidentally removed the validates_presence_of :name declaration, was it worth it? Of course not (yes, yes, if you were working on an airport control system for launching rockets to Mars and the rockets would hit the White House if they weren’t scheduled with a name, you can test it—but you aren’t, so forget it).

The problem with calling out over-testing is that it’s hard to boil down to a catchy phrase. There’s nothing succinct like test-first, red-green, or other sexy terms that helped propel test-driven development to its rightful place on the center stage. Testing just what’s useful takes nuance, experience, and dozens of fine-grained heuristics.

Seven don’ts of testing

But while all that nuance might have a place in a two-hour dinner conversation with enlightened participants, not so much in a blog post. So let me firebomb the debate with the following list of nuance-less opinions about testing your typical Rails application:

1. Don’t aim for 100% coverage.
2. Code-to-test ratios above 1:2 is a smell, above 1:3 is a stink.
3. You’re probably doing it wrong if testing is taking more than 1/3 of your time. You’re definitely doing it wrong if it’s taking up more than half.
4. Don’t test standard Active Record associations, validations, or scopes.
5. Reserve integration testing for issues arising from the integration of separate elements (aka don’t integration test things that can be unit tested instead).
6. Don’t use Cucumber unless you live in the magic kingdom of non-programmers-writing-tests (and send me a bottle of fairy dust if you’re there!)
7. Don’t force yourself to test-first every controller, model, and view (my ratio is typically 20% test-first, 80% test-after).

Given all the hundreds of books we’ve seen on how to get started on test-driven development, I wish there’d be just one or two that’d focus on how to tame the beast. There’s a lot of subtlety in figuring out what’s worth testing that’s lost when everyone is focusing on the same bowling or bacon examples of how to test.

But first things first. We must collectively decide that the TSA-style of testing, the coverage theater of quality, is discredited before we can move forward. Very few applications operate at a level of criticality that warrant testing everything.

In the wise words of Kent Beck, the man who deserves the most credit for popularizing test-driven development:

I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence (I suspect this level of confidence is high compared to industry standards, but that could just be hubris). If I don’t typically make a kind of mistake (like setting the wrong variables in a constructor), I don’t test for it.

We’re finally ready to unveil the API for the new Basecamp. The documentation lives on Github and we encourage developers to help us improve it with pull requests.

The new API covers most of what’s available in the web UI and whatever is currently not there, we’ll be sure to add in the future. You can create projects, add people to those projects, work the calendar, upload files, the works.

We’re really excited to see where people will take the API. Especially the Events API that allows you to poll for changes since a given timestamp. This should make it easy to have local notification style integrations and much easier for people who want to build synching services.

So have at it!

We’ve been busy tightening the nuts and bolts on the all-new Basecamp in the wake of last week’s launch. As part of the process, I decided to take a closer look at client-side page load performance. Shaving even a tenth of a second off of page load time can have a big impact on perceived performance, so it’s a worthwhile investment.

Profiling the JavaScript page load event led to a surprising revelation. On pages with many to-dos, an overwhelming majority of the time was spent initializing sortable behavior. The culprit looked something like this:

$.ready(function() {
  $("article.todolist, section.todolists").sortable();
});

In other words, when you’d load any page with to-dos on it, Basecamp would make the items of each to-do list reorderable, then make the whole collection of to-do lists reorderable. That could take up to half a second on heavy pages.

Here’s the thing: you probably aren’t reordering to-dos every time you visit a project. It would be best if we could avoid initializing sortables until the last possible moment, just before you start to drag a to-do.

Deferring Initialization

That got me thinking. What would happen if we tried waiting until the first mouse press before initializing sortables? The mousedown event gets us a lot closer to the intention of reordering than the page load event. I tried it out:

$(document).on("mousedown", ".sortable_handle", function() {
  $("article.todolist, section.todolists").sortable();
});

The new code says that when we receive a mousedown event on the page that comes from somewhere inside a sortable drag handle, we should go ahead and initialize all the sortables on the page.

Sadly, this code doesn’t work. The sortable() initialization installs a mousedown handler of its own, but by that time it is too late. The event has already bubbled its way up to our delegated event handler.

If only there were a way to catch the mousedown event just before it begins its ascent up the DOM…

There are only two hard things in Computer Science: cache invalidation and naming things — Phil Karlton

Doing cache invalidation by hand is an incredibly frustrating and error-prone process. You’re very likely to forget a spot and let stale data get served. That’s enough to turn most people off russian-doll caching structures, like the one we’re using for Basecamp Next.

Thankfully there’s a better way. A much better way. It’s called key-based cache expiration and it works like this:

When we started working on Basecamp Next last year, we had much internal debate about whether we should evolve the existing code base or rewrite it. I’ll dive more into that debate further in a later post, but one of the key arguments for a rewrite was that we wanted a dramatic leap forward in speed — one that wouldn’t be possible through mere evolution.

Speed is one of those core competitive advantages that have long-term staying power. As Jeff Bezos would say, nobody is going to wake up 10 years from now and wish their application was slower. Investments in speed are going to pay dividends forever.

Now for the secret sauce. Basecamp is so blazingly fast for two reasons:

It used to be one of the biggest pains of web development. Juggling different browser versions and wasting endless hours coming up with workarounds and hacks. Thankfully, those troubles are now largely optional for many developers of the web.

Chrome ushered in a new era of the always updating browser and it’s been a monumental success. For Basecamp, just over 40% of our users are on Chrome and 97% of them are spread across three very recent versions: 16.0.912.75, 16.0.912.63, and 16.0.912.77. I doubt that many Chrome users even know what version they’re on — they just know that they’re always up to date.

Firefox has followed Chrome into the auto-updating world and only a small slice of users are still sitting on old versions. For Basecamp, a third of our users are on Firefox: 55% on version 9, 25% on version 8. The key trouble area is the 5% still sitting on version 3.6. But if you take 5% of a third, just over 1% of our users are on Firefox 3.6.

Safari is the third biggest browser for Basecamp with a 13% slice and nearly all of them are on some version of 534.x or 533.x. So that’s a pretty easy baseline as well.

Finally we have Internet Explorer: The favorite punching bag of web developers everywhere and for a very good reason. IE represents just 11% of our users on Basecamp, but the split across versions is large and depressing. 9% of our IE users are running IE7 — a browser that’s more than five years old! 54% are running IE8, which is about three years old. But at least 36% are running a modern browser in IE9.

7% of Basecamp users on undesirables
In summary, we have ~1% of users on an undesirable version of Firefox and about 6% on an undesirable version of IE. So that’s a total of 7% of current Basecamp users on undesirable browser versions that take considerable additional effort to support (effort that then does not go into feature development or other productive areas).

So we’ve decided to raise the browser bar for Basecamp Next and focus only on supporting Chrome 7+, Firefox 4+, Safari 4+, and, most crucially, Internet Explorer 9+. Meaning that the 7% of current Basecamp users who are still on a really old browser will have to upgrade in order to use Basecamp Next.

This is similar to what we did in 2005, when we phased out support for IE5 while it still had a 7% slice of our users. Or as in 2008, when we killed support for IE6 while that browser was enjoing closer to 8% of our users.

We know it’s not always easy to upgrade your browser (or force an upgrade on a client), but we believe it’s necessary to offer the best Basecamp we can possibly make. In addition, we’re not going to move the requirements on Basecamp Classic, so that’ll continue to work for people who are unable to use a modern browser.

Basecamp Next, however, will greet users of old browsers with this:

When I did all the programming for the original version of Basecamp back in 2003, we ended up shipping with just about 2,000 lines of code. A lot has happened in those eight years and we’ve acquired a more delicate taste of just how beautiful we want the basics executed.

This means significantly more code. Here are the stats from running “rake stats” on the Rails project:

On top of that we have just over 5,000 lines of CoffeeScript — almost as much as Ruby! This CoffeeScript compiles to about twice the lines of JavaScript.

Basecamp Next is running Rails 3.2-stable and we’ve got a good splash of client-side MVC in the few areas where that makes sense through Backbone.js and various tailor-made setups.

Got any questions about our stack or code base? Post them in the comments and we’ll fill you in.

Three years ago, I wrote about how improvements in technology keep allowing us to punt on sharding the Basecamp database. This is still true, only more so now.

We’ve grown enormously over the last three years but RAM keeps getting cheaper and FusionIO SSD’s keep getting faster. If anything, it seems like recent advances in SSD technology are accelerating and it’s ever more unlikely that we’ll need to shard Basecamp.

Basecamp remains a perfect candidate for sharding. Isolated accounts, no sharing between them. Yet the cost in increased complexity is constant while the cost of throwing hardware at the problem keeps dropping.

It’s like how some old people have a hard time dealing with inflation and “they want how much for a gallon of milk these days?”. Technologists who grew up when RAM cost $1,000 per megabyte can have a hard time dealing with the luxury of RAM being virtually free (we just bought about a terabyte worth of RAM for a Basecamp Next caching system that cost just around $12,000).

The progress of technology is throwing an ever greater number of optimizations into the “premature evil” bucket never to be seen again.