Last week I set out to improve the performance of the Dashboard and Contacts tabs in Highrise. Both tabs would frequently be much too slow. Especially the Contacts tab, which for our own account some times could take upwards two seconds to load.

The number one rule for improving performance is to measure, the number two rule is to measure some more, and the third rule is to measure once again just to be sure. Guessing about performance never works, but it’s a great excuse to get you out in the weeds chasing phantom ponies.

Looking outside the epicenter
So I measured and found that part of the problem was actually not even part of the epicenter, the notes and the contacts. In fact, we were wasting a good 150ms generating New Person/Company form sheets all the time (through a complicated Presenter object that’s high on abstraction and low on performance). Even though these sheets were the same for everyone.

That left me with two choices: Either I could try to speed up the code that generated the forms or I could cache the results. Since speeding up the code would require taking everything apart, bringing out the profiler, and doing lots of plain hard work, I decided to save myself a sweat and just cache. People using Highrise couldn’t care one way or the other as long as things got faster and frankly, neither could I.

I ended up with this code:

<% cache [ 'people/new/contact_info', image_host_differentiation_key ] do %>
  <%= p.object.contact_info.shows.form %>
<% end %>

This cache is hooked up to our memcached servers for Highrise. The image_host_differentiation_key makes sure that we don’t serve SSL control graphics to people using Safari/Firefox, but still do it for IE, in according to our asset hosting strategy.

Good enough performance
But saving 150ms per call wasn’t going to do it. So I added memcached caching to the display of the individual contacts and notes as well. The best thing would of course be if I could cache the entire page, but since Highrise is heavy on permissions for who can see what, that would essentially mean per-user caching. Not terribly efficient and hard to keep in synch. So instead we just cache the individual elements and still run the queries to check what you can see.

It’s not the fastest approach in the world, but remember that performance optimization is never about the optimal, it’s about the good enough. Performance is a problem when it’s a problem, but otherwise it’s just not relevant. People are not going to feel the difference between a page rendered in 50ms and one rendered in 100ms, even though that’s a 100% improvement. Especially not when you consider that each Highrise page also loads a bunch of styles, javascripts, and images. It’s just not relevant at that point.

All that was needed in the end to make Highrise considerably faster was these five caching calls we do in the view:

This helped bring pages that before could easily take over a second down to 100-400ms range. Much more acceptable. Our general rule of thumb is that most pages should render their HTML on the server in less than 200ms and almost all in less than 500ms. That feels like a good compromise of good enough performance. Of course we have lots of actions rendering in way less than that and also some that are still above that range.

Accidental gains
As I pushed these improvements live, I was tailing the production logs to get a cursory overview of how the caching was improving repeated calls. That turned out to be proven nicely so, but I also noticed something else. Generating the Atom feeds that I kept seeing in the log was taking an awful long time. Many would take 500ms or so. Nasty when you see the same request come in again and again!

Thankfully Highrise had just been updated to Rails 2.2 as part of this improvement run anyway, which meant that we had access to the new HTTP freshness features. I quickly added a few ActionController::Base#stale? calls and immediately saw the beauty of “304 Not Modified” responses flying back over the wire. Meaning that we were no longer regenerating a response for a client that already had the latest version. HTTP is peach!

I also noticed that we were fielding a lot of sorta-expensive API calls from a known 3rd party and gently wrote them an email asking for etag and last-modified header respect, so they wouldn’t tax our servers if they already had the latest info.

Together all of these changes lead to a ~30% drop in average response times as measured by New Relic. Not too shabby for a handful of caching calls.