Sharding is a database technique where you break up a big database into many smaller ones. Instead of having 1 million customers on a single, big iron machine, you perhaps have 100,000 customers on 10 different, smaller machines.

The general advise on sharding is that you don’t until you have to. It’s similar to Martin Fowler’s First Law of Distributed Object Design: Don’t distribute your objects! Sharding is still relatively hard, has relatively poor tool support, and will definitely complicate your setup.

Now I always knew that the inevitable day would come where we would have no choice. We would simply have to shard because there was no more vertical scaling to be done. But that day seems to get pushed further and further into the future.

Last week I set out to improve the performance of the Dashboard and Contacts tabs in Highrise. Both tabs would frequently be much too slow. Especially the Contacts tab, which for our own account some times could take upwards two seconds to load.

The number one rule for improving performance is to measure, the number two rule is to measure some more, and the third rule is to measure once again just to be sure. Guessing about performance never works, but it’s a great excuse to get you out in the weeds chasing phantom ponies.

Looking outside the epicenter
So I measured and found that part of the problem was actually not even part of the epicenter, the notes and the contacts. In fact, we were wasting a good 150ms generating New Person/Company form sheets all the time (through a complicated Presenter object that’s high on abstraction and low on performance). Even though these sheets were the same for everyone.

That left me with two choices: Either I could try to speed up the code that generated the forms or I could cache the results. Since speeding up the code would require taking everything apart, bringing out the profiler, and doing lots of plain hard work, I decided to save myself a sweat and just cache. People using Highrise couldn’t care one way or the other as long as things got faster and frankly, neither could I.

I ended up with this code:

<% cache [ 'people/new/contact_info', image_host_differentiation_key ] do %>
  <%= p.object.contact_info.shows.form %>
<% end %>

This cache is hooked up to our memcached servers for Highrise. The image_host_differentiation_key makes sure that we don’t serve SSL control graphics to people using Safari/Firefox, but still do it for IE, in according to our asset hosting strategy.

Good enough performance
But saving 150ms per call wasn’t going to do it. So I added memcached caching to the display of the individual contacts and notes as well. The best thing would of course be if I could cache the entire page, but since Highrise is heavy on permissions for who can see what, that would essentially mean per-user caching. Not terribly efficient and hard to keep in synch. So instead we just cache the individual elements and still run the queries to check what you can see.

The media is biased towards stories that want to be told. This is especially true about the media covering the web world. It’s much easier to write a story about Facebook, Google, Youtube, or any of the other limelight stories where much of the juicy details are willingly shared.

We have so many visitors!
We make so much money on advertising!
We got so much funding!

Inflating evaluations are great when you’re listed on the market, gunning for more venture capital, or trying to bail with the biggest parachute. So naturally these companies drip and drop the honey for the journo worker bees and we get all these silly stories about younger and younger people being worth more and more monopoly money.

In the mean time, many of the real stories are never told. The quiet successes by small teams who stand little if anything to gain by sharing their numbers and telling about their success. Lest they attract competitors or other unwanted interest. They’re just happy making millions quietly from happy customers.

I’ve talked to so many of these entrepreneurs in private and have often been shocked by how well they’re doing. And I always think to myself: 1) why didn’t I know about this?, and 2) if only everyone else knew too.

This is especially evident in discussions about successful businesses online. People point at the big media stories and perhaps a few more and think disparagingly to themselves that this is probably it. If you’re not one of the high profile media darlings, you’re never going to get that trip to space. Bullcrap!

Know that the world of successful businesses online is much larger than that tiny tip that peaks above the surface for a reporter to find. There’s incredible wealth being created below. Sure, they’re probably not making billions (that’s hard to keep quiet), but there are plenty of untold millions.

My Macbook Pro 2.2 just decided to kill the display and all I get is blank on boot. Apparently, it’s a known issue relating to the 1.5/1.5.1 firmware update. When the software fixes don’t work (and they didn’t), the solution is to change the logic board. Nice.

This incident made me think about all the Macs I’ve had for the past few years and how they’ve all failed in various ways. I often donate older machines to family, so I get to hear about how they do 2-3-4 years into the process. And it’s usually not pretty.

My old G4 iMac had it’s DVD drive fail on it after two years. Three of my older laptops have had their hard drives fail (with painful data loss in one instance). One of my Macbook pros that my brother is using lost its firewire ports and its DVD drive (it’s just 2 years old). Mary’s fairly new Macbook air is making weird noises occasionally.

It’s a pretty terrible statistic. Jason’s current Macbook Pro also has a variety of issues and he ordered a new one just to be sure he wasn’t left stranded. Luckily, I also just got myself a new MBP (and the dual-DVI cable is supposed to arrive tomorrow!).

I wonder if this is just par for course. Or if I, and many people I know, have just been incredibly unlucky with Apple gear. But it has happened enough times that it seems statistically unlikely.

So please share your successful run of Apple machines that have been able to last 3-4 years without breaking down. I need to regain some faith.

To truly be inspired for great work, you need to know that you’re making a difference. That you’re putting a meaningful dent in the universe. That you’re part of something that’s making a difference and that your role in that something is significant.

This doesn’t have to be grand at all. You don’t have to be looking for the cure for cancer. It could be done by a waitress at a neighborhood cafe that’s the gathering point of local artists. The key is that your efforts would be missed, your customers would have a sense of loss, if you stopped doing what you’re doing.

If you’re void that sense of purpose, the pleasure in your work will eventually wane and ultimately feel hollow. I’ve lived that sensation more than once. Working with tools and techniques and even people that I enjoyed, but where the end did not justify the journey.

You can only hide in shadows of the circumstantial for so long before your passion begins to fade. You can only excuse your lack of impact on the world with “but it’s great money” or “at least we’re doing agile” or even “this way I get to use Rails” until the playlist of stories repeat and it just all sounds the same.

Remember that your time is limited. By the time you discover that you’ve been coasting on empty calories, the pale face staring you back in the mirror might be hard to recognize.

I remember waking up to such a face on day long ago and thinking “the world would have been no different if I had not been here the past six months”. That’s a terrible feeling of regret.

But the good story is that it’s never too late to do something about it. I’d give up a cozy working atmosphere and using tools I enjoyed if it meant having to do work that just didn’t matter. You should too.

There’s a fine line between being indifferent with the state of things and using Reddit to express your every displeasure with all facets of life. In between is the discontentment you can use to light a fire under your productivity.

The key is to focus on the discontent with things that you can actually change. Get riled up about your programming environment and submit a patch. Become annoyed with how the text flows on your company homepage and rewrite it. Feel guilty about the UI of a common action in your application and redesign it.

God, grant me the serenity to accept the things I cannot change, the courage to change the things I can, and the wisdom to know the difference.
— Reinhold Niebuhr

When you find people who embrace this idea, you’ll usually find people with exactly the pointed drive that gives them the power to Get Things Done. Hire them.

Browsers try their best to protect users from the evils of the internet, but some times their eager kills good use cases all the same. The “mixed content warning” that IE and Firefox throws when you reference a non-SSL asset from an SSL page is one of those cases.

The problem is that browsers don’t like caching SSL content. So when you have an image or a style sheet on SSL, it’ll generally only be kept in memory and may even be scrubbed from there if the user is low on RAM (though you can kinda get around that).

Even when you do your best to limit the number of style sheets and javascript files and gzip them for delivery, it’s still mighty inefficient and slow to serve them over SSL every single time the user comes back to your site. Even when nothing changed. HTTP caching was supposed to help you with that, but over SSL it’s almost all for naught.

Now I understand why SSL content is not kept on a disk cache. You don’t want the financial-results-2008.png lying around on a public computer that you thought was safe to use because of SSL.

This is why it would be ever so wonderful if you could refer to a non-SSL asset from an SSL page. That way you could say “this stuff is private, don’t share it” when serving over SSL and at the same time take advantage of HTTP caching for things that pose no security risk, like drag_handle.gif.

But oh no. If you try to do that with IE, you’ll get a “mixed content warning” that’ll scare the bejesus out of your users, so that won’t go. On Firefox, you’ll often (depending on settings) get the same warning if you try to serve JavaScript or style sheets like that.

Only Safari stands out as the hero with no mention of “mixed content warning”. So please, IE and Firefox, can you learn from Safari and kill it too? Then we wouldn’t have to adjust our asset rules depending on the user agent and everyone would be able to enjoy the extra speed of properly caching assets.

Note: There’s a reasonable argument for warning on JavaScript includes as man-in-the-middle attacks can do nasty things, but that’s not true for CSS (on anything but IE) or images

Apple must be smiling whenever one of their competitors bill their next MP3-player as an iPod killer. It reminds everyone who the market leader is and invites them to evaluate the product on Apple’s terms.

If you’re going to be an iPod killer — and not just a great new music player — you have to first out-Apple Apple at all the things that makes the iPod special. That means superior industrial design, an iTMS-beating catalogue of content, and a better desktop experience than iTunes.

That’s almost an impossibly tall order. Which is of course why it hasn’t happened yet and probably won’t happen any time soon.

That doesn’t mean that there’s not room for other entries in the music player department, just that you’re going to have a hell of a time making it if you think cloning the market leader and saying you did it better is the way to do it.

Now replace Apple and the iPod with the gorilla and blockbuster in your niche. Are you setting yourself up to be a Zune?

Having great taste is one of the most important characteristics of designers, programmers, and managers alike. Being able to discern what’s good from that which is bad. Which of the thousands of possible little details are key to make whatever you’re working on just right.

The problem with the concept of taste is that it’s so ephemeral. One view of the world is that some people just have it and others don’t. Either you’re lucky enough to be born with it and you’ll be forever awesome or you’re a tasteless sod doomed to create crappy work. I don’t subscribe.

I think taste is mostly about developing an eye for the details that matter and that it’s absolutely something that can be learned. The best way to learn what details that matter is to examine the details of great and not-so-great work and contrast and compare.

It requires determination and dedication to develop an eye, to develop your taste, but it’s absolutely possible. Some may be predisposed, but anyone willing to be a student can get there. So what are you waiting for?