Three years ago, I wrote about how improvements in technology keep allowing us to punt on sharding the Basecamp database. This is still true, only more so now.
We’ve grown enormously over the last three years but RAM keeps getting cheaper and FusionIO SSD’s keep getting faster. If anything, it seems like recent advances in SSD technology are accelerating and it’s ever more unlikely that we’ll need to shard Basecamp.
Basecamp remains a perfect candidate for sharding. Isolated accounts, no sharing between them. Yet the cost in increased complexity is constant while the cost of throwing hardware at the problem keeps dropping.
It’s like how some old people have a hard time dealing with inflation and “they want how much for a gallon of milk these days?”. Technologists who grew up when RAM cost $1,000 per megabyte can have a hard time dealing with the luxury of RAM being virtually free (we just bought about a terabyte worth of RAM for a Basecamp Next caching system that cost just around $12,000).
The progress of technology is throwing an ever greater number of optimizations into the “premature evil” bucket never to be seen again.
Anonymous Coward
on 27 Jan 12“You kids today, with your hip-hop music and saggy-baggy pants, are lazy. Why, in my day, I had to code the entire internet in 2K of RAM. That’s the way it was and we liked it.”
I much prefer the 21st century.
Paul D
on 27 Jan 12It truly is amazing that Michael Moore has not only made the greatest movies ever made, but has single-handedly slved the sharding problem!
Bryan Larsen
on 27 Jan 12“Why, in my day, I had to code the entire internet in 2K of RAM . That’s the way it was and we liked it.”
We liked it because it was fun. The coolest project I ever did was a 17-function 40-bit precise motor controller with 4K of ROM and 63 bytes of RAM.
Jason
on 27 Jan 12So if you know your investment would pay off, why wouldn’t you do it? You wouldn’t immediately earn your investment back, but you’d amplify your gains while riding the wave of technology.
The Spelling Brigade
on 27 Jan 12One ‘r’ in terabyte.
terabyte
on 27 Jan 12terabyte
Rudiger
on 27 Jan 12That’s Dr. Moore.
Eric
on 27 Jan 12The only thing that gives me caution is that your resistance to sharding/distributing keeps putting larger and larger eggs in one basket.
Carl Mercier
on 27 Jan 12It would be interesting if you could write about your whole infrastructure, servers, how it all works together, etc.
DHH
on 27 Jan 12Eric, you don’t do stuff like this without having backup database servers ready. That would be terrible. So we have hot backups standing ready in case of failure.
Tom
on 27 Jan 12Just out of interest, can you share the current size of the Basecamp production database?
Karthik Hariharan
on 27 Jan 12Too many of the startups today are pushing their databases to cloud servers, not realizing how much performance they are missing out on. A little hardware goes a long way with a database.
It’d be nice if there were more hosts that seamlessly married cloud/VPS for web servers with dedicated hardware for I/O intensive servers like Database servers. A lot of them claim to support this, but simply aren’t priced right for bootstrapped startups.
Duff OMelia
on 27 Jan 12It is pretty excellent that hardware advances have allowed you to punt on the complexity of sharding.
The question I have is around how to easily run database migrations when there’s a ton of data. So if you’re using mysql and you have 20 million rows in your messages table, adding or renaming columns without significant downtime isn’t very cinchy.
I’m thinking that one of the reasons that schema-less eventual consistency db’s like Riak have a bright future is that it makes zero-downtime deployment easier even when there’s a ton of data.
If you’re using mysql, then perhaps effectively using the new ActiveRecord::Base.store in Rails 3.2 will help.
Thanks for the post – I’d love to hear your thoughts on how to keep our mysql schemas malleable over time.
Douglas F Shearer
on 27 Jan 12A few years ago your post here mentioned you purchased some Schooner MySQL appliances. Still going with those, or have commodity servers proved to be better in the long run?
Eric
on 27 Jan 12But it’s not just the database that can cause trouble. Storage capacity keeps growing, but IOPS/capacity is not and neither are communications links ( internal or external ). Hot standby’s are great, but are no substitute for a full backup which gets harder and harder to squeeze out. Hybrid approaches take the load off of predictable workloads, but can fall over for random workloads.
I’ll presume you have your bases covered, but you are really the exception not the rule. With these problems. With advances in sharded DB tech ( choices and simplicity ) the last few years the “cost” of switching is coming down, not staying the same. At some point it very well be the right choice even if Dr Moore has your mainframe covered.
Joe Ruby
on 27 Jan 12“Michael Moore has not only made the worst movies ever made”
FIFY.
Troy
on 27 Jan 12Do those beefy servers with super fast SSD’s make schema changes a non issue as well? How long does it take to alter one of basecamp’s larger tables? I’ve always wondered this, hopefully one day I’ll have an app big enough to find out for myself, but until then hopefully those that are paving the way before us can share. Thanks alot!
Taylor
on 27 Jan 12@Troy,
They definitely make schema changes much much faster. It varies depending on load / size of the table obviously. We usually try to make sure as much of the table is cached in memory as possible—again that helps.
Taylor
on 27 Jan 12@Doug
Schooner no longer makes / supports their appliances. They were just “custom” IBM servers anyways.
We still use Schooner for one of our databases. Although they’ve developed some great stuff, we’ve run into some problems using their Synchronous replication. We are eagerly awaiting engineering improvements.
Anonymous Coward
on 27 Jan 12@Troy I’m another happy FusionIO customer, when I do schema changes we’re now CPU bound while the IO subsystem is more or less idling (MySQL is single threaded on schema changes).
Dan
on 27 Jan 12It’s not just sharding. When I was in college just ten years ago and cutting my teeth on C++, grades and a good portion of class time emphasized the importance of memory management. Nowadays with the emergence of garbage collection and cheap memory, this only applies really to mobile applications. Even normal forms, of which I am a fan simply because of the cleanliness of the code, really aren’t as important as they used to be.
I’ll always bet on hardware outpacing software. Well written software that takes into account hardware performance seems to be important only in the short term to get the earliest versions of hardware out the door and usable. But over time, the hardware blows past the software and the design considerations that were once vital are just legacy bullshit (generally speaking).
EH
on 27 Jan 12Duff: Smoothing features in database technologies are nice, but downtime should be no taboo when you want to do things right.
Joe Van Dyk
on 27 Jan 12Duff—if you use postgresql, you can add a column or rename a column in a 20,000,000 row table in a few milliseconds. (assuming you don’t have to insert new data, of course)
Scott
on 27 Jan 12That’s DR. Moore, you hooligans.
Kyle Drake
on 27 Jan 12I did a presentation on a game I made at Geoloqi that talks about the Fusion IO drives and why I think local hardware with SSD is the best solution for persistent data stores right now: http://www.slideshare.net/KyleDrake/building-mapattack
Kyle Drake
on 27 Jan 12Sorry, it didn’t link: http://www.slideshare.net/KyleDrake/building-mapattack
The one warning I want to provide though is that not all SSDs are created equal: Make sure you get one that writes its cache to the disk on power failure, or you’re going to be in a world of hurt.
Douglas Lee
on 31 Jan 12A lot of systems which were built when RAM and SSD were orders of magnitudes more expensive than they are today could be profitably “un-sharded” to simplify their design and cut maintenance/operations costs.
Merle
on 31 Jan 12Good design means you’re not tiptoeing on the highwire.
Ed
on 01 Feb 12Yes, and they have Memory Cube coming in two to three years time. ( OK may be four to five years ) that will increase the capacity by ten times!!!!
We are possibly looking at a 1U Server with 1TB RAM and Superfast SSD built in!
Yousaf
on 02 Feb 12Twitter does a good job at database sharding.
This discussion is closed.