With all the recent talk about the fabulous new office space that the Chicago crew just moved into, I wanted to share a little bit about another long term move that is nearing completion. For the last four years our infrastructure has been hosted with Rackspace. As of last weekend, the vast majority of our traffic is now being served out of our own colocated server cluster.
Some of our Dell R710 Servers, we have a bunch of these.
Hosting with Rackspace over the last four years has allowed us to ignore hardware to a large degree. When I joined 37signals, I was the first system administrator in the company and we didn’t have the resources to realistically manage our own physical infrastructure. Choosing a provider like Rackspace made a great deal of sense. We also had a lot of unknowns about what our growth profile was going to be and were able to take advantage of the ability to more organically grow our infrastructure.
Fast forward to today and we have a team of three people who are responsible for the care and feeding of the 37signals infrastructure. Our applications are mature and we understand our growth path very well. In those four years, we’ve grown from around 15 physical machines to a mixture of around 150 physical and virtual machines. We’ve also grown from having less than 1TB of data to on the order of 80TB of data today. Our needs have evolved a great deal as we’ve grown and we reached the point where it made sense for us to acquire our own hardware and manage our own datacenter infrastructure. The amount of flexibility that we have with our own environment makes it much easier for us to use some specialized equipment that meets our needs better than the solutions that Rackspace generally supports.
Isilon clustered storage servers.
Over the next few days I hope to cover some of the architectural choices we made when we were designing the new platform. If you have anything in particular you’d like to hear more about, please drop me a line via email to mark [at] 37signals.com and I’ll try to cover a few of them.
ploogman
on 23 Jul 10Did you colocate into a datacenter in the Chicago area or did you remain in Rackspace’s datacenter? Or maybe none of the above? This is a very interesting thread. Please talk a bit more about how you found using storage servers better than a SAN like EMC or why you decided to aggregate storage instead of sharding out storage and separating users on separate storage/servers. Thanks!
MI
on 23 Jul 10We did relocate into a datacenter in the Chicago area, though the fact that it was in Chicago was a pretty secondary concern. Rackspace doesn’t offer colocation servers for third party servers, generally speaking.
I plan to talk some about our storage environment next week.
ploogman
on 23 Jul 10also in regards to the Dell servers pictures, do you make use of embedded hypervisor/virtualization or any virtualization at all?
BillP
on 23 Jul 10Bring it on! Can’t wait to hear about it.
I’d love to hear about the database server layout as well…
anon
on 23 Jul 10was there downtime during the switch away from rackspace? (I didn’t notice any) And may this be a problem if you want geographically disperse servers in the future?
MI
on 23 Jul 10anon: We did take fairly brief scheduled outages to move each of the applications to the new datacenter. The longest one was Basecamp, which took us a total of around 70 minutes of downtime. Backpack, Campfire, and Highrise took somewhat less time, averaging on the order of 30-40 minutes I believe.
None of the decisions we made for this site will negatively impact any future options for setting up geographically distributed facilities. That was definitely in mind when we were planning the move.
dotemacs
on 23 Jul 10Just wondering, what made you go for Isilon boxes? Thanks
Greg Laws
on 23 Jul 10Looking forward to your storage discussion as well. I remember you talking about that on the Podcast a few months ago.
Thanks for the post Mark. It’s nice to hear about the infrastructure side of 37s.
Robert Sullivan
on 23 Jul 10In light of the fact that certain Dell servers (not yours) were recently in the limelight because Dell was shipping a replacement board with malware, I would be interested in (1) What prompted the move from RackSpace (2) What security measures do you have in place? (Knowing, of course, that you probably wouldn’t want to get too detailed on either of those questions)
Robert Sullivan
on 23 Jul 10I see on closer reading that you indeed mention the reason (extra flexibility) for moving away from RackSpace, my mistake!
Patrick
on 23 Jul 10Once you get a handle on your growth moving to your own servers makes sense since you can get some tax breaks on capital investment.
Jonrus
on 24 Jul 10There’s so much stuff in this thread that I don’t understand. Looks impressive though :)
Ted
on 24 Jul 10@Mark
Any reason why you’re not using EC2?
(Especially given the investment by Bezos into 37signals).
MI
on 24 Jul 10Ted: The primary reason is around our data needs. We tend toward monolithic databases that grow to fairly large sizes. EC2 instances don’t have the I/O capacity and memory to run them at acceptable performance levels. We also have really aggressive performance targets for all of our applications that are easier to hit when we can control the entire environment.
EC2 and other cloud hosting services (at least for now) are a much better target for greenfield projects than for porting existing applications. I don’t expect that to be the case forever, and it’s possible (even likely) that in a few years an infrastructure cloud will be a better solution for our production systems. Until then, we optimize for now.
The Bezos investment isn’t a factor in any of our decision making, so that didn’t have any bearing.
Henrique
on 24 Jul 10Cool post. I mentioned the other day here at SVN that would be interesting getting some insight on 37signal’s infra-structure and growth history, and what was learned during the years.
Congratulations on the new datacenter! ;)
Sarah
on 24 Jul 10Does anyone else find ,b>150 virtual images A LOT for basecamp, tada, campfire, backpack, highrise and jobboard?
anon
on 24 Jul 10I remember a post from a while back where 37s converted some virtuals to physical for performance reasons, it would be interesting to know the breakdown of P vs V.
MI
on 24 Jul 10Sarah: The actual number is between 100-150, and includes physical + virtual. There is a lot more going on than what you listed. We support at least 18 different applications, all of which are built in a highly available way with multiple instances, staging environments, search servers, background processing, and so on. What seems simple on the outside is somewhat more complex behind the scenes,
Anon: We have just shy of 50 physical servers, the remainder is virtualized. We run physical application servers for Basecamp, Backpack, Campfire, and Highrise as well as for databases, Solr search servers, and other similar services with either very high CPU or I/O requirements.
Ryan
on 24 Jul 10Patrick: Tax-wise, monthly charges ala Rackspace would seem far preferable to ownership. Monthly hosting fees are 100 percent deductible as a business expense, immediately. When you acquire assets like servers, you have to depreciate over say 5 years, so you’re paying up front with after-tax dollars.
37signals will still be able to deduct the new monthly charges for colo, bandwidth etc. And of course there are more important considerations in a decision like this than taxes (i.e. serving customers). But as a pure tax play Rackspace would seem superior because it’s 100 percent deductible up front.
gf ui
on 25 Jul 10Have you considered an XServe-based solution?
Walt
on 26 Jul 10Xserve? You’re kidding, right?
Apple’s play in the enterprise/big iron space is embarrassing. Dell, if you can stand to deal with their bureaucracy, actually has the enterprise hardware space pretty much figured out.
Surprised they went with Isilon storage though – Dell branded EMC is some pretty nice stuff, and you can bundle the discount in with your server purchases.
What is 37signals running for their virtual infrastructure? Dell is a VMware reseller btw.
Shane
on 26 Jul 10Just wondering if you contemplated blade servers at all? If you did, what pushed you away from them?
Brian Armstrong
on 26 Jul 10Would love to see the cost breakdown and where the crossover point was for you guys. Thanks!
Joker
on 26 Jul 10Considered Azure? (no, I’m not kidding).
http://is00hcw.spaces.live.com/Blog/cns516.entry
gf ui
on 26 Jul 10OK all, just to contain what my XServe comment seems to have started:
I did not mean to propose anything, including XServe. I also do not think it makes any difference to the customer what the infastracture is, as long as it works as expected. But I do think that 37signals, being users of apple products, would have at least briefly considered such a solution and I wanted some insight into that.
plaqq
on 26 Jul 10@ GF UI i agree but maybe the reason is a preference to use Linux/BSD Unix and not OS X – so using the XServe may be ill-advised or cost prohibitive
@ Shane I suspect they need a lot of CPU power for RoR and maybe blades just wouldn’t give them them big monolithic power they need? I have no idea just a guess
this is a great thread and looking forward to more posts on the iinfrastructure – thanks to Mark for sharing this stuff with us!!
Shane
on 26 Jul 10@plaqq I assume the same, their processing needs must be great, but they can be met by blades. The only downside I can see to blades is the initial cost, you have to buy the chassis. But since they were buying 150 physical servers anyways, they could have purchased 10 blade chassis. Blades seem like they may have been a good fit (just from being a casual observer). It would be interesting to hear if the 37 Signals guys weighed them at all. This certainly isn’t a concern for most people, but if you are lucky enough to get to the point where serious hardware purchases are in the works, it would be beneficial to all to know if and why they (blades) were not chosen.
Florin
on 27 Jul 10What operating system do you use for RoR stack and MySQL ?
This discussion is closed.