All the 37signals properties were offline for two hours this morning between 10AM and 12PM CST (16:00 to 18:00 GMT) as our load balancer blew out and knocked out the network connection for all our servers. No data was lost and the machines all kept running, but they weren’t accessible from the internet.
We’re very, very sorry for this interruption of service. While we were able to report on the progress of this interruption through our http://status.37signals.com (all the products and 37signals.com pointed to that site during the majority of the outage) that’s a small consolation when you want to access data stored on our services right now. It was just not good enough.
While we don’t have a formal service-level agreement (SLA), we still want to compensate anyone who felt they were negatively affected in their work because of this outage. Please write firstname.lastname@example.org (and include your application URL) and we’ll get that taken care of.
Naturally, we’re going to have a long, serious talk with our service provider (Rackspace). They’re supposed to be the best in the business, but in this instance they failed us, so we in turn failed you. We’ll do everything we can to make sure that something as simple as a load balancer (or firewall or switch or any other network equipment) going bad does not cause two hours of downtime.
Again, we’re truly sorry for this interruption. This is not how Fridays are supposed to be.