Criminals attacked the Basecamp network with a distributed denial-of-service attack (DDoS) this morning. The attackers tried to extort us for money to make it stop. We refused to give in and worked with our network providers to mitigate the attack the best we could. Then, about two hours after the attack started, it suddenly stopped.
We’ve been in contact with multiple other victims of the same group, and unfortunately the pattern in those cases were one of on/off attacks. So while things are currently back to normal for almost everyone (a few lingering network quarantine issues remain, but should be cleared up shortly), there’s no guarantee that the attack will not resume.
So for the time being we remain on high alert. We’re collaborating with the other victims of the same group and with law enforcement. These criminals are sophisticated and well-armed.
Still, we want to apologize for such mayhem on a Monday morning. Basecamp, and our other services, are an integral part of how most of our customers get work done. While no data was compromised in this attack, not being able to get to your data when you need it is unacceptable.
During the attack we were able to keep everyone up to date using a combination of status.basecamp.com, Twitter, and an off-site Gist (thank you GitHub!). We’ll use the same channels in case we’re attacked again. If the attack does not resume, we will post a complete technical postmortem within 48 hours.
We want to thank all our customers who were affected by this outage for their patience and support. It means the world to us. Thank you.
Jodi
on 24 Mar 14The fact that you have been open, honest and have kept us all up to date is invaluable and speaks volumes. Thank you very much.
Evan Volgas
on 24 Mar 14You guys kept everyone up to date, spoke frankly and candidly to the problem, and just generally responded to this very professionally. Thanks for staying on top of this, and for responding to this kind of crisis the way that you should. I was a loyal Basecamper anyway, but it’s the way you guys handle problems like these that make it even less likely I’ll ever leave
Jeremiah
on 24 Mar 14Our mutual customers have asked me to advise them on the possibility of a data breach today. Other customers commenting on this blog seem confident in your claims that no customer data was compromised and this was just an extortion attempt. But DDoS attacks are sometimes cover to exfiltrate data in an APT attack. Our customers’ source code is a prime target for such attacks. so I feel some concern is warranted. In my reading of the Basecamp service, it seems that some customers may have had SSL disabled by default, or based on recommendations to enhance speed or export data to other services. SSL was provided to all Basecamp customers in 2009, but Basecamp required the users to enable it on their own at that time. What percentage of your users have not done so, yet? How are you so certain that customer code repos were not compromised during this attack? May I be pointed towards customer documentation regarding your architecture and how our mutual customers can ensure they are properly implementing database-level encryption of their data on your service? Does Basecamp retain the keys to a customer’s encrypted repos? Forgive me if these answers are found on the website, a pointer to them will more than suffice. Thank you…
Jeremiah
on 24 Mar 14Learning as I go here—I was misinformed that this was a CVS service, my apo0logies. However, as a PM platform with file sharing features we are concerned with the integrity of all data stored on the service, and the methods which other CVS services hook into Basecamp…
The Insider
on 24 Mar 14Jeremiah – If I was your customer, I wouldn’t be paying you to investigate my concerns by commenting on the company website, but by getting in touch via support channels behind closed doors, like a gentleman.
Keep it up, Basecampers.
Brett Daniels
on 24 Mar 14Thanks for keeping us up to date – your transparency is absolutely appreciated.
Mads Hjorth
on 24 Mar 14Well navigated!
Thanks for that extra effort, it really makes a big difference for us – the customers.
I have just recently argued that cloud services like yours have better uptime than in house solution. I think that is still the case. But the way you handle communications and restoring services is well above any in house operation I have met.
Greetings from a long time follow! (I attended one of your first presentations of rail in Roskilde some 15 years ago, and I am glad to see your ambitions are still high)
Jay Stockwell
on 25 Mar 14We’ve gone through the same deal before and it sucks. We now use Cloudflare at the DNS level to proactively mitigate this. It’s an excellent first line of defense.
Michael
on 25 Mar 14Thanks for the good communication all around, and thanks to Jeremiah for providing the unintentional comedy during the wrap-up.
Jordan
on 25 Mar 14I’m excited to read the writeup – thanks for being so transparent about things. We were offline all morning, but only felt sorry for the Basecamp team. Not an angry word was muttered; hopefully that speaks volumes!
Chris
on 26 Mar 14Really appreciated up front communication and honesty about the uncertainties. Even more appreciative of your architecture keeping those goons outside of the house!
Jessica
on 29 Mar 14Thank you for the communication, quick thinking, and honesty. Would love to hear how you survived the attack. What are your server security procedures?
This discussion is closed.