Yesterday at 12:45pm central time, our ops team detected a dramatic spike in login requests to Basecamp. More than 30,000 login attempts were made in the hour that followed from a wide array of IP addresses. Our first line of defense was to block the offending addresses, but ultimately we needed to enable captcha to stop the attack.
After the attack was over, we diagnosed that 124 accounts had unauthorized access from the attack. We immediately reset the password for these accounts, logging out any intruders, and emailed the affected account holders with all the relevant information.
All of the unauthorized access was gained using the correct username and password for the account. It’s highly likely that these credentials were obtained from one of the big breaches, like those collected in combos like Collection #1, Anti Public, or Exploit.in. All the affected accounts showed as “owned” on haveibeenpwned.com.
Our preliminary investigation shows that none of the unauthorized access actually performed any actions within the accounts. It seemed like the attack focused on first validating which accounts were vulnerable, perhaps with a plan to later exploit these vulnerable accounts. Thankfully we were able to detect and stop the attack very quickly, and also ensure that any intruders were prevented further access.
Never the less, this is a serious reminder that you should never share the same password between multiple services. Particularly services such as Basecamp that may contain sensitive information. Here’s what we recommend you do to stay safe:
1.) Use a password manager to ensure you’re using different, secure passwords on every service you use. Then if one service is breached, you don’t have to worry about the rest. We use 1Password at Basecamp and recommend it.
2.) Subscribe to a breach notification service, like the one offered by haveibeenpwned.com. Then you’ll be alerted if your credentials are part of hack known to the public.
3.) Turn on two-factor authentication (2FA) wherever you can! We offer 2FA protection for Basecamp using Google Sign-In. Most services that deal with sensitive information offer 2FA these days. It’s especially important that you enable this for critical services, like your email address.
Our ops team will continue to monitor and fight any future attacks. They did an excellent job detecting and addressing this particular attack. But if someone has your username and password, and you don’t have 2FA protection, there are limits to how effective this protection can be.
Protecting yourself against attacks like this is important. Take the time to learn the basics, and take the steps outlined above to limit the risk.
Update: On January 31st, the mass-attack resumed in much greater strength than before. More than 5,000 IP addresses were used to test stolen credentials. 89 proven correct, but no content was accessed on these accounts, and we followed the same procedure of resetting all logins and writing the people affected. We’ve since beefed up our CAPTCHA protection across all applications and all clients, which has been effective at stopping the attack. CAPTCHA isn’t perfect, and some times it’s annoying, but it has provided effective protection against this wave of attack. We continue to work on shoring up defenses, but do follow the steps outlined above to protect yourself!
David — Thanks for this. We’ve been using Basecamp for a long time, and we’re consumers of your book! We also do crisis communications, and I appreciate your speed and transparency. I wanted to recommend that you consider a partnership with the Cyber Readiness Institute. (https://www.cyberreadinessinstitute.org/) With co-chairs Ajay Banga and Samuel J. Palmisano, this free resource can help businesses become cyber secure. As a believer in both this Institute and Basecamp, I’d love to see about a co-sponsored educational seminar for your small business users. It could be a nice follow up to your already upstanding response. Thoughts?
I’ll take a look. Thanks!
Thanks for the proactive work in protecting these accounts. One of the feature requests from us has been to enable admins to force all users to use google sign-in instead of leaving the choice to users to pick that over basecamp native login. Can this feature be prioritized?
We’re looking into a bunch of ways to beef up self-defense powers. That one is definitely on the list.
Did you know that you can post to post to Basecamp by email using someone else’s account so that it looks like the post came from them? You might want to fix that security hole while we’re talking about security…
Will investigate that. Thanks for reporting!
We don’t have a known vulnerability here. Could you help us understand the issue you found by submitting a security report: https://basecamp.com/about/policies/security/response? Thanks again!
The link to enabling 2FA in Basecamp 2 in the 2FA post doesn’t work – it just leads to the help homepage.
Something must have happened to that Basecamp 2 guide. Hmm! We’ll get it restored. In the meantime, you can switch to Google 2FA when you’re logged in from this link: https://launchpad.37signals.com/identity/login_option. Sorry for the confusion, thanks for reporting!
This has now been properly fixed, so the link for Basecamp 2 in the 2FA post works again. Thanks for catching this!
Thanks for the heads-up and recommendations.
Are logins from Google accounts affected ?
We have contacted all account holders if they were affected. If you have 2FA turned on, you would not be vulnerable to these types of attacks. Here’s a guide to turning on 2FA for Gmail: https://support.google.com/accounts/answer/185839?co=GENIE.Platform%3DDesktop&hl=en
How do we enable 2FA in Basecamp 3 signed in through E-mail address?
You have to switch to Google Sign-In to use 2FA with Basecamp. See all details in /svn3/protect-your-basecamp-login-with-googles-two-factor-authentication/.
Besides not sharing the same password for different services, it is also highly recommended to have a very strong password for your e-Mail account. The reason is, once the e-Mail account is hacked, that most sites allow you to reset the password through your e-Mail. If that is not yours anymore, you’re in deep trouble.
With this type of attack, information of multiple German politicians have been tweeted a while ago (see https://www.heise.de/newsticker/meldung/Parteihack-Persoenliche-Dokumente-Hunderter-deutscher-Politiker-veroeffentlicht-4265180.html).
Hey David,
Any chance of getting letting people use a 2FA app like Google Authenticator?
In order to use 2FA with Basecamp you have to log in using your Google account. Google 2FA supports Google Authenticator, Duo, etc, so you’re all good.
The answer given here, that you have to use a Gmail account to setup MFA, is a non-answer. For corporate, licensed users using their domain address, they have no MFA option, which seems pretty unreasonable.
Please set up a platform neutral MFA solution such as Google Authenticator, Duo, something.
I agree with Pingyu, a number of our employees don’t have gmail and we have our own domain address, we’d love to see a 2FA app like Microsoft Authenticator, Duo, etc.
Thanks!
You don’t need to use Gmail or Corporate Google to use Google 2FA sign-in with Basecamp. You can signup for a Google account here: https://accounts.google.com/SignUp?hl=en.
That being said, I’m very sympathetic to the idea of “not wanting Google in our life at all”, and we’re looking into ways of satisfying that in the future.
I know this likely doesn’t entirely satisfy your requirements Pingyu and Scott, but you can create a Google account without Gmail.
https://accounts.google.com/SignUpWithoutGmail?hl=en
That being said, it’d be nice for Basecamp to support native 2FA using an authenticator app.
My access to Basecamp 3 was deleted. I did not receive notification of any kind. Please let me know if I need to be aware of any more relevant information.
That sounds like a separate issue from the mass-login attack, Margaret!
Can you send an email to [email protected] with more information about the problem? We’ll be happy to help you out there!
-Jabari
30K attempts with 124 compromises? That’s so cute! Seriously, have a chat with security teams of mid-sized cloud companies some time for perspective. It’s not uncommon to see 3k+ attacking IPs doing 500k+ auth attempts in ~12h with over 10-30k account compromises (with attackers that come back every day for multiple months). Captchas work well unless you have business/users in CN or until attackers find your APIs and alternate auth methods. Consider yourself lucky you weren’t targeted by a more powerful adversary but, be ready for them when they come knocking. [email protected] if you want to chat more candidly.
Your made-up example of “500k+ auth attempts in ~12 hours” is equivalent to 41k per hour. Seeing as how this attack was stopped after 1 hour, it’s really not that far off. Your patronizing tone comes off as somewhat insulting. Also if they’re getting 30k account breaches, they’re doing something wrong.
Hello, since the captcha was initiated, I’ve been unable to log in on mobile (only on laptop). Is this something others are experiencing?
I do the captcha correctly and it then just sits back at the log in screen. Hoping to get back to mobile use soon. Thank you.
So sorry, Laura!
Can you send an email to [email protected] with a bit more information about whether you’re using one of the mobile apps, or the browser on your phone?
Thank you!
Hey, David
There is no way to cancel third-party applications access to the Basecamp account. I can’t even see which applications have access to my account.
One day it could be a big security hole.
This moment, you have to change your password to lock out third-party integrations. But we’re working on feature to make sure that’s not necessary 🙏
I use google 2FA. No basecamp password at all.
In that case, please write support for instructions on how to expire third-party integrations. We’re working on a self-serve option, but at the moment it’s embarrassingly complicated.
I was trying to find this screen the other day!
Does changing your password reset the third party oauth token though? I’ve seen cases (not with Basecamp) where I’ve changed my password but app access has remained.
We’ve started using Sqreen (https://www.sqreen.io) to offer runtime app protection. It’s really good. There are a number of alternatives but definitely something to look at from both a monitoring and bot/attack mitigation perspective.
How were people notified? I logged in yesterday and saw a message about the hack that said I need to change my password and that if I used the password on any other account, I was extremely vulnerable, or something like that. Was I one of the 124? I received a threatening email this a.m. referencing this password (no mention of Basecamp of course) and trying to blackmail me for money. How can I know if I was one of the unlucky few?
David mentioned every compromised account was found on haveibeenpwned.com so you could check there. That doesn’t necessarily mean you were affected by the Basecamp attack but if you’re on the list then you should start changing your passwords ASAP.
Jonathan, we emailed all 124 account holders directly. We’ve since verified that none of these accounts actually had any content accessed. And all passwords were reset. So there’s nothing to blackmail you about regarding Basecamp!
Hey Dave,
My company WWPass is a BaseCamp customer. Our team absolutely loves your product. As security experts, we compliment your handling of the attack.
As you probably know, these attacks will continue as long as usernames and passwords are used for authentication. This is why we created an authentication solution that replaces human-readable credentials with machine-readable credentials that are used for authentication and/or encryption, mainly used by the military. That’s right, the WWPass solution has no usernames and no passwords. It is also independent of email providers (Google, Microsoft, corporate domains), so that all of your customers can use WWPass.
We would love to provide an option for your customers (especially ourselves) to use WWPass authentication with BaseCamp. It is a relatively simple integration – we’ve done it many-many times. In fact, I am willing to dedicate our resources to help integrate WWPass authentication with BaseCamp for free, so that our team can start using it. We can also integrate client-side encryption for a truly zero-trust system. None of your competitors offer these secure features. I know, I’ve looked. This could present a new market opportunity for BaseCamp, to offer customers additional security options for your service, for pennies a day. You can learn more at wwpass.com. If you are interested, let’s chat and see what makes sense.
Thanks, Perry. We’ll check it out! We’re working on a variety of security upgrades, including investigating FIDO/U2F and WebAuthn for the future as well.
As of 3:24 this morning this basecamp user and other users in our group began receiving sextortion emails through Basecamp. You have a problem!