Last week we were dragged over the coals — and rightfully so — for failing to communicate clearly regarding a security exploit that was discovered in early August.
It turned out that the issue was related to Ruby on Rails. The exploit affected multiple products (ours and others). After the initial investigation, and escalation to the Rails security team, the root cause was patched within a few days of the initial report. We then updated our apps, tested, and deployed updates.
Fixed promptly, communicated poorly
Problem was, we did a terrible job communicating from start to finish. We didn’t communicate well with the person who initially reported the vulnerability, we didn’t communicate well internally, and we didn’t communicate well publicly.
Perfect security is a moving target. New exploits and security discoveries pop up over time. They occur in OSes, web browsers, frameworks, embedded systems, and commercial software. Anyone who’s in the software business has to deal with these issues from time to time. What matters is that issues are taken seriously, delegated properly, handled appropriately based on severity and priority, and communicated clearly with all parties involved. When someone reports a security issue, they’re reporting it because they want to help. It’s important for us to keep that in mind.
Getting better
After ultimately reviewing what went wrong, we began to rework our internal process for dealing with security reports. This is a longer term project which we’ve just begun. Something we could do in the short term was review how we communicate publicly about security on our main security page on 37signals.com.
After a fresh slap in the face (which is definitely a healthy thing from time to time), it became clear that the words we were using were the wrong words. We weren’t setting the right expectations. Some of the lines were cringeworthy. It just wasn’t us. I don’t think any of us ever liked the way this page was written, but we never got around to changing it. Now was the time.
The old version
Phrases like “You’ll never lose important data” and “Your data won’t be compromised” and “You’ll always have access to your data” are easy things to say, but there’s no such thing as never, won’t and always in the world of security. Even government computers and banks with billion-dollar security systems can be compromised. Hyperbole may sell, but we don’t want to sell it.
What we can promise is that we take security seriously, that we’ve invested big in systems that keep data safe and secure, and that we’re continually focused on improving security. Internal audits, external audits, applying patches, following reports by security researchers, and keeping our ears open are just some of the ways we stay focused.
With that in mind, we spent some time rewriting the headlines, statements, and general customer-facing security messages. We called on some advice from a security company we’ve worked with in the past. The key points remain, but the language is better. We can stand behind these words.
The new, live version
Instead of “You’ll never lose” we say “We protect”. Instead of “Won’t be compromised” we talk about physical security (the old headline really didn’t pertain to the copy below the headline). Instead of “you’ll always have” we say “Redundancy for all major systems”. These are better words because they are more meaningful words.
In addition to revamping the standard security page, we also created a security response page. This page details the process for submitting a secure security report and explains how these reports will be handled. We will also be listing the names of the people who’ve reported vulnerabilities and helped us improve our security.
A continual process
Some steps we’ve already taken include assigning a dedicated security contact (before we had a few different people handle inbound security issues, but a single point of responsibility is better), creating a special address for reporting security issues (with a public key to keep those reports safe), establishing a better internal process for routing the issues to the right people, and implementing a better external process for reporting on issues.
Of course updating language on a marketing page doesn’t have anything to do with actual security, but the way we communicate has a lot to do with trust. We’ve made missteps before and we’ve gotten better each time. We’re not perfect, and we’ll never be perfect, but we’re still aiming for it. Thanks to everyone who’s gone out of their way to work with us to find, fix, and disclose security flaws.
JF
on 09 Sep 09I forgot something else… We’ve also linked up the security page at the bottom of the footer on all the product sites and 37signals.com site. This makes it easier to find and get in touch with us if someone needs to report a security issue.
Jeffrey
on 09 Sep 09Do you still store user’s passwords in plaintext, instead of a password hash? (Source: http://www.jgc.org/blog/2009/05/can-you-trust-37signals-with-your.html)
Do you still consider the disclosure of security vulnerabilities to be “feature requests”, as Thomas Ptacek of Matasano Security reported?
JF
on 09 Sep 09Jeffrey: Once we move to 37signals Accounts we will be hashing passwords. 37signals Accounts represents a complete rewrite of our username/password system done right.
We don’t consider the disclosure of security vulnerabilities to be feature requests. Our support team has been re-instructed not to treat them as such. That was more a lapse of proper training of our support folks (I’ll take the heat for that) than commentary on internal attitudes about security reports.
Jeppe
on 09 Sep 09Jason, great handling and great post. This is yet another example of why learning from your failures is important. No disrespect :)
Kevin
on 09 Sep 09Also no disrespect intended and please correct me if I am wrong, but I didn’t see an apology.
Personally, I don’t think an apology is needed. Your actions for handling any future issues is what I would expect. Good post.
FJ
on 09 Sep 09My experience with 37Signals products has been overwhelmingly positive in terms of features, layout and general workflow philosophy.
However, my experience with 37Signals support has been consistently negative, varying from the rudely dismissive to the canned useless.
While I empathise with growing pains, and the difficulties of setting up a proper security reporting process, I do hope this recent grilling will lead to a greater review of company policies: beyond the security reporting process, it is the entire dialogue between company and clients that needs to be revised.
Calling 37Signals often feels like calling Cox Digital Cable, which is a pity considering the quality of the products.
JF
on 09 Sep 09FJ, sorry you had a bad experience with our support team. Can you email me directly and tell me about specific exchanges that have given you this impression? I definitely want to know about it. jason@37signals…
Thanks.
Pablo
on 09 Sep 09I see the whole information more technical than the previous version. Some users might want to hear that their information is secured rather than “Sophisticated Physical Security”... what a mouthful.
sloan
on 09 Sep 09The problem I see here isn’t that there was no apology, but this is pretty basic security feature that JF says they were not doing. Honestly, I could not recommend their services at this point based on this post. Naturally, I won’t be using these services for anything confidential until it is proven that it is actually secure.
Gudbergur
on 09 Sep 09Hi, great post and as with many posts here I agree with your approach. Staying honest without exaggeration or hype is the way to go. On my blog i posted a funny screenshot from IE7, hope you’ll check it out – http://gudbergur.wordpress.com/2009/09/09/ui-ie7-information-bar/.
Anonymous Coward
on 09 Sep 09Ok so if the new version is more honest… how does storing passwords in plaintext relates to this phrase (from your new statement): “We work with security researchers to keep up with the state-of-the-art in web security”
I don’t intend to troll, but you are not being as honest as you claim.
Laurent
on 10 Sep 09As a response to the Anonymous Coward…
The phrase “We work with security researchers to keep up with the state-of-the-art in web security” doesn’t imply that they are always at the state-of-the-art in web security. It simply says that they are always working on it.
According to the previous response from JF, they are actually working on the password issue you mentioned, so that statement is true, even if you want to imply otherwise…
AtleastBetterThanNothing
on 10 Sep 09Staying ‘honest’ with exaggeration and hype is the way to go
JF
on 10 Sep 09@sloan: If you don’t think we’re secure somewhere, and you have a security issue to report, we’d like to know about it. That’s why we set up the security response page.
kb
on 10 Sep 09I think its a psychological matter, too. When somebody reports a security vulnarability, then you feel hit yourself. Obvious your product is not so perfect as you think it is. Thats a possible reason why, for myself, can explain why the communication with somebody who reports a security issue is bad.
zorg
on 10 Sep 09For a bit of a perspective on this: http://brian.mastenbrook.net/display/36
Ben
on 10 Sep 09Have to agree, plain text passwords is not acceptable. I’m surprised 37signals would even be there, and in 2009 no less. This is security 101 and has been for decades.
JC
on 10 Sep 09Am I the only one with a problem with the expression “24/7/365”, surely it should be 24/365 or 24/7/52 – unless there are now 61320 hours in a year!
Daniel Tenner
on 10 Sep 09I don’t think password hashing is a security feature. It’s a “keep the users happy” feature.
Unhashed passwords is only an issue for you if you’re using the same password across multiple sites. And if you’re doing that, then you’re the security issue, not the systems you’re using.
It’s a shame that most people jump on the “passwords should be hashed” bandwagon without much thought, which means every web app out there (mine included) needs to hash passwords to keep them happy. A lot of time wasted building a relatively pointless feature.
Don’t use the same password across multiple sites (which should be a basic security practice, the online equivalent of brushing your teeth). Then you won’t care about hashed passwords anymore.
math0ne
on 10 Sep 09I just wanted to chime in on this and sort off address a couple of the things Danial said above.
I was initially concerned when i heard about the fact that 37signals doesn’t hash their passwords and I’m quite frankly astonished that it’s still an issue to this day.
My security concerns about this specifically relate not to my own security but the security of my clients. People using the same passwords on multiple sites is a fact. I don’t personally do it, but I can almoast guarantee that the majority of my client, which have accounts with 37signals do.
All it takes is one devious engineer at 37signals to compromise the security of my clients. The purpose of hashing passwords, in my opinion, is not to protect the passwords from outside of the organization but to protect the passwords from being misused within the organization.
Hannes
on 10 Sep 09WTF? You store plain passwords? WTF? That is just… I am speechless.
Within a sec all the things you’re saying on your security site become happy talk. Damnit.. I’m hurt – love your products btw…
MI
on 10 Sep 09math0ne: At some point you have to trust the vendors you use. Hashed passwords or not, if your service provider wants to steal your password they can — they generally have access to the code that accepts the unencrypted password before it gets hashed, after all.
That said, I completely agree that hashed passwords are a better solution, and we’re actively working on it, but it’s not a panacea.
Colin Prince
on 10 Sep 09You say “Full redundancy for all major systems”.
I’d drop the Full; just say your systems are redundant.
JP
on 10 Sep 09MI:
The issue of hashing passwords has little to do with corporate trust. I mean, if I’m already giving you my credit card number I must trust you somewhat. It has more to do with the concern of your servers being hacked. Given that a lot of people use the same password on multiple sites, I’m surprised this was ever an issue.
Here’s the reality of the situation. You guys spend so much on design, and your products are great… but that won’t mean shit if your server ever gets hacked.
I tweeted about this in June, http://twitter.com/jprichardson/status/2099023024 . I’m surprised it’s taken this long for this issue to come to light.
-JP
Greg
on 10 Sep 09@MI
But that’s the point we are all trying to make related to hashed passwords. It’s been a known issue with 37signals since day 1.
Implementing hashed passwords is EXTREMELY easy and yes, it is a best business practice.
What we are all trying to communicate to 37signals is that if simply things like hashed passwords are not being performed … how can we “trust the vendor [you]” with our data?
It goes along the lines of thinking, wow – hashed password is so basic … what else is 37signals not doing?
How this security threat was handled only puts the nail in the coffin for me. Sorry guys but you’ll be seeing a canceled account in the next week or so once I migrate my data off of your network.
Micheal
sloan
on 10 Sep 09@math0ne you are referencing what my concern is. no one should be able to tell you your password. it is just a bad base practice and honestly any book or quick search about password security gives you quick and easy solutions for it that are better than nothing. for those saying that you have to trust sites you use, there is trust, and there is bad security. which leads me to… @JF the point for me is that at this point, do we need to send a security checklist and have you guys check it off? do we need to give you a plan for how to build secure software? to rebuild trust I think you have to be more transparent than “working with…” and then saying ask us if you have any concerns. given what has happened, I think your company should do more than that and put together a list of all the things you are doing to rectify security holes like this and harden your systems. I appreciate the open lines of communication, but I feel the ball is more in the court of your company and you are in the unenvious position of having to prove yourselves in the realm of security.
TL
on 10 Sep 09Really guys? I just looked at the new security response page and here is my opinion. Offering a “public key” to send an email puts another, how do I say, step or potential for confusion between the client and you. I guess you are assuming that people that would report security problems know how to use the key and that their email software even supports that.
Why not make that page SSL and provide a security response form in line? Not only would that be easier for the user but it would show that you take this seriously.
JF
on 10 Sep 09TL: The public key is for security researchers. This is their recommendation for submitting security reports with sensitive details about exploits.
We will also look into providing a secure form for other folks.
BJS
on 10 Sep 09Do the recent fixes cover the security breach shown in this video http://evilpacket.net/2009/jul/9/basecamp-one-wrong-click/ ??? This was not an issue with ROR. This guy did raise an eyebrow.
Chris Carter
on 10 Sep 09I would say go beyond “talking to security researchers”, I would send your developers to some security classes, workshops and conferences. Security is not something you install or that someone comes in and gives you. Security is an organizational mindset that has to permeate everything you do, otherwise you’re only as secure as your weakest system.
That’s the point of the outrage over non-hashed password. If you guys are still operating with essentially the same skillset and organizational mindset that led to a very amateur design like non-hashed passwords, and kept the same implementation in place for years, then you really haven’t done a whole lot extra. “Working with security researchers” doesn’t do anything if you don’t change your company’s mindset. And I mean that – your whole company, from the guy at the top on down has to orient their mindset toward security.
Does that mean a little inconvenience here and there? Yes. Does it mean adding a little extra paranoia into your daily life? Yes. However, security is not free, and again, not something you just buy. Firewalls are not security.
Francis
on 10 Sep 0912 ninjas can’t handle an onslaught of an army of customers. :)
JF
on 10 Sep 09Chris: Good suggestion re: classes and workshops. Do you have any recommendations?
JH
on 10 Sep 09BJS: Yes, the XSS issue identified by evilpacket.net was fixed soon after it was reported.
Thomas Ptacek
on 10 Sep 0937signals didn’t make up the idea that vulnerabilities should be submitted via PGP. That’s also Microsoft’s policy, Apple’s policy, and Google’s policy. Say what you will about their respective corporate cultures, Microsoft, Apple, and Google deal with a lot more inbound security advisories than 37signals ever will.
The reason you want messages PGP’d instead of submitted via SSL is that PGP keeps the messages encrypted at rest as well as in motion. The web form to submit security flaws doesn’t, without still more crypto code.
It’s also the case that security advisories tend to get discussed in email and forwarded between concerned problem, which is another problem PGP handles well and SSL handles not at all.
Shaz
on 10 Sep 09Guess what BaseCamp sends back when you try to recover your password? Yup, your plaintext password. Incredible.
Jim Gay
on 10 Sep 09@JF Since you’re talking about using meaningful words: “I’ll take the heat for that” is lame. It sounds like “well, I guess nobody else is going to step up and even though it wasn’t my fault, you may blame me.”
Something like this would be better, and shows that you mean it: “It was my responsibility and I failed.” Saying “I’ll take the blame” isn’t actually taking the blame.
Even if you weren’t directly responsible for certain actions, you’re responsible for the outcome of your team’s performance.
The re-written security page is much better.
@Jim Gay
on 10 Sep 09Christ, what damn difference does it make? He took the blame. That means he accepts responsibility. What’s ambiguous about that? Sheesh.
Chris Carter
on 10 Sep 09JF:
http://www.sans.org/
They’re one of the larger training organizations for security and can get you certified as well. In addition, it looks like they have Chicago workshop coming up at the end of October, too.
JF
on 10 Sep 09Thanks Chris, we’ll check it out.
Jim Gay
on 10 Sep 09Not sure how to reply to @Jim Gay, but like I mentioned: since we’re talking about meaningful words it does make a difference, just like the examples of the changed content on the security page.
Words matter. That is part of the point of this post.
Paul
on 14 Sep 09When I launch Basecamp in Chrome (https mode) I can see the message that there are some untrusted elements on the page.
anonymous
on 14 Sep 09Great changes, I feel much more secure with that wording. Everytime I see, 100% security, I read, we haven’t done anything, we don’t understand our security posture, but we still need to sell, so there you go, we are 100% secure. Now it’s clear you are committed to security, and you have spent enough time to understand that you are not 100% safe.
Vitezslav Valka
on 15 Sep 09Thank you guys, really interesting read. Keep up the good work. At Pixmac.com “Getting real” was a great inspiration since we started and now I see many other sites that just look like they’ve read your book :-) I’m proud to have it at home…
This discussion is closed.