Last week we were dragged over the coals — and rightfully so — for failing to communicate clearly regarding a security exploit that was discovered in early August.
It turned out that the issue was related to Ruby on Rails. The exploit affected multiple products (ours and others). After the initial investigation, and escalation to the Rails security team, the root cause was patched within a few days of the initial report. We then updated our apps, tested, and deployed updates.
Fixed promptly, communicated poorly
Problem was, we did a terrible job communicating from start to finish. We didn’t communicate well with the person who initially reported the vulnerability, we didn’t communicate well internally, and we didn’t communicate well publicly.
Perfect security is a moving target. New exploits and security discoveries pop up over time. They occur in OSes, web browsers, frameworks, embedded systems, and commercial software. Anyone who’s in the software business has to deal with these issues from time to time. What matters is that issues are taken seriously, delegated properly, handled appropriately based on severity and priority, and communicated clearly with all parties involved. When someone reports a security issue, they’re reporting it because they want to help. It’s important for us to keep that in mind.
After ultimately reviewing what went wrong, we began to rework our internal process for dealing with security reports. This is a longer term project which we’ve just begun. Something we could do in the short term was review how we communicate publicly about security on our main security page on 37signals.com.
After a fresh slap in the face (which is definitely a healthy thing from time to time), it became clear that the words we were using were the wrong words. We weren’t setting the right expectations. Some of the lines were cringeworthy. It just wasn’t us. I don’t think any of us ever liked the way this page was written, but we never got around to changing it. Now was the time.
The old version
Phrases like “You’ll never lose important data” and “Your data won’t be compromised” and “You’ll always have access to your data” are easy things to say, but there’s no such thing as never, won’t and always in the world of security. Even government computers and banks with billion-dollar security systems can be compromised. Hyperbole may sell, but we don’t want to sell it.
What we can promise is that we take security seriously, that we’ve invested big in systems that keep data safe and secure, and that we’re continually focused on improving security. Internal audits, external audits, applying patches, following reports by security researchers, and keeping our ears open are just some of the ways we stay focused.
With that in mind, we spent some time rewriting the headlines, statements, and general customer-facing security messages. We called on some advice from a security company we’ve worked with in the past. The key points remain, but the language is better. We can stand behind these words.
The new, live version
Instead of “You’ll never lose” we say “We protect”. Instead of “Won’t be compromised” we talk about physical security (the old headline really didn’t pertain to the copy below the headline). Instead of “you’ll always have” we say “Redundancy for all major systems”. These are better words because they are more meaningful words.
In addition to revamping the standard security page, we also created a security response page. This page details the process for submitting a secure security report and explains how these reports will be handled. We will also be listing the names of the people who’ve reported vulnerabilities and helped us improve our security.
A continual process
Some steps we’ve already taken include assigning a dedicated security contact (before we had a few different people handle inbound security issues, but a single point of responsibility is better), creating a special address for reporting security issues (with a public key to keep those reports safe), establishing a better internal process for routing the issues to the right people, and implementing a better external process for reporting on issues.
Of course updating language on a marketing page doesn’t have anything to do with actual security, but the way we communicate has a lot to do with trust. We’ve made missteps before and we’ve gotten better each time. We’re not perfect, and we’ll never be perfect, but we’re still aiming for it. Thanks to everyone who’s gone out of their way to work with us to find, fix, and disclose security flaws.