Taylor’s post about our growth in 2011 included a bunch of numbers showing how the pistons inside the 37signals engine are pounding faster, but it all got swept away by what seemed like an innocent side-note: The 100 millionth file was called cat.jpg.
Being as it is that the internet is constantly accused of being just an elaborate way of sharing pictures of cats, sharing pictures of cats, we thought that was funny. But it wasn’t. We shouldn’t make jokes about anything even remotely related to people’s data.
Because the natural train of thought from there goes: Hey, if they saw the file name cat.jpg and shared it with the world, what’s to prevent them from sharing other data? Actual sensitive data, like Downsizing-Plans-2012.pdf? Hell, what if they’re actually looking at my secret new logo and leak it to the press?
That’s a completely legitimate train of thought to ride and it was our mistake to get it on track. So let’s start with first things first: We’re sorry. We made a mistake. We should have thought it through and remembered that storing your data with someone else in the cloud hinges on a fragile layer of trust. We poked that trust in the eye and it was wrong. We shouldn’t have checked the log files to see the name of the 100 millionth file.
So what’s a business to do from here?
Well, we could:
- Let heads roll, either through a public reprimand or even outright firing. Pinpoint blame, call it an isolated rogue employee, and dish out hard punishment.
- Form new institutional scar tissue and require that all future SvN posts would be reviewed by senior management and approved before posted.
- Try to cover up that it even happened by removing the remark from the original post, censor comments, or even delete the whole thing and pretend that would make it go away.
History is littered with examples of all three approaches, but none of them felt just, moral, or us. So we came back to our guiding principles on all things 37signals: What would we feel would be proper if we were wronged? First, apologize like a grown-up and own up to the mistake.
Second, how can we turn this mistake into a point of pride? A good place to start would be our privacy policy, which we basically haven’t touched in years. While not as bad as some, it’s surely not a page that has us radiating with pride. If I was a customer, I’d want to know 1) who can access my data, 2) under what circumstances will it happen, 3) what about law enforcement requests?, and the like.
So that’s what we’re going to do. We’ve started a project on Basecamp Next to reform the privacy policy to make it current, relevant, and as human as possible.
What would you like to see in it?
Ben Garvey
on 16 Jan 12How about transparency on security issues? If someone breaks in an potentially steals information, tell us about it.
How is the data stored? Plain text or encrypted?
Des Traynor
on 16 Jan 12Nice post – well handled.
If you’re interested, 500Px do a great job of being “as human as possible”. See the right hand column here: http://500px.com/terms
Tomas Sancio
on 16 Jan 12OK, the damage is done & blame is assigned. Can you now share with the rest of us the picture of the cat?
DHH
on 16 Jan 12Ben, that’s a good one. We haven’t had any system-wide breaches (knock on wood!), but if we were to have one, we’d definitely share that. There should be a point and a promise about that.
The database is encrypted when it’s backed up, but the live database that’s serving the site is not (since we have to send it unencrypted in order for you to read it). The transport is of course encrypted using SSL (we moved all plans to SSL, including free, a while back).
Jigar
on 16 Jan 12@Ben: +1
I never used to give importance to log files. I guess that in web apps, security of log files (both while storing or moving) must be given as much importance as storing passwords or other sensitive data.
Adding this to my 2012 resolutions. :-]
joe larson
on 16 Jan 12simply tweak the policy to say that any data containing the words “cat”, “teh”, or “lol” is considered to be marked “public” and may be freely shared and distributed by 37s.
Ben
on 16 Jan 12@DHH Thanks. I just want to make sure all my cat projects are safe. hahaha j/k Makes sense on the live database.
I think most users weren’t too concerned, but the privacy revamp is a good idea in its own right.
Erik
on 16 Jan 12“Bad” companies would go with #s 2 and 3 while most “good” companies would go for #s 1 and 2. Almost all, however, would follow #2 as a reflex because this one thing happened one time.
And then your blog posts get less interesting, and less relevant, and less frequent, until it requires a policy that the blog must be updated once per week…
Michael
on 16 Jan 12I’d like your privacy policy to say when employees are allowed to see customers’ account data. I’m fine with that being an extensive list (support, developers, sysadmin at minimum would all see it) but I’d like to know.
Matthew
on 16 Jan 12Why is the fact that the name was cat.jpg even an issue? People are SO SENSITIVE for the DUMBEST REASONS. Obviously if the filename was “super-secret-project-name-for-well-known-company.ppt” they wouldn’t have shared it.
Tim Grimsditch - Six3
on 16 Jan 12Evernote set the bar for clear privacy policies, and really helped us. Here’s a blog post on our experience:
http://www.kernelmag.com/features/building-a-company/668/should-we-worry-about-user-privacy/
Piskvor
on 16 Jan 12@joe larson: Very useful, especially when this leaks “Patent appliCATion #12345.pdf” to the world at large.
Lukas Rieder
on 16 Jan 12You can’t do anything better than apologizing and taking the chance to improve.
“Ben Garvey” asks legitimate questions, but I’d say he’s wrong about “stealing” information. You cannot steal information, more likely is that you can get unauthorized access to information.
I think this is important, because the blame is to put on companies that leak information instead of people who achieved access to that information.
In terms of a traditional robbery, the blame is to put on the robbers.
Therefore public understanding of such privacy issues would be much better.
Just my 2cents.
Deltaplan
on 16 Jan 12I am unsure if the uproar came from the fact that they had shared the “cat.jpg” filename, or rather by the sole fact that people have realized that 37signals employees could have access to some of their “private” data.
Which leads me to a point : when using some cloud application, you should always remain aware that the people behind it can have access to what you store into it. Regardless of the privacy policies, you should always know that if you store sensitive information, you have to acknowledge that some people will still be able to access it. Having them promise you to follow the strictest privacy policy is simply no guarantee that no leak will happen : all applications have bugs, all companies may hire someday a rogue employee that won’t comply with the companies’ policies, not to mention your own (potentially tragic) mistakes (like forgetting to restrict access to a sensitive file, or putting it in the wrong project, etc…)
Therefore, we, users, are ultimately the only ones who are able to make security decisions about our data. If the disclosure of some information would put your business at risk, then don’t store it anywhere in a readable form. Use strong encryption, choose your filenames carefully… don’t simply rely on any promises from an application provider to keep your data secure.
Bill Wilson
on 16 Jan 12Meh, I gave up trusting this blog and 37signals when they went political with their postings about a year ago. The previous posting just reinforced that they’ll use your data however they want, especially if it makes their political point or benefits their own business.
They can change all the worthless ‘privacy’ statements they want, but they lost my trust when they declaring themselves leftists on this blog. Every leftist I’ve ever known has situational ethics that say they can do what they want as long as it doesn’t hurt themselves, your stuff be darned.
When I saw the previous post, I knew it was typical. When I saw the apology, I knew they realized they’d hurt themselves, and suddenly the situational ethics kicks in (oops, my behavior hurts me now, not just someone else, so I should stop), and they backtrack.
Morality means thinking of others regardless of the consequence to yourself. That attitude doesn’t exist at 37signals.
I still read these postings because of the non-political design ideas, but I quit trusting them with my data back when the decided to write off half their market.
jurgen
on 16 Jan 12Maybe, just maybe, when the 100 millionth file was uploaded someone looked at the log to see what time the post was uploaded to make a note of that and along side the log file was the name of the file. Had it been “corporatePasswords.xls” or something, I trust it would be ignored. But when you see something like “cat.jpg” next to the 100 millionth log entry, it is humorous, non-descript and a more interesting note than saying, “The 100 millionth file posting happened at 7:44am on September 13th, 2011”. It was a harmless human decision that doesn’t effect my perception of 37signals trustworthiness or credibility.
Bill Wilson
on 16 Jan 12You’d engender greater trust if you’d quit deleting critical comments from your postings. I recognize you have that right, but it certainly reinforces the idea that you have no desire to listen to honest feedback.
Mark Frankel
on 16 Jan 12I like what you guy do in general, but being right so often makes it very hard for you to act appropriately when you’re wrong.
This is not about privacy policies and I think that’s an attempt to spin the real problem and an attempt to turn a bad act into a good “better privacy” cause.
What would be helpful, since you guys are so open, is show us the thinking, besides self-confidence, that led you to make an error like this.
Ben Garvey
on 16 Jan 12@Lukas I agree 100%. I should have used “unauthorized access.” I cringe every time I hear the phrase, “identity theft.”
Tom
on 16 Jan 12i’d like a way to download data similar to G+ or Facebook
George Gecewicz
on 16 Jan 12I think just a nice summary at the top of the page would be good. You’ve gotta have the verbose legal stuff, but something at top like:
That’s just gibberish but I think you get the point…Just condensed, maybe one-sentence long bullets of the main points of the Policy, then the full policy underneath it.
Mike
on 16 Jan 12At a way to own up to something…yes it was funny. When I read it again…I still think it’s funny. But I appreciate that you see not all think this way, and that there is a cost for the jab. Thanks for taking the 4th option. It’s an example of how to run a great business.
Edvinas
on 16 Jan 12Have not followed comments in post about that file, but maybe I will get answer here: was “cat file” owner notified about all the buzz, if so, how client felt about all this?
DHH
on 16 Jan 12Ben, we do not delete critical comments. Your leftist hand waving is right above jurgen’s comment for all to see.
Mark, I don’t think there’s a deep analysis available on why this happened besides what’s been posted already. It was the 100 millionth file and the file name was cat.jpg. That appeared funny and we posted it without thinking it through. We make mistakes like all humans but we try to own up to them when they happen and look for ways to improve.
DHH
on 16 Jan 12Edvinas, we didn’t actually trace who owned the cat.jpg file. That information doesn’t appear neatly in the log files (although you could cross-reference it if you were on a mission).
Rex
on 16 Jan 12Thirdly, how can we advertise our next product with the apology?
I didn’t really think it was that big of a deal…
Dave
on 16 Jan 12@Bill Wilson:
So you’re going to throw away what could be a profitable, useful business relationship over politics?
And people wonder why the US sucks so badly.
*Posted from Michigan!
Glenn
on 16 Jan 12I appreciate this post. When I read the original post I did notice it and it did raise a red flag for me, but I didn’t say anything because it seemed a fairly small thing…but there was a little twinge of doubt in the back of my head. Now this doubt has been erased because 1) you noticed it, 2) you were straightforward and 3) you took responsibility. Thanks!
Tom Andersen
on 16 Jan 12Why should customers data be on your servers. As an option, at least, they should be able to use your tools on their files with out your knowledge.
For example, is there a way to build your products so that they are ‘merely’ tools, in the way that MS can’t read the documents I write with Word? The web tool equivalent of this is that your tools operate by directly connecting with a users own files, hosted on say S3 or Rackspace, without going through your servers. Why should you need to see the files?
Mark Frankel
on 16 Jan 12David, thanks for the response.
Is it possible that if you did think it through, you would still post it because it’s funny? In fact on HN many people think you did nothing wrong based on the fact that it was just a cat.jpg file and no real harm was done.
I’ve learned a lot from you guys, so let me share one observation. There are many companies great companies, like Asana, entering your space for various reasons.
What worked well in the past, in terms of attitude, might not work as well in the future. A little more introspection and a examination of your well earned self-confidence might be just what the market ordered.
Bill Wilson
on 16 Jan 12@Dave, No, it’s thrown away over trust. How many times does 37signals have to prove they can’t be trusted before you move on?
Look, good for them for ‘learning’. But you know, for a person who lives to serve others, such an act would never have occurred to them, and the apology wouldn’t be necessary.
It’s the difference between understanding ethics by learning them, and having them fully integrated into your life and business so that you don’t make such a mistake in the first place.
By the way, there are places where you can go to learn ethics without having to first make the mistakes and costing you clients. They have these classes all over the country, and the meet for about an hour every week. On Sundays, usually.
Bill Wilson
on 16 Jan 12“What worked well in the past, in terms of attitude, might not work as well in the future. A little more introspection and a examination of your well earned self-confidence might be just what the market ordered.”
@MarkFrankel – Well said. I’m not discounting the quality of the product, only the moral integration of those who want to handle my data. And yes, in my experience, people whose politics lean left (as 37signals has declared themselves in their postings) tend to make these kind of ‘mistakes’ again and again.
You don’t have to do on-the-job training for morals and ethics. And I wish you wouldn’t with my data. You can learn it outside the work place.
Mark Frankel
on 16 Jan 12I just want to clearly state that my comments were in no way a political statement, just some advice for 37signals.
Sunir Shah
on 16 Jan 12While I don’t think there was great harm done, in this particular case, what was alarming was that 37signals didn’t recognize the mistake and said, “oops.” Instead, there was some initial deflection by DHH, which this post does a lot to overcome.
Privacy policies are only as good as your staff’s judgment, and sometimes there are slip ups. Given that there was no malicious intent, in this case the only expectation was to recognize the slip up in order to reinforce publicly and internally what’s expected of your people.
Why the huge outcry? The industry as a whole has a huge credibility problem with managing people’s private data. We all hear that from customers. Therefore, careful protection of private customer data is a touch point and for good professional reasons.
Ben Garvey
on 16 Jan 12@DHH doh, don’t throw me in with Bill Wilson.
Trevor
on 16 Jan 12@37signals
Securing your customer data is specifically why companies use Oracle Database.
Did you know that Oracle provides Database Vault. What it all allows you to do is set it up to prevent event DBAs from viewing or modifying data.
Idea being, DBAs should be able to “administator” the database, but should not be allow to either VIEW or even MODIFY customer/employee data (e.g. credit card #, SSN, salary data, etc..)
There is another product Oracle provides which is called Transparent Database Encryption . What it does is encrypt your customer data on disk, but then when a database select is issued – it unencrypts the data on the fly without needing to modify your application code.
Unfortunately, no such products like this exists for MySQL.
Given the size of your company now and how much sensitive customer data you are now storing, might be worthwhile for you guys to seriously consider using Oracle now.
Disclaimer: I’m employed by Oracle.
Rush
on 16 Jan 12You didn’t say what customer it was. And who gives a shit about a fucking cat picture? I would concentrate on stop being pussies in 2012.
Russell
on 16 Jan 12I cringed when I saw the original post say “it was a picture of a cat!” because that implies someone looked at it. But gleaming cat.jpg from the log files is much more innocent, almost downright harmless. Regardless, 37’s response to the issue is nothing short of ideal. Options 1,2,3 are too often the typical reaction companies make.
@Bill – Would rather 37s stay opinionated and true to themselves than try to give HJ’s to everyone. Owning up to imperfections is the best response you’ll get from a company. Most try to sweep it under the rug. A former employer of mine managed to leak client cpanel credentials to the web through a series of poor wiki security management choices. Not a peep was made to the customer, or on their blog. Truth is all people/ companies screw up, but few have the balls to be honest about it.
Bill Wilson
on 16 Jan 12It’s worth asking yourself: If 37signals had declared themselves conservatives, said that your fees would fund efforts to stop planned parenthood and support conservative causes, would it give you pause?
Then, if you overlooked it, and you found out that your personal data was accessible to conservative 37signals employees, and that someone had ‘innocuously’ described your data, what would you do?
When a company declares their politics openly, as 37signals did a year ago, no one should be surprised that people they don’t know will attach their own experiences and opinions about those political ideologies to that company.
Would you use this product if it were made by Haliburton? If it were purchased by Bain Capital? The Koch Brothers?
Let’s at least be honest with ourselves here.
Many of the ideas of 37signals are brilliant, and that’s why I keep them in my RSS feed. For instance, I applaud their stance against the anti-capitalism in the industry that says that all products must be free.
But I have experiences with leftists that prevent me from doing business with 37signals. This incident reinforced it.
DHH
on 16 Jan 12Trevor, I don’t really see how that helps things as long as we are able to log into accounts to debug them. The data will be decrypted at that point. Maybe such a split makes more sense in a large corporation where system administrators don’t even have access to the application level to help resolve issues
Ben, yikes, sorry! This was obviously directed at ethics-only-come-through-Sunday-church Bill.
@Ben Garvy
on 16 Jan 12Don’t worry, no one is throwing you in with Mr. Wilson. Your beer glasses are safe.
ploogman
on 16 Jan 12@DHH
Will you be like Switzerland used to be – i.e. not providing account info to the US government for bank accounts? (now they do)
Or would you freely share client and files info with a government agency request? Or would you wait until a court ordered it? And would you then comply?
Has that ever happened with 37? Just curious.
Good honest post BTW to address the cat. Forever known as the feline incident of 2012.
Dan
on 16 Jan 12One advantage to adding physical barriers between employees and user data is to remind employees when they are accessing user data. For a small company, perhaps it doesn’t have to be a VP-approval account permission that prevents you from providing good service, but it could still be a speed bump. Adding these to your infrastructure now would also prepare you for the possibility of getting large later, when the number of employees justifies more significant internal limits to access (assuming you agree it does).
I think this incident is an example of how easy it is to forget what counts as private user data when there are no doors employees need to open to access systems. A door could be useful even if it isn’t locked.
Bob Flowerdew
on 16 Jan 12@ploogman it should be known as “The Cat Flap”
GeeIWonder
on 16 Jan 12I agree that encryption seems a natural fit here. If that doesn’t fit the existing support/debug paradigm, it may be time for a paradigm shift.
Of course it may not be worth it, or that sort of level of privacy might just be incompatible with the sort of level of support/debug you seek to provide, which is fair enough.
Certainly it would seem a conversation worth having. Because an answer isn’t immediately obvious doesn’t really seem a good enough reason to throw the question out.
Respect for posting an apology though. Honestly.
Christopher B
on 16 Jan 12Good job on this post.
All in all, I don’t think any of us are worried you will leak a sensitive file name in a blog post. The concern is that we have no idea who has access to our files, and when they are allowed to look at our files, and what precautions or internal audits you have on your employees.
The company has a whole wouldn’t do something wrong but a rogue employee could and we just want to know that if someone does look at our information its because we requested it, and someone high enough up, and he/she has to go through a process to do so that is reviewed by your team.
At the end of the day we don’t know what happens in your office, not matter what you tell us may happen or should happen, so it’s all about trust. We trust in you, and we trust that you won’t put too much trust into any single employee.
Thank you again for this post guys.
mahyarm
on 16 Jan 12As I said on HN, it would of been fine if you just asked by email first and gotten that users permission and then said so, after the fact since you forgot to mention it or in the post itself if you remembered to mention it. “Your the customer who purchased item 100’000’000” is so common marketing wise that it doesn’t surprise people or raise eyebrows.
Nick
on 16 Jan 12I understand people’s fears, but truth be told they are just that: fears. Because nothing bad happened at all.
How many files named cat.jpg do you have on your servers? How many .jpg files do you have? How many c.*g files do you have? How many files do you have?
Now we can answer ‘at least one’ to all of those questions, but could you give precise figures? Would it be statistics or confidence betrayal? You gave an answer to the last question and I wouldn’t see anything wrong if you published exact answers to all of them.
The problem, I think, is that it is this exact cat.jpg, the one and only. And you COULD look at it and you COULD find out who uploaded it. So everybody felt like a lottery player where the prize is privacy invasion.
So, the problem arises when a piece of user data is singled out and somehow identified. I don’t know how to word it, but that would be a good clause for your privacy policy, maybe the only one making such situations violation of it in the future.
Joe Misiti
on 16 Jan 12“ethics-only-come-through-Sunday-church Bill” hahah
Miz
on 16 Jan 12How about we get access to data log specific to our accounts?
See who in your company viewed (even the file name) of our data.
I want to have access to this information.
DHH
on 16 Jan 12ploogman, we occasionally get requests from law enforcement to hand over user data and we categorically reject it every time. The only way they’re getting access is by having a court order that compels us to do it, which means that they have a to get a judge to sign off on it. Their request usually ends there.
Kevin
on 16 Jan 12David, this post is just another example that there are few, if any, companies other than you guys who can say:
1. We make good products our customers like to use. 2. We talk openly about everything we do. If you learn something from us, feel free to use it in your business. 3. We let all of our employees write unedited blog entries about what they’re working on and how they do what they do. Feel free to read and learn from them what you can. 4. We leave comments open so blog our readers and customers can participate in the discussion, even though doing so opens us up to both thoughtful criticism and negative trolls who will never like what we have to say. 5. When we screw up, one of our big cheeses will apologize like a human and own up to it. And that apology from a human won’t be written by someone in a corporate communications office.
Thanks for doing what you guys do and for setting such a great example.
Shane
on 16 Jan 12All this over looking at the filename! This was honest and introspective, you guys do a really good job. Mistakes will be made, own them, learn from them and move forward.
BTW: This sort of thing is why I recommend you to everyone that asks…
Bill Wilson
on 16 Jan 12@DHH – Yes, mock church-goers. You’re really winning customers there.
Not Bill Wilson
on 16 Jan 12@Bill I don’t know who you are trying to pick a fight with here. No one seems to be biting
Screw Bill Wilson
on 16 Jan 12@Bill Wilson: I’m a conservative and you’re embarrassing me.
I want to make very clear that Bill Wilson is a nut job and he does not represent the entire right. I do not believe, at all, that ethics require religion; or that they’re even meaningfully correlated. It seems that good people always find ways to do good, and lousy people find ways to be lousy, no matter what.
Nor do I believe that people who support slightly larger social safety nets than I prefer are evil folks who care nothing for others. I tend to disagree with where they draw the lines, but I’ve seen nothing indicating any sort of lack of ethics.
Bill Wilson is something that all parties have, and that are a bit embarrassing: he’s a nut job. He’s one of those our team versus your team assholes who thinks that the other team has no redeeming qualities when honestly, we all have a lot more in common than we have held differently.
I’m sorry that Bill Wilson is choosing to make a mockery out of conservatism. Please know that he does not represent the right. He is a lone looney-bin, whose irrational hatred serves only to embarrass those of us with working heads and working hearts.
It’s clear you weren’t attacking people who go to church, but rather you were attacking assholes who claim that you NEED to go to church to have ethics.
Sadly, Bill Wilson’s thin skin and small brain weren’t capable of determining the difference.
Bill Wilson is an embarrassment to all Christians, to all Republicans and to all Conservatives. Hopefully he will go do something more useful with his time, rather than spend it making Christians and Conservatives look like fucking idiots.
Dozier Hudson
on 16 Jan 12At WeTalky, we posted our Privacy Policy on GitHub in markdown format in the hopes of encouraging community contribution and complete transparency. Every change is viewable.
Thomas
on 16 Jan 12I’ve just finished reading all the comments, and all I’ve got to say is don’t worry about people like Bill Wilson. He seems to be following this post rather closely, giving way to the fact that he’s not so much a unsatisfied, untrusting user, as most likely a jealous competitor or someone who has a vendetta against 37signals.
DHH & 37signals crew, rock on.
John
on 16 Jan 12@Thomas: not to mention the reality that virtually anything that is loved by some is hated by others. It’s difficult to evoke emotional response only on one end of the spectrum.
Peter
on 16 Jan 12All of Bill Wilson’s wonderful comments made me think about the positive side of censorship. Those thoughts made me wonder how you would respond in the following situation:
If you find someone is using your products for inappropriate or illegal projects (think child pornography or drug trafficking) how would you respond? Would you share information that you had with the proper authorities.
I am assuming you would find out through a media story, someone sending you an email, or stumbling upon it when your digging through customers confidential files for fun ;). Do you have any filters to monitor for this sort of thing?
P.S. I am not involved in any inappropriate or illegal acts. Just wanted to clear that up in case Bill wants to start accusing me of trying to legalize drugs.
Michael
on 16 Jan 12What was the political post? I don’t think 37signals is exactly run by Abbie Hoffman.
Matt
on 16 Jan 12I liked the way you put it in your post:
For me, it would be helpful if your privacy policy spelled it out in plain English, explicitly in the form of answering those questions.
Simon
on 16 Jan 12@Trevor
Interesting to have the classical “Enterprise” opinion. (and Oracle opinion, which is quite the symbol of the “enterprise”)
Yes ideally only the user should have access to its data.
But the solutions proposed by Oracle are not convenient.
I mean if you go the soft way (removing view permissions) this is just dumb as the DBAs can grant themselves any permissions.
So you go the “real” way (eg encryption). You impact performance, raise the complexity of debugging, and add the cost to setup & maintain the solution.
Another (important) cost is that you will need to segment your admin team. If you have only one team to do all administration (network & software) they will always have a way to break in. They are ADMINISTRATORS!
These costs are huge if you need to update your application all the time. But yes I guess it is a “good” choice for some Enterprise which doesn’t care about the cost & updating frequently.
There is another way (which follows the companies that actually live from their applications). That is to employ people they TRUST.
Simon
on 16 Jan 12@DHH
You made a mistake and recognize it, so for me it is fine.
Anyway I hope you are not coming with some huge EULA just to show you’ve done something ;-)
Matt Boynes
on 16 Jan 12The comment “it was a picture of a cat!” did make me take pause of its implications, and I do appreciate your honest reflection. We all make mistakes (all of us, Bill) and the best we can take from them is an opportunity to grow.
Here’s my two cents on what I’d like to see done as a result of this. I do trust you with my data because you have a heavy investment in the success of your company. I don’t trust new-comer employee xyz who doesn’t have that same investment. I’d ideally like to know that employees have zero access to any of my data unless by my request and/or you have software that monitors data access and automatically raises red flags. I’d like file names, user names, etc. to be encrypted in logs so they don’t appear in plain text to potentially tempt an employee to look further into them (e.g. “User tcook uploaded image iphone5.jpg”).
Again, thanks for addressing and not dismissing privacy concerns. It’s easy to think like @Rush about an innocent comments, but it’s far more valuable to your customers to show that you don’t take privacy implications lightly.
Jay
on 16 Jan 12@Bill Wilson
How did DHH mock all church goers? Wait, he didn’t. He just mocked you (and he even singled your name out) – for implying that one can only operate a trustworthy business if you have church taught ethics. We all know that’s not true, as I suspect do you too deep down.
The idea that the general institution known as “the church” is a completely trustworthy entity is laughable. Implying that the church, and those taught by it, are the only ones possible of running trustworthy institutions is also ridiculous. There are wonderful leaders in churches, and bad ones. Same outside of the church. If you aren’t sitting in your church pew with any more discernment than that, that is scary. The church ultimately is operated by flawed people, even those that have good intentions. Why do you think Jesus saved his most cutting observations and teachings for church leaders of his day? He was brutal to them, and for good reason – they were prideful, holier-than-thou hypocrites. Be careful you aren’t one of them.
And really… Why are we discussing this in a business blog? 37s politics or religious beliefs have nothing to do with what they are capable of as a business. They are working to get better, like all of us (or, apparently, most of us).
Tom Andersen
on 16 Jan 12I again ask – why does 37signals need to access my data? It seems that the 37 signals apps mostly (all?) fall into the ‘silo’ pattern – unlike say a LinkedIn or Facebook that needs to have one ‘big data place’, as there are interactions.
When I open vi and use it to edit a file, I don’t expect to have the authors of vi need to store the file for me. So don’t store my files.
What do you see as the value your company adds – to provide file security for users, or to give them tools to manage projects? The front page says ‘rely on our web apps’.
Along the route to this is to not use your own private data formats, but rather open ones like text, png, mp3, etc. Then store those files with companies who have a primary mission to store files for users.
Its not 37signals that is at fault here – its the whole ‘hundred silos’ problem – each web app company seems to want to store your files in its own special corner. So we have a mess with photos on Flikr and Picassa, word docs on Google Docs and Microsoft’s cloud, projects on 37 signals, etc. etc. All with terms of service. How is a company supposed to manage all that? It can’t – and so the LAN – VPN – office share continues to creak along.
The internet needs a hard drive.
http://bit.ly/xpaUhs
Leave Bill Alone
on 16 Jan 12Ok, ok – leave Bill Wilson alone.
He may be bigoted. He may be misguided. He may not appreciate the irony of his own statements. He may even be insane. But he still benefits from the design advice from 37signals, so he’s still one of us – like a drunken, racist uncle that we still have to pretend to like at Thanksgiving.
Curtis
on 16 Jan 12@37signals: In the original post you said “It was a picture of a cat!”. Now in this post, you’re saying “The 100 millionth file was called cat.jpg.” These two statements don’t exactly mean the same thing. Did you actually open the 100 millionth file and see that it was a cat or did you just read the log file and see that it was cat.jpg?
Pablo
on 16 Jan 12wow, you stored users passwords in plain text for years and now you’re all ecstatic about a random file called cat.jpeg?
Just wow.
David
on 17 Jan 12We are a publicly traded corporation and have been using basecamp for critical projects for the past three years. The seemingly silly cat comment triggered a review of the privacy policy relative to our legal and fiduciary obligations under SEC regulations and find ourselves in a bind. The issue is less about 37 signals disclosing a file name and more about the “it was a picture of a cat” – not the file was named “cat”.
SEC regulations require that we take all reasonable steps to secure data to prevent potential non-public disclosures of insider information. It’s not clear from DHH’s statements and definitely not called out in the privacy policy that our proprietary information is kept securely and not visible to 37 Signals employees that could use such information to trade or otherwise profit from our use of basecamp and campfire.
While we love basecamp – this “event” and DHH’s apology has had the effect of officially notifying us of a potential security issue with our data, documents and other “insider” information.
We admire collecting information from the community but we now are obligated by the SEC to put basecamp’s privacy policy through a full legal and regulatory review. While the community suggestions are great, the SEC is very specific and therefore having your attorney draft a rigorous privacy policy is the only way we ( and many of your public clients – either direct or indirect ) will be able to use basecamp without restriction.
Today our legal department put six limits on our corporate use of basecamp and campfire until there is clarity about security of information. These limits debilitate most projects using basecamp ( almost all ) and we have already requested new internal systems to fill in. I loath this return to the bad old days over a stupid cat comment – but the real issue is about clarifying what 37 signals security and privacy policy truly is and drafting it so it can be used to demonstrate appropriate due diligence. Unfortunately the SEC simply won’t take DHH’s or our word for it. There must be a policy and procedures to limit 37 Signals employee visibility to proprietary information and logs that document which employee have accessed what information should a breach of trust ( and law) occur.
We were wrong to assume this was in place but now that we have been informed that it isn’t – we cannot use basecamp or campfire until it is. That is going to be hell. Speed with thorough legal review is essential.
Blues4Free
on 17 Jan 12@David Good thing the SEC is keeping a really close eye on this and you guys. Otherwise they might find other things to do like investigate wall street investment firms that were fraudulently selling shitty mortgage bonds as AAA rated bonds or something stupid like that.
David
on 17 Jan 12Ha. Yeah we actually make real products in six states and employ 14,000. We spend a ridiculous amount of treasure complying with regulations. But it does appear that if we traded in smoke and mirrors – we would be beyond reproach.
Sadly, this is the way it is and in the end we will spend a month putting a new system in place and re-training simply so a lawyer can check a box – all because I made an assumption three years ago and that assumption was exposed by a silly cat flap and a potential risk confirmed by a well meaning apology.
Dan
on 17 Jan 12Bad news. There are other publicly traded companies listed on Basecamp’s marketing website, like Adidas. Will they be notified by SEC too?
ploogman
on 17 Jan 12@ David (from the publicly traded company)
-seems like an inappropriate place to address your perspective even it it is actually true
- and BTW all the wonderful rules and regulations and the SEC and the insiders appointed to high posts in the SEC and Treasury Dept and all the huge failures and dishonesty like Arthur Anderson, Enron, AIG, Banks, etc. “too big to fail” the truth is this… the markets are basically pure gambling now – a company that is doing well does not necessarily do well in the markets – the market makes choices beyond what individuals can research and beyond insider info that can be stolen – its worth very little – big investment holdings trade by computers doing no research, just running off algorithms and looking to make money at other’s expense – the stock market has been perverted and is not so different from going to a casino and gambling that was a bit of a rant but back to David…
the SEC is not targeting you because of your use of Basecamp – I don’t believe that and I am calling your bluff
- and every SAAS vendor can access your data and thousands of techs and sysadmins in data centers around the county can access data on servers in those data centers – they have to for maintenance purposes – so you can cut the crap and scare tactics
in terms of security, 37s’ move to their own equipment co-located was actually a big step in reducing the number of people with any access to the systems and user data, and it was probably a tough decision and more like a traditional company mindset that was ballsy for 37 to do
there are plenty of undercapitalized and understaffed operations providing “secure” SAAS – and 37 is not one of them
Mike Unwalla
on 17 Jan 12George Gecewicz wrote: You’ve gotta have the verbose legal stuff…
Why? Good lawyers can write clearly (www.clarity-international.net).
David
on 17 Jan 12@ploogman
Hilarious!
Inappropriate place – seems like DHH asked for input. My input is we need more than community input, we need 37 signals to get it right with professional legal input.
Indeed the SEC is not “targeting” us but it is the law and our access to the public capital market is dependent on compliance with the law. Are you actually suggesting that since some companies flaunt the law we all should? Bluff? Which bluff was that? SEC regulations – are you really suggesting we all ignore them even after your rant on Wall Street corruption? With all due respect, even with the hardship compliance creates, I don’t like the world you appear to suggest.
I question your premise about all Saas offerings – several use encrypted databases and others have take clear steps to protect client data. It obviously wasn’t clear to you – I am a big fan and champion of basecamp but true advocates are frank with their partners particularly when input is requested. Of course our lawyers and consultants will address this but from a user point-of-view, we need haste and a professional response that will restore confidence. Again, with respect, your dismissal do nothing to restore confidence in the real world.
On second thought, after reading other posts in this thread I agree with you. Sadly SVN does seem like the wrong place for the basecamp community to discuss challenges (or even 37 Signals posts ) because a few lack professionalism and experience. I took the time expecting suggestions on how to handle this – instead I got this. Sadly hilarious.
Regardless, I look forward to DHH restoring trust, definitely not you.
Mr FUD
on 17 Jan 12“We admire collecting information from the community but we now are obligated by the SEC to put basecamp’s privacy policy through a full legal and regulatory review.”
If you are so diligent about SEC regulations surely you have a policy of carrying out such legal reviews before you decide to use a third party online tool…
David
on 17 Jan 12@MrFud
Yes I messed up. We started using basecamp with a vendor and it was such a great experience it spiraled into many other projects and eventually internal project. But again, you are correct, we/I messed up.
@37 Signals
I’d like to correct one thing I said. I mentioned “other services that encrypt”. Encryption is not actually a requirement – not even close. We use several service that do not encrypt. Press releases for example are priviledged information that are shared with partners ( PRNewswire, WSJ, CNBC for example) prior to public release under the condition of an public embargo. For a period of time ( during the embargo ) employees of these services have access to insider information. This is all meets regulatory requirements because our partners and/or services have documented that they have trained their employees on how to handle client confidentiality. With this training, if a violation occurs, its an act of volition, not ignorance. This is baked into our service agreement – not privacy policy – and provides all the diligence required.
Just wanted to correct the impression that encryption is the only solution. In fact 30 minutes of documented employee training and modification of the service agreement is likely all that is required.
Ploogman
on 17 Jan 12@ David
You are hilarious and your “perspective” is FUD and competition to 37. The SEC does not restrict ANY company from using Basecamp just as they do not dictate what color paper clips you must buy. Many government agencies and large companies happen to use Basecamp and other Saas even though Basecamp targets the Fortune 5 million.
Also your comment about 37’s legal review was a little silly because 37 has their own lawyers etc. They have also published books. Check yourself please and stop the SEC scare tactic which is transparently phoney
Erich
on 17 Jan 12I think I’m going to be one of the only ones to come to @David’s defense and most likely will be excoriated, but here it goes.
I worked in a company that was regulated by the SEC, both because it was publicly traded and because it sold a products used in the trading of securities at almost all levels. In this case, it’s not that they were being targeted by the SEC as result of this incident, but rather that the legal staff got wind of the issue and “took action.” I have no problem believing that. Even if it’s not true, it’s completely and absolutely plausible. I would be surprised if there weren’t more cases of this going on among the Basecamp user-base, this morning.
It can be highly demoralizing environment for developers. What is rational and reasonable is not the rule of the day when it comes to compliance. The utterance of the compliance lawyers and the SEC can wipe away weeks or months of work or in this case, three years of established practice that has yielded much greater productivity.
It is reality, however, and wailing about the injustices of the CDO scandal and lack of oversight by the SEC is not at all productive and, ulimately, moot. The folks up the line (division managers, CEO, CCO, et. al.) stop caring and stop thinking rationally when the word ‘compliance’ comes up.
Ryan
on 17 Jan 12Wow. Very strong antagonism to David’s very valid points. I didn’t really care much about this “cat.jpg” incident myself, but David’s quite right that this is a serious matter for 37signals if they want people like him as a customer. How can some people here possibly suggest he ignores the law simply because other companies do so? He still has a professional reputation and business to maintain; the people associated with the recent mortgage swindle may not be in jail (yet) but there are still real and hard consequences associated with their behaviour (and some still to be played out).
Worse, David admitted making a mistake in not performing due diligence originally by having a proper review of 37signal’s privacy policy. Ironically this very blog post is about admitting mistakes and then doing the right thing, yet David is being criticised for now for doing just that!
BradM
on 17 Jan 12Isn’t this getting a little of out hand?
I do understand everyone’s concern, but wasn’t the filename ‘cat.jpg’ just logged? Meaning, they didn’t view the actual JPG of a cat?
Servers have log files. They log many aspects of a software program. If they didn’t, customer service would be practically non-existent I think.
It was logged as an uploaded file that just so happened to be called ‘cat.jpg’.
What is the big deal?
Joe
on 17 Jan 1237signals said “It was a picture of a cat”. If it weren’t for saying what the contents of the file were, there may not have been such an issue.
BradM
on 17 Jan 12I understand that, but I think it’s reasonable to assume that it was ‘A picture of cat’ when the file is called cat.jpg
If they said, it was a picture of a dog … well then I would speculate that they indeed viewed the contents of the file.
Nathan
on 17 Jan 12I agree BradM, it is definitely out of hand. But there are also valid points. As an accountant that has performed audits, I definitely understand where @David is coming from on the SEC regulations. It just goes to show how such an innocent comment can lead to a sh*t storm of after effects
If anybody is at all in disbelief of @David’s statements, please read the first paragraph of the following link. This is very real.
http://sas70.com/sas70_overview.html
Derek
on 17 Jan 12Probably not a bad idea for 37S to become SAS70 compliant, but this whole thing is ridiculous.
josh
on 17 Jan 12In my opinion SAS70 and now SSAS 16 (what replaced it) are fairly worthless from a real customer standpoint. All they say is that company X has a control in place and they say they follow it.
I wouldn’t want 37Signals to pay their accounting firm $25,000 more for that designation…
If they were going to throw money away, I’d rather have them spend money on real security.
Nathan
on 17 Jan 12@Josh… you are 100%, absolutely correct on this. Here’s the other problem I have with these type of audits: the person you’re auditing is the person that is paying you. Think that’s a conflict of interest? I sure do.
I have no doubts about 37s security. I’m an accountant and use basecamp just like other accountants. I trust it. I trust 37s.
Unfortunately though, 37s may have to go bring out the dog and pony show to please the “compliance guys”. At that point it’s just simple economics. If it cost you 25K to save a few hundred thousand (or millions) in business while improving the lives of your customers, you just do it.
I truly feel bad that this has turned into such a huge deal. It really shouldn’t have been. It was an innocent attempt at trying to add some personality to a blog post.
ploogman
on 17 Jan 12seems like “David” has posted as multiple people above – pretty lame – the Fortune 5 million don’t want to deal with things that are just pointless roadblocks – and the backdrop to his comments are that “legal” is getting involved, in other words that “lawyers” are the real problem – that joke is pretty old and worn folks
Would someone use Basecamp as the definitive way to track their own clients’ financial portfolio of stocks and bonds? Of course not, its not made for that, all 37 stuff is just to get shit done. Talk about “edge cases” – that would be totally off the edge. Period. They don’t facilitate insider trading or risk of info going to the wrong hands any more than a pad of paper that someone leaves on a subway by accident – only with 37 there is no pad of paper that can be left by accident – in other words, got something sensitive to keep? you can try and write it down and hide it but guess what is a million more times secure? Basecamp – and its not even made for that – again, its made to get things done and keep track of projects.
Martin
on 18 Jan 12Now we see the cat… .jpg!
Chris
on 18 Jan 12We have some big clients who simply will not use basecamp because you don’t provide enough security information.
In fact in the past, (you might even remember my persistence with one bank in particular) its been like getting blood out of a stone. These are the 4 main reasons basecamp failed to be adopted with some of our clients:
1/ You don’t allow remote Pen Testing 2/ You don’t provide certifications (SAS70II etc) 3/ You don’t provide information about at rest encryption 4/ Your site fails cross-site scripting vulnerabilities.
Glenn
on 18 Jan 12DHH. You did good. Now it’s time to post another couple posts and have this discussion and the over analysis and the crazy people go away.
God
on 18 Jan 12Apologee Assepted
Steve
on 18 Jan 12Really good site, please take a look at www.designyourown-steve.blogger.com It’s not very good at the moment as I’ve just made it and I’m only a teen, please email me your ideas so I can feature them in my blog, thanks.
Michael
on 18 Jan 12Chris, serious clients like that build their own internal tool. Otherwise they’re just trying to waste your time/their time. It’s not a problem.
Matt Lee
on 19 Jan 12You could install mod_removeip on your Apache servers, and not log IP address information.
David
on 22 Jan 12Im def reporting this to the media. Good luck. You are going to need it.
Pg
This discussion is closed.