Nearly every boss has said it. And just about every employee has heard it. Yet it’s one of the most meaningless lines ever spoken in the office:

“My door is always open.”

The statement usually is followed up with some version of, “If you ever have an issue with anything, please come talk to me.”

What’s wrong with this? Isn’t it important for your employees to know that you are open to hearing their suggestions, concerns, and criticisms? Of course it is.

But let’s be real here: In most cases, “My door is always open” isn’t really an invitation to speak up. It’s a cop-out. It makes the boss feel good but puts the onus entirely on the employees. You might as well say, “You find the problems and then take all the risk of interrupting my day and confronting me about them.” How many people have taken you up on that offer?

Your employees have lots of opinions about everything—your strategy and vision; the state of the competition; the quality of your products; the vibe in the workplace. There are tons of things you can learn from them.

But how many of these ideas and opinions have you actually heard? A tiny fraction, I’d bet. The reality is that companies are full of things that are left unspoken. And even when they are out in the open, the CEO is almost always the last to know.

I like to think of myself as a leader whose door is always open. But I recently learned that an open door isn’t enough…

Read the rest of the article at Inc.com

Let’s take a break from the business news cycle. I love the news and I’ve covered it for a decade: what new startup is launching, why a stock price just moved, who’s being hired or fired. But there’s a whole universe of fascinating stories waiting to be covered about what’s old in business.

That’s why we’re launching The Distance, a new online magazine featuring original journalism about bootstrapped businesses that are at least 25 years old. If you’ve ever been curious about your favorite family-owned restaurant or that little shop on the corner, this is the publication for you. These businesses might not make headlines, but their owners have compelling stories about how they started, what they’ve learned, and why they keep doing it.

This is a heady time for people interested in great stories, whether it’s telling them or reading them. From newer sites like Vox and FiveThirtyEight and Narrative.ly to legacy media outlets that keep producing indispensable work (I still subscribe to two print newspapers), today’s readers have a lot of choices competing for their limited time. The Distance offers its own kind of storytelling – enjoyable reads about long-lasting businesses and the people behind them.

We’ll be publishing one story a month starting in May. We hope you find the companies of The Distance as interesting as we do and come back each month for more. (And while Basecamp is sponsoring the magazine, The Distance is editorially independent and we will not write about Basecamp customers.)

We’d also love to hear from you. If you know of any companies that would make good profile subjects, please let me know!

The annual Chicago Comics and Entertainment Expo is this weekend and we’re really excited to announce that Basecamp will be sponsoring designer toy maker, Shawnimals at the show!

Shawnimals uses Basecamp to make awesome stuff, from plush ninjas to recycled sketchbooks. The show runs from April 25th–27th, so come by booth 655 to say hi and pick up a limited edition Happy Sherpa plush!

Performance tuning is a fun sport, but how you’re keeping score matters more than you think, if winning is to have real impact. When it comes to web applications, the first mistake is start with what’s the easiest to measure: server-side generation times.

In Rails, that’s the almighty X-Runtime header — reported to the 6th decimal of a second, for that extra punch of authority. A clear target, easily measured, and in that safe realm of your own code to make it appear fully controllable and scientific. But what good is saving off milliseconds for a 50ms internal target, if your shit (or non-existent!) CDNs are costing you seconds in New Zealand? Pounds, not pennies, is where the wealth is.

Yet that’s still the easy, level one, part of the answer: Don’t worry too much about your internal performance metrics until you’ve cared enough about the full stack of SSL termination overhead, CDN optimization, JS/CSS asset minimization, and client-side computational overhead (the latter easily catching out people following the “just do a server-side API”, since the json may well generate in 50ms, but then the client-side computation takes a full second on the below-average device — doh!).

Level two, once reasonable efforts have been made to trim the fat around the X-Runtime itself, is getting some big numbers up on the board: Mean and the 90th percentile. Those really are great places to start. If your mean is an embarrassing 500ms+, well, then you have some serious, fundamental problems that need fixing, which will benefit everyone using your app. Get to it.

Keep going beyond even the 99th

Just don’t stop there. Neither at the mean or the 90th. Don’t even stop at the 99th! At Basecamp, we sorta fell into that trap for a while. Our means were looking pretty at around 60ms, the 90th was 200ms, and even the 99th was a respectable 700ms. Victory, right?

Well, victory for the requests that fell into the 1st to 99th percentile. But when you process about fifty million requests a day, there’s still an awful lot of requests hidden on the far side of the 99th. And there, young ones, is where the dragons lie.

A while back we started shining the light into that cave. And even while I expected there to be dragons, I was still shocked at just how large and plentiful they were at our scale. Just 0.4% of requests took 1-2 seconds to resolve, but that’s still a shockingly 200,000 requests when you’re doing those fifty million requests.

Yet it gets worse. Just 0.0025% of requests took 10-30 seconds, but that’s still a whooping 1,250 requests. While some of those come from API requests that users do not see directly, a fair slice is indeed from real, impatient human beings. That’s just embarrassing! And a far, far away land from that pretty picture painted by the 60ms mean. Ugh.

Finally, there was the true elite: The 0.0001%, for a total of 50 instances. Those guys sat and waited between 30 and 60 seconds on their merry request to complete. Triple ugh.

Dragon slaying

Since lighting the cave, we’ve already been pointed to big, obvious holes in our setup that we weren’t looking at before. One simple example was file uploads: We’d stage files in one area, then copy them over to their final resting place as part of the record creation process. That’s no problem when it’s a couple of 10MB audio files, but try that again with 20 400MB video files — it takes a while! So now we stage straight in the final resting place, and cut out the copy process. Voila: Lots of dragons dead.

There’s still much more work to do. Not just because it sucks for the people who actually hit those monster requests, but also because it’s a real drain on the rest of the system. Maybe it’s a N+1 case that really only appears under very special circumstances, but every time the request hits, it’s still an onslaught on the database, and everyone else’s fast queries might well be slowed down as a result.

But it really does also just suck for those who actually have to sit through a 30 second request. It doesn’t really help them very much to know that everyone else is having a good time. In fact, that might just piss them off.

It’s like going to the lobby of your hotel to complain about the cockroaches, and then seeing the smug smile of the desk clerk saying “oh, don’t worry about that, none of our other 499 guests have that problem… just deal with it”. You wouldn’t come back next Summer.

So do have a look at the far side of your histogram. And use actual request counts, not just feel-good percentiles.

Before he made The Simpsons, Matt Groening’s famous comics and illustrations graced the covers of Apple brochures. The writing inside—from 1989, mind you—still does a great job selling the Mac.

Instead of blanket marketing a one-size-fits-all message, Apple took the time to speak to every situation a person is in. If you’re feeling overwhelmed, the Mac is there to put order back in your life. If you’re unemployed, the Mac is there to help you chase a career. If you’re a habitual procrastinator, the Mac is there for your spark of productivity.
They’re listening to us, and our problems. Talk about empathy.
Plenty of marketing today, especially in software, is a package of feature sets, bells, whistles, and some boasting about how they’re better than the next guy. Perhaps there’s mention of “benefits,” sure, but we’re always left figuring out how a product is supposed to fit in our own lives.
Chances are, your product isn’t for everybody. It doesn’t have to be, either. Really listen to your audience—you’re lucky to have them. Instead of assuming what they need, ask where they’re coming from. How did they get to the point of finally asking your product for help? If you can figure that out, all of a sudden your marketing changes from “making sales” to “being there for friend.” That feels good.

So, a few years ago Dana Brunetti at Trigger Street Productions (Social Network and Captain Philips) got in touch with us after reading our book, REWORK. He loved the themes and the overall story of how our company came to be.
Some time passed and we hadn’t heard anything… Until a week ago. We’re super-excited to let you know that Netflix Originals has decided to take on the project and turn our little book into a feature film!
Martyn Burke was brought on to write the screenplay and Gwyneth Horder-Payton will direct. Filming is supposed to start sometime this spring and we just got these beautiful promo posters in the mail.

It’s rare that companies exercise options like the one they had on this book and we really can’t say how unbelievably excited and flattered we are to have our story told in a whole new way.

As we detailed in Basecamp was under network attack, criminals assaulted our network with a DDoS attack on March 24. This is the technical postmortem that we promised.

The main attack lasted a total of an hour and 40 minutes starting at 8:32 central time and ending around 10:12. During that window, Basecamp and the other services were completely unavailable for 45 minutes, and intermittently up and down or slow for the rest. In addition to the attack itself, Basecamp got put in network quarantine by other providers, so it wasn’t until 11:08 that access was restored for everyone, everywhere.

The attack was a combination of SYN flood, DNS reflection, ICMP flooding, and NTP amplification. The combined flow was in excess of 20Gbps. Our mitigation strategy included filtering through a single provider and working with them to remove bogus traffic.

To reiterate, no data was compromised in this attack. This was solely an attack on our customers’ ability to access Basecamp and the other services.

There are two main areas we will improve upon following this event. Regarding our shield against future network attacks:

1. We’ve formed a DDoS Survivors group to collaborate with other sites who’ve been subject to the same or similar attacks. That’s been enormously helpful already.
2. We’re exploring all sorts of vendor shields to be able to mitigate future attacks even faster. While it’s tough to completely prevent any interruption in the face of a massive attack, there are options to minimize the disturbance.
3. Law enforcement has been contacted, we’ve added our statement to their case file, and we’ll continue to assist them in catching the criminals behind this attack.

Regarding the communication:

1. There was a 20-minute delay between our first learning of the attack and reporting it to our customers via Twitter and status. That’s unacceptable. We’ll make changes to ensure that it doesn’t take more than a maximum of 5 minutes to report something like this again.
2. Although we were successful at posting information to our status site (which is hosted off site), the site received more traffic than ever in the past, and it too had availability problems. We’ve already upgraded the servers that power the site and we’ll be conducting additional load and availability testing in the coming days.

We will continue to be on high alert in case there is another attack. We have discussed plans with our providers, and we’re initiating new conversations with some of the top security vendors.

Monday was a rough day and we’re incredibly sorry we weren’t more effective at minimizing this interruption. We continue to sincerely appreciate your patience and support. Thank you.

Criminals attacked the Basecamp network with a distributed denial-of-service attack (DDoS) this morning. The attackers tried to extort us for money to make it stop. We refused to give in and worked with our network providers to mitigate the attack the best we could. Then, about two hours after the attack started, it suddenly stopped.

We’ve been in contact with multiple other victims of the same group, and unfortunately the pattern in those cases were one of on/off attacks. So while things are currently back to normal for almost everyone (a few lingering network quarantine issues remain, but should be cleared up shortly), there’s no guarantee that the attack will not resume.

So for the time being we remain on high alert. We’re collaborating with the other victims of the same group and with law enforcement. These criminals are sophisticated and well-armed.

Still, we want to apologize for such mayhem on a Monday morning. Basecamp, and our other services, are an integral part of how most of our customers get work done. While no data was compromised in this attack, not being able to get to your data when you need it is unacceptable.

During the attack we were able to keep everyone up to date using a combination of status.basecamp.com, Twitter, and an off-site Gist (thank you GitHub!). We’ll use the same channels in case we’re attacked again. If the attack does not resume, we will post a complete technical postmortem within 48 hours.

We want to thank all our customers who were affected by this outage for their patience and support. It means the world to us. Thank you.

Much intellectual capital is spent examining the logical advantages and disadvantages of our programming tools. Much ego invested in becoming completely objective examiners of productivity. The exalted ideal: To have no emotional connection to the workbench.

Hogs and wash. There is no shame in being inspired by your tools. There is no shame in falling in love with your tools. Nobody would chastise a musician for clinging to their favorite, out-dated, beat-up guitar for that impossible to explain “special” sound. Some authors even still write their manuscripts on actual type writers, just for the love of it.

This highlights the tension between programmers as either engineers or craftsmen. A false dichotomy, but a prevalent one. It’s entirely possible to dip inspiration and practice from both cans.

I understand where it’s coming from, of course—strong emotions often run counter to good arguments. It’s hard to convince people who’ve declared their admiration or love of something otherwise. Foolhardy, even. It can make other types of progress harder. If we all fell madly in love with Fortran and punch cards, would that still be the state of the art?

I find the benefits far outweigh the risks, though. We don’t have to declare our eternal fidelity to our tools for them to serve as our muse in the moment. And in that moment, we can enjoy the jolt of energy that can come from using a tool fitting your hand or mind just right. It’s exhilarating.

So much so that it’s worth accepting the limitations of your understanding. Why do I enjoy Ruby so very much? Well, there’s a laundry list of specific features and values to point to, but that still wouldn’t add up to the total sum. I’ve stopped questioning it constantly, and instead just embraced it.

Realizing that it’s not entirely rational, or explainable, also frees you from necessarily having to push your muse unto others. It’s understandable to be proud and interested in inviting others to share in your wonder, but mainly if they haven’t already found their own.

If someone is already beholden to Python, and you can sense that glow, then trying to talk them into Ruby isn’t going to get you anywhere. Just be happy that they too found their workbench muse.

At the end of the day, nobody should tell you how to feel about your tools (let alone police it out of you, under the guise of what’s proper for an engineer). There’s no medal for appearances, only great work.

One of the fun aspects of illustrating for the new basecamp.com marketing site is getting assignments from Jamie, Jason, and Mig. One of my favorite briefs I got was “can you illustrate browser trouble?”

Upon getting art requests, I usually search to see if there’s a standardized image for the concept. In this case I didn’t find any imagery that rose to the level of iconic, or was particularly interesting or clear. I opted to start from scratch.

Hmmm, browser trouble…

Broken windows, bugs, injuries, cracked screens, dizzy people? I drew a page of visual brainstorms and posted it on our Basecamp project.

Whoever assigns me a drawing—in this case, Jamie—reviews my explorations and then ask me to flesh out one of the directions. Jamie liked the guy with the computer head and suggested that there be a browser window with a frown face.

I drew up another page with color.

With that, Jamie got back to me with “Awesome.”

I waited to see how the image would be implemented. Working with the Basecamp design team is great for me because I’m not particularly strong in Photoshop or Illustrator—those guys are all ninjas. I like to draw, and that is how my time is spent most efficiently.

Usually I have no idea how the images are going to be used until they are implemented live on the website, which is fine by me. I like the surprise of finding a fully rendered web page with my drawings.

Hopefully it is a page you will never have to find.