You’re reading Signal v. Noise, a publication about the web by Basecamp since 1999. Happy .

Dragons on the far side of the histogram

David
David wrote this on 8 comments

Performance tuning is a fun sport, but how you’re keeping score matters more than you think, if winning is to have real impact. When it comes to web applications, the first mistake is start with what’s the easiest to measure: server-side generation times.

In Rails, that’s the almighty X-Runtime header — reported to the 6th decimal of a second, for that extra punch of authority. A clear target, easily measured, and in that safe realm of your own code to make it appear fully controllable and scientific. But what good is saving off milliseconds for a 50ms internal target, if your shit (or non-existent!) CDNs are costing you seconds in New Zealand? Pounds, not pennies, is where the wealth is.

Yet that’s still the easy, level one, part of the answer: Don’t worry too much about your internal performance metrics until you’ve cared enough about the full stack of SSL termination overhead, CDN optimization, JS/CSS asset minimization, and client-side computational overhead (the latter easily catching out people following the “just do a server-side API”, since the json may well generate in 50ms, but then the client-side computation takes a full second on the below-average device — doh!).

Level two, once reasonable efforts have been made to trim the fat around the X-Runtime itself, is getting some big numbers up on the board: Mean and the 90th percentile. Those really are great places to start. If your mean is an embarrassing 500ms+, well, then you have some serious, fundamental problems that need fixing, which will benefit everyone using your app. Get to it.

Keep going beyond even the 99th

Just don’t stop there. Neither at the mean or the 90th. Don’t even stop at the 99th! At Basecamp, we sorta fell into that trap for a while. Our means were looking pretty at around 60ms, the 90th was 200ms, and even the 99th was a respectable 700ms. Victory, right?

Well, victory for the requests that fell into the 1st to 99th percentile. But when you process about fifty million requests a day, there’s still an awful lot of requests hidden on the far side of the 99th. And there, young ones, is where the dragons lie.

A while back we started shining the light into that cave. And even while I expected there to be dragons, I was still shocked at just how large and plentiful they were at our scale. Just 0.4% of requests took 1-2 seconds to resolve, but that’s still a shockingly 200,000 requests when you’re doing those fifty million requests.

Yet it gets worse. Just 0.0025% of requests took 10-30 seconds, but that’s still a whooping 1,250 requests. While some of those come from API requests that users do not see directly, a fair slice is indeed from real, impatient human beings. That’s just embarrassing! And a far, far away land from that pretty picture painted by the 60ms mean. Ugh.

Finally, there was the true elite: The 0.0001%, for a total of 50 instances. Those guys sat and waited between 30 and 60 seconds on their merry request to complete. Triple ugh.

Dragon slaying

Since lighting the cave, we’ve already been pointed to big, obvious holes in our setup that we weren’t looking at before. One simple example was file uploads: We’d stage files in one area, then copy them over to their final resting place as part of the record creation process. That’s no problem when it’s a couple of 10MB audio files, but try that again with 20 400MB video files — it takes a while! So now we stage straight in the final resting place, and cut out the copy process. Voila: Lots of dragons dead.

There’s still much more work to do. Not just because it sucks for the people who actually hit those monster requests, but also because it’s a real drain on the rest of the system. Maybe it’s a N+1 case that really only appears under very special circumstances, but every time the request hits, it’s still an onslaught on the database, and everyone else’s fast queries might well be slowed down as a result.

But it really does also just suck for those who actually have to sit through a 30 second request. It doesn’t really help them very much to know that everyone else is having a good time. In fact, that might just piss them off.

It’s like going to the lobby of your hotel to complain about the cockroaches, and then seeing the smug smile of the desk clerk saying “oh, don’t worry about that, none of our other 499 guests have that problem… just deal with it”. You wouldn’t come back next Summer.

So do have a look at the far side of your histogram. And use actual request counts, not just feel-good percentiles.

Marketing around situations

Mig Reyes
Mig Reyes wrote this on 6 comments

Before he made The Simpsons, Matt Groening’s famous comics and illustrations graced the covers of Apple brochures. The writing inside—from 1989, mind you—still does a great job selling the Mac.


Instead of blanket marketing a one-size-fits-all message, Apple took the time to speak to every situation a person is in. If you’re feeling overwhelmed, the Mac is there to put order back in your life. If you’re unemployed, the Mac is there to help you chase a career. If you’re a habitual procrastinator, the Mac is there for your spark of productivity.
They’re listening to us, and our problems. Talk about empathy.
Plenty of marketing today, especially in software, is a package of feature sets, bells, whistles, and some boasting about how they’re better than the next guy. Perhaps there’s mention of “benefits,” sure, but we’re always left figuring out how a product is supposed to fit in our own lives.
Chances are, your product isn’t for everybody. It doesn’t have to be, either. Really listen to your audience—you’re lucky to have them. Instead of assuming what they need, ask where they’re coming from. How did they get to the point of finally asking your product for help? If you can figure that out, all of a sudden your marketing changes from “making sales” to “being there for friend.” That feels good.

Another Chapter

Basecamp
Basecamp wrote this on 13 comments

So, a few years ago Dana Brunetti at Trigger Street Productions (Social Network and Captain Philips) got in touch with us after reading our book, REWORK. He loved the themes and the overall story of how our company came to be.
Some time passed and we hadn’t heard anything… Until a week ago. We’re super-excited to let you know that Netflix Originals has decided to take on the project and turn our little book into a feature film!
Martyn Burke was brought on to write the screenplay and Gwyneth Horder-Payton will direct. Filming is supposed to start sometime this spring and we just got these beautiful promo posters in the mail.


It’s rare that companies exercise options like the one they had on this book and we really can’t say how unbelievably excited and flattered we are to have our story told in a whole new way.

Basecamp goes back to basics

Jonas Downey
Jonas Downey wrote this on 11 comments

If there’s one thing everyone can agree on, it’s that computers and user interfaces peaked in the 1980s. Everything was clear and simple, without all of our modern annoyances like portability, color, touchscreens, and the Internet.

With that in mind, we took a long, hard look at Basecamp. We asked questions like:

  • Is Basecamp as simple as it can be?
  • Should we make Basecamp less colorful?
  • What if Basecamp was stored on 800kb floppy disks?
  • When is Knight Rider starting?

So today, after months of work, we’re excited to announce that Basecamp is going back to basics. We’ve rolled out an update to all our users that’s available right now. Just visit Basecamp in your web browser, and type the code dogcow to get a sneak preview of our new direction. We think you’re going to love it.


Basecamp’s new system requirements:

  • Macintosh System 6 or higher
  • 512kb of RAM
  • 800kb floppy disk drive
  • 1200bps dial-up modem (recommended)

P.S. refresh your browser to exit the preview.

Basecamp network attack postmortem

David
David wrote this on 13 comments

As we detailed in Basecamp was under network attack, criminals assaulted our network with a DDoS attack on March 24. This is the technical postmortem that we promised.

The main attack lasted a total of an hour and 40 minutes starting at 8:32 central time and ending around 10:12. During that window, Basecamp and the other services were completely unavailable for 45 minutes, and intermittently up and down or slow for the rest. In addition to the attack itself, Basecamp got put in network quarantine by other providers, so it wasn’t until 11:08 that access was restored for everyone, everywhere.

The attack was a combination of SYN flood, DNS reflection, ICMP flooding, and NTP amplification. The combined flow was in excess of 20Gbps. Our mitigation strategy included filtering through a single provider and working with them to remove bogus traffic.

To reiterate, no data was compromised in this attack. This was solely an attack on our customers’ ability to access Basecamp and the other services.

There are two main areas we will improve upon following this event. Regarding our shield against future network attacks:

  1. We’ve formed a DDoS Survivors group to collaborate with other sites who’ve been subject to the same or similar attacks. That’s been enormously helpful already.
  2. We’re exploring all sorts of vendor shields to be able to mitigate future attacks even faster. While it’s tough to completely prevent any interruption in the face of a massive attack, there are options to minimize the disturbance.
  3. Law enforcement has been contacted, we’ve added our statement to their case file, and we’ll continue to assist them in catching the criminals behind this attack.

Regarding the communication:

  1. There was a 20-minute delay between our first learning of the attack and reporting it to our customers via Twitter and status. That’s unacceptable. We’ll make changes to ensure that it doesn’t take more than a maximum of 5 minutes to report something like this again.
  2. Although we were successful at posting information to our status site (which is hosted off site), the site received more traffic than ever in the past, and it too had availability problems. We’ve already upgraded the servers that power the site and we’ll be conducting additional load and availability testing in the coming days.

We will continue to be on high alert in case there is another attack. We have discussed plans with our providers, and we’re initiating new conversations with some of the top security vendors.

Monday was a rough day and we’re incredibly sorry we weren’t more effective at minimizing this interruption. We continue to sincerely appreciate your patience and support. Thank you.

Basecamp was under network attack this morning

David
David wrote this on 12 comments

Criminals attacked the Basecamp network with a distributed denial-of-service attack (DDoS) this morning. The attackers tried to extort us for money to make it stop. We refused to give in and worked with our network providers to mitigate the attack the best we could. Then, about two hours after the attack started, it suddenly stopped.

We’ve been in contact with multiple other victims of the same group, and unfortunately the pattern in those cases were one of on/off attacks. So while things are currently back to normal for almost everyone (a few lingering network quarantine issues remain, but should be cleared up shortly), there’s no guarantee that the attack will not resume.

So for the time being we remain on high alert. We’re collaborating with the other victims of the same group and with law enforcement. These criminals are sophisticated and well-armed.

Still, we want to apologize for such mayhem on a Monday morning. Basecamp, and our other services, are an integral part of how most of our customers get work done. While no data was compromised in this attack, not being able to get to your data when you need it is unacceptable.

During the attack we were able to keep everyone up to date using a combination of status.basecamp.com, Twitter, and an off-site Gist (thank you GitHub!). We’ll use the same channels in case we’re attacked again. If the attack does not resume, we will post a complete technical postmortem within 48 hours.

We want to thank all our customers who were affected by this outage for their patience and support. It means the world to us. Thank you.

Finding your workbench muse

David
David wrote this on 21 comments

Much intellectual capital is spent examining the logical advantages and disadvantages of our programming tools. Much ego invested in becoming completely objective examiners of productivity. The exalted ideal: To have no emotional connection to the workbench.

Hogs and wash. There is no shame in being inspired by your tools. There is no shame in falling in love with your tools. Nobody would chastise a musician for clinging to their favorite, out-dated, beat-up guitar for that impossible to explain “special” sound. Some authors even still write their manuscripts on actual type writers, just for the love of it.

This highlights the tension between programmers as either engineers or craftsmen. A false dichotomy, but a prevalent one. It’s entirely possible to dip inspiration and practice from both cans.

I understand where it’s coming from, of course—strong emotions often run counter to good arguments. It’s hard to convince people who’ve declared their admiration or love of something otherwise. Foolhardy, even. It can make other types of progress harder. If we all fell madly in love with Fortran and punch cards, would that still be the state of the art?

I find the benefits far outweigh the risks, though. We don’t have to declare our eternal fidelity to our tools for them to serve as our muse in the moment. And in that moment, we can enjoy the jolt of energy that can come from using a tool fitting your hand or mind just right. It’s exhilarating.

So much so that it’s worth accepting the limitations of your understanding. Why do I enjoy Ruby so very much? Well, there’s a laundry list of specific features and values to point to, but that still wouldn’t add up to the total sum. I’ve stopped questioning it constantly, and instead just embraced it.

Realizing that it’s not entirely rational, or explainable, also frees you from necessarily having to push your muse unto others. It’s understandable to be proud and interested in inviting others to share in your wonder, but mainly if they haven’t already found their own.

If someone is already beholden to Python, and you can sense that glow, then trying to talk them into Ruby isn’t going to get you anywhere. Just be happy that they too found their workbench muse.

At the end of the day, nobody should tell you how to feel about your tools (let alone police it out of you, under the guise of what’s proper for an engineer). There’s no medal for appearances, only great work.

Drawing Trouble

Nate Otto
Nate Otto wrote this on 13 comments

One of the fun aspects of illustrating for the new basecamp.com marketing site is getting assignments from Jamie, Jason, and Mig. One of my favorite briefs I got was “can you illustrate browser trouble?”

Upon getting art requests, I usually search to see if there’s a standardized image for the concept. In this case I didn’t find any imagery that rose to the level of iconic, or was particularly interesting or clear. I opted to start from scratch.

Hmmm, browser trouble

Broken windows, bugs, injuries, cracked screens, dizzy people? I drew a page of visual brainstorms and posted it on our Basecamp project.

Whoever assigns me a drawing—in this case, Jamie—reviews my explorations and then ask me to flesh out one of the directions. Jamie liked the guy with the computer head and suggested that there be a browser window with a frown face.

I drew up another page with color.

With that, Jamie got back to me with “Awesome.”

I waited to see how the image would be implemented. Working with the Basecamp design team is great for me because I’m not particularly strong in Photoshop or Illustrator—those guys are all ninjas. I like to draw, and that is how my time is spent most efficiently.

Usually I have no idea how the images are going to be used until they are implemented live on the website, which is fine by me. I like the surprise of finding a fully rendered web page with my drawings.

Hopefully it is a page you will never have to find.

Yelp's nice settings toggle

Jamie
Jamie wrote this on 5 comments

Yelp updated their Privacy Policy so now businesses can get some insight on their customers. The nice thing is Yelp gives you a preview of how your information would appear to these businesses.

You can Change Privacy Settings to toggle between how much info you want to reveal. As a Yelp user I’m not passive in this. I’m given a choice. I thought this was a pretty cool interaction—a nicely done modal.

The Category Moat

Ryan
Ryan wrote this on 15 comments

A few of us at Basecamp became fans of the “job to be done” framework taught by Clay Christensen, Bob Moesta and Chris Spiek. The core idea is that what you are selling and what people are buying are two different things. Understanding what people are trying to do with your product helps you know whether you’re getting hotter or colder as you consider changes to your product.

For example, we think we’re selling a project management product. But some people really use our tool and pay us every month to manage their clients. The projects were always fine—it’s the clients that are a challenge! That’s just one example.

Clay has suggested (eg. here) that when you identify what people truly use your product to accomplish, you protect yourself from competition. He’s a smart man, so when he says something odd like that I try to dig in. I’m starting to see what he means.

It’s natural to identify with a product category. You think “we make product management software” or “we make candy bars” because you have to explain yourself over and over. It’s always easier to use available categories than to invent new ones. It’s just like language. We speak the lexicon instead of inventing words.

But for people who want to innovate, this is a problem. Identifying with a product category is outsourcing your strategy to the past. Is the world really carved up into allowable product categories? No. We are all figuring this stuff out every day. Experience shows that amazing breakouts and surprise successes competed on unorthodox dimensions (see Blue Ocean Strategy for examples).

Bob tells the story of a clock maker. They sell an alarm clock for small kids who started sleeping in their own room. It’s not a normal alarm clock. It has an arrow that points to whether the kid is supposed to be in bed or whether he is allowed to get up. That way he doesn’t go running into his parents’ room until after a reasonable hour.

If you think this product is a clock then it’s in the clock category in the clock aisle with a clock price. But parents who bought the clock said they would pay $100 or more for it because it keeps the kid out of their room. It’s a sleep protector.

So how does thinking outside the category protect us from competition? I’ve been conducting interviews with Basecamp customers, and I’m feeling first hand how tricky it is to think outside of a category. You don’t have a shorthand. You don’t have words and feature lists given to you. It’s like you’re floating out in space with nothing to grab onto.

That’s the key. The fact that it’s so hard to think outside of a category is the moat. Staying focused on why you made the features you did, what specific situations call for them, and how that combo creates progress for people requires diligence and confidence and unyielding attention to actual behavior. Sticking to the truth of the matter instead of the walls of a category keeps you on your own path and away from the pitfalls of conventional thinking. That’s hard to compete with.