There used to be a panicked feeling that would set in when we’d have any sort of outage or issue in Basecamp past — that stomach-dropping, heart-palpitating, sweaty-palmed feeling. But on November 8th when I awoke to a 6am text spelling out Basecamp’s downtime, I wasn’t worried. Before I finished reading the full text, I remember thinking, “Oh, they’ll have it sorted out before I can finish making coffee.” But as I continued reading and began to understand the estimated downtime to be at least two hours, my adrenaline hit.
The first thing I wanted to do was check on the support team. Were they in panic-mode? How sweaty were their palms? How many customers had they talked to already today? How close to capacity were they?
And by the time I received the alert and logged on (coffee brewing while I said Good Morning, thank glob for remote work), Basecamp had been in read-only for about 30 minutes, three times my prediction. Despite the stress of a lengthy downtime, knowing that we’d have a few hours of this status allowed us to settle in and accept our predicament. We had time to get into a flow and trust ourselves to talk our customers through this.
Really, what I realized when I logged on was that everything was absolutely under control on the support team. And of course it was: for the past two years, our team has been conducting crisis drills with each other. Once a month, we rotate responsibility for these drills and each person is responsible for coming up with their own style of drill. They’ve become quite the gif-filled, fun time! We work from a playbook (hosted on GitHub in case Basecamp is down) that acts as a living document we can update as-needed. We’re currently in the process of using our experience from the read-only outage to revamp and reassess the playbook to make it even more accessible, comprehensive, and succinct — no small task, mind you!
With my initial concern for the team quelled, I could then address my second concern: our customers. (Yes, team first and customers second: put your own oxygen mask on before you put it on your charge.) As I read through emails and tweets our customers sent us, I expected significantly more upset than we received. My expectation was that we had let down millions of people, prevented them working, created more undue stress in an already stressful life. While there were upset folks, it was only a handful: of the 2000 emails and tweets we responded to during those five hours of downtime, only a handful of people were upset. Everyone else sent us well-wishes, told us to take care of ourselves, thanked us for the decade of consistent uptime. Here are four of those emails we received:
I’ve been a Basecamp customer for most of my adult life across various projects and organizations. I just wanted to say that I wasn’t even 1% bothered by the fact that there was an outage. I appreciate your candor and earnestness in responding, and your commitment to total uptime is laudable. But you’ve more than earned some leeway. Chronic downtime is annoying and should cause outrage. But a mistake every year or two, let alone once a decade, should be given a pass. You all do beautiful work, and I’m very grateful to have access to Basecamp the vast majority of life’s hours. If you can, don’t be hard on yourselves!
I just want you to know that I appreciated the explanation for the down time. I am so used to having that issue with other software program[s] I use, that it didn’t even phase me. I just did something else for awhile and forgot about it. Thanks for the nice explanation, and you don’t need to worry about rebuilding my trust. Also, I’m really sorry about the fires!
Just wanted to let you know as a client I really appreciate the outage report and taking full responsibility. In all of the years I have used BC, it is the only time I can think of that there was ever an issue. So really you can pat yourselves on the back. Keep up the good work! I love BC and support is always amazing. There are many, many digital/cloud companies who could take a page from the Basecamp playbook! And hopefully everyone in the fire areas, are safe. No need to respond.
I want to reiterate what I said on Twitter. We’ve been with you guys since classic and while it was definitely a crazy time here, it shows us how much we rely on Basecamp to run our agency. And I know it was a way crazier time there! Thanks for your open, honest communication throughout it all.
The more I read from our customers, the more I realized how true it was that we’ve been building real relationships with them over the years. Our team has a refrain for when we’re busy that It’s just email, but we know that’s not the total truth — it’s people who care about us as much as we care about them. It was really wonderful to see that validated, as it made our very stressful day feel, for a lack of a better word, bittersweet. We ended the day feeling pride despite our adrenaline and exhaustion. It was a testament to the fact that we created an empowered support team that responds in a timely and human way, and so when we let our people down they were able to show us grace. For that, we are outrageously grateful.
As Basecamp sat in a read-only state, the Camp, Hill, and Woolsey fires were spreading throughout California, and several customers took the downtime as an opportunity to write in to spread well-wishes. Please consider donating to help families who survived the fires.
