Signal v. Noise

From 2:13am GMT March 13 / 9:13pm Central March 12 until around 4:10am GMT / 11:10pm Central, Basecamp 3 was mostly offline and Basecamp 2 unable to process file uploads and downloads, as our cloud storage provider had a severe, sustained outage.  We continued to have minor disruptions in service from 4:10am GMT / 11:10pm Central until everything was cleared at 6:53am GMT / 1:53am Central.

This is the second time in a week that I’m forced to write “I’m so sorry”. That’s incredibly painful. Both because it’s because we’re failing our customers for the second time in a week, but also because it’s showing us just how unprepared we’ve been as an organization to deal with these cloud challenges, despite our belief otherwise.

I’m not going to bother you with platitudes about “lessons to be learnt”, because I’ve already done that just a few days ago. This goes much deeper than just a few lessons. It has called into question our entire risk management and operational structure at Basecamp.

It’s also been a mighty fall. From reaching for 99.999% in uptime – the hallowed five nines! – we’re now scrambling for two of them. From riches of reliance to rags of shambles. To say this is humbling is an epic understatement.

We’re stopping all major product development at Basecamp for the moment, and dedicating all our attention to fixing these single points of failure that the recent cloud outages have revealed. We’re also going to pull back from our big migration to the cloud for a while, until we’re able to comfortably commit to a multi-region, multi-provider setup that’s more resilient against these outages.

I’m sorry. I’m really sorry (and ashamed).

Keep reading “Basecamp outage: When it rains, it pours”

Based on data from 597 people, the best ways to build trust as a leader aren’t what you think they are.

How do you build trust as a leader? The answer seems intuitive enough.

For many of us, we hold company off-sites and run team-building activities. Informal lunches, monthly social get-togethers, and one-on-one meetings are part of how we build trust at work.

We also thank our team publicly and give employee recognition for a job well done. And, we strive to be transparent with company information during all-team meetings.

These are among the most popular ways to build trust because they work… Right?

Wrong.

Keep reading “The 3 most effective ways to build trust as a leader”

Why are manhole covers round? How many golf balls can fit in a 747? Why are job interviews so terrible? In the latest episode of the Rework podcast, Aja Hammerly, a developer advocate at Google, talks about the drawbacks of common tech interview techniques like whiteboard coding and trivia questions, and shares her tips for improving the process by making it about discovering the candidate’s best qualities.

For further reading on interviewing, check out Aja’s post “There Is No Perfect Interview” and these posts by Jim Mackenzie of Basecamp’s Support team: “Interview or Interrogation?” and “Under Pressure.”

From 4:30am GMT March 7 / 10:30pm Central March 6 until 1:02pm GMT / 7:02am Central, Basecamp 2 and the search feature in Basecamp 3 were mostly offline due to a catastrophic network failure with our cloud provider. Both our primary network link, our backup network link, and several additional ad-hoc network links between critical services needed to run Basecamp 2 were forced offline, as the cloud provider sought to deal with underlying network problems they were having.

Both Basecamp 2 and the search feature in Basecamp 3 are now fully back online.

But this was one of the worst outages we’ve had in the history of Basecamp. We’re incredibly sorry about just how long and broad of an interruption this caused, especially for our European customers of Basecamp 2. We’re so very sorry about this. We know this caused real and deep interruption to many people’s workflow from the early morning to the early afternoon on the main European timezones.  And of course to any other customers around the world, including the US, who were also affected.

We’ve learned some hard lessons about network availability, the limitations of redundant, and double redundant backup connections. We’ll be working diligently to change how we work with cloud providers in the future, and how we can insulate ourselves and our customers from any future incidents like this. While this incident may have been triggered by network issues outside of our immediate control, it’s always within our control how we architect our systems, how we prepare for disasters, and how we ensure something like this never has the power to inflict such a traumatic outage.

So I want to make absolutely clear that this is our failure. Even in this new world of cloud services, it’s still always our fault when Basecamp isn’t available. Whatever the underlying problem for an outage, there’s always something you could have done to prevent it. And our list in this case includes a number of both obvious and not-so-obvious steps we could have taken. We will now take them.

Once again, I’m deeply sorry for this terrible outage. We will work as diligently we can to ensure that this doesn’t happen to any version of Basecamp again, neither past or present. Thank you for understanding, thank you for your patience, and thank you for being a customer, even if you with all justification ran out of both understanding and patience during this utterly unacceptable outage.

Keep reading “Basecamp 2 and Basecamp 3 search outage report”

CONSISTENCY

When looking at a single screen, the button shape and centering is consistent. Further, a primary button is called out using size, color, and placement – in line with interface guidelines.

CONFUSION

Stop up top on the Timer, stop down at the bottom on the Alarm. The Timer and Alarm designs look so similar that visual/muscle memory can lead you to tap the wrong button. Confusing!

CONTEXT

Ah, this is what you really want. A design steeped in context. It’s an alarm clock, and you often want to snooze one of those – especially early in the morning when you’re randomly smacking at things, hoping to hit the right thing. So make Snooze huge – no more hunting for that small button in a field of black. And make stop larger too. You could use this with your eyes closed as long as you know the shape of the phone, and which side is up/down. (Note: this is a conceptual design by Alex Cornell)

When it comes to delegating, invite your team into both the thinking and the doing.

Do you consider yourself “a doer”?

That person who enjoys doing the work, fine-tuning the details, meddling in the weeds of how it’ll all work? Then you probably have trouble delegating as a leader.

I know I do.

For so many managers and leaders — especially those of us who are used to be the person doing the work and are now handing off the work to others — learning to delegate is, well, tricky, if not painful.

Keep reading “Delegate outcomes, not activities.”

Basecamp 3’s to-do lists keep you in the loop when you’re working closely with other members of your team. You get notified when someone assigns you a to-do and that person gets notified when you check it off.

This works great when it’s clear who needs to be assigned, but that’s not always the case. Sometimes you don’t know who should do the work, other times a to-do isn’t for anyone in particular and just needs to be logged.

Take a bug list, for example. People across your company might log software bugs, carefully documenting what’s broken and how to recreate it. In the moment, they might not be sure who to assign, so they log the bug and move on. But if you didn’t want those bugs to fall through the cracks, you’d have to monitor the list yourself.

It shouldn’t be on you to reload a to-do list every hour to see what’s changed. Starting today, you won’t have to!

A new notification

If you’d like to receive notifications when to-dos are added to a specific list, just go to that list in Basecamp. Inside the right-hand menu, you’ll see a new option to receive these notifications:

Once you’ve turned on notifications, you’ll see messages in your Hey! menu every time someone adds a new to-do to that list:

Prefer to get email notifications? No problem — we’ll bundle up these notifications so your inbox doesn’t get clobbered every time someone adds a to-do:

Just for you

These notifications are only for you and only for a particular list. Other people will have to opt-in if they’d like to receive notifications, too. Want to stop receiving notifications? Just visit the to-do list, open the menu, and turn off the notification.

Give it a try!

We hope this update makes it easier to keep track of bugs, QA issues, and other unassigned tasks. Let us know what you think!

Thanks again for being a Basecamp customer.

In the latest episode of the Rework podcast, we place a long-distance phone call to Antarctica to chat with Kathrin Mallot, an astrophysicist who works at the IceCube Neutrino Observatory in the South Pole. In this episode, Kathrin talks about preparing for a work assignment in a super remote part of the world; practicing self-care during the punishing Antarctic winter; getting along with coworkers that you also live with in close quarters; frozen nose hairs, snacks, Internet access, and more!

A bad boss shapes our leadership style more than we realize.

Who’s the worst boss you ever had? Your answer to this question matters. It influences your leadership style in more ways than you think.

For myself, I can answer that question, “Who’s the worst boss you ever had?” almost immediately.

Keep reading “The Anti-Mentor”

Did you receive this email from us today?

Subject: Basecamp 3 security alert: You must set a new password ⚠️

You’re not alone! You’ve done nothing wrong, and you’re in good company. 1 in 20 Basecamp users got this email alert. Here’s the full story.

---

Last month’s mass-login attack was a sobering reminder that passwords just aren’t cutting it online. It’s getting worse. We all feel it.

Without a password manager app, it’s just impossible to use passwords securely online. Every bank-level secure web site we log into with a super strong password (One of our “main” passwords, maybe? With a ‘4’ instead of an ‘a’, am I right?) is only as secure as the flimsiest fly-by-night.com where we signed up with the exact same password.

When the weakest link gets hacked and fly-by-night.com data is breached, the whole chain falls apart. Anyone can pluck our email and passwords out of the rubble and waltz right through the front door of our other bank-level secure web sites. And who would bat an eye? They’ve got our email and our super strong password. Roll out the red carpet, Ft. Knox.

That’s so frustrating. Any motivated hacker can go grab a giant list of logins stolen in data breaches, scan down the list to look for your email address, maybe find some passwords you’ve used before, and go try to log in to other sites right away. If you reused the breached password on another site—BAM, they’re in as you. It’s that easy.

Ooo, that burns. It’s not right. And what can we do!?

As internet users, there’s much we can do to protect ourselves online. We can start using password managers (like 1Password) for every single site. Every single time. We can use a free password breach notification service like Have I Been Pwned for early warnings of data breaches that could reveal our personal data. And we can take care to enable two-factor authentication (2FA) everywhere it’s available to us, guaranteeing that a breached password won’t be enough to log in on its own.

At Basecamp, there’s much we can do, too. We can require strong, hard-to-guess passwords. We can offer two-factor authentication. But it’s not enough to celebrate sophisticated best practices when our average, everyday passwords are still 100% at risk of data breaches. We can do better.

Dial back a month. After January’s mass-login attack, we asked

1. How can we put a stop to this attack? We did this! ✅
2. Now, how can we prevent attacks like this from happening again? 🤔

We can prevent attacks like this—that reuse breached passwords to log in to Basecamp—by keeping breached passwords out of Basecamp in the first place. We can’t get out ahead of every new data breach, but we sure can catch their scent, track them down, and kick them out.

We started by checking that our users aren’t already caught up in a number of high profile, widely available data breaches. We went straight to the source: we scanned the data breaches themselves for user emails, extracted any associated plain-text passwords, and securely checked whether they match user logins.

(We use bcrypt to securely salt & hash passwords, which is a one-way encryption of sorts that’s extremely computationally costly to try to guess. We never compare unencrypted/unhashed passwords, or even have them available in the first place.)

Today we alerted every Basecamp user whose email and password we could find in a data breach, in full view of everyone online, that we removed the unsafe password and disabled login until a new password is set.

Next, we ensured that breached passwords can’t find their way in to Basecamp from here on out. Basecamp now validates that passwords are not breached. For this we thank the fantastic Have I Been Pwned API, a free service provided by a concerned citizen for checking that a password does not appear in its massive, growing trove of known data breaches. Basecamp securely asks whether a password appears in a breach and marks it as invalid if it does. Breached passwords simply aren’t allowed into Basecamp anymore.

Basecamp elbowed out breached passwords today, and we’re keeping them out.

If you received an email alert today, we’re taking a big step together. Start by setting a new Basecamp password. You’ve got this, and we’re here to help.

---

If you came here wondering whether you just got a phishing email: great instincts! Any email asking you to set a password or mess with your login is super slippery dubious. This is legit. We invite you to email us directly at [email protected] for a gut check about any dubious email.