Our team adds new checks and alerts every week so that we can stay ahead of new issues. We try very hard to make sure that each alert is configured and tested such that it provides timely and credible evidence of a real problem. Sometimes though, when things go wrong we are inundated with alert information which actually hinders and confuses our problem identification and resolution.

A real world example

A server with two 10 Gigabit network connections experiences a hardware failure and spontaneously reboots. Our Campfire room is filled with alerts not just for the host being down, but also for the switch (ports) the host is connected to.

We monitor the switch ports because we want to know that they are at the correct speed, that there are no individual failures, and that no “foreign” devices have been plugged into the network. In the case of a host failure, the information about the switch ports is secondary to the information about the host—but it represents 2x the volume of alert data we receive.

In cases like this we need to make our monitoring system more aware of the dependencies exist between these checks so that we can eliminate the noise. To do so we use a number of open source technologies:

Yesterday I did an extended Q&A session at Techstars Chicago. Great group, great questions. I really enjoyed it.

Before the talk, Troy, the guy who runs Techstars Chicago, showed me this board they had propped up on an easel outside the office.

The board listed every company in the current Techstars Chicago class, along with some numbers. The columns included inventory, inventory sold, remaining inventory, and net profit.

Here’s a picture of said board:

These are the results from the challenge. But what was the actual challenge?

The challenge

Each company was told to go sell bottled water. Each company had to decide how many bottles of water they wanted for their inventory. They couldn’t get more later. They had one set of inventory and that was it. They could charge whatever they wanted per bottle.

A few other rules… They couldn’t sell them in the Merchandise Mart (which is the massive building where Techstars Chicago is headquartered), so they had to hit the streets to find buyers.

I believe they had one day to sell their water.

Observations from the results

• The companies that were over-confident lost the most money. In this case I define over-confident as taking on too much inventory.
• 75% of the 40% of the companies that were profitable ended up with zero inventory. If they had a second chance, I wonder if they’d increase the price of their water. It’s impossible to tell from the board when the companies with zero inventory ran out of inventory, but they may have been better off selling their bottles for more and ending up with just a few extra at the end rather than zero. Does zero mean they underpriced their product?
• It’s a lot better to only sell 110 bottles and make a profit of $108.60 than it is to sell 868 bottles and end up losing $331.20. Again, impossible to tell from the chart, but I wonder how much work went into selling 868 bottles only to lose $331.21 compared to how much work went into selling 110 bottles and ending up with a $108.60 profit.
• The top two sellers (Peoplematics and Project Fixup) both lost money.
• SocialCrunch, the company that ended up with the highest profit, were sitting in the front row at my talk ;)

I wonder if the results in the water challenge will mirror the results of the companies themselves if/when they get their own actual products to market.

Overall, I love this exercise. I think this is a great idea. No matter what you do in life, selling is a core skill. And there’s nothing quite like having to hit the bricks and sell your wares. It’s the best teacher you’ll ever have.

Basecamp Classic’s database is actually split across two sets of servers. One set contains a single table which is approximately 430 Gbs or more than half the entire volume of data (across both sets) in total.

Two years ago we separated this table because of its growth and size compared to the other tables. By separating the table we could scale the database hardware more closely to data growth, and we kept InnoDB buffer pool evictions to a minimum which made performance more stable.

Recently our monitoring showed some less than desirable metrics regarding this database pair: the least of which was that free storage would be exhausted in about 90 days. There was also a number of slow queries due to insufficient buffer pool space and slow queries from data “on disk”. We had already exhausted the normal tuning approaches and we needed to find a solution for these problems that didn’t involve significant time or money expenditures.

There are two common methods used to keep growing MySQL databases peforming optimally: buying new hardware or reducing the volume of data such that approximately 80% fits in memory. Buying new hardware is expensive and usually incurs a high time and staffing penalty. In most situations reducing the amount of the data is impossible because the database is actually growing through active use.

In customer support, empathy is everything—we need to be able to put ourselves in our customers’ shoes, feel their pain, give them the kind of help we’d want to receive whenever we have a problem. Potential hires are even tested for empathy before joining the team. (In case you missed it, Jim wrote a lovely post about empathy yesterday.)

Empathy is a skill—like any other, a skill that can be fostered and developed and built into a culture. (Dev Patnaik’s “Wired to Care: How Companies Prosper When They Create Widespread Empathy” explores the role empathy plays in successful organizations.)

Every once in a while, admittedly, my empathetic skills can be … challenged. And sometimes I make mistakes.

It happens. When a person we are genuinely trying to help is being unnecessarily rude or refusing to listen, it can be frustrating. In those situations, our capacity for empathy is compromised. The temptation can be to “stoop to their level” because “they’re just not listening” or “I can’t get my point across any other way.”

I recently worked with a customer who wasn’t able to sign into her website. Not our product, Basecamp—her own website. I did my best to politely explain they were different tools, and that she’d want to contact the admin for her website. She “frowned” me, and left angry feedback because she still wasn’t able to sign into her website.

I checked this customer’s history—she had written in about the same issue before, and another teammate had already politely explained she was barking up the wrong tree. I tried another reply, explained a different way. She left more angry feedback, because she still couldn’t sign into this other tool.

At that point, frustration got the better of me. “I’m really sorry for the misunderstanding,” I wrote. “I can only help you with Basecamp, though. I’m unable to help you get into your website. That is not our tool, or our company. Does that makes sense? It is as though you have contacted support for your vacuum cleaner, when your dishwasher is what is broken.”

I went to her website and found the company who supports it, and gave her the link to their customer support site. Turns out, her website was for her petsitting business. I read her “About Me” page. She has two boxers.

I have two boxers.

Her humanity slapped me right across the face.

What was I doing, scolding this fellow animal lover (this fellow human!) about vacuum cleaners versus dishwashers? Why didn’t I try harder to explain things clearly and patiently, like I would with my mom and dad? (Come to think of it, why aren’t I more patient with my mom and dad when they have tech questions?) I felt about two inches tall.

She didn’t write back. I hope she got the help she needed.

From now on, in my mind: Every time there’s a challenging case, the customer owns at least two boxers.