As you may have noticed, we’re working on an all new version of Basecamp called Basecamp 3. We want to build the fastest version of Basecamp yet, so we’ve been hard at work on optimizing performance. One of the things we’ll be doing to improve both performance and reliability is to serve Basecamp 3 from two datacenters: one in Chicago, IL, where our other applications are all located, and our second site in Ashburn, VA that we’ve brought online over the past couple of years. We recently optimized how traffic reaches these datacenters by using Anycast BGP.
Here’s how it works in some more detail.
When you make a request for a page in Basecamp 3, that request leaves your computer, goes through your home or office Internet router, and ends up in your internet service provider’s network, whether that’s Comcast, Google Fiber, or your mobile phone carrier. In most cases, your ISP will send that request along to another upstream ISP (though a process called peering), eventually reaching one of the networks that connect to our datacenters (XO and ServerCentral).
Once it reaches our providers, the ✨magic ✨ of anycast starts to happen. Because we advertise the IP address for Basecamp 3 as being served from either datacenter, the provider can choose to send that traffic in either direction, so it looks for the lowest “cost” (fastest) path to get that traffic delivered. When it picks the shortest route, you win: you get to the datacenter that’s closest to you in Internet terms, which may or may not actually be physically closer to you.
We previously utilized a DNS service to send traffic to both data centers for Basecamp 3 and our other multiple-datacenter applications. Below you can see the traffic for one of our other applications over the last two days. Initially traffic was split evenly between Chicago and Ashburn. After the change was pushed yesterday, Chicago started seeing progressively more traffic as the day went on and our traffic mix shifted further to customers on the west coast. This morning, Ashburn has seen the bulk of the traffic since it is closer to customers in Europe and the east coast.
Anycast routing provides some other benefits as well. In contrast with modifying DNS records (even with very low TTLs) changes to anycast routes result in a much shorter “long tail” of traffic hitting the datacenter that has been removed from production service.
Compare the above graph of a DNS based change with the graph below showing anycast route changes :
You can see that traffic shifts happen much faster using anycast.