Rails ships with a default configuration for the three most common environments that all applications need: test, development, and production. That’s a great start, and for smaller apps, probably enough too. But for Basecamp, we have another three:

  • Beta: For testing feature branches on real production data. We often run feature branches on one of our five beta servers for weeks at the time to evaluate them while placing the finishing touches. By running the beta environment against the production database, we get to use the feature as part of our own daily use of Basecamp. That’s the only reliable way to determine if something is genuinely useful in the long term, not just cool in the short term.
  • Staging: This environment runs virtually identical to production, but on a backup of the production database. This is where we test features that require database migrations and time those migrations against real data sizes, so we know how to roll them out once ready to go.
  • Rollout: When a feature is ready to go live, we first launch it to 10% of all Basecamp accounts in the rollout environment. This will catch any issues with production data from other accounts than our own without subjecting the whole customer base to a bug.

These environments all get a file in config/environments/ and they’re all based off the production defaults.


So we have something like this for config/environments/beta.rb:

# Based on production defaults
require Rails.root.join("config/environments/production")

beta_host_name = `hostname -s`.chomp[-1]

BCX::Application.configure do
  # Beta namespace is different, but uses the same servers
  config.cache_store = :mem_cache_store, PRODUCTION_MEM_CACHE_SERVERS,
    { timeout: 1, namespace: "bcx-beta#{beta_host_name}" }

  # Each beta server gets its own asset environment
  config.action_controller.asset_host = config.action_mailer.asset_host =
    "https://b#{beta_host_name}-assets.basecamp.com" 
end

This gives each beta server its own memcache namespace and asset compilation, so we can run different feature branches concurrently without having them trample over each other.

Since many of our associated services are shared between production and the beta/staging/rollout environments, we take advantage of the YML reference feature to avoid duplication:

production: &production
  url: "http://10.0.0.1:9200" 
beta:
  <<: *production
rollout:
  <<: *production
staging:
  url: "http://10.0.1.2:9200"


Custom Configuration
To run six environments like we do, you can’t just rely on Rails.env.production? checks scattered all over your code base and plugins. It’s a terrible anti-pattern that’s akin to checking the class of an object for branching, rather than letting it quack like a duck. The solution is to expose configuration points that can be set via the environment configuration files.

Lots of plugins do this already, like config.trashed.statsd.host, but sometimes you need a configuration point for something existing in your code base or for a plugin that wasn’t designed this way. For that purpose, we’ve been using a tiny plugin called Custom Configuration. It allows you to do configuration points like this:

# Use cleversafe for file storage
config.x.cleversafe.enabled = true

# Use S3 for off-site file storage
config.x.s3.enabled = true

It simply exposes config.x and allows you to set any key for a namespace and then any key/value pair within that. Now you can set your configuration point in the main environment configuration files and pull that data off inside your application code. Or use a initializer to configure a plugin that didn’t follow this style.

In-app stage switcher
For 37signals employees, we expose a convenient in-app stage switcher to jump between the different environments and setups. That’s mighty useful when you want to checkout a new feature branch or ensure that everything got rolled out right.

Rollout to 10%
While the rollout servers are always ready, we only use them when a feature is about to go live. The process is to deploy the feature branch you’re about to merge to master to the rollout environment. Then you flip the switch with cap rollout tenpercent:enable, which instructs the load balancers to send 10% of accounts to the rollout servers. When you’re content that all is well with the feature branch, you merge it into master, deploy to production, and turn off the rollout again with cap rollout tenpercent:disable.

The great thing about doing it like this is that the enable/disable action is very fast. It’s not like the scramble to do a full capistrano rollback. This just ticks the load balancer to send some traffic or not. So the second you catch an issue, you can get the 10% back on regular production, fix the problem, and then try again. Great for your blood pressure levels!

Just do it
For a long time, all we had was the staging environment. But the addition of multiple, dedicated beta servers to test feature branches concurrently, and the rollout environment to deploy with more confidence, has been a big boost to our workflow. There’s not a lot of work in setting this up and Rails was built for it from the beginning. The defaults are just a starting point.