We’re constantly working on new features, improvements, and technical upgrades for Basecamp. Many of these changes need to be experienced in the wild to guide their evolution. We need to live with them in our daily use of Basecamp to see whether what seemed like a good idea is actually a good idea.
To this end, we run six different beta servers that all point to the same production database. We’ve found it impossible to accurately evaluate a feature unless it’s being used in anger with real data that actually matters. Evaluating changes against a staging server that’s running an old copy of the database just doesn’t cut it.
The way we book a beta server for use with a certain branch is simple: It’s just in the title of the BCX Campfire room.
When you’re done with a beta server, you change the title to “available”. When you need a beta server, you change it to the name of the running branch.
To select a given beta server, we’ve added a drop-down to all 37signals’ staff accounts within Basecamp itself.
You just pick the server you want to run on and it’ll route you to the beta environment running the branch you’re looking to try out.
Long-term exposure to upcoming features is a great way to get them just right. But it’s also a great way to realize that what you thought was a great idea just wasn’t good enough to ship. We’ve killed many features and changes to Basecamp after living with them for a few weeks. Often times, the most important features are the ones you don’t ship.
Justin Reese
on 11 Sep 12“used in anger”
Love this.
Dr Nic Williams
on 11 Sep 12Are all DB migrations pushed into master so that DB schema changes on the shared DB are common? Any other specific commits pushed to master early?
Tadas
on 11 Sep 12I tried changing subdomain to the ones visible in the picture – they all work. Does it mean everyone can try out beta branches? :)
Josh Hamilton
on 11 Sep 12Is each beta server entirely isolated such that it runs its own caches, workers, background processes and whatnot, or do you find that it’s enough to just go with the app servers?
Taylor
on 11 Sep 12@Drnic
All beta envs and production share the same prod database. We use a production db in staging to test migrations a few times, then we roll them out in production.
JohnO
on 11 Sep 12How do you handle situations where the database needs to change for a new feature?
One of the stickiest parts I’m encountering more and more with using lots of feature branches is the fact that my migration system doesn’t know a thing about source-control branches :/
Taylor
on 11 Sep 12@JohnQ
We test them in staging, then we push them to beta / production. In the case of those migrations usually production can run fine with the changes in place.
Ed James
on 11 Sep 12We have something very similar, although not quite as sophisticated. We have a single Beta environment which uses our live db, with 6 production servers running our app. So far this has been very useful, although we do find it tricky to test payments (using Spreedly callbacks) and background processes (which can only have a single instance of the process running across all servers) in our Beta environment.
How do you guys approach these problems in your setup?
Arik Jones
on 11 Sep 12Any plans to offer a customer-facing beta versions for those interested in the risk? (ala Gmail Labs)
Brian Bravo
on 11 Sep 12Thank you for your insightful and brief post. I love how you folks at 37Signals work.
Anonymous
on 11 Sep 12This is going to be unpopular.
Dan
on 12 Sep 12Does just one person hit the beta server they are “testing” against or do you switch it up so that you each use each other’s features/additions for feedback?
kstcable
on 12 Sep 12Maybe is not very popular but thanks your post.
Aurélien
on 12 Sep 12Interesting!
The purpose of the testing environment is to protect users & data from incorrect behavior of the new code. Examples: - Data are available to anyone due to malfunction. - Error leading to erase a project when the user wants to delete a file in the project. - Drop of performances on the database for everyone due to a new query.
How do you prevent such events without the dedicated database? Do you raise the level of test before adding the modification? Do you raise the level of monitoring and your capacity of recovering (standby database, automated backout solution etc…) ?
Thanks.
Humidity Meter
on 12 Sep 12Thanks for sharing such a useful post with all the readers.
Kim
on 12 Sep 12@37signals
This scares the heck out of me.
So production data is in your DEV & TEST environments? Environments which typically have little controls over who has access to it and can view such data.
Jonas
on 12 Sep 12@Kim
DEV & TEST is, at least in my view, very much pre-beta stages.
DHH
on 12 Sep 12Kim, our beta environments have exactly the same access controls as production does. You can’t see any data you’re not allowed to see because you won’t be able to log into those accounts without knowing user name and password - just like on production.
Bryan
on 12 Sep 12Hey Kim,
I think the best way to put it is instead of immediately moving things that are finished, tested and ready to go into production, they publish to a beta server for a period of time.
So, all pending DB changes have been applied as they would for any production roll out, but the UI changes/enhancements are not there yet. This way they can preview how the new UI changes feel before rolling out the front end.
The only time an issue may arise is if a table column was removed that the current production business layer/UI relied on, which I doubt happens very often. 37s is pretty savvy and probably has a way to deal with that as well.
It is a great idea, one that I would like to utilize myself.
Kim
on 12 Sep 12@DHH
But don’t 37signals employees have access directly to the database and can view customer data from directly within the database.
That’s what scares me
Calm voice of reason
on 12 Sep 12Kim,
Uh, um, yeah…. boy I don’t even know where to start.
Bruno Miranda
on 13 Sep 12You guys actually use campfire in the browser? I love the service but without Pyro or Flint it’s pretty easily forgotten behind other tabs.
This discussion is closed.