We’ve been slowly trickling some of our internal projects onto GitHub, making them more widely available in the hopes that (for one) they’ll be as useful to others as they are to ourselves, and (for another) that people will contribute patches back to make the projects even better.
Today I moved our CachedExternals plugin there. You can read all about it in the README, but read on for an overview (and justification).
The road to fast deploys
A few months ago I was focused on making our deployments much, much faster. I was tired of waiting 5-10 minutes when deploying Basecamp (Yes, I know others have it much, much worse, but that shouldn’t prevent me from wanting it to be even better for myself.) Fast deploys are really critical to rapid and agile iterations: if it is too painful to deploy your application, you’re more likely to wait until later than deploy sooner.
The first tweak I made was a custom Capistrano deployment strategy: FastRemoteCache. However, this still left a lot of data to be copied on each deploy, since we had Rails and all of the plugins used by our applications checked into our repository, directly.
Now, bundling your dependencies with your application does reduce the pain associated with managing different versions of those dependencies. No two of our applications are running precisely the same version of Rails, and having each app maintain it’s own Rails version is a pretty simple way to avoid dependency hell.
However, it means that each deploy has to copy all of rails, and all of its plugins, every time, even though they are very unlikely to change often. It would be nicer if those dependencies could be cached somewhere else, outside of the app, and just have the app point to them. Obviously, RubyGems is one candidate for that, since you can install multiple versions of a single library, but it would mean (first of all) building custom Rails gems, since we often run on Rails versions that aren’t sanctioned releases (yes, we’re that brave!), and (secondly) remembering to install the correct versions of the gems on all of our servers. (We’ve got, um, several of those.)
Caching external dependencies
This CachedExternals plugin basically lets you define your library dependencies in a YAML configuration file in your application. Then, on deploy, those dependencies are checked out into a location specific to the application (the “shared path”, for those with Capistrano-fu). If the needed revision of a library is already checked out there, it is used instead, and since that is the common case, you need move only a much smaller subset of data. By bundling the dependency info with the application, you keep that knowledge where it belongs, and you let the deployment process itself keep track of what needs to be updated, when it needs to be updated.
We use FastRemoteCache and CachedExternals to great effect internally. It has reduced our deploy time for some of our applications to a matter of some 15-30 seconds (though Basecamp is still a minute or two to deploy—better, but plenty of room for improvement, yet).
So, give it a spin and let us know how it works for you!. Better yet, submit patches for the things you think are missing!
kevinseo
on 31 Oct 08Great product!
Mick Staugaard
on 31 Oct 08Brillant and simple. Can’t wait to try it out.
Thanks for sharing!
bryanl
on 31 Oct 08We just use a remote cache and rsync. Speeds up time considerably.
fractious
on 31 Oct 08Doesn’t vendoring everything and then using capistrano’s cached copy option not already solve this problem? Or have I missed something?
ActsAsFlinn
on 31 Oct 08@fractious , my thought initially also but looking a little deeper it looks like a braid/piston-like mixture. My team recently updated to 2.1.1 on a large project then 2 days later 2.1.2 is released with only a handful of changes. Unfortunately I’ve got to unfreeze 2.1.1 and freeze 2.1.2 to get out a completely new vendor/rails. This looks like it might solve that problem.
Jamis
on 31 Oct 08@fractious, no, it doesn’t really solve it. This is because first, you still have to check everything out to make the cached copy, and second, because you then have to move that cached copy to every server, which is expensive (especially if you’re vendoring Rails, since that’s a lot of data to move).
The purpose of the CachedExternals plugin is to make it so that you don’t have to move Rails (or any other external dependency) at all (or rather, only when you need a new version of them). Once checked out on each server, subsequent deploys will just reuse the previous checkout of each dependency, meaning you’re moving just your application code on each deploy. It really does make a big difference.
fractious
on 31 Oct 08OK gotcha, thanks for the clarification guys. I think I’ll have to check it out then :)
Scott
on 31 Oct 08Jamis,
You absolutely Rock! This will make life much simpler (when life requires you to code for a living).
Thank you for sharing.
This discussion is closed.