One highlight of Amazon’s EC2 is having a wide range of generally available services to help reduce moving parts.
We store part of our cluster configuration in S3. The server instances pull this configuration and bootstrap from there using a simple set of rake tasks and a server provisioning tool, Sprinkle. You could use SimpleDB for a similar purpose. One could serve as a backup of the other, given their similar APIs. Either way means fewer moving parts.
Another vital EC2 feature is passing arbitrary data to an instance. Many bundled images now automatically execute a blob of text you pass to the instance on boot as a shell script, like those supplied by Alestic. We use this to sync configuration scripts and packages from S3.
While reading Tim Dysinger’s article on using EC2 as a simple DNS, I thought this was a great way to remove the need for an internal DNS server on EC2 for smaller setups. We use a similar technique: specifying a single EC2 Security Group for a host as its identifier. Each server generates its hosts file every minute. Simple, effective and one fewer moving part.
Security groups are useful for describing roles and other identifying information about each host. We use this information to generate Nagios monitoring configuration files. For example, a security group named “role: app” will automatically enable HTTP checks and Passenger memory checks.
All this means less dependence on a centralized configuration server or pushing large sets of commands over SSH manually. While these techniques are effective, they require more moving parts and their own care and maintenance.
As your application’s complexity increases, you’ll thank yourself for the opportunity to reduce the complexity underneath it.
Eric-Olivier Lamey
on 02 Dec 08Always interesting to hear about production setups from sysadmins, thank you. I don’t understand why you don’t talk about the single most important part of any architecture: configuration manager. The tool doesn’t matter (as long as it is puppet ;-) but the principle is of the most importance: self-documenting, formal, reproducible recipes for your systems and services. Oh, and sprinkle? Do you recompile everything? Why not use (and love) your distribution packages?
Marko
on 02 Dec 08To get any decent version of ruby (or rails) recompiling is the simplest thing that can possibly work. At least it’s easier then building your own packages. :)
Joshua Sierles
on 02 Dec 08Eric-Olivier,
We create a base image with our required packages already installed. We actually very few dependencies that aren’t included with each app, including Rails. Some of those are rolled into packages, or installed from source by Sprinkle before creating the base image.
After that, sprinkle does any other package installations, and a set of rake tasks are run depending on the role of the machine.
I’ll go out on a limb to say that I find Puppet and tools like it unnecessarily complex for most environments. They require a centralized service. Puppet’s choice to avoid using normal Ruby for its DSL seems unnecessary. I love using tools whose code I can read over in a few hours so I might extend it as needed. A lot of operations don’t really need a DSL wrapper.
Also, working with EC2 and virtualization changes your way of thinking about provisioning. Throwaway servers don’t necessarily need long-term provisioning. If your base config changes, you can boot some new instances and even create a new bundled image to use. We’re experimenting with this flow now: stage your environment with a series of install tasks. When it’s perfected, bundle it and deploy it into production.
That said, we do love Ruby and use Sprinkle plus Rake. They are simple, extensible and document things clearly. In the end, the majority of work is in getting the dependencies right. That work is different for every environment.
ian
on 02 Dec 08Are you all using EC2 to supplement your servers at Rackspace, or are you moving off of Rackspace all together?
DHH
on 02 Dec 08Tada on EC2 is a supplement and an experiment. We are not looking to move away from Rackspace.
Andy
on 07 Dec 08guys, I am a mISV with a product that helps to work with Amazon S3 called CloudBerry Explorer. I was following your blog for “inspiration” and was pleased to know that you were using S3 service. Why don’t you give a try to my product? http://www.cloudberrylab.com/
This discussion is closed.