Ta-da List just moved to three exciting platforms: Rails 2.2, Phusion Passenger, Amazon EC2.
This is an important milestone for 37signals. As proponents of simplicity, we really felt the difference the latter two technologies made for server provisioning and deployment. As we grow more comfortable with these services, we’ll be moving other applications this way and writing more about the results.
Phusion Passenger
After we comissioned Phusion to add global queueing to Passenger, we felt it was time to give it a try. Since we were already on Apache, it proved a simple transition. We’re really impressed with the ease of deployment and stability under Passenger. The app now requires less than 10 lines of configuration to launch and deploy. Passenger handles its own process spawning: done. Its command line tools for monitoring requests and memory usage complete the package for easy integration into monitoring tools.
Amazon EC2
This has been my ongoing project at 37signals. Avoiding the extra layer of low-level setup involved with our current Xen-based virtualization system brought me closer to the core concerns of our environment – how to best automate support for the applications from the systems side. More often than not, a traditional server deployment consists of a range or organically provisioned services and environments. Ours is no exception, due to the rapidly-changing requirements of each application. EC2’s lack of persistence forces you to think about automating this from the start. This turns out to be a blessing in disguise. Setting up a full environment consisting of dedicated instances for mail, Nagios/remote logging, application serving, a master and a slave now takes about 5 minutes.
In future posts we’ll detail how we used combination of image bundling and a custom EC2 deployment tool to build the Ta-da List environment.
If you haven’t documented your server deployment process in code or experimented with these technologies, now is the time.
Ric from Swirrl
on 27 Nov 08We’ve just moved over to Passenger too. It rocks.
Adam
on 27 Nov 08I tried Passenger a couple of months back in testing, I found that after a while it just crashed my system to the point where I had to power it off and power it on, its good news to see you adopting it.
Have you noticed any savings in memory on your systems with using it?
Joshua Sierles
on 27 Nov 08Since we moved from 64 to 32-bit instances, it’s hard to measure whether we are saving memory per-process.
On a related note, since Passenger comes with a script to report memory usage, we use this little script from cron to watch and kill processes that pass a certain limit. A SIGUSR1 signal will gracefully exit a process once its current request has finished.
http://gist.github.com/29838
Cameron Westland
on 27 Nov 08We’ve been using passenger for a while. It’s even great for development. I’m definitely going to check out EC2 in more depth.
Cheers!
Sean
on 27 Nov 08Could you share your passenger configuration?
Aron Pilhofer
on 27 Nov 08Adam, I’d encourage you to give it a try again, and if you still have problems, join the passenger Google group and post there. We have been using it in production for months now, and the latest versions we have found to be absolutely rock solid.
Joshua Sierles
on 27 Nov 08Sean,
In a standard setup Passenger requires no configuration at all. Here are the global options we use, since we need rewrites, global queue and want a decent-sized pool of workers when needed.
RailsAllowModRewrite on
PassengerUseGlobalQueue on
PassengerMaxPoolSize 30
Matt Geary
on 27 Nov 08Would love to hear more about what your custom image looks like and how you deploy? I know you mentioned another post but and quick and dirty info would be helpful. Are you managing your deployments yourself or do you use something like RightScale?
StartBreakingFree.com
on 28 Nov 08Would love to hear more about Passenger, this is the first I’ve heard of it.
I’ve been deploying my app with Capistrano. I don’t really understand the difference after reading the Passenger website.
Any help? Thanks! Brian
Paul Smith
on 28 Nov 08I know that one of the big new featues of Rails 2.2 is thread safety. Does passenger support thread_safety and what are your takes on thread safety. Are you going to try moving some apps to the thread safe route?
Joost Diepenmaat
on 28 Nov 08Joshua, I believe the newest development version of Passenger comes with build in memory limits. It can kill processes that pass a certain limit without the need of a script (which is the dirty way I believe).
Yaroslav
on 28 Nov 08Could you please share your experience with using storage on EC2—Elastic Block Store, I believe? It was a bit frightening to deploy to EC2 before, when there was no “real” persistent storage option for databases.
Ric
on 28 Nov 08@Brian You can still use Capistrano to deploy. Passenger is just a substitute for Mongrels (for example).
Joshua Sierles
on 28 Nov 08Matt,
Our custom image is based on the Ubuntu Intrepid images from Alestic. We install useful EC2 gems and base packages, bundle, then provision each instance by role. Working with EC2 is so easy, we don’t see much value in using a third-party provider for our scale.
Brian,
Passenger doesn’t replace capistrano. Rather, it might replace mongrel in your setup. You still deploy the same way via capistrano, with a few small changes. I suggest trying it out on your local machine first. Use the Passenger Prefpane if you’re on a Mac.
Paul,
We don’t have much use for the threading improvements at the moment, as we scale fine with single-threaded dispatch. Passenger doesn’t support this.
Joost,
Those improvements were removed. The script works great for us so far.
Yaroslav,
EBS and Elastic IPs were the primary motivation for moving to EC2. We use EBS extensively: for MySQL data, logs and local repository mirrors. Performance so far is excellent. Snapshotting volumes is a breeze and makes setting up MySQL slaves and staging environments really easy.
Nic
on 28 Nov 08And they’re using Dynect for DNS provisioning. Featured on the home page today 2008/11/28 (that is 11/28/2008 in the States) and with a case study.
Paul Smith
on 28 Nov 08Joshua Sierles: Thanks for the response. Would thread safety hurt performance? I’m just thinking it looks pretty easy to implement so would it hurt to use a thread safe version of some of your apps? I would think performance would only get better. I don’t know though. That’s why I’m asking you :p
Carsten Nielsen
on 29 Nov 08We’re actually in the process of implementing this exact setup right now for LearnHub. The less moving parts you have, the better.
Paul Campbell
on 30 Nov 08How are you finding EC2 for email delivery? EC2 is notorious for being blacklisted when sending emails from an EC2 instance … do you try to get your instance IPs whitelisted? Or are you using a third party SMTP service?
DHH
on 30 Nov 08Paul, thread safety holds no great promise on C Ruby 1.8.x when you can use Passenger and a Copy-on-write Ruby like REE. You get roughly the same memory savings and the processes are automatically managed and uses all the cores in your machine.
Joshua Sierles
on 30 Nov 08Paul,
Mail delivery on EC2 is fine when using an Elastic IP. You own them for as long as you like and can move them from instance to instance as necessary. Most of these are not blacklisted because they are limited and cost money. We assigned ourselves a few and none of them are blacklisted.
twojeanonse
on 30 Nov 08Very good text, and valuable informations. Thank you.
Sloan
on 01 Dec 08This really was a helpful post and the comments were helpful as well. Any thoughts on NeverBlock for even faster requests? I know that you guys scale just fine, but I wonder if it would make the pages render any faster…?
Sloan
on 01 Dec 08Link to NeverBlock just in case you don’t know what I’m talking about: http://www.espace.com.eg/neverblock
Jason
on 01 Dec 08Thanks for letting me know you are using Amazon EC2. Now I have to write my own solution for the same stuff. Because my employer doesn’t allow cloud computing without the possibility to issue a security audit to check the safety of our company information.
Bruce
on 01 Dec 08Mail delivery from EC2 is not fine, even with properly configured Elastic IPs (we’ve tested it extensively this year). Note: this is a topic discussed at length in the Amazon EC2 forums, where the current recommendation is to proxy mail service off-cloud.
Also, Amazon does offer Persistence Storage for EC2, which we’ve found useful on our production servers. It’s a NAS setup that we use to store some of our base setup and MySQL instances. Bringing up a new server, now, is reduced to the requisition and boot time. Backup, too, is simple (and works against live data stores).
Joshua Sierles
on 02 Dec 08Bruce,
While browsing the forums, there seemed to be a lot of confusion about this topic. While reverse DNS is often cited as a problem, EIPs have reverse entries. Simply opting for a 3rd-party service is definitely a good move, but costly. Do you have any references or blog posts about this particular issue?
Bob
on 02 Dec 08I would love to see more from you guys on your moves over to Passenger. Right now I run it on my test server here at my house, but have not deployed anything of consequence (that should come with the New Year). I’d be interested to hear the struggles you’ve had, but also the good things as well.
StartBreakingFree.com
on 02 Dec 08Thanks I appreciate the follow up! Brian
Matt Puchlerz
on 03 Dec 08I couldn’t seem to find any bug trackers for Ta-Da List, so this seems as good a place as any to post a bug.
Reordering a list appears to work fine, until you reload the page and find that your new order wasn’t saved. I can see that the AJAX requests are firing, and there appears to be a response. A blank response, but a response nonetheless.
Could it have been something with the migration of Ta-Da Lists to Passenger?
This discussion is closed.