For a long time we’ve been struggling to manage the growing presence of JavaScript at 37signals. We needed a way to share common JavaScript components and behaviors across our applications and web sites. And we wanted the ability to organize our JavaScript source code into multiple files and folders without the overhead of including dozens of <script> tags on every page.
So we scratched the itch by creating Sprockets.
Sprockets is a Ruby library that preprocesses and concatenates JavaScript source files. It takes any number of source files and preprocesses them line-by-line in order to build a single concatenation. Specially formatted lines act as directives to the Sprockets preprocessor, telling it to require the contents of another file or library first or to provide a set of asset files (such as images or stylesheets) to the document root. Sprockets attempts to fulfill required dependencies by searching a set of directories called the load path.
It helps you turn messy JavaScript into clean modules for development and a single .js file for deployment.
Here are some of the benefits of Sprockets:
Extract reusable code and share it across multiple web sites or applications. Sprockets makes it easy to extract JavaScript plugins from your site or application and share them across your portfolio. Use your SCM to check out plugin repositories directly into your site or application. Then tell Sprockets to add the plugins to your load path. Support for asset provisioning lets you bundle CSS and images with JavaScript plugins, too.
Speed up your site by automatically concatenating JavaScript into a single file for production. Concatenating your site’s JavaScript means all your source code is cached in the browser on the first hit. It also means you reduce the number of HTTP requests necessary to load a single page. When combined with gzip compression on the web server, concatenation is the fastest way to serve JavaScript to your users.
Organize your JavaScript source code into multiple commented files and directories. Finally, an end to navigating long, difficult-to-maintain JavaScript source files. With Sprockets, JavaScript source code can live anywhere on your system, even outside your site’s or application’s document root. You’re free to split source code up into multiple files and organize those files into directories during development. You can also add as many comments as you want—they’ll be stripped from the resulting output.
Use bleeding-edge framework and library code in your application. If you use and contribute to open-source JavaScript frameworks and libraries that use Sprockets, like Prototype and script.aculo.us, the build processes for those scripts can be integrated directly into your application. That makes it possible to track the latest development versions of your framework and library dependencies by adding their repositories to your application’s Sprockets load path.
Sprockets is compatible with the PDoc JavaScript documentation system and the JavaScript framework of your choice. If you document your JavaScript source code with PDoc, Sprockets will automatically strip documentation comments from the resulting concatenated output. You can use any JavaScript framework you like in your site or application—Sprockets is framework-agnostic.
Screencast: How we’re using Sprockets in our new marketing sites
I’ve recorded a screencast showing how we use Sprockets in our new marketing sites.
(You can watch the full-size verison on Vimeo.)
How to get Sprockets
Visit the Sprockets web site for installation and usage instructions. The source code is available on GitHub, and you can install the RubyGem with gem install sprockets.
Sprockets can be used as a Ruby library or with a simple CGI script. There’s also a Rails plugin available separately.
Mislav
on 19 Feb 09We’ve been struggling with managing all the scripts in our latest project … And now it’s here finally :) thanks for releasing!
Casper
on 19 Feb 09You f—king rock!! Looks very promising, thx for sharing.
off to convince our developers to give it a go*Timothy
on 19 Feb 09Seems like this would be really useful. Thanks for the info!
Dieter
on 19 Feb 09Now is the time on sprockets vhen ve danse.
Dave Strus
on 19 Feb 09You read my mind. Bless your heart, Sam.
Tim
on 19 Feb 09How did Sprockets know that
“behavior/hover_observer”
Is in the 37ui parent directory when “prototype” and “extra” were not in the 37sui directory
John Tantalo
on 19 Feb 09In the demo, how did Sprockets know to rebuild the concatenated file when the required file changed? Was the cached file invalidated off stage?
SS
on 19 Feb 09@Tim: Our Sprockets load path is “vendor/sprockets/*/src” – that is, every “src” subdirectory of every directory in “vendor/sprockets.” So that matches “vendor/sprockets/37ui/src,” “vendor/sprockets/dim_zum/src,” “vendor/sprockets/extra_extra/src,” and “vendor/sprockets/prototype/src.”
@John Tantalo: Caching was turned off in the screencast, so /sprockets.js was transparently proxying the CGI. (More info in the nph-sprockets.cgi source code.)
Justin
on 19 Feb 09It would be cool if it could optionally minify the concatenated js as well.
Bob (Buffone)
on 19 Feb 09Concatenation of JavaScript and CSS can improve site performance dramatically.
If you are looking to do JavaScript and CSS concatenation and compression and you aren’t using Ruby, look that the RockstarApps’ tools that plugin into Eclipse and Aptana.
rockstarapps.com
Jeff Putz
on 19 Feb 09Sounds a lot like the ASP.NET ScriptManager, which does exactly this.
SS
on 19 Feb 09@Justin: I agree! There’s a ticket about minification on Lighthouse. But for now, configuring your server to gzip JavaScript should get you 80% of the way there.
Erik Frick
on 19 Feb 09This is awesome! wonders if there are PHP equivalents
Radoslav Stankov
on 19 Feb 09I was just starting to make something similar, but sprockets come just in time :) Great tool, I even started to use it for my mini-prototyped base framework and a project for work.
Daniel Øhrgaard
on 19 Feb 09That’s pretty awesome. Great work as always, Sam! Congrats!
@Erik Frick: I actually build something similar just a few weeks ago in PHP. I’m sure several such scripts already exist, but I wanted to roll my own, just for fun.
My script is a simpler JS/CSS (or anything really) concatenator in that it can’t handle nested require-statements like Sprockets. Rather, it moves the concern of including scripts out of the head-tags and into the script’s configuration. It concatenates the files it’s told to concatenate, and caches the result. It’ll also gzip its response if possible.
Again, it’s not as complete as Sprockets, but if you want, I’ll document the code a little better, and post a link where you can download the script.
SS
on 19 Feb 09@Erik Frick: All you need is a basic Ruby installation on your server to use Sprockets as a CGI script. That’s what we’re doing for our marketing sites, which are otherwise written in PHP.
Radoslav Stankov
on 19 Feb 09Bw, are you planing on integrating sprockets yui compressor or something like this to reduce the file size of the resulting concatenated file.
John B
on 19 Feb 09Interesting – I never realized there was such a demand for this.
I’ve been using a similar system that I developed in PHP that takes care of concatenates, caches, and minifies or runs yui compressor on the JS. I haven’t released it publicly, but if there’s interest, I will. Just let me know.
Joe Van Dyk
on 19 Feb 09This is exactly what Fixie Shrinker does. It’s a Rails plugin for smartly compressing your javascript and css.
http://github.com/joevandyk/fixie_shrinker/tree/master
Was released a few months ago.
SS
on 19 Feb 09@Joe Van Dyk: Cool plugin! Looking at the source, though, I don’t believe it has the same functionality as Sprockets.
With Sprockets, you define a load path to tell it where to find dependencies. That means you don’t have to keep all your JS in public/ – you can have code that lives in Sprockets libraries (checked out into vendor/sprockets/) or even Rails plugins.
Sprockets preprocesses its source files to find specially-formatted comments called “require directives.” So you can have Sprockets libraries that depend on other libraries, and they’ll be concatenated in the right order.
Finally, Sprockets offers asset provisioning, which means your Sprockets library can offer up a set of images, stylesheets or even HTML files that will be automatically copied into the document root.
Mike Riley
on 19 Feb 09Someone should have done this a long time ago. I know it’s a problem I’ve encountered before and a solution like this was always one of those things on the tip of my tongue that I never really put the time or effort into implementing. It’s great to see someone hold the torch and lead the way, I really hope that frameworks will begin to standardize dependency checking on this.
Evan
on 20 Feb 09So, I’m assuming that you concatenate all the script files server-side by creating a new source file and appending the subsequent source files and dependencies.
If it does this on the fly, wouldn’t the source file have to be recompiled for each user that doesn’t already have it cached? And, if you changed the source of one of the sub-files or dependencies wouldn’t you need to indicate a change of version so the browser auto-caching abilities don’t overlook that file for download?
This looks like it would work wonders for small sites, but it looks like scalability may become an issue as the sites grow bigger. For instance 100000 unique users in a day would require this process bing done for each accessed source file that uses sprockets. That’s a lot of unnecessary server overhead (especially if there are a lot of levels of dependencies included). I know GET requests are expensive in terms of loading time, but server-side IO access can be really expensive if pushed to a larger scale.
Caching and versioning the pre-generated files might be good idea to consider.
SS
on 20 Feb 09@Evan: No need to assume, just check the docs or source. :) Caching is built in.
The sprockets-rails plugin uses Rails page caching and provides a Rake task for performing precompilation in a post-deploy hook. In addition, the Sprockets concatenation object is cached in development mode based on the latest mtime of all your source files.
The CGI script will cache its output if the “sprockets_generate_output_file” environment variable is set to “true.” Use this in conjunction with a rewrite rule that proxies to CGI when the output file doesn’t exist.
Rakesh Pai
on 20 Feb 09How is this any different from say Dojo’s build system? Or is Prototype just catching up?
jonnii
on 20 Feb 09@SS I think this is absolutely fantastic. My javascript files were starting to hit thousands of lines and were becoming unmanageable. I’ve come across one bug, which I’ve added to lighthouse.
My only feature request would be the ability to have multiple sprocket ‘applications’. For example, I’d like to sprocketise some of the javascript files on my main webpage and additionally be able to sprocketise all the javascript for my editor.
So you could do something like: <%= sprockets_include_tag ‘editor’ %>
which use the load editor.js instead of application.js.
I hope this makes sense.
Stuart Loxton
on 21 Feb 09@Erik and all those who want something similar in PHP I’ve started a clone at http://github.com/stuartloxton/php-sprockets/tree/master
Robin Hood
on 23 Feb 09Make sure to load your javascript at the end of your html page. This one liner could get large in size (even if gzipped + minified) if you are using prototype, scriptaculous, custom js, etc…
Don’t want to block other components from loading with the maximum of 2 concurrent HTTP connections per browser.
Jim Higson
on 25 Feb 09This is very cool. I’ve been writing my own scripts (I write about this here: http://jimhigson.blogspot.com/2009/02/im-microslotting-at-ogn.html) for this – it is good to finally have a general purpose framework.
If it does everything I need I should switch to Sprockets at some point.
Jim Higson
on 25 Feb 09Forgive the misspost. Link properly this time.
This discussion is closed.