I’m curious what software test methodologies and tools you use. Are you religious about Test Driven Development (TDD)? Do you use Rails’ Test::Unit, or do you prefer alternate tools such as Behavior Driven Development with rspec? A list of the test-related plugins you use would be interesting to see.
We’re big believers in the benefits of testing, but I wouldn’t call it religious. I personally rarely get into the flow of writing my tests first, except for transformation-style methods that takes known A and turns it into known B. I find that test-first often doesn’t fit my brain all that great for molding beautiful APIs. I know, though, that it works wonders for others and I deeply respect that approach. On our team, it seems to work especially well for Jamis at times.
I’m perhaps also a little on the fringe in regards to how much I test. I don’t believe that getting 100% coverage is worth it in most cases. Especially not when it comes to testing view code. For our applications, it simply changes too often and the criticality of display bugs is low and very rarely realized anyway.
The areas of testing where I personally derive the most benefit is from object-interaction tests. Like testing the signup form that creates a handful of records at once and need to spread error checking across the lot. Or dealing with project limits in Basecamp that’ll interrogate multiple other classes. Or API interactions for the creation of Writeboards.
For people new to testing, though, I would recommend going all-out from the start. The only way to get a good grasp on when you can be more lax in your testing is to have done “too much” for a while and identified areas that could go with less without materially hurting your productivity now and in the future. So in that sense, easing your testing rigor is a late-stage optimization that needs to be done with great care. If you’re ever in doubt whether to spend time testing or not, test.
In terms of tooling, we’re using the vanilla Test::Unit flavor that ships with Rails. I’m personally not convinced that rspec holds much improvement over it and I tend to think the DSL-style has been taken a tad too far. But I do like the BDD-style focus of naming your tests in terms of “should” and I use that in my own style. In the large scheme of things, it really doesn’t matter one way or another. The important thing is that you test and if rspec and BDD makes testing more enjoyable for you, rock on.
Also, while I don’t think striving for 100% test coverage is worth it in most cases, it’s nice to know where you stand none the less. The rcov package provides a great way of seeing how you tests are holding up and where you might be lacking.
Nate
on 14 Nov 07Great post. I was really curious where you were with the rspec stuff and if you were falling out of love with Rails native stuff. Because I too, just haven’t been able to get on the rspec train.
Keep these posts coming, David on the tech stuff please. Love to hear your thoughts on this stuff.
Ryan Norbauer
on 14 Nov 07Hey David,
Testing can certainly be taken too far.
I’ve been playing a lot with rspec lately (the syntax of which I love and the spec-first approach somewhat less so), and I often notice examples with code that tests things like the built-in behavior of Rails. There are already tests within Rails to account for that functionality, so it’s a waste of time to verify that validates_presence_of actually behaves as expected. If testing means painstakingly re-coding and testing every method call in your application (and sometimes both Ruby and Rails themselves!) in this way, it significantly increases the psychological barriers to testing, and we all know what that means.
So for my own products and projects, I’ve finally arrived at a happy intermediary place where I test what matters first (as you say, interfaces, interactions, and data transformations) and then add edge-case tests for bugs that I encounter over time or potential snags I envision from the outset.
However, for what it’s worth, super thorough specs/tests can be helpful when delivering a base of code to a client (for those of us who work with clients from time to time). Sometimes you want a document that lays out all of the behavior of the application (that there are certain associations created, etc.) for someone who isn’t familiar with Rails and doesn’t necessary understand “validates_presence_of” “has_many :through” in the way that makes Ruby and Rails fairly self-documenting for those of us who are familiar with it. In this way, tests become a communication tool more than anything else. Rspec makes that super-easy and actually quite pretty. But it’s still a lot of work.
And my philosophy is you should never do a lot of work without a damn good reason. :)
sandofsky
on 14 Nov 07I suspect 1:1 coverage is a silly ideal, but you should minimally have a smoke check for every page to your site.
Also, every time you fix a bug, you should include a test. Regressions are unacceptable.
Not long ago I switched to rSpec. I’ve found its syntax makes it living documentation, and I’ve already heard of developers sitting down with clients to write specs. I’d say those benefits far outweigh the cost of changing your workflow.
Dan Gebhardt
on 14 Nov 07I recently discovered the shoulda testing plugin from thoughtbot and find it to be a happy medium between vanilla Test::Unit and the rather complex rSpec. Shoulda brings many of the benefits of rSpec, such as readable test names and contexts (even nested contexts), to Test::Unit, along with a number of helpful macros for testing ActiveRecord. After implementing shoulda, my lines of test code have decreased but actual test coverage has improved and the tests themselves are more readable.
Henrik Lied
on 14 Nov 07Ah, finally some tech-stuff! :-) Keep those up, please!
theJareCare
on 14 Nov 07i’d say be wary of heavy interaction-based testing, the code just gets too brittle.
and yeah try not to drink too much of the bdd/rspec kool-aid as well.
hey free active record book contest over at
http://giantrobots.thoughtbot.com/2007/11/13/the-contest
freebie!
- jc
Benjamin Jackson
on 14 Nov 07Rspec was the catalyst which got me testing consistently. Something just clicked once I was forced into using the word “should” to describe my apps’ behavior.
Since then as I’ve worked with teams which rely on test/unit I’ve gotten more comfortable with the old-school TU syntax, as well as opening up to assertions like assert_select which are a huge help in making sure the essential parts of my views (think form actions!) are in the state they should be.
I recently switched over from Rspec to test/spec. I had a bitch of a time using rspec and test/unit in the same projects, and using autotest and heckle out of the box closed the deal. I can’t go back to running my tests by hand.
I’m beginning to use shoulda but am not totally convinced that assertions like ‘should_belong_to’ provide real value beyond upping your assertion count.
As far as test-first development, I’ve tried it both ways and find that for testing AR models and non-rails projects it’s a great way to develop and helps ensure that I’m writing my code to spec instead of the other way around. When it comes time to test controllers I invariably switch over to the browser and fill in the tests once I know for sure what I need to be testing.
I don’t shoot for 100% coverage, but I get jittery when I see model methods of more than one line without tests. I’ll second Ryan on not testing AR and also on making sure you have a smoke test for every page. Nothing worse than opening up your app to find a 500 error on the signup page caused by a slipped instance var. In the end the degree of thoroughness really depends on how much I expect to use (and consequently refactor) the codebase.
Mocks kick ass, and mocha’s syntax (defining expects, etc directly on Object) is much more intuitive than flexmock or the other libraries I’ve used. After you’ve added fixtures for 5 different authentication cases, it’s amazing to mock all of that out with 5 lines of code and watch your tests speed up with the reduced database hits.
brad
on 14 Nov 07Nice to know that even the big names have trouble getting into the testing mindset at times. I am definitely in that boat, more often than I should be.
Bart ten Brinke
on 14 Nov 07The syntax of rSpec is way easier to read than any unit test. That is the main reason we are using rSpec at the moment. Also the new plain text stories are great! My mom can now write test plans! http://www.movesonrails.com/articles/2007/11/06/rspec-plain-text-stories
Doug
on 14 Nov 07@Ryan Norbauer:
It’s interesting that you use the example of validates_presence_of. I actually found out the hard way that it doesn’t work as I expected. validates_presence_of on a boolean will fail if the boolean is false, even though technically, a value is present.
Sam Barnum
on 14 Nov 07I’ve found that too many tests early on can really hinder developing an API. The tests introduce dependencies on the API, so when you want to totally change something you need to tweak the tests also.
I totally agree on transformative cases, however, where there is a very clearly defined input and output. Tests are great in those cases, and provide some real peace of mind, particularly when you’re making major changes to the internals (but the API remains the same).
Yossef
on 15 Nov 07To chime in on the validates_presence_of and “testing things that are already tested” bandwagon — BDD, BDD, BDD.
Real, useful tests come in the form of specs, and what they should specify is the behavior, not the implementation. You don’t write a spec that says your model includes a call to validates_presence_of (and I’d like to see you try anyway, since there’s no validation reflection); you write a spec that says these values over here are valid and those values over there are not. Whatever you do to make that spec pass is fair game.
anonymous
on 17 Nov 07While I like rspec, I can’t stand the “should x…” naming convention for the specs. When every spec/test description starts with “should”, the word becomes useless noise. Why say that an object “should do x”, when saying the object “does x” expresses the intent of spec more directly and succinctly? Yes, we understand that it should do x. That’s why there’s a spec for it.
This discussion is closed.