brad 07 Feb 06

My colleagues and I have a client that we LOVE working for because she provides this kind of “celebratory” feedback all the time. We are always motivated to do our best for her, and to go out of our way (working nights, weekends, whatever) to handle her requests, because she provides constructive and positive feedback and praises us enthusiastically. She is walking proof that if you treat your employees or contractors with respect, give them encouragement, and show them that you’re listening to them, you’ll get their loyalty and their best work.

Vinn77 07 Feb 06

I celebrate every breath.

Jamie Tibbetts 07 Feb 06

If you do what you love, and are working with other like-minded individuals, work should be a constant celebration. So I agree, it’s important not to forget that. I just don’t see what that has to do with quality control. And I’m not sure this post draws the connection between the two.

Celebrations help the quality of team morale, but have little or nothing to do with the quality of the product. You and your team could be perfectly happy and celebrating every step of a project, and it could turn out to be a total piece of crap.

I think this posts needs to better explain how the two concepts are connected.

Anonymous Coward 07 Feb 06

Jaime: quality control doesn’t have anything to do with the interpretation of crap or not crap, it just focuses on quality, making sure things work. QA/QC lets out a ton of crap products, but they’re “quality”

Jamie Tibbetts 07 Feb 06

re: Anonymous Coward

I understand that. I meant “crap” as in “works like crap (i.e. buggy),” not “crap” as in “it’s a crappy idea.” ;)

I just think this post contains two different concepts that don’t have much to do with each other. Celebrating tiny milestones is great for keeping morale up. But it’s the constant use of your your own constantly-evolving product during development that increases quality control - not the celebrations.

RS 07 Feb 06

Thanks Jamie. I made some changes. What I’m pointing out is that when you share an accomplishment, it provides a chance for your team to look at what you did and evaluate it. That way you check the thing as it’s built on the fine-grain level as well as the big everyday level.

Kyle 07 Feb 06

So, let me get this straight: you don’t user test, you just test your products by using them. My friend, that is user testing.

While I agree with your point, it’s definately user testing. Just because you don’t have use cases, or don’t have a “QA team” doesn’t mean you can’t test your own products :)

I think this is why home grown products are often better than contracted products — the home grown ones are used by their creator over and over again.

RS 07 Feb 06

My friend, that is user testing.

Ask most builders what “user testing” is and they will have a different idea.

Brandon 07 Feb 06

Ryan, I know you guys have a small team, but how do you go about “signing off” on modifications along the way? For instance, if someone suggests a change, is it then discussed in the office and passed to those on the team that are located outside of Chicago or is there an order in which this “change” must be funneled through the team?

RS 07 Feb 06

but how do you go about �signing off� on modifications along the way?

If we’re not sure about a change, we’ll run it by each other in chat. “Hey how about this?” There’s no structure to it.

JF 07 Feb 06

And to follow up on RS’s comment…

If we have the wrong decision, we fix it by making the right one next. When you make a bunch of small decisions you’re never going to make a decision that you can’t change quickly if you were wrong.

scottbp 07 Feb 06

I think this kind of guerilla user testing (that is what we call it) works for you because you guys are fairly close to your target audience.

Do you think it would work as well if you were designing something for an audience which was very different from the guys in your office?

In my work I am an expert, but that means I have to know when to ask questions, to share with my colleagues, or to go out and get feedback from the intended audience.

I’m not saying that your celebration model is bad, I’m just not sure it would work for many dev teams not in your position

Jamie Tibbetts 07 Feb 06

Thanks Jamie. I made some changes. What I�m pointing out is that when you share an accomplishment, it provides a chance for your team to look at what you did and evaluate it. That way you check the thing as it�s built on the fine-grain level as well as the big everyday level.

Bingo. That’s the glue this post needed. Much better now. :)

Paul Ingles 08 Feb 06

Regular deployments are a key practice within agile development — as is open (and regular) communication with customers. Complete end-to-end walkthroughs (both automated and non) are a good way to spot things.

It’s something we’ve recently started doing more of where I work — every Monday the whole development/testing/QA team sit down and go through the product, spotting anything we could improve/change etc.

It’s also one of the things I aim to do with the stuff I do outside of work — the sooner you deliver, the earlier you get feedback, and the quicker you are to improve.

Gayle 08 Feb 06

I once had a boss who, when I suggested they give positive reinforcement instead of threats and negative reinforcement, said, “Oh, right, I forgot, we have to be nice to you, you’re sensitive.”

Yikes.

I would love to work in a place of enthusiasm. It would make me work three times as hard, that’s for damn sure.

Zane Rockenbaugh 08 Feb 06

WHOA! User testing by the developers and “eating your own dog food” and all that jazz is great, as is enthusiasm, but don’t fool yourself. It’s not a replacement for user testing. I’ve used some of your products and some of the buttons, or how to make certain things happen were far fromclear. Since it’s so free/cheap, I accept that, but the reason it was so muddy was because you don’t user test.

You can’t seriously expect to get good user feedback from such a biased group as the people that work on a product. If the person using the product has been told what the product does, how it works, what the concepts are, and has seen it develop through successive stages, it’s going to be obvious to them how things work. Of course it works how they expect it to work. They made it work like that! (Or were close to those that did.)

Take that same product, and give it to someone else—even someone from the same industry with more or less the same expectations and view of how things work—and you’re almost sure to find at least a couple of places where they say, “This button makes no sense,” or “Why did that happen when I clicked this.”

You should say “Spotting the big, obvious problems is cake…”, but you should also understand that having a self-selected, inherently biased, and pre-trained group of people say, “I love it, works great” is not the same thing as a general, properly selected user/usability test.

RS 08 Feb 06

… a self-selected, inherently biased, and pre-trained group of people …

That’s exactly our strength. We make opinionated software by having our own ideas about what should be done.

JF 08 Feb 06

I�ve used some of your products and some of the buttons, or how to make certain things happen were far fromclear. Since it�s so free/cheap, I accept that, but the reason it was so muddy was because you don�t user test.

Everything can always be better, but if you think user testing is the only path to perfection, you’re way off. If user testing made perfect products we’d all be using perfect products. We’re not. The only thing that nears perfection is constant iteration based on real-world use and feedback.

willy wonka 08 Feb 06

Totally agree with JF on this one. Feedback and iteration are the *essence* of user testing. Thus if one had another mean to do so, then no problems.

Also - if you and your team are the major users of the product your building, then you are in fact user testing in an informal manner. User testing seems to be most helpful when UI & others are testing nomenclature, functionality, and other stuff about something they are not common user at.

37Signals is a unique situation - they build for themselves and then share with others.

woody 08 Feb 06

This conversation has been useful for me. I’m in an environment that has a workflow that, when it exists at all, is dysfunctional … we don’t test, it simply isn’t budgeted in the development spec. We all kind of work in parallel paths that merge when the project is (past) due. At that time we throw everything over the wall to the client.

It’s a nightmare for everyone and somewhat comical … a sad, farcical kind of comedy.

Rick 08 Feb 06

37Signals is a unique situation - they build for themselves and then share with others.

Ryan, has 37Signals created a web application specifically for an individual customer? If so, do you have any insight into how you managed the testing considering budgets (probably limited), deadlines (probably aggressive) and user expectations (probably ever changing).

RS 08 Feb 06

For all of our client work, whether it’s been interface design or something more involved, the following has worked best: Fix time and price, and keep scope flexible. Then work on the Next Most Important thing thoughout the time you have.

Dan Boland 08 Feb 06

So stop saving the bubbly for launch and pour some at every step.

Damn, you guys must be drunk as hell when a product finally launches. =D

Zane Rockenbaugh 08 Feb 06

RS: That�s exactly our strength. We make opinionated software by having our own ideas about what should be done.

Haha. Okay… I guess if you define “what I do” as “what should be done” then you will always do exactly what the right thing. But unless you just happen to be have the most brilliant idea about X, then you’ll also have seriously limited yourself unnecessarily by seeing “opiniated” as a virtue.

My assumption is that feedback is a good thing, and it’s the feedback that disagrees with you that’s the most valuable. If someone tells you “this is exactly what I wanted!” it certainly makes you feel fuzzy, but if someone else tells you an idea that could make whatever it is your doing 10 times better, isn’t that better feedback, even if it disagrees with your opinion? Sure the external feedback will sometimes be “wrong” but all the strictly internal feedback is always of low value.

You’re process may be totally sufficient for your ends. Of course you know more about your goals and market than I do. My only point is that I don’t see the value in what you suggest. Hearing that the other guy that you already talked to about what you want to do likes what you did doesn’t add anything; you already knew that.

JF: Everything can always be better, but if you think user testing is the only path to perfection, you�re way off.

Slow down Hoss. I never said nor implied anything about perfect software. Indeed, there’s no such thing; all software is going to be sub-optimal for most users. As you say, the way to better software is “constant iteration and user feedback,” and that’s exactly the point I’m making. If you’re feedback is limited to the other people who already share your own idea of what the software should be, then you’re not getting feedback so much as stroking each other’s ego.

willy wonka: Also - if you and your team are the major users of the product your building, then you are in fact user testing in an informal manner.

Absolutely, but if your team are the major users of your product, you’re going to have a hard time paying the bills! So, I was assuming there are other users out there, and that it may be a good idea to get feedback from them.

Zane Rockenbaugh 08 Feb 06

To quote myself: “My only point is that I don�t see the value in what you suggest.” I should have said “I don’t see the value in what you suggest as far as quality control goes. I certainly see the value as far as keeping morale high and fixing the obvious problems and such. But it seems that what you’re talking about has more to do with keeping spirits high than quality control in the common sense of the word.

RS 08 Feb 06

I was assuming there are other users out there, and that it may be a good idea to get feedback from them.

We get feedback in the form of emails and forum posts every day from our customers, and we read them. For v1, however, we trust our own decisions. We’d rather find out what real customers (not lab rats) think about our decisions and then make changes based on that.

willy wonka 09 Feb 06

Yo Ryan, why we got be lab rats? :)

Zane Rockenbaugh 09 Feb 06

We get feedback in the form of emails and forum posts every day from our customers…

Okay, then I guess I just don’t understand what you mean when you say “we don’t user test”. Do you mean that you don’t user test v1?

JF 09 Feb 06

Okay, then I guess I just don�t understand what you mean when you say �we don�t user test�. Do you mean that you don�t user test v1?

We never do explicit user testing as is commonly defined in the web industry. We build something that we are comfortable with and then release it. Then we listen and iterate in the wild.

Kira 14 Feb 06

I’m glad you found something that works so well for you. But please don’t suggest that this organic feedback model can be transplanted willy-nilly!

As an example of a place where this can’t be transplanted: We build Web apps for groups completely unlike us. They spend very little time with our apps, they spend less time than we do on computers in general, and they aren’t familiar with our jargon (even the words we *think* they know).

We build the best products we can, and find as many stumbling blocks as we can, and fix them. And then we watch actual users to see all the things we missed.

In fact, we just did this last week. Good thing we brought 10 of them in to user test for us — we have a long list of things that aren’t stumbling blocks for us, but are for them. And now we can fix the problems before launching the product and pissing them off.