Please note: This site's design is only visible in a graphical browser that supports Web standards, but its content is accessible to any browser or Internet device. To see this site as it was designed please upgrade to a Web standards compliant browser.
Signal vs. Noise

Our book:
Defensive Design for the Web: How To Improve Error Messages, Help, Forms, and Other Crisis Points
Available Now ($16.99)

Most Popular (last 15 days)
Looking for old posts?
37signals Mailing List

Subscribe to our free newsletter and receive updates on 37signals' latest projects, research, announcements, and more (about one email per month).

37signals Services
XML version (full posts)
Get Firefox!

Remote vs. Co-Located Design Testing

15 Jul 2003 by Matthew Oliphant

Because we don’t test users, we test designs.

We do remote testing sometimes. We’ve used Vividence, and NetRaker. We’ve NetOp‘ed into the participants computer and talked with them over the phone. But is it good enough? Does remote testing really get good data? Or better data than co-located testing (assuming co-located testing has value)?

Have you used remote testing? How well has it worked for you? Did it depend on what type of testing you did?

11 comments so far (Post a Comment)

15 Jul 2003 | Steve Hoffman said...

I've been wondering about this too. We've historically only done face to face testing, but I've considered Vividence. Do you really get actionable data out of it?

15 Jul 2003 | JF said...

As with any testing, it's not the data that makes the difference, it's the interpretation of the data. My favorite quote regarding data comes from David Ogilvy: "I notice increasing reluctance on the part of marketing executives to use judgment; they are coming to rely too much on research, and they use it as a drunkard uses a lamp post for support, rather than for illumination."

15 Jul 2003 | fajalar said...

I agree, JF. But I also think that the methodology used feeds into the data that gets collected, which then feeds into how the data is interpreted.

What is especially true for usability testing is that you have people who are making educated guesses about testing data. We can only hope that the education is valid.

15 Jul 2003 | Darrel said...

I'd say remote testing is better than no testing, but nothing beats actually watching a site visitor (attempt to) use your creation.

15 Jul 2003 | jharr said...

Matthew (MO), I work designing interfaces for a large PDM company and we've had really good success with remote testing. We needed an alternative when travel budgets were slashed and we've found remote testing - using webex or similar products - work quite well.

It's like all testing, it comes down to planning, planning, planning. As long as you plan for all scenarios and make the proper accommodations for latency, refresh and even voice delay you can effectively test. We're still looking for effective recording technologies to capture screen activity, but beyond that we're pretty satisfied.

Have to agree that nothing beats being there in person, but for most testing this gets the jobs done and provides worthwhile data.

Feel free to email if you want to discuss further.

16 Jul 2003 | Simon said...

I was just thinking yesterday that I wanted to create some web apps for running Card Sorts and Freelisting remotely. I've got a project with very little to no budget and need some way of getting everything done.

Does anyone know of any quality Open Source, Freeware or cheap applications which will do a good job for this or other usability/user tests?

16 Jul 2003 | fajalar said...

You can try IBM's Ease of Use area's EZSort.

EZSort is helpful, though the UI needs some serious help. I have used it to complile data from a card sort. So I entered the information later, rather than having the participants do it during the session. I'd just test it first, of course.:)

And, yup, it's free.

16 Jul 2003 | Steve said...

I haven't tried or observed remote testing. It would certainly be a big help in terms of some logistics, especially in terms of getting either testers or subjects to the test location. But, I'd be a bit worried about losing some of the things you get from in-person testing. There's a lot that hard data can tell you, but there are more subjective things that can prove to be just as important. Watching someone's reactions, hearing how they're talking themselves through a task, etc. often tell as much as or more than the actual data. Perhaps some of the remote testing solutions offer that sort of experience; I haven't researched them to know.

It will be interesting to see if this is an area that continues to be developed and expanded. The benefits of it are quite obvious. If some of the potential shortcomings can be addressed, I could see remote testing becoming almost de rigeur.

28 Nov 2004 | click said...

6118 Very well said chappy.

31 Jan 2005 | compatelius said...

bocigalingus must be something funny.

Comments on this post are closed

Back to Top ^