Sortfolio uses Amazon’s Mechanical Turk (“a global, on-demand, 24×7 workforce”) to make sure no NSFW images make it into the galleries. It’s pretty amazing how quickly the processing happens. A visitor uploads an image. Above is the review screen the “turks” see in order to approve/reject. Then the image goes live. The whole thing takes just a few seconds and costs us only a penny per image.
Full disclosure: Amazon’s Jeff Bezos is an investor in 37signals.
AK
on 26 Mar 10Seconds? Wow. A very good idea.
How to you handle mistakes (i.e. when porn is approved, or when an acceptable image is rejected)? I imagine that getting each image classified by a few turks would do it.
Alex M
on 26 Mar 10Online slave labor. Nice guys.
The Real Josh
on 26 Mar 10Looks like the Mechanical Turk is quite efficient. I just noticed I live in one of the only states without a provider listed. So sad.
James S
on 26 Mar 10@Alex M, how many images can you look at in an hour? Assuming each image takes 3 seconds (load, look, click), that’s 20 a minute, or $12.00/hr. Pretty decent pay for just clicking Yes/No. Mindnumbing, sure, but hardly slave labor.
I like the solution, well thought out.
Sam
on 26 Mar 10I also question the ethics of this.
Alex M
on 26 Mar 10@James S
I’ve been testing it out so far and you can’t look at 20 images in an hour. Many of the jobs fall far below minimum wage, at least in the state I’m in.
Anonymous Coward
on 26 Mar 10I see this as a win-win. How many other jobs are there where you can potentially get paid to look at porn?
Alex M
on 26 Mar 10Look, some of the jobs are better than others. But, the problem is the service is skirting labor laws by classifying what is really hourly labor as per item contract work, which it isn’t.
Example: Instead of hiring someone $20 bucks to cut my lawn (takes 2 hours, let’s say). I work out how many square feet of lawn I have and then charge per square foot, each foot of grass being “per item contract work,” making sure that this never amounts to more than the $20 bucks I would normally charge to cut my lawn.
Antti
on 26 Mar 10I think these turks are working on their best idea right now.
Stephanie
on 26 Mar 10It is a great task for AMT and a needed task for people who manage UGC sites. We at CrowdFlower have been doing this task for our customers with extra, statistically backed quality control (just in case a worker flags good images or doesn’t flag porn).
The other reason we like using Amazon Turkers for this type of task is we can set the parameters of what is acceptable. For instance, suggestive pictures might be ok for one site, but bad for another.
“I know it when I see it,” surely applies more to humans than computers in this case.
ridata
on 26 Mar 10Slavery: A form of forced labor in which people are considered to be the property of others.
No one is forcing you. Playing checkers is more of a form of slavery than this—you have to work completely without compensation.
Online “slave labor” is harmless. There is no boss threatening you if you quit. No one saying you should continue doing it. Only yourself. :-)
Alex M
on 26 Mar 10Okay Ridata, if you like playing semantics. Let’s call it unethical labor.
I think it’s just plain f*cked up that 37signals leases out these images for $99 bucks and pays people literally pennies to approve them, instead of hiring someone for a fair hourly wage to do the work of approving images.
Sean
on 26 Mar 10When I first heard about Amazon’s Mechanical Turk, I thought it was unethical, too. But even if you can manage $1 per hour, that’s a pretty fucking good wage in some parts of the world.
Beats working at a Nike sweat shop.
ALF
on 26 Mar 10In regards to slave labor, most “turkers” are not from a 3rd world country. A majority of the demographic consists of professionals who are occupying their free time at work and home with extremely simple tasks that pay something.
Alex
on 26 Mar 10Am I the only one who sees a usability issue here?
1. For a tidious mind-numbing tasks of looking at hundreds of images the question should be shortened significantly to sound more like “Any porn or gore?” 2. Buttons should be simple “Yes” and “No” without additional descriptions like Approve, Reject, etc.
Anonymous Coward
on 26 Mar 10@ALF – Do you have any stats to support that claim?
TylerB
on 26 Mar 10Surprised no one has commented on the poor UI yet. “Click YES to Approve” ... why wouldn’t the button itself just say “approve” ? Oh wait it does….
Frankerson P
on 26 Mar 10I would have signed up to help you guys out, but my offensive image recognition skills are honed so sharply as to earn me a cool 2 pennies per image. Sorry 37Signals, you just can’t afford talent like me.
ALF
on 26 Mar 10http://www.ics.uci.edu/~jwross/pubs/SocialCode-2009-01.pdf
(and my own personal experiences, we do a ton of data validation using turkers)
Sean
on 26 Mar 10Wait, you have investors? Somehow I got the impression you wouldn’t take outside investment.
Berserk
on 26 Mar 10@Sean:
From the archives.
Doug
on 26 Mar 10Amazon Turk Machine is a great idea IMO. I have a friend with some mild developmental disabilities. I’ve been thinking of pointing him at this—it’s something he is capable of, and he has a ton of free time sitting at home all day. At least it’s productive work.
Mike Hagstrom
on 26 Mar 10@Alex M
Wait, so you think its unethical for another person to do what they want for what wage they see fit for the task?
But its ethically correct for you to tell them what to do with their lives?
If it cost more 37signals probably wouldn’t pay it therefore there would be no service, and whoever, instead of getting paid something, would be getting nothing.
Bottom Line: It’s not your life, don’t try to run it.
Justin
on 26 Mar 10Another benefit of Mechanical Turk mentioned in the post but overlooked in the comments: its speed. Let’s imagine 37signals hired an employee with the sole task of approving Sortfolio images. If a company submits an image at 3:00 a.m. Chicago time, its approval would most likely have to wait until that person starts their work day (people have to sleep sometime, you know). By using Mechanical Turk, approvals can happen at any time on any day.
From a Sortfolio user’s perspective, I’d rather see “your image will appear in a couple of minutes” than “your image will appear whenever our approval team gives it the OK.”
Tom Powell
on 26 Mar 10I did an experiment last year to see how much I could make: turned out to be about $2.18 an hour. http://coinnovative.com/the-mechanical-turk-experiment-how-i-made-218-an-hour-and-how-you-can-too/
I would also add that, if this is available to workers in other countries with lower per capita incomes, the hourly rate could work well for them.
Sam
on 26 Mar 10@TylerB: That’s not what it says. It says ‘YES to reject’, not ‘Click YES to approve’. Yes and no are answers to the question, and approve/reject are the associated actions.
Yes/no in the context of buttons are ambiguous. What does a button that says ‘yes’ do? ‘Approve’ and ‘reject’ are much clearer.
Mike Hagstrom
on 26 Mar 10@Sam
I think I would have skipped out on this layer of complexity though, people associate no with rejection and yes with acceptance, to turn them around is confusing.
Just have the buttons be yes and no, and skip out the part on acceptance rejection.
Especially when you might have people looking at these vary fast, or who don’t speak english that well.
Scott
on 26 Mar 10I agree that the UI language is goofy. Both affirmatives (“Yes”/Approve) should be together by rephrasing the question OR the Approve/Reject step should be removed and let them just answer the one question. You’re paying them to answer the question, not to decide approve/reject.
J
on 26 Mar 10So now we know how to game the system. Upload dodgy image. Sign up to Turk. Approve our own image.
Peng Zhong
on 27 Mar 10I’m not a fan of the ambiguous UX language either. I knocked out a quick mockup with better usability here:
http://nylira.com/p/turk
Qz
on 27 Mar 10Why not just put two buttons at the top of the page:
[this looks fine!] [ACK—PORN!]
Steven Wagner
on 27 Mar 10Jason and team, I assume you describe to the Turks what pornographic or offensive images are?
We have been looking at using it to check pages for our clients to ensure their dealers are properly using their brand. But the challenge is identifying proper product or brand images if you are unfamiliar with the offering.
Could you overlay the page with a reference image of for say, does this page have a 37signals logo (loge thumbnail in the header)?
fbm
on 28 Mar 10So do you get a lot of offensive images? Care to share some stats?
Tathagata
on 28 Mar 10Major UX goof up as pointed out by many above, including @TylerB @Sam @Mike Hagstorm, and @Scott. This was the first thing that caught my eye.
How could you guys have messed this up :) You probably were in a hurry of some sort :)
JD
on 28 Mar 10Hey guys, thanks for the feedback. It is quite possible that your suggestions might work even better.
We spent about 10 minutes creating this UI. It has worked for approving thousands of images. That’s all that really matters.
Adam
on 29 Mar 10@Peng Zhong
Nice mock up.
Michael S
on 29 Mar 10I don’t get the slave labor comments. What’s unethical about it? Workers know what they get paid and aren’t coerced into anything.
As for usability, Peng Zhong’s idea is an improvement by getting rid of the “translation” but how about also moving the question so it’s near the buttons (or the other way around) for a more direct correlation?
This discussion is closed.