Poll: What experiment to run on AWS mechanical turk?

Posted by adatta02 on Monday December 15, 2008 @04:15AM
from the self-post dept.

After our experience using the mechanical turk we've become a bit obsessed with trying to find a really awesome use case for the service - even if its just experimental.

We have some downtime right now and we are narrowing down the experiment between the following:

1. Some sort of cooperative art project - inspired by http://www.tenthousandcents.com 2. A massive chess game that forks every time a move is made (probably a poor idea) 3. Some implementation of the Prisoner's dilemma game. 4. Any time of game to get a sense of how risk seeking or risk averse the Turks are - possibly a tie in with the Monty Hall Problem.

Most of these ideas are centered around game theory but we are pretty much open to anything (within reason). Any votes or opinions on what would be the most interesting/entertaining?

-Thanks

( 13 comments | 11 points | on HN )

Re:Poll: What experiment to run on AWS mechanical turk? by AlexeyMK on Monday December 15, 2008 @07:16AM (#398266)

I'd love to see a prisoner's dilemma experiment via MTurk. These tend to be financial in nature, though; can you set variable payment rates per task, depending on external factors?

[ Parent | on HN ]

Re:Poll: What experiment to run on AWS mechanical turk? by sigstoat on Tuesday December 16, 2008 @06:35PM (#400274)

you can pay out arbitrary bonuses once they've answered.

so you could easily do single-round prisoner's dilemma, and have amazon pair up 2 (different) folks at a time, and when you get the results back, look to see what the folks in each pair did, and award bonuses appropriately.

i'm not sure if there is a way to do an iterated version.

[ Parent | on HN ]

Re:Poll: What experiment to run on AWS mechanical turk? by iamelgringo on Monday December 15, 2008 @08:08AM (#398287)

A friend last week told me about a business he's working with. They do Natural Language Processing to scan thousands of articles about a specific topic/product to see if they are positive, negative or neutral. They are then hoping to turn around and sell the business intelligence.

I was wondering how it would compare on a cost/effectiveness basis with the Turk.

Let us know what your experiment is and how you fared.

[ Parent | on HN ]

Re:Poll: What experiment to run on AWS mechanical turk? by MaysonL on Monday December 15, 2008 @05:34PM (#398933)

See if you can use MT to provide the fitness function for some genetic algorithm, evaluate the results, tune it, and profit!

[ Parent | on HN ]

Re:Poll: What experiment to run on AWS mechanical turk? by Deadsunrise on Monday December 15, 2008 @11:57AM (#398484)

Some months ago I tried to set up a project to transcribe Alexander Shulgin's Lab books but Mturk wasn't available outside US.

http://www.erowid.org/library/books_online/shulgin_labbooks/

[ Parent | on HN ]

Re:Poll: What experiment to run on AWS mechanical turk? by lacker on Monday December 15, 2008 @08:51PM (#399092)

Go meta - use Mechanical Turk to ask what would be a really cool idea to run using Mechanical Turk.

[ Parent | on HN ]

Re:Poll: What experiment to run on AWS mechanical turk? by andr on Monday December 15, 2008 @09:34AM (#398352)

Liar's poker. Ask 2 people to pick a number and the one that picks the lower doesn't get paid. The number of people that actually play the game is the result of the experiment.

[ Parent | on HN ]

Re:Poll: What experiment to run on AWS mechanical turk? by eru on Monday December 15, 2008 @10:21AM (#398386)

How about a dollar auction?

[ Parent | on HN ]

Re:Poll: What experiment to run on AWS mechanical turk? by zitterbewegung on Monday December 15, 2008 @06:05AM (#398237)

How about creating an experiment and designing it right on turk? You create a rough sketch and then you auto-correct it using people?

[ Parent | on HN ]

Re:Poll: What experiment to run on AWS mechanical turk? by antigravity on Tuesday December 16, 2008 @08:34AM (#399459)

You mean keep tinkering with the study until it "works"?

If you're not careful, you could get http://en.wikipedia.org/wiki/Publication_bias on a massive new scale...

It would be interesting to try to find scientifically optimal ways of doing study iteration, or approaches to interpreting the complete (experiment_rev, results)-list from another researcher...

[ Parent | on HN ]

Re:Poll: What experiment to run on AWS mechanical turk? by bjclark on Tuesday December 16, 2008 @12:37AM (#399236)

I'd pay $.05 to get the them to up vote my site for certain search terms on Google's new "Search Wiki". See what happens.

[ Parent | on HN ]

Re:Poll: What experiment to run on AWS mechanical turk? by shafqat on Monday December 15, 2008 @07:16AM (#398267)

What was your previous experience?

[ Parent | on HN ]

Re:Poll: What experiment to run on AWS mechanical turk? by adatta02 on Monday December 15, 2008 @02:11PM (#398644)

we used the Turk service to basically do OCR on about 3k images - http://shout.setfive.com/2008/11/20/artificial-artificial-in...

[ Parent | on HN ]