- Title:
- TurKit: Human Computation Algorithms on Mechanical Turk
- Reference Information:
- Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller. 2010. TurKit: human computation algorithms on mechanical turk. In <em>Proceedings of the 23nd annual ACM symposium on User interface software and technology</em> (UIST '10). ACM, New York, NY, USA, 57-66. DOI=10.1145/1866029.1866040 http://doi.acm.org/10.1145/1866029.1866040
- UIST 2010 New York, New York.
- Author Bios:
- Greg Little is affiliated with both Arizona State University as well as Massachusetts Institute of Technology. In the last 6 years he has published 22 papers through the ACM.
- Lydia Chilton is a graduate student at the University of Washington. She has interned for Microsoft Research in Beijing. She received both her Bachelor's degree as well as her Master's degree at the Massachusetts Institute of Technology.
- Max Goldman is a professor at the Massachusetts Institute of Technology. Goldman has 12 publications over the last 4 years and is associated with the User Interface Design Group
- Rob Miller is a professor at the Massachusetts Institute of Technology. Miller is the leader of the User Interface Design Group at the University. Miller is interested in web automation.
- Summary
- Hypothesis:
- Researchers hypothesized that abstracting human computation, through MTurk, as a function call and applying a crash-and-rerun programming style allows users to design experiments that can be duplicated and modified in a uniform method.
- Methods
- Researchers developed an Integrated Development Environment that allowed users to create and run their algorithms that involve human computation. The system uses wrappers around the MTurk API and added functionality such as a keyword 'once'. The additional functionality and underlying database enabled the crash-and-rerun programming model by only incurring costly function calls (a call to MTurk for example) only once, simply retrieving results stored in the database.
- Results
- The researchers have explored the use of Turkit over the past year. When some of the experiments were run they were utilizing an early version of the product so many of the beneficial aspects were not implemented yet. Their results show that the crash-and-rerun model can handle smaller projects. The time it takes for the entire script to rerun is faster than nearly all of the human function calls. The researchers also posted their own experiments and databases online so that anyone can replicate their results.
- Contents
- Turkit automates the use of MTurk to allow for ease of use and repetition. Key functionality for this project is the ability to do post-operation line debugging and modifications between reruns. This allows users to produce a small program and edit it during operation to improve it on the next run.
- Discussion
- Researchers certainly met some of their goals, proving that it is possible to include human computation inside of a generic algorithm. The slow time response of human computation works well with their crash-and-rerun programming model. This is an interesting approach to systematically use the MTurk capabilities provided by Amazon. I like how this can be used for Research but, as they stated, I doubt that it would be very useful for large scale production projects. However, their idea is a good start and I hope that they continue to do research to ramp up their project for longer programs.
Picture Source: "TurKit: Human Computation Algorithms on Mechanical Turk"
No comments:
Post a Comment