Paper Reading #6: TurKit: Human Computation Algorithms on Mechanical Turk

 References
Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller.  "TurKit: human computation algorithms on mechanical turk". UIST '10 Proceedings of the 23rd annual ACM symposium on User interface software and technology.  ACM New York, NY, USA ©2010.

 Author Bios
Greg Little is a graduate of MIT.

Lydia Chilton is a graduate student at the University of Washington.  She has interned for Microsoft Research in Beijing.  She is also a graduate of MIT.

Max Goldman is a professor at MIT and is part of the User Interface Design Group.

Rob Miller is also a professor at MIT.  Miller is the leader of the User Interface Design Group.

Summary 

• Hypothesis - The researchers hypothesize that using TurKit they can aid in the development of algorithmic tasks for mTurk at the cost of efficiency.
  
• Method/Content - They based their design on what they called the "crash and re-run" paradigm. This is an advanced implementation of dynamic programming; in essence, it allows the expensive functions to run only once and subsequent callings of the same function simply retrieves data stored in a database. Thus when the program crashes, the user can pick and choose which functions to run again, without needing to run the entire thing.
  
• Results - The results were overall positive. The main complaints were that the scripts TurKit could run needed to be deterministic, for if it were to change with different inputs it would need to be run again anyway. Another complaint was that some users did not know of some of the features, but this also may be that they were using it in the early stages of TurKit's development, when some had not been implemented yet. They did, however, discover that the running time of all of the TurKit script is faster than nearly all of the human function calls.
• Content - TurKit is a tool that automates mTurk for ease of use and repetition. It allows users to specify which parts of the program to rerun in the event of a crash. It goes in depth for some of the features. It also describes user reactions to TurKit.
  

 Discussion
The researchers successfully developed a useful tool to help with mTurk. Its crash and re-run design paradigm complements the slow human function runtimes very well. They did state, however, that while it is useful for research, it is doubtful that it will be useful in any large-scale project. While it is hard to say whether human computational resources are truly a valuable resource are not is up to mTurk, but TurKit is indeed a useful tool to be used with mTurk. I hope they can continue their development of TurKit so they may scale the tool for large-scale projects.