Paper Reading #21: Human model evaluation in interactive supervised learning

References

Rebecca Fiebrink, Perry R. Cook, and Dan Trueman.  "Reflexivity in digital anthropology". UIST '11 Proceedings of the 23rd annual ACM symposium on User interface software and technology.  ACM New York, NY, USA ©2011.

 Author Bios

Rebecca Fiebrink is an assistant professor in Computer Science and affiliated faculty in Music. Until recently she was a postdoc at the University of Washington.

Perry Cook earned his PhD from Stanford University.  His research interests lie in Physics-based sound synthesis models.

Dan Trueman is a professor at Princeton University.  In the last 12 years he has published 6 papers through the ACM. He is also a musician, primarily with the fiddle and the laptop.

Summary 

• Hypothesis - The researchers hypothesized that Interactive Machine Learning (IML) would be a useful tool to improve the current generic machine learning processes currently used.
• Method -The researchers first developed a system of IML to help with music composition, called Wekinator. They then conducted three studies. The first study included several PhD students aimed towards improving the system itself. They used the software while composing their own music, and met regularly to discuss their experiences and suggest improvements for the software. The second study involved undergraduates. They were told to use the software in an assignment specifically geared towards supervised learning in interactive music performance systems. The third and final study had a professional cellist use the system to create a gesture recognition system. The gestures were to provide correct musical notation, such as staccato.
• Results -  Although some results were expected, they also ran into a few things they had not. For one, users tended to overcompensate; that is, they provided more than enough information to make sure the system got it right. Also, the system's performance sometimes surprised users, encouraging them to expand their ideas of the desired goal.
  
• Content - The researchers observed as users interacted with the machine learning software. They found that while users liked the cross-validation, most of them preferred direct validation. The IML was determined to be useful because of its ability to continuously improve the effectiveness of the learning model itself.

 Discussion

This paper was very well done. The experiments were well thought out, carried out, and explained. The proved their hypothesis and were successful in explaining why. Using three independent studies, they were able to compile a large amount of data to use. I think that these results will be very useful, not just in the application they chose but in a widespread realm of problems.