CHS: Small: Collaborative Research: Optimizing the Human-Machine System for Citizen Science

This research aims to improve the efficiency, accuracy, and usability of online systems supporting citizen science, in which communities organized around serious scientific research projects combine the contributions of amateurs and professionals. In order to respond most efficiently to the increasing data deluge across multiple domains, citizen science platforms need to be more dynamic and complex - incorporating intelligent task assignment and machine learning strategies. Systems that make use of both human and machine intelligence are of interest to scientists from a wide range of disciplines. Whether viewed as social machines or as active learning systems in which progressive input from humans improves machine learning, these hybrid systems exhibit complex behavior which needs to be understood for effective system design. For example, machine learning researchers have concentrated on using the large training sets produced by citizen science projects in order to train algorithms that are later applied to a full dataset. Yet this serial processing may not be the most efficient use of the human or machine effort. The main research goal of this project is to investigate how the overall efficiency of the combined human-machine system is impacted by the separate components and their related properties and what the implications are for either human or machine classifiers or both. This process will test the hypothesis that improved overall efficiency will actually reduce the load on expert human classifiers instead of, as currently required, needing larger expert training sets for machines.

This project will investigate the dynamic combination of human and machine classifiers, gaining for the first time knowledge of how load can be optimally shared in a real, flexible citizen science platform. This research effort will be supported by building and deploying software modules on the existing Zooniverse infrastructure, the world-leading platform for online citizen science. It will (1) carry out efficient and dynamic task assignment, distinguishing in near-real time between experienced and inexperienced, and between skilled and less skilled classifiers; and (2) combine human and machine classifications dynamically, periodically training automatic classification routines on the increasing volume of training data produced by volunteers. This new software will then be utilized in a novel 'cascade filtering' mode that reduces complex classification problems into a series of single binary tasks. The software developed in this project will provide domain scientists and social machine researchers who wish to exploit the new infrastructure with a fully flexible suite of functions appropriate to the needs defined by their specific problems.