CHS: Small: Collaborative Research: Optimizing the Human-Machine System for Citizen Science

This project investigates optimal human-machine collaboration for analysis of large, complex data sets. The project uses advances in machine learning to develop new citizen science system infrastructure to be built into Zooniverse --- a large, open-source platform for online citizen science. Citizen Science is an established method for distributed analysis of large quantities of data in which online volunteers help with tasks requiring human pattern recognition. An example is identifying morphology of galaxies in the Galaxy Zoo project. Much larger data sets are looming on the horizon. Designing a human-machine system to accelerate labeling of known classes at the same time as solving the problem of detecting interesting anomalies (suggesting new phenomena) requires answering several crucial research questions about how humans and machines best complement one another. Since the project's new techniques will be incorporated into Zooniverse, they will be available to all for use in citizen science projects across many disciplines. Additional benefits of this project include engaging over 2 million members of the public who participate in citizen science through Zooniverse, engaging young women in University of Minnesota computer science coding camps, and providing year-long capstone projects for Data Science Masters program students to engage in real-world research while preparing them for careers in data science.

This project will carry out a detailed investigation of load balancing between human and machine classifiers, optimizing for the speed, accuracy, completeness or purity required by the domain research for a given task. The research program follows two thrusts: (1) Classification Efficiency Studies to optimize the classification efficiency of known classes across multiple domains and task types; and (2) Systematized Serendipity Studies to increase the efficiency of discovery, including detection of rare instances, unusual findings, and new classes. The project develops new infrastructure that builds on the existing capabilities of the Zooniverse citizen science platform through two modules: (1) Machine Integration Infrastructure to readily incorporate and combine machines on projects to increase classification efficiency as well as explore the machine-driven component of systematized serendipity; and (2) Leveling-up Strategies for Volunteers to enable human-driven identification of emergent classes. Ultimately, the human- and machine-driven mechanisms will be joined to form a combined human-machine system that will be tested for its ability to identify unknown, rare, or difficult to identify classes.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.