CRII: RI: Towards Learning Skills from First Person Demonstrations

Humans learn a skill from an expert's demonstrations such as playing tennis, which requires understanding subtle details of sequential actions, e.g., eye-hand coordination across swing motion. This project develops technologies to learn such skills by observing demonstrations from first-person videos. The first-person videos are highly dynamic, local, and person-biased due to severe head movements, which generates a larger variation of visual data. Analyzing the videos produced by the head-mounted camera system is challenging because state-of-the-art computer vision systems built upon third-person videos cannot be directly applied. The research team addresses these challenges by developing both hardware and computational models. The principal investigator of the project will integrate the research results into a sequence of newly designed computer vision courses in the University of Minnesota. The research team will publicly share the dataset, representation, and trained models, and organize workshops and tutorials to broader audiences in computer vision and robotics.

This research investigates problems in learning skills from first person demonstrations. This project designs a head-mounted camera system composed of a first-person camera and multiple proxemic cameras that can fully cover the space of interactions. The project also develops a new representation specific to the head-mounted camera system, called proxemic affordance map, to efficiently represent visual scene and action in 3D. The proxemic affordance map encodes 3D visual semantics in a form of 3D depth map, visual attention, and body pose, which enables measuring the correlation between action and its surroundings. This allows learning the dynamics of proxemic affordance map to model diverse physical activities, e.g., how an action will change the state of its surrounding contexts.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.