Collaborative Research: NRI: INT: Dense 3D Reconstruction of Dynamic Actors in Natural Environments using Multiple Flying Cameras

While large-scale multi-camera domes have been developed for data collection in controlled laboratory settings it is not possible to achieve a similar level of measurement quality outdoors where there is much potential benefit to such data collection. For example, use of such measurements include the body dynamics of a running cheetah, or people, or analyzing herding behaviors of animals or birds. This leads to scientists relying on extremely inefficient and dangerous data collection methods. For example, biologists studying the behaviors of wild animals try to predict where the animals will be and place some cameras which only give some limited data at specific locations. This project addresses such challenges by exploring the research of methods and development of a large-scale data collection tool for high-resolution and multi-viewpoint visual recording and motion analysis of natural group behaviors (e.g., herds of animals or groups of people) in-the-wild over very large environments (e.g., desert plains or mountain sides) using a team of flying robots.

This project develops computational models that integrate the fundamentals of computer vision and multi-agent control to measure the group of actors in 3D. Through the development of this system, this project will make major advances in technology at the intersection of perception and control that include: (1) a new study of methods for precise, rapid, and robust target motion forecasting and relative state estimation that estimates the 3D motion of the robots and actors quickly with strong uncertainty estimates; (2) a new decomposition of the perception-aware multi-objective multi-UAV safe motion planning problem, that allows long-term planning based on consistent actor forecasting uncertainty models and coverage objectives; (3) a new guaranteed safe but adaptive paradigm for reactive flight control that is able to generate safety maneuvers even under large disturbances and vehicle dynamics changes, and that can leverage prior flight experience for real-time adaptation; (4) new theory of 3D reconstruction for dynamic scenes captured by UAVs that will enable high-resolution mesh and skeletal reconstruction of the groups of actors. The research outcome will be disseminated through multiple educational activities.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.