CAREER: Spatial Awareness for Machine Perception

For many applications in health, security, and robotics, the capability for computers to track objects in video is critical. From autonomous vehicles that reliably avoid obstacles to security cameras that swiftly respond during emergencies, artificial perception systems need to remain accurate during poor visibility situations, such as with cluttered rooms, dark streets, or occluded objects. This integrated research and education project aims to develop machines that are able to spatially sense their visual surroundings, even when the surroundings have low visibility with significant obstructions. On multiple levels, the investigators will furthermore use their integrated approach to advance education in artificial intelligence, such as outreach activities for K-12 students and streamlined curriculums for both undergraduate and graduate education in computer vision.

The research program introduces a framework that tightly integrates geometry, motion, and acoustics in order to learn rich spatial representations of natural scenes from unlabeled visual data. The research team will study how learning from the incidental structures of unlabeled videos will cause rich spatial representations to emerge without human supervision. The framework will leverage the propagation of sound through objects to learn visual tracking under occlusion. In other cases, the project will use scene geometry to transfer incidental supervision between camera views in order to learn spatial memory models. A key advantage of the approach is the efficiency to operate on videos that span both large physical spaces and long temporal horizons. Motivated by implicit surfaces in mathematics and graphics, the algorithm will analytically represents videos as continuous functions, which are compact and robustly embed video dynamics in 3D space.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.