NRI: Hierarchical Representation Learning for Robot Assistants

More than eighteen million people in North America have a physical disability due to limited mobility, restricting their independence, lifestyle, and ability to perform daily activities. One in five older adults struggle with mobility, millions of people with limited mobility are veterans, and a significant number of people have limited mobility because of diseases and accidents. Due to recent advances in artificial intelligence, robots hold great promise to provide timely assistance to people with disabilities, and drive improvements to their quality of life, independence, and productivity. This project introduces an automated robot assistant that is able to recognize a person's goal, and provide them the right object at the right time, thereby helping people perform complex activities, such as cooking, object repair, and housekeeping. From both sight and dialogue, the research products will be able to anticipate what objects a person will need in the near future, and deliver it at exactly the right moment. Furthermore, the project will generate new educational opportunities at the intersection of robotics, computer vision and natural language processing through a series of systematically designed curriculum and annual capstone projects for assistive robotics. Due to the tight integration of multiple disciplines and the large practical impact, these educational programs will serve as an excellent platform for training the next generation of roboticists and increasing the diversity in the field.

This research project introduces a novel hierarchical representation learning framework for assistive robots, which serves as a common interface to drive integration between robotics, computer vision, and natural language understanding. The project includes three thrusts. First, the team will develop hierarchical task representations. Second, human intention prediction and verification will developed. The final thrust will address intention-aware planning. Unlike established state representations in robotics, the new representation leverages non-Euclidean geometry, such as hyperbolic manifolds. Since hyperbolic space is a continuous analog of a tree, it provides new opportunities for learning task hierarchies from large-scale unlabeled instructional videos. This task representation is able to anticipate the activities of people, steer dialogue to reduce uncertainty, and provide dense rewards for long-horizon planning. This representation is learned from large-scale unlabeled instructional videos, making this approach flexible and adaptable to the many real-world applications of just-in-time object delivery for people with limited mobility.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.