CAREER: Learning Predictive Models for Visual Navigation and Object Interaction

在新的环境中有效地移动并与对象交互需要建立对人的期望（例如，迎面而来的人将经过哪一侧），地点（例如，车钥匙可能在家里的什么地方），以及物品（例如，门是怎么开的？”然而，手动将这些期望构建到决策系统中是具有挑战性的。与此同时，机器学习已被证明是成功的，在许多相关的应用领域从训练数据集提取代表性的模式。虽然使用机器学习来学习决策预测模型似乎很有前途，但设计选择（数据源、监督形式、预测模型的架构以及预测模型与决策的交互）是紧密交织在一起的。作为该项目的一部分，研究人员将确定机器学习有利于导航和对象交互的确切方面;并共同设计数据集，模型和学习算法，以构建实现这些好处的系统。该项目将通过设计可以利用大规模不同数据源进行训练的方法，改进导航和对象交互的预测推理的最新技术。该项目开发的模型、数据集和系统将提高导航和移动的操作能力。这将使实际的下游应用成为可能（例如，辅助机器人，远程呈现），并为后续研究开辟途径（例如，人机交互）。该项目将通过课程开发、参与研究项目和研究成果的可访问传播为学生和更广泛的社区教育做出贡献。该项目将共同设计数据收集方法、学习技术和政策架构，以实现对涉及导航和移动的操作问题的人、地和物的预测模型的大规模学习。研究人员将解决以下三个研究任务：（1）为决策所需的人员，地点和对象设计预测模型;（2）识别数据源并生成监督，以大规模学习这些预测模型;以及（3）有效使用学习的预测模型的分层和模块化策略架构。研究者将在适用的情况下重复使用现有的感觉-计划-控制组件（运动规划器、反馈控制器）（例如，自由空间中的运动），并在需要推测的模块中引入学习（即，高级决策模块，例如，识别有希望的探索方向，预测迎面而来的人下一步会去哪里，什么是打开抽屉的好位置）。调查人员将通过比较系统的效率与不预测reasoning.This奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准的评估支持的效率进行评估的方法。

Efficiently moving around and interacting with objects in novel environments requires building expectations about people (e.g., which side will an oncoming person pass by), places (e.g., where are car keys likely to be in a home), and things (e.g., which way will a door open). However, manually building such expectations into decision-making systems is challenging. At the same time, machine learning has been shown to be successful in extracting representative patterns from training datasets in many related application domains. While the use of machine learning to learn predictive models for decision-making seems promising; the design choices (data sources, forms of supervision, architectures for the predictive models, and interaction of the predictive models with decision-making) are deeply intertwined. As part of this project, investigators will identify the precise aspects in which machine learning benefits navigation and object interaction; and co-design datasets, models, and learning algorithms to build systems that realize these benefits. The project will improve the state-of-the-art of predictive reasoning for navigation and object interaction by designing approaches that can leverage large-scale diverse data sources for training. Models, datasets, and systems developed in this project will advance navigation and mobile manipulation capabilities. These will enable practical downstream applications (e.g., assistive robots, telepresence), and open up avenues for follow-up research (e.g., human-robot interaction). The project will contribute to the education of students and the broader community through curriculum development, engagement in research projects, and accessible dissemination of research.The project will co-design data collection methods, learning techniques, and policy architectures to enable large-scale learning of predictive models for people, places, and things for problems involving navigation and mobile manipulation. Investigators will tackle the following three research tasks: (1) designing predictive models for people, places, and objects that are necessary for decision making; (2) identifying data sources and generating supervision to learn these predictive models at-scale; and (3) hierarchical and modular policy architectures that effectively use the learned predictive models. Investigators will re-use existing sense-plan-control components (motion planners, feedback controllers) where applicable (e.g., motion in free space), and introduce learning in modules that require speculation (i.e., high-level decision-making modules, e.g., identifying promising directions for exploration, predicting where will an oncoming human go next, what is a good position to open a drawer from). Investigators will evaluate the effectiveness of proposed methods by comparing the efficiency of systems with and without predictive reasoning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.