CAREER: Learning Predictive Models for Visual Navigation and Object Interaction

在新颖的环境中高效地四处移动和与物体交互需要建立对人(例如，迎面而来的人会从哪一边经过)、地点(例如，汽车钥匙可能放在家里的哪里)和物品(例如，门将向哪一方向打开)的预期。然而，在决策系统中手动构建这样的预期是具有挑战性的。同时，机器学习已经被证明能够成功地从许多相关应用领域的训练数据集中提取具有代表性的模式。虽然使用机器学习来学习用于决策的预测模型似乎很有前途，但设计选择(数据源、监督形式、预测模型的体系结构以及预测模型与决策的交互)是紧密交织在一起的。作为该项目的一部分，研究人员将确定机器学习在哪些方面有利于导航和对象交互；并共同设计数据集、模型和学习算法，以建立实现这些好处的系统。该项目将通过设计能够利用大规模不同数据来源进行培训的方法，改进导航和对象交互预测推理的最新水平。该项目中开发的模型、数据集和系统将提升导航和移动操作能力。这些将实现实际的下游应用(例如，辅助机器人、远程呈现)，并为后续研究(例如，人与机器人的交互)开辟道路。该项目将通过课程开发、参与研究项目和可访问的研究传播来促进对学生和更广泛社区的教育。该项目将共同设计数据收集方法、学习技术和政策架构，以实现对涉及导航和移动操作的问题的人、地点和事物的预测模型的大规模学习。研究人员将处理以下三项研究任务：(1)为决策所需的人、地点和对象设计预测模型；(2)识别数据源并生成监督，以大规模学习这些预测模型；以及(3)有效使用学习的预测模型的分层和模块化政策架构。研究人员将在适用的情况下重新使用现有的感觉-规划-控制组件(运动规划器、反馈控制器)(例如，在自由空间中的运动)，并在需要推测的模块中引入学习(即，高级决策模块，例如，确定有希望的探索方向，预测迎面而来的人下一步将去哪里，从什么位置打开抽屉)。调查人员将通过比较有和没有预测性推理的系统的效率来评估建议方法的有效性。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Efficiently moving around and interacting with objects in novel environments requires building expectations about people (e.g., which side will an oncoming person pass by), places (e.g., where are car keys likely to be in a home), and things (e.g., which way will a door open). However, manually building such expectations into decision-making systems is challenging. At the same time, machine learning has been shown to be successful in extracting representative patterns from training datasets in many related application domains. While the use of machine learning to learn predictive models for decision-making seems promising; the design choices (data sources, forms of supervision, architectures for the predictive models, and interaction of the predictive models with decision-making) are deeply intertwined. As part of this project, investigators will identify the precise aspects in which machine learning benefits navigation and object interaction; and co-design datasets, models, and learning algorithms to build systems that realize these benefits. The project will improve the state-of-the-art of predictive reasoning for navigation and object interaction by designing approaches that can leverage large-scale diverse data sources for training. Models, datasets, and systems developed in this project will advance navigation and mobile manipulation capabilities. These will enable practical downstream applications (e.g., assistive robots, telepresence), and open up avenues for follow-up research (e.g., human-robot interaction). The project will contribute to the education of students and the broader community through curriculum development, engagement in research projects, and accessible dissemination of research.The project will co-design data collection methods, learning techniques, and policy architectures to enable large-scale learning of predictive models for people, places, and things for problems involving navigation and mobile manipulation. Investigators will tackle the following three research tasks: (1) designing predictive models for people, places, and objects that are necessary for decision making; (2) identifying data sources and generating supervision to learn these predictive models at-scale; and (3) hierarchical and modular policy architectures that effectively use the learned predictive models. Investigators will re-use existing sense-plan-control components (motion planners, feedback controllers) where applicable (e.g., motion in free space), and introduce learning in modules that require speculation (i.e., high-level decision-making modules, e.g., identifying promising directions for exploration, predicting where will an oncoming human go next, what is a good position to open a drawer from). Investigators will evaluate the effectiveness of proposed methods by comparing the efficiency of systems with and without predictive reasoning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.