CAREER: Interaction-oriented 3D Representation Learning on Point Cloud

三维（3D）视觉领域的中心目标是利用感知来有效地计划和执行行动。考虑一下基于视觉观察制定行动计划的过程。人们可能希望了解对象在经历特定动作后可能发生的变化。有趣的是，人类通常通过体验式学习获得这些知识，这意味着感知、认知和互动之间存在强烈的相互作用。受这种相互交织的关系的启发，该项目努力在这种感知-认知-交互循环的背景下研究3D视觉的深度学习方法。该项目将利用计算机视觉、机器学习和计算机图形学在三个关键领域的最新进展：3D点云数据学习、闭环策略学习框架和现实模拟环境。主要的方法论将需要仔细分析三维理解和交互之间的关系，并设计创新的学习框架。这些将包括来自3D点云的表示学习，用于预测行动和行动的后果。该项目将增强对物理具体化人工智能系统（embodied AI）内三维世界的理解。最终目标是构建能够从交互式体验中最佳学习的人工智能系统。该研究对智能制造、探索性机器人、自动驾驶、增强现实生活和工作辅助设备等许多应用都是有益的。这个项目将对三维点云数据的理解分为三个不同的类别：对对象结构的理解，对运动学和动力学的掌握，以及对交互的感知。对于每个维度，团队将开发新的框架来封装感知-认知-交互循环。此外，该研究将探索能够无缝集成到循环中的学习算法和3D神经网络架构。鉴于本研究的视角，该项目还有望发掘出3D视觉文献中尚未广泛研究的一系列挑战。该团队将努力发现创新的、基于原则的解决方案，以应对这些挑战。由于该项目涉足新领域，现有数据集不足，该团队还将冒险在虚拟和现实环境中进行交互数据收集。研究理念建立在研究者之前在3D深度学习，可推广的政策学习和机器人模拟器设计方面的工作基础上。这项基础性工作为3D表示铺平了道路，可以为设计和学习适应不同环境和任务的交互策略提供信息。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The central objective in the field of three dimensional (3D) vision is to leverage perception to plan and execute actions effectively. Consider the process of creating action plans based on visual observations. One might wish to understand the possible alterations to an object after it undergoes a specific action. Interestingly, humans often acquire this knowledge through experiential learning, implying a strong interplay between perception, cognition, and interaction. Inspired by this intertwined relationship, this project endeavors to examine deep learning methodologies for 3D vision within the context of this perception-cognition-interaction cycle. The project will capitalize on the recent advancements in Computer Vision, Machine Learning, and Computer Graphics across three pivotal areas: 3D point cloud data learning, closed-loop policy learning frameworks, and realistic simulation environments. The principal methodology will entail a careful analysis of the relationship between 3D understanding and interaction, and design innovative learning frameworks. These will incorporate representation learning from 3D point clouds for the prediction of actions and by the consequences of actions. The project will enhance the understanding of the three-dimensional world within physically embodied artificial intelligence systems (embodied AI). The ultimate goal is to construct AI systems that can optimally learn from interactive experiences. This research is beneficial for many applications such as smart manufacturing, exploratory robotics, autonomous driving, and augmented reality devices for life and work assistance.This project breaks down the understanding of 3D point cloud data into three distinct categories: comprehension of object structure, grasp of kinematics and dynamics, and perception of interaction. For each dimension, the team will develop novel frameworks that encapsulate the perception-cognition-interaction cycle. Moreover, the research will probe into learning algorithms and 3D neural network architectures that can seamlessly integrate into the cycle. Given the perspective of this research, the project is also expected to unearth a series of challenges not yet extensively researched in 3D vision literature. The team will strive to uncover innovative, principle-based solutions to these challenges. As the project treads new ground and existing datasets are insufficient, the team will also venture into interaction data collection in both virtual and real-world settings. The research philosophy builds on the investigator's prior work in 3D deep learning, generalizable policy learning, and robot simulator design. This foundational work paves the way for 3D representations that can inform the design and learning of interaction policies adaptable across diverse environments and tasks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.