CAREER: Interaction-oriented 3D Representation Learning on Point Cloud

职业：点云上面向交互的 3D 表示学习

• 批准号: 2240160
• 负责人: Hao Su
• 金额: $ 60万
• 依托单位: University of California-San Diego
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-08-15 至 2028-07-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2240160&HistoricalAwards=false
• 关键词: CAREER; Interaction; oriented; 3D; Representation

The central objective in the field of three dimensional (3D) vision is to leverage perception to plan and execute actions effectively. Consider the process of creating action plans based on visual observations. One might wish to understand the possible alterations to an object after it undergoes a specific action. Interestingly, humans often acquire this knowledge through experiential learning, implying a strong interplay between perception, cognition, and interaction. Inspired by this intertwined relationship, this project endeavors to examine deep learning methodologies for 3D vision within the context of this perception-cognition-interaction cycle. The project will capitalize on the recent advancements in Computer Vision, Machine Learning, and Computer Graphics across three pivotal areas: 3D point cloud data learning, closed-loop policy learning frameworks, and realistic simulation environments. The principal methodology will entail a careful analysis of the relationship between 3D understanding and interaction, and design innovative learning frameworks. These will incorporate representation learning from 3D point clouds for the prediction of actions and by the consequences of actions. The project will enhance the understanding of the three-dimensional world within physically embodied artificial intelligence systems (embodied AI). The ultimate goal is to construct AI systems that can optimally learn from interactive experiences. This research is beneficial for many applications such as smart manufacturing, exploratory robotics, autonomous driving, and augmented reality devices for life and work assistance.This project breaks down the understanding of 3D point cloud data into three distinct categories: comprehension of object structure, grasp of kinematics and dynamics, and perception of interaction. For each dimension, the team will develop novel frameworks that encapsulate the perception-cognition-interaction cycle. Moreover, the research will probe into learning algorithms and 3D neural network architectures that can seamlessly integrate into the cycle. Given the perspective of this research, the project is also expected to unearth a series of challenges not yet extensively researched in 3D vision literature. The team will strive to uncover innovative, principle-based solutions to these challenges. As the project treads new ground and existing datasets are insufficient, the team will also venture into interaction data collection in both virtual and real-world settings. The research philosophy builds on the investigator's prior work in 3D deep learning, generalizable policy learning, and robot simulator design. This foundational work paves the way for 3D representations that can inform the design and learning of interaction policies adaptable across diverse environments and tasks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

三维（3D）愿景领域的核心目标是利用感知来有效地计划和执行行动。考虑基于视觉观察创建行动计划的过程。人们可能希望了解对象进行特定动作后可能的更改。有趣的是，人类经常通过体验式学习获得这些知识，这意味着感知，认知和互动之间存在很强的相互作用。受这种相互交织的关系的启发，该项目努力在这个感知认知相互作用周期的背景下检查深度学习方法的3D视觉。该项目将利用三个关键领域的计算机视觉，机器学习和计算机图形方面的最新进步：3D点云数据学习，闭环策略学习框架和现实的仿真环境。主要方法将需要仔细分析3D理解与互动之间的关系，并设计创新的学习框架。这些将从3D点云中纳入代表性学习，以预测动作和行动的后果。该项目将增强对身体体现的人工智能系统（体现AI）中三维世界的理解。最终目标是构建可以从交互式体验中最佳学习的AI系统。这项研究对许多应用，例如智能制造，探索性机器人技术，自动驾驶以及增强的生活和工作帮助现实设备。该项目将3D点云数据的理解分为三个不同的类别：对对象结构，运动和动态的理解，以及相互作用的感知。对于每个维度，团队将开发封装感知认知相互作用周期的新颖框架。此外，该研究将探究学习算法和3D神经网络体系结构，这些算法可以无缝整合到周期中。从这项研究的角度来看，该项目还有望发掘一系列尚未在3D Vision文献中进行的挑战。该团队将努力揭示针对这些挑战的创新，基于原则的解决方案。由于该项目踩新的地面和现有数据集不足，因此团队还将冒险进入虚拟和现实世界中的交互数据收集。研究理念是基于研究者在3D深度学习，可推广的政策学习和机器人模拟器设计中的先前工作。这项基本工作为3D表示形式铺平了道路，可以在各种环境和任务中适应互动策略的设计和学习。该奖项反映了NSF的法定任务，并被认为是值得通过基金会的知识分子的评估来提供支持的，并具有更广泛的影响。