FMRG: Adaptable and Scalable Robot Teleoperation for Human-in-the-Loop Assembly

FMRG：用于人在环装配的适应性和可扩展的机器人远程操作

• 批准号: 2037101
• 负责人: Steven Feiner
• 金额: $ 374.92万
• 依托单位: Columbia University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-01-01 至 2025-12-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2037101&HistoricalAwards=false
• 关键词: FMRG; Adaptable; Scalable; Robot; Teleoperation

The COVID-19 pandemic has accelerated the adoption of remote working in many industries. The ability for employees to work remotely, often from home, has become crucial to an organization's long-term resilience and growth potential. However, while advances in software and networking have made it possible for information workers to work remotely, most manufacturing workers cannot, because the infrastructure that is needed doesn't exist. This Future Manufacturing (FM) project will research an adaptable and scalable robot teleoperation system that allows factory workers to work remotely. The research will benefit both the manufacturing industry and the workforce by increasing access to manufacturing employment and improving working conditions and safety. By combining human-in-the-loop design with machine learning, this research can broaden the adoption of automation in manufacturing to new tasks. Beyond manufacturing, the research will also lower the entry barrier to using robotic systems for a wide range of real-world applications, such as assistive and service robots. The research team is collaborating with NYDesigns and LaGuardia Community College to translate research results to industrial partners and develop training programs to educate and prepare the future manufacturing workforce.This research suggests three key ideas to enable human-in-the-loop assembly: First, the system uses a physical scene understanding algorithm that converts the real-world robot workspace into a virtual manipulable three-dimensional scene representation. Next, a three-dimensional Virtual Reality user interface will be used to allow users to specify high-level task goals using this scene representation. Finally, the system uses a goal-driven reinforcement learning algorithm to infer an effective planning policy, given the task goals and the robot configuration. This system can overcome several limitations of existing teleoperation systems. By separating high-level task planning from low-level robot control using a physical scene representation, the system allows the operator to specify task goals without having expert knowledge of the robot hardware and configuration. By using reinforcement learning for low-level control, the system is more generalizable to new tasks and hardware.This award is co-funded by the Divisions of Civil Mechanical and Manufacturing Innovation, Electrical, Communications and Cyber Systems, Computer and Network Systems, Undergraduate Education, and Behavioral and Cognitive Sciences and the Cyber Physical Systems, NSF Scholarships in Science, Technology, Engineering, and Mathematics, and Advanced Technological Education Programs.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

COVID-19大流行已加速了在许多行业的远程工作。员工经常在家中远程工作的能力对组织的长期韧性和增长潜力至关重要。但是，尽管软件和网络的进步使信息工作者可以远程工作，但大多数制造业工人都不能，因为不存在所需的基础架构。这个未来的制造项目（FM）项目将研究一个适应性和可扩展的机器人遥控系统，该系统使工厂工人可以远程工作。这项研究将通过增加获得制造业就业机会并改善工作条件和安全性来使制造业和劳动力受益。通过将人类的设计与机器学习相结合，这项研究可以扩大制造业中自动化的采用。除制造业外，该研究还将降低进入障碍，以在辅助和服务机器人等广泛的现实应用程序中使用机器人系统。 The research team is collaborating with NYDesigns and LaGuardia Community College to translate research results to industrial partners and develop training programs to educate and prepare the future manufacturing workforce.This research suggests three key ideas to enable human-in-the-loop assembly: First, the system uses a physical scene understanding algorithm that converts the real-world robot workspace into a virtual manipulable three-dimensional scene representation.接下来，将使用三维虚拟现实用户界面来允许用户使用此场景表示形式指定高级任务目标。最后，考虑到任务目标和机器人配置，系统使用目标驱动的增强学习算法来推断有效的计划政策。该系统可以克服现有的远程操作系统的几个局限性。通过使用物理场景表示形式将高级任务计划与低级机器人控件分开，该系统允许操作员在不具有机器人硬件和配置的专家知识的情况下指定任务目标。通过使用强化学习来进行低级控制，该系统更容易被新任务和硬件概括。该奖项由民用机械和制造创新，电气，通信和网络系统，计算机和网络系统，教育以及行为和认知科学以及网络物理系统，NSF技术，技术，机构，机构，机构，机构，机构，机构，机构，机构，机构，机构和机构的奖项共同资助。奖项反映了NSF的法定任务，并通过使用基金会的智力优点和更广泛的影响审查标准评估值得支持。

项目成果

A Testbed for Exploring Virtual Reality User Interfaces for Assigning Tasks to Agents at Multiple Sites

用于探索虚拟现实用户界面以将任务分配给多个站点的代理的测试平台

• DOI: 10.1145/3607822.3618004
• 发表时间: 2023
• 期刊: SUI '23: Proceedings of the 2023 ACM Symposium on Spatial User Interaction
• 作者: Liu, Jen-Shuo;Wang, Chongyang;Tversky, Barbara;Feiner, Steven
• 通讯作者: Feiner, Steven

TANDEM: Learning Joint Exploration and Decision Making With Tactile Sensors

TANDEM：使用触觉传感器学习联合探索和决策

• DOI: 10.1109/lra.2022.3193466
• 发表时间: 2022
• 期刊: IEEE Robotics and Automation Letters
• 影响因子: 5.2
• 作者: Xu, Jingxi;Song, Shuran;Ciocarlie, Matei
• 通讯作者: Ciocarlie, Matei

Using Multi-Level Precueing to Improve Performance in Path-Following Tasks in Virtual Reality

使用多级预提示提高虚拟现实中路径跟踪任务的性能

• DOI: 10.1109/tvcg.2021.3106476
• 发表时间: 2021
• 期刊: IEEE Transactions on Visualization and Computer Graphics
• 影响因子: 5.2
• 作者: Liu, Jen-Shuo;Elvezio, Carmine;Tversky, Barbara;Feiner, Steven
• 通讯作者: Feiner, Steven

Precueing Sequential Rotation Tasks in Augmented Reality

在增强现实中预先执行顺序旋转任务

• DOI: 10.1145/3562939.3565641
• 发表时间: 2022
• 期刊: VRST '22: Proceedings of the 28th ACM Symposium on Virtual Reality Software and Technology
• 作者: Liu, Jen-Shuo;Tversky, Barbara;Feiner, Steven
• 通讯作者: Feiner, Steven

Cueing Sequential 6DoF Rigid-Body Transformations in Augmented Reality

增强现实中的连续 6DoF 刚体变换

• DOI: 10.1109/ismar59233.2023.00050
• 发表时间: 2023
• 期刊: Proceedings of the 2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR
• 作者: Liu, Jen-Shuo;Tversky, Barbara;Feiner, Steven
• 通讯作者: Feiner, Steven