RI:Small: Capturing, Perceiving, and Rendering of Artistic Skills for Real-time Interactive Creation of Art

RI:Small：捕捉、感知、渲染艺术技巧，实现艺术的实时互动创作

• 批准号: 2008302
• 负责人: Seth Hutchinson
• 金额: $ 44.95万
• 依托单位: Georgia Tech Research Corporation
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2008302&HistoricalAwards=false
• 关键词: RI; Small; Capturing; Perceiving; Rendering

Today’s rapid technological advances have raised fears that humanity will lose relevance, that automated machines will eventually outpace human physical capabilities, human thought, and even human creativity, that humanity will be on the losing side of the emerging technology gap. This project will provide new methods for human-robot collaboration in artistic expression, as a step toward bridging this technology gap in domains where creativity is paramount. For instance, consider a master painter with decades of practice on canvas; preserving their mastery of skills and creativity is extremely challenging, let alone replicating their masterpieces. The team will address a myriad of technical challenges associated with capturing artist skills, recognizing their creative intents, and robotically generating art in the artist’s own styles and techniques. Calligraphy presents a complex set of technical challenges, as it involves contact of a soft, deformable brush with a paper surface to apply liquid ink. The team will investigate algorithmic solutions for skillful use of such complex artist mediums by robots. The developed robotic systems will be publicly demonstrated in collaborative performances with professional artists; these dissemination activities will demonstrate how a proficient technology could preserve and amplify the creative status of the human artist.The long-term goal of the project is to achieve genuine human-robot collaboration during the creative, artistic process. Progress toward this goal will proceed along four separate thrusts, each focused on pushing the state of the art in a particular direction. The first step is to build systems that, in some sense, understand the artist’s creative process. For the purposes of this project, understanding can be demonstrated by capturing, perceiving, and rendering the dexterous skills of human artists. The artist’s motions will be recorded using motion capture technology (similar to methods used in modern cinema), and the captured data will then be used to design mathematical models that will form the basis for reasoning about the artist’s actions, and for building perception algorithms that can understand the artist’s motion in real time. Hence, the first research thrust, capture, will develop representations of artistic skills, and algorithms to enable the recording of the creation of an artifact via skillful manipulation of a tool. The second thrust, perception, will develop algorithms that can, in real-time, sense the artistic intent of an artist, using learned representations, requiring much less comprehensive sensing modalities. The third thrust, rendering, will develop identification and control algorithms that allow a robot to render artistic actions in different contexts, e.g., after creation of an artwork is captured or in the context of an interactive system. The final step is designing robotic systems that are capable of rendering these skills, with the goal of enabling genuine human-robot collaboration during the creative process. This symbiotic relationship preserves the creative status of the human artist, while providing augmented capabilities to enhance the creation process. Finally, the fourth thrust is to integrate all of these results into a truly interactive system, where both perception and control are leveraged to enable human-robot collaboration in creating novel works of art.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

今天的快速技术进步引起了人们的担忧：人类将失去相关性，自动化机器最终将超过人类的身体能力、人类的思想，甚至人类的创造力，人类将在新兴的技术差距中处于劣势。这个项目将为人类和机器人在艺术表现方面的合作提供新的方法，作为在创造力至关重要的领域弥合这一技术差距的一步。例如，考虑一位在画布上练习了几十年的大师画家；保留他们掌握的技能和创造力是极具挑战性的，更不用说复制他们的杰作了。该团队将解决与捕捉艺术家技能相关的无数技术挑战，识别他们的创作意图，并以艺术家自己的风格和技术自动生成艺术。书法呈现出一套复杂的技术挑战，因为它涉及到软的、可变形的毛笔与纸表面的接触，以应用液体墨水。该团队将研究算法解决方案，让机器人熟练地使用这种复杂的艺术媒介。开发的机器人系统将在与专业艺术家的合作表演中公开展示；这些传播活动将展示一种熟练的技术如何能够保持和扩大人类艺术家的创造性地位。该项目的长期目标是在创作和艺术过程中实现真正的人机协作。实现这一目标的进程将沿着四个不同的方向进行，每个方向都专注于推动技术的发展。第一步是建立系统，在某种意义上，理解艺术家的创作过程。为了这个项目的目的，理解可以通过捕捉、感知和渲染人类艺术家的灵巧技能来展示。艺术家的动作将使用动作捕捉技术（类似于现代电影中使用的方法）进行记录，然后捕获的数据将用于设计数学模型，这些模型将形成关于艺术家动作的推理基础，并用于构建可以实时理解艺术家动作的感知算法。因此，第一个研究重点，捕捉，将发展艺术技能的表现，以及通过熟练操作工具来记录人工制品创造的算法。第二个重点，感知，将开发算法，可以实时感知艺术家的艺术意图，使用学习表征，需要更少的综合传感模式。第三个重点，渲染，将开发识别和控制算法，使机器人能够在不同的环境中呈现艺术动作，例如，在艺术品创作被捕获后或在交互式系统的环境中。最后一步是设计能够呈现这些技能的机器人系统，目标是在创作过程中实现真正的人机协作。这种共生关系保留了人类艺术家的创作地位，同时提供了增强创作过程的能力。最后，第四个重点是将所有这些结果整合到一个真正的交互式系统中，在这个系统中，感知和控制都被利用来实现人机协作，以创造新颖的艺术作品。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

A Hybrid Cable-Driven Robot for Non-Destructive Leafy Plant Monitoring and Mass Estimation using Structure from Motion

• DOI: 10.1109/icra48891.2023.10161045
• 发表时间: 2022-09
• 期刊: 2023 IEEE International Conference on Robotics and Automation (ICRA)
• 作者: Gerry Chen;Venkata Harsh Suhith Muriki;Cédric Pradalier;Yongsheng Chen;F. Dellaert

Locally Optimal Estimation and Control of Cable Driven Parallel Robots using Time Varying Linear Quadratic Gaussian Control

• DOI: 10.1109/iros47612.2022.9981144
• 发表时间: 2022-08
• 期刊: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
• 作者: Gerry Chen;S. Hutchinson;F. Dellaert

GTGraffiti: Spray Painting Graffiti Art from Human Painting Motions with a Cable Driven Parallel Robot

• DOI: 10.1109/icra46639.2022.9812008
• 发表时间: 2021-09
• 期刊: 2022 International Conference on Robotics and Automation (ICRA)
• 作者: Gerry Chen;Sereym Baek;J. Flórez;Wanli Qian;Sang-won Leigh;S. Hutchinson;F. Dellaert

Constraint Manifolds for Robotic Inference and Planning

用于机器人推理和规划的约束流形

• DOI: 10.1109/icra48891.2023.10161024
• 发表时间: 2023
• 期刊: IEEE
• 作者: Zhang, Yetong;Jiang, Fan;Chen, Gerry;Agrawal, Varun;Rutkowski, Adam;Dellaert, Frank
• 通讯作者: Dellaert, Frank

Equality Constrained Linear Optimal Control With Factor Graphs

带因子图的等式约束线性最优控制

• DOI: 10.1109/icra48506.2021.9562000
• 发表时间: 2021
• 期刊: 2021 International Conference on Robotics and Automation (ICRA
• 作者: Yang, Shuo;Chen, Gerry;Zhang, Yetong;Choset, Howie;Dellaert, Frank
• 通讯作者: Dellaert, Frank