RI:Small: Capturing, Perceiving, and Rendering of Artistic Skills for Real-time Interactive Creation of Art

今天的快速技术进步让人们担心人类将失去相关性，自动化机器最终将超过人类的身体能力，人类的思想，甚至人类的创造力，人类将在新兴技术差距中处于失败的一方。该项目将为人类与机器人在艺术表达方面的合作提供新的方法，作为弥合创造力至关重要的领域中这一技术差距的一步。例如，考虑一位在画布上有数十年实践经验的大师画家;保持他们对技能和创造力的掌握是极具挑战性的，更不用说复制他们的杰作了。该团队将解决与捕捉艺术家技能相关的无数技术挑战，识别他们的创作意图，并以艺术家自己的风格和技术自动生成艺术。书法提出了一系列复杂的技术挑战，因为它涉及柔软、可变形的画笔与纸张表面接触以涂上液体墨水。该团队将研究算法解决方案，以便机器人巧妙地使用这种复杂的艺术家媒介。开发的机器人系统将在与专业艺术家的合作表演中公开展示;这些传播活动将展示熟练的技术如何保护和放大人类艺术家的创造性地位。该项目的长期目标是在创作和艺术过程中实现真正的人机合作。 朝着这一目标的进展将沿着沿着四个独立的方向进行，每个方向都集中在将技术推向一个特定的方向。 第一步是建立系统，在某种意义上，理解艺术家的创作过程。在这个项目中，理解可以通过捕捉、感知和渲染人类艺术家的灵巧技能来展示。艺术家的动作将使用动作捕捉技术（类似于现代电影中使用的方法）记录下来，然后捕捉到的数据将用于设计数学模型，这些模型将构成对艺术家动作进行推理的基础，并用于构建能够在真实的时间内理解艺术家动作的感知算法。因此，第一个研究重点，捕捉，将开发艺术技能的表示，和算法，使记录的创造一个工件通过熟练的操作工具。 第二个重点是感知，它将开发出一种算法，可以实时感知艺术家的艺术意图，使用学习到的表征，不需要太全面的感知方式。第三个重点是渲染，将开发识别和控制算法，使机器人能够在不同的环境中渲染艺术动作，例如，在捕获艺术品的创建之后或者在交互式系统的上下文中。 最后一步是设计能够呈现这些技能的机器人系统，目标是在创作过程中实现真正的人机协作。 这种共生关系保持了人类艺术家的创造性地位，同时提供了增强的能力来增强创作过程。最后，第四个重点是将所有这些成果整合到一个真正的交互式系统中，在这个系统中，感知和控制都被利用来实现人类与机器人的合作，以创造新颖的艺术作品。这个奖项反映了NSF的法定使命，并被认为是值得支持的，通过使用基金会的智力价值和更广泛的影响审查标准进行评估。

Today’s rapid technological advances have raised fears that humanity will lose relevance, that automated machines will eventually outpace human physical capabilities, human thought, and even human creativity, that humanity will be on the losing side of the emerging technology gap. This project will provide new methods for human-robot collaboration in artistic expression, as a step toward bridging this technology gap in domains where creativity is paramount. For instance, consider a master painter with decades of practice on canvas; preserving their mastery of skills and creativity is extremely challenging, let alone replicating their masterpieces. The team will address a myriad of technical challenges associated with capturing artist skills, recognizing their creative intents, and robotically generating art in the artist’s own styles and techniques. Calligraphy presents a complex set of technical challenges, as it involves contact of a soft, deformable brush with a paper surface to apply liquid ink. The team will investigate algorithmic solutions for skillful use of such complex artist mediums by robots. The developed robotic systems will be publicly demonstrated in collaborative performances with professional artists; these dissemination activities will demonstrate how a proficient technology could preserve and amplify the creative status of the human artist.The long-term goal of the project is to achieve genuine human-robot collaboration during the creative, artistic process. Progress toward this goal will proceed along four separate thrusts, each focused on pushing the state of the art in a particular direction. The first step is to build systems that, in some sense, understand the artist’s creative process. For the purposes of this project, understanding can be demonstrated by capturing, perceiving, and rendering the dexterous skills of human artists. The artist’s motions will be recorded using motion capture technology (similar to methods used in modern cinema), and the captured data will then be used to design mathematical models that will form the basis for reasoning about the artist’s actions, and for building perception algorithms that can understand the artist’s motion in real time. Hence, the first research thrust, capture, will develop representations of artistic skills, and algorithms to enable the recording of the creation of an artifact via skillful manipulation of a tool. The second thrust, perception, will develop algorithms that can, in real-time, sense the artistic intent of an artist, using learned representations, requiring much less comprehensive sensing modalities. The third thrust, rendering, will develop identification and control algorithms that allow a robot to render artistic actions in different contexts, e.g., after creation of an artwork is captured or in the context of an interactive system. The final step is designing robotic systems that are capable of rendering these skills, with the goal of enabling genuine human-robot collaboration during the creative process. This symbiotic relationship preserves the creative status of the human artist, while providing augmented capabilities to enhance the creation process. Finally, the fourth thrust is to integrate all of these results into a truly interactive system, where both perception and control are leveraged to enable human-robot collaboration in creating novel works of art.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.