Collaborative Research: Participatory Sensemaking with Embodied Co-Creative Agents

协作研究：通过具体的共同创意代理进行参与式意义建构

• 批准号: 2123597
• 负责人: Brian Magerko
• 金额: $ 57.36万
• 依托单位: Georgia Tech Research Corporation
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2123597&HistoricalAwards=false
• 关键词: Collaborative; Research; Participatory; Sensemaking; Embodied

This award supports research to develop a computational architecture (called PACE) to model embodied and co-creative behavior between humans and embodied intelligent machines. Humans collaborate creatively (co-create) in many settings—for example, dancing with a partner, playing pretend with a child, or brainstorming in an engineering design meeting. As common as this experience is, it rarely defines our interactions with intelligent machines. This project explores improvisational, collaborative, and co-creative dance as a form of non-contact physical interaction between co-creative partners. The project is based upon creative sensemaking theory, which casts the creative practice as a dynamic social process in which individuals alternate between different cognitive states as they make sense of and respond to the actions of others within new environments. This project will promote the progress of science by studying human co-creative practice between advanced dancers to better understand human co-creativity in action and to inform the design and development of co-creative AI technology. The project team will use that understanding to develop AI agents that can co-create with humans. Project outcomes will include novel machine algorithms capable of replicating aspects of human-human co-creativity, as well as public performances of human dancers engaging in co-creative dance with the embodied machine intelligence. This work will inform the long-term development of more effective interfaces in a wide range of applications that involve integrated mind, machine, and motor function—such as physical therapy, design brainstorming, or future experiences with robots in the home. This project also supports K-12, undergraduate, graduate, and public education through outreach and mentorship, and via public human-AI performances.The goal of this project is to develop a modular, reusable system for building embodied co-creative AI. The project team will use contemporary dance as an application domain, as its practitioners are formally trained in exploring, expressing, and collaborating through embodied non-contact physical interactions. The project team will first conduct a qualitative analysis of video and gesture data from human dancer dyads in improvisation sessions. This analysis will be focused on understanding how dancers build understanding through interaction with each other and the environment (i.e. through a process called participatory sensemaking). The team will then formalize ther findings within an architecture for creating virtual, embodied agents that can sense, learn, and generate movement during improvisation. To demonstrate and refine this architecture, the team will develop a co-creative AI that approaches expert-level participatory sensemaking in contemporary dance, and train this agent to create a curated improvisational partner. The agent will be evaluated in rehearsal and in performance. The main contributions of this project will be a) the first open-access annotated dyadic movement dataset; b) a better understanding of how human dancers co-create; c) a interface for dancers to train and improvise with AI; and d) an architecture for making co-creative AI in motor-related domains based on empirical studies of co-creativity.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该奖项支持开发一种计算体系结构(称为PACE)的研究，以模拟人类和具身智能机器之间的具身和共同创造行为。人类在许多环境中进行创造性的合作(共同创造)--例如，与伙伴跳舞，与孩子玩假装游戏，或在工程设计会议上进行头脑风暴。尽管这种体验很常见，但它很少定义我们与智能机器的互动。这个项目探索即兴、协作和共同创作的舞蹈，作为共同创作伙伴之间非接触式身体互动的一种形式。该项目基于创造性感官创造理论，该理论将创造性实践塑造成一个动态的社会过程，在这个过程中，个人在不同的认知状态之间交替，因为他们在新的环境中理解并回应他人的行动。该项目将通过研究高级舞者之间的人类共同创造实践来推动科学进步，以更好地了解人类共同创造的行动，并为共同创造AI技术的设计和开发提供信息。该项目团队将利用这种理解来开发可以与人类共同创造的人工智能代理。项目成果将包括能够复制人与人共同创造的各个方面的新型机器算法，以及公开表演人类舞者与体现的机器智能进行共同创造的舞蹈。这项工作将为更有效的接口在涉及集成大脑、机器和运动功能的广泛应用中的长期发展提供信息-例如物理治疗、设计头脑风暴或未来在家中使用机器人的体验。该项目还通过外展和指导，以及通过公共的人类-人工智能表演，支持K-12、本科生、研究生和公共教育。该项目的目标是开发一个模块化的、可重用的系统，用于构建具体化的协同创造人工智能。项目团队将把现代舞作为一个应用领域，因为它的实践者经过正式培训，通过具体化的非接触性物理交互来探索、表达和合作。该项目团队将首先对即兴表演环节中的人类舞者二人组的视频和手势数据进行定性分析。这一分析将侧重于了解舞者如何通过彼此和环境的互动(即通过一种称为参与式感觉制造的过程)建立理解。然后，该团队将在一个体系结构中正式确定其他发现，以创建虚拟的、具体化的代理，这些代理可以在即兴发挥期间感知、学习和生成运动。为了展示和完善这一架构，该团队将开发一种共同创作的人工智能，这种人工智能接近现代舞中的专家级参与式感觉制作，并培训这名代理来创建一个精心策划的即兴合作伙伴。将在排练和表演中对代理进行评估。该项目的主要贡献将是a)第一个开放获取的注释二元运动数据集；b)更好地了解人类舞者如何共同创造；c)舞者训练和即兴使用人工智能的界面；以及d)基于共同创造的经验研究，在运动相关领域进行联合创造人工智能的架构。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

AI Meets Holographic Pepper’s Ghost: A Co-Creative Public Dance Experience

AI 遇见全息 Pepper 的幽灵：共同创意的公共舞蹈体验

• DOI: 10.1145/3563703.3596658
• 发表时间: 2023
• 期刊: DIS '23 Companion: Companion Publication of the 2023 ACM Designing Interactive Systems Conference
• 作者: Trajkova, Milka;Deshpande, Manoj;Knowlton, Andrea;Monden, Cassandra;Long, Duri;Magerko, Brian
• 通讯作者: Magerko, Brian

Observable Creative Sense-Making (OCSM): A Method For Quantifying Improvisational Co-Creative Interaction

可观察的创意意义构建（OCSM）：一种量化即兴共同创意互动的方法

• DOI: 10.1145/3591196.3593514
• 发表时间: 2023
• 期刊: Observable Creative Sense-Making (OCSM
• 作者: Deshpande, Manoj;Trajkova, Milka;Knowlton, Andrea;Magerko, Brian
• 通讯作者: Magerko, Brian