HCC: Small: Multi Modal Music Intelligence for Robotic Musicianship

PI 在该项目中的目标是通过实现人类和机器人音乐家之间的多模式通信以及在实时音乐交互中嵌入基于知识的音乐智能来推进机器人音乐领域的发展。 为此，PI 将寻求了解人类如何表示和处理音乐中的时间结构，他将开发一种视觉系统，使机器人音乐家能够预测人类音乐家的手势。 该人工视觉系统以人与人的音乐互动为模型，将补充PI之前开发的音乐聆听系统，从而使机器人能够更好地与人类即兴演奏者同步演奏。 此外，PI还将转录和分析（使用隐马尔可夫模型等统计工具）伟大的古典作曲家以及爵士乐大师的大量作品；离线分析，加上访谈、调查和焦点小组，将增进我们对团体游戏的认知和机械方面以及现场表演中视觉和身体线索的作用的了解，并使PI能够构建一个长期的文化知识库，该知识库将添加到之前已经开发的短期实时即兴创作技术中。 总的来说，项目成果将形成人类和人工音乐才能的综合模型，PI 将使用行为测量（例如同步的时间差）、主题问卷和类图灵测试来评估该模型，以评估音乐交互的质量。 更广泛的影响：该项目将为我们在音乐才能、人机交互、计算机辅助协作和即兴创作等领域的知识做出根本性贡献。 它将通过高知名度的音乐会，帮助将人机互动带给公众，通过在人与机器之间创造新颖的音乐合作，吸引那些不经常被数学、科学或工程学吸引的学生的兴趣和想象力。 该项目将作为未来音乐互动形式的测试平台，将计算机音乐的感知和算法方面在视觉和听觉上带入物理世界，这可能会激发人们以新的方式演奏和思考音乐。 最终，这项研究预计将揭示更广泛的概念，例如人类和人工的创造力和表达，以及机器创造或协助创造有价值的美学和艺术产品的可行性。

The PI's objective in this project is to advance the field of robotic musicianship by enabling multi-modal communication between human and robotic musicians, and by embedding knowledge-based musical intelligence in real-time musical interactions. To these ends the PI will seek to understand how temporal structures in music are represented and processed by humans, and he will develop a vision system that enables a robotic musician to anticipate a human musician's gestures. Modeled after human-human musical interaction, the artificial vision system will complement the musical listening system developed in the PI's prior work, thereby allowing a robot to better synchronize its playing with human improvisers. In addition, the PI will transcribe and analyze (using statistical tools such as Hidden Markov Models) a large corpus of works by the great classical composers as well as by masters of jazz; the off-line analysis, coupled with interviews, surveys, and focus groups, will advance our knowledge regarding cognitive and mechanical aspects of group play and the role of visual and physical cues in live performance, and will enable the PI to construct a long-term cultural knowledge base that will be added to the short-term real-time improvisation techniques already developed previously. Collectively, project outcomes will lead to a comprehensive model of human and artificial musicianship, which the PI will evaluate using behavioral measures (such as time differences in synchronization), subject questionnaires, and a Turing-like test to evaluate the quality of the musical interaction.Broader Impacts: This project will make fundamental contribution to our knowledge in areas such as musicianship, human-robotic interaction, computer assisted collaboration, and improvisation. It will help bring human-robot interaction to the general public through high visibility concerts that capture the interest and imagination of students who are not regularly drawn to math, the sciences, or engineering by creating novel musical collaborations between humans and machines. The project will serve as a testbed for future forms of musical interactions, bringing perceptual and algorithmic aspects of computer music into the physical world both visually and acoustically, which may inspire people to play and think about music in new ways. Ultimately, the research is expected to shed light on broader concepts such as human and artificial creativity and expression, and the feasibility for machines to create, or assist in creating, meritorious aesthetic and artistic products.