CAREER: Human-Computer Collaborative Music Making

音乐是地球上每一种文化的一部分，音乐的享受几乎是普遍的。音乐表演通常是高度协作的；音乐家们协调他们的音高，协调他们的时间，加强他们的表现力，使音乐击中观众的心。这项研究设想了一个人机协作音乐制作系统，它允许人们以类似于我们彼此合作的方式与机器合作。这是非常重要的，因为我们生活在一个人与机器之间的互动变得越来越深入和广泛的世界，所以开发允许我们与机器合作的系统是研究网络人类系统、机器人和人工智能的主要目标。项目成果将通过赋予机器更强的音乐感知能力（在合奏表演中关注单个部分的视听与单音聆听），更具表现力的音乐表演技能（富有表现力的视听呈现与音频的定时适应），以及对音乐理论和作曲规则的更深入理解（作曲和即兴技巧与乐理新手），推动自动化伴奏系统的最新发展。这个项目将展示音乐和技术之间的强大联系，这种联系激发了一代又一代伟大的多学科思想家，如毕达哥拉斯、伽利略、达·芬奇和富兰克林。该项目开发的技术将通过与伊士曼音乐学院和罗切斯特中国合唱协会的合作，应用于增强音乐会体验。对大学预科生和大学生的拓展将通过各种活动来完成，包括实验室参观，“音乐和数学”的夏季迷你课程，以及罗切斯特大学独特的跨学科音频和音乐工程项目的教学和建议。该项目有四个研究重点，预期成果如下：1)关注人类表演：多乐器复调音乐表演的机器听音和视觉分析算法；2)表现力机器表演渲染：表现力计算模型和表现力表演的视听渲染技术；3)为即兴创作建模音乐语言：作曲规则的计算模型，以及音乐生成、和声和即兴创作的算法；4)系统集成：一个人机协同的音乐制作系统，一套以主观评价为支撑的设计原则。该研究将推动现有交互机制向人机协作方向发展。它还将把当前静态对象显示类型的增强现实推进到更加智能、动态和协作的音乐表演增强现实。对视听分析的研究将促进机器听力和对音乐语境中视听场景的视觉理解。表现性表演的视觉渲染研究将为音乐表演视觉表现力的计算建模开辟新的领域。计算音乐语言模型的研究是音乐信息学中许多任务的基础，包括转录、作曲和检索。将分析、表演和音乐语言建模集成到一个实时协作系统中，代表了智能实时计算的一个新水平。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Music is part of every culture on earth, and the enjoyment of music is nearly universal. Music performance is often highly collaborative; musicians harmonize their pitch, coordinate their timing, and reinforce their expressiveness to make music that strikes the hearts of the audience. This research envisions a human-computer collaborative music making system that allows people to collaborate with machines in a manner similar to that in which we collaborate with each other. This is of great significance, as we live in a world where the interaction between humans and machines is becoming deeper and broader, so developing systems that allow us to collaborate with machines is a primary goal of research into cyber-human systems, robotics, and artificial intelligence. Project outcomes will advance the state of the art in automated accompaniment systems by empowering machines with much stronger music perception skills (audio-visual attending to individual parts in ensemble performances vs. monophonic listening), much more expressive music performance skills (expressive audio-visual rendering vs. timing adaptation of audio only), and much deeper understanding of music theory and composition rules (composition and improvisation skills vs. music theory novice). This project will showcase the powerful connection between music and technology, which has inspired generations of great multidisciplinary thinkers such as Pythagoras, Galilei, Da Vinci, and Franklin. The techniques developed in this project will be applied to augmented concert experiences through collaborations with the Eastman School of Music and the Chinese Choral Society of Rochester. Outreach to pre-college and college students will be accomplished through a variety of activities, including lab visits, a summer mini-course on "music and math" and teaching and advising in the unique and interdisciplinary Audio and Music Engineering program at the University of Rochester. The project has four research thrusts with the following expected outcomes: 1) Attending to Human Performances: algorithms for machine listening and visual analysis of multi-instrument polyphonic music performances; 2) Rendering Expressive Machine Performances: computational models for expressiveness and audio-visual rendering techniques for expressive performances; 3) Modeling Music Language for Improvisation: computational models for compositional rules, and algorithms for music generation, harmonization, and improvisation; 4) System Integration: a human-computer collaborative music making system, and a set of design principles backed by subjective evaluations. The research will advance existing interaction mechanisms toward human-computer collaboration. It will also advance the current static-object-displaying type of augmented reality to more intelligent, dynamic and collaborative augmented reality in music performances. The research on audio-visual analysis will advanes both machine listening and visual understanding of audio-visual scenes in the music context. The research on visual rendering of expressive performances will open a new field of computational modeling of visual expressiveness in musical performances. And the research on computational music language models is fundamental for many tasks in music informatics, including transcription, composition, and retrieval. The integration of analysis, performance and music language modeling towards a real-time collaborative system represents a new level of intelligent real-time computing.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.