CAREER: Human-Computer Collaborative Music Making

音乐是地球上每一种文化的一部分，对音乐的享受几乎是普遍的。音乐表演通常是高度合作的；音乐家们协调他们的音高，协调他们的时机，并加强他们的表现力，以创造出打动观众心灵的音乐。这项研究设想了一个人机协作音乐制作系统，允许人们以类似于我们彼此协作的方式与机器协作。这一点意义重大，因为我们生活在一个人与机器之间的交互变得越来越深入和广泛的世界，因此开发允许我们与机器协作的系统是网络人类系统、机器人和人工智能研究的主要目标。项目成果将通过赋予机器更强的音乐感知技能(在合奏表演中对单个部分的视听关注与单声道聆听)，更具表现力的音乐表演技能(富有表现力的视听呈现与仅对音频的计时改编)，以及对音乐理论和作曲规则的更深层次的理解(作曲和即兴技巧与音乐理论新手)来推动自动伴奏系统的发展。这个项目将展示音乐和技术之间的强大联系，这激励了毕达哥拉斯、伽利莱、达芬奇和富兰克林等一代又一代伟大的多学科思想家。这个项目中开发的技术将通过与伊士曼音乐学院和罗切斯特中国合唱学会的合作，应用于增强音乐会体验。面向大学预科和大学生的外展活动将通过各种活动实现，包括参观实验室，举办夏季“音乐和数学”迷你课程，以及在罗切斯特大学独特的跨学科音频和音乐工程项目中提供教学和咨询服务。该项目有四个研究方向，预期成果如下：1)关注人类演奏：多乐器复调音乐表演的机器聆听和视觉分析算法；2)渲染富有表现力的机器表演：表现力的计算模型和表现力表演的视听渲染技术；3)即兴演奏的音乐语言建模：作曲规则的计算模型，以及音乐生成、和声和即兴演奏的算法；4)系统集成：人机协同音乐制作系统，以及一套以主观评价为支持的设计原则。这项研究将推动现有的交互机制朝着人机协作的方向发展。它还将推动目前静态对象显示类型的增强现实在音乐表演中向更智能、更动态和更协作的增强现实发展。对视听分析的研究将有助于机器听懂和视觉理解音乐语境中的视听场景。表现性表演的视觉渲染研究将开辟音乐表演视觉表现力计算建模的新领域。而计算音乐语言模型的研究是音乐信息学中的许多任务的基础，包括转录、作曲和检索。将分析、表演和音乐语言建模集成到一个实时协作系统中，代表了智能实时计算的一个新水平。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Music is part of every culture on earth, and the enjoyment of music is nearly universal. Music performance is often highly collaborative; musicians harmonize their pitch, coordinate their timing, and reinforce their expressiveness to make music that strikes the hearts of the audience. This research envisions a human-computer collaborative music making system that allows people to collaborate with machines in a manner similar to that in which we collaborate with each other. This is of great significance, as we live in a world where the interaction between humans and machines is becoming deeper and broader, so developing systems that allow us to collaborate with machines is a primary goal of research into cyber-human systems, robotics, and artificial intelligence. Project outcomes will advance the state of the art in automated accompaniment systems by empowering machines with much stronger music perception skills (audio-visual attending to individual parts in ensemble performances vs. monophonic listening), much more expressive music performance skills (expressive audio-visual rendering vs. timing adaptation of audio only), and much deeper understanding of music theory and composition rules (composition and improvisation skills vs. music theory novice). This project will showcase the powerful connection between music and technology, which has inspired generations of great multidisciplinary thinkers such as Pythagoras, Galilei, Da Vinci, and Franklin. The techniques developed in this project will be applied to augmented concert experiences through collaborations with the Eastman School of Music and the Chinese Choral Society of Rochester. Outreach to pre-college and college students will be accomplished through a variety of activities, including lab visits, a summer mini-course on "music and math" and teaching and advising in the unique and interdisciplinary Audio and Music Engineering program at the University of Rochester. The project has four research thrusts with the following expected outcomes: 1) Attending to Human Performances: algorithms for machine listening and visual analysis of multi-instrument polyphonic music performances; 2) Rendering Expressive Machine Performances: computational models for expressiveness and audio-visual rendering techniques for expressive performances; 3) Modeling Music Language for Improvisation: computational models for compositional rules, and algorithms for music generation, harmonization, and improvisation; 4) System Integration: a human-computer collaborative music making system, and a set of design principles backed by subjective evaluations. The research will advance existing interaction mechanisms toward human-computer collaboration. It will also advance the current static-object-displaying type of augmented reality to more intelligent, dynamic and collaborative augmented reality in music performances. The research on audio-visual analysis will advanes both machine listening and visual understanding of audio-visual scenes in the music context. The research on visual rendering of expressive performances will open a new field of computational modeling of visual expressiveness in musical performances. And the research on computational music language models is fundamental for many tasks in music informatics, including transcription, composition, and retrieval. The integration of analysis, performance and music language modeling towards a real-time collaborative system represents a new level of intelligent real-time computing.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.