CAREER: Human-Computer Collaborative Music Making

职业：人机协作音乐制作

• 批准号: 1846184
• 负责人: Zhiyao Duan
• 金额: $ 49.92万
• 依托单位: University of Rochester
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-06-01 至 2024-05-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1846184&HistoricalAwards=false
• 关键词: CAREER; Human; Computer; Collaborative; Music

Music is part of every culture on earth, and the enjoyment of music is nearly universal. Music performance is often highly collaborative; musicians harmonize their pitch, coordinate their timing, and reinforce their expressiveness to make music that strikes the hearts of the audience. This research envisions a human-computer collaborative music making system that allows people to collaborate with machines in a manner similar to that in which we collaborate with each other. This is of great significance, as we live in a world where the interaction between humans and machines is becoming deeper and broader, so developing systems that allow us to collaborate with machines is a primary goal of research into cyber-human systems, robotics, and artificial intelligence. Project outcomes will advance the state of the art in automated accompaniment systems by empowering machines with much stronger music perception skills (audio-visual attending to individual parts in ensemble performances vs. monophonic listening), much more expressive music performance skills (expressive audio-visual rendering vs. timing adaptation of audio only), and much deeper understanding of music theory and composition rules (composition and improvisation skills vs. music theory novice). This project will showcase the powerful connection between music and technology, which has inspired generations of great multidisciplinary thinkers such as Pythagoras, Galilei, Da Vinci, and Franklin. The techniques developed in this project will be applied to augmented concert experiences through collaborations with the Eastman School of Music and the Chinese Choral Society of Rochester. Outreach to pre-college and college students will be accomplished through a variety of activities, including lab visits, a summer mini-course on "music and math" and teaching and advising in the unique and interdisciplinary Audio and Music Engineering program at the University of Rochester. The project has four research thrusts with the following expected outcomes: 1) Attending to Human Performances: algorithms for machine listening and visual analysis of multi-instrument polyphonic music performances; 2) Rendering Expressive Machine Performances: computational models for expressiveness and audio-visual rendering techniques for expressive performances; 3) Modeling Music Language for Improvisation: computational models for compositional rules, and algorithms for music generation, harmonization, and improvisation; 4) System Integration: a human-computer collaborative music making system, and a set of design principles backed by subjective evaluations. The research will advance existing interaction mechanisms toward human-computer collaboration. It will also advance the current static-object-displaying type of augmented reality to more intelligent, dynamic and collaborative augmented reality in music performances. The research on audio-visual analysis will advanes both machine listening and visual understanding of audio-visual scenes in the music context. The research on visual rendering of expressive performances will open a new field of computational modeling of visual expressiveness in musical performances. And the research on computational music language models is fundamental for many tasks in music informatics, including transcription, composition, and retrieval. The integration of analysis, performance and music language modeling towards a real-time collaborative system represents a new level of intelligent real-time computing.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

音乐是地球上每一种文化的一部分，音乐的享受几乎是普遍的。音乐表演通常是高度合作的;音乐家协调他们的音高，协调他们的时间，并加强他们的表现力，使音乐打动观众的心。这项研究设想了一个人机协作音乐制作系统，允许人们以类似于我们彼此协作的方式与机器协作。这具有重要意义，因为我们生活在一个人类与机器之间的互动越来越深入和广泛的世界中，因此开发允许我们与机器协作的系统是网络人类系统，机器人技术和人工智能研究的主要目标。项目成果将通过赋予机器更强的音乐感知技能来推动自动伴奏系统的发展（在合奏表演中视听注意单个部分与单声道聆听），更有表现力的音乐表演技巧（表达性视听渲染与仅音频的定时适配），以及对音乐理论和作曲规则更深入的理解（作曲和即兴创作技巧vs.音乐理论新手）。这个项目将展示音乐和技术之间的强大联系，这激发了几代伟大的多学科思想家，如毕达哥拉斯，伽利略，达芬奇和富兰克林。通过与伊士曼音乐学院和罗切斯特中国合唱协会的合作，本项目中开发的技术将应用于增强音乐会体验。对大学预科和大学生的外联将通过各种活动来完成，包括实验室参观，夏季迷你课程“音乐和数学”以及罗切斯特大学独特的跨学科音频和音乐工程课程的教学和咨询。该项目有四个研究方向，预期成果如下：1）关注人类表演：多乐器复调音乐表演的机器聆听和视觉分析算法; 2）渲染表现力机器表演：表现力的计算模型和表现力表演的视听渲染技术; 3）即兴音乐语言建模：作曲规则的计算模型，以及音乐生成，协调和即兴创作的算法; 4）系统集成：人机协作音乐制作系统，以及一套由主观评估支持的设计原则。这项研究将推动现有的人机协作交互机制。它还将推动当前静态对象显示类型的增强现实在音乐表演中更加智能，动态和协作的增强现实。视听分析的研究将同时兼顾音乐语境中视听场景的机器聆听和视觉理解。表现性表演的视觉渲染研究将为音乐表演视觉表现力的计算建模开辟一个新的领域。音乐语言模型的研究是音乐信息学中许多任务的基础，包括转录、作曲和检索。将分析、表演和音乐语言建模集成到一个实时协作系统中，代表了智能实时计算的新水平。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

BeatNet: A real-time music integrated beat and downbeat tracker

BeatNet：实时音乐集成节拍和强拍跟踪器

• 发表时间: 2021
• 期刊: International Society for Music Information Retrieval
• 作者: Heydari, Mojtaba;Cwitkowitz, Frank;Duan, Zhiyao
• 通讯作者: Duan, Zhiyao

SingNet: a real-time Singing Voice beat and Downbeat Tracking System

SingNet：实时歌声节拍和强拍跟踪系统

• DOI: 10.1109/icassp49357.2023.10096580
• 发表时间: 2023
• 期刊: Speech and Signal Processing (ICASSP
• 作者: Heydari, Mojtaba;Wang, Ju-Chiang;Duan, Zhiyao
• 通讯作者: Duan, Zhiyao

BachDuet: A deep learning system for human-machine counterpoint improvisation

BachDuet：人机对位即兴创作的深度学习系统

• 发表时间: 2020
• 期刊: Proceedings of the International Conference on New Interfaces for Musical Expression
• 作者: Benetatos, Christodoulos;VanderStel, Joseph;Duan, Zhiyao
• 通讯作者: Duan, Zhiyao

Skipping the Frame-Level: Event-Based Piano Transcription With Neural Semi-CRFs

• 发表时间: 2021
• 作者: Yujia Yan;Frank Cwitkowitz;Z. Duan

Draw and Listen! A Sketch-Based System for Music Inpainting

画和听！

• DOI: 10.5334/tismir.128
• 发表时间: 2022
• 期刊: Transactions of the International Society for Music Information Retrieval
• 作者: Benetatos, Christodoulos;Duan, Zhiyao
• 通讯作者: Duan, Zhiyao