Collaborative Research: FW-HTF-RM: Artificial Intelligence Technology for Future Music Performers

美国有近30万人在音乐领域工作。人工智能(AI)的最新进展对音乐创作者产生了深远的影响，但尚未对专业音乐表演者的实践产生太大影响。这个项目将调查人工智能技术如何改变未来音乐表演者的工作，无论是个人练习(通过开发分析音乐表演的指导工具)还是协作练习(通过开发可以取代团队中缺失的表演者的系统)。该团队将从两个主要问题来评估这些工具：(1)AI技术何时能为专业音乐人的练习和表演提供可衡量的好处？(2)哪些因素会影响未来音乐人在工作中对AI技术的接受？最初的工具将专注于弦乐乐器，但基础技术可能会适应艺术和娱乐业的各种表演背景。该项目团队还将探索如何使用这些工具来造福弱势群体的学生；这些工具可能会改善获得音乐教学机会有限的学生的学习机会。该团队将从STEM中代表性不足的群体中招聘学生研究人员。该项目将开发和整合计算机视觉、自然语言处理和音频分析的技术，以创建两个支持人工智能的工具来支持弦乐表演者。第一个工具是评估者，目的是改善个人实践和业绩。它分析音乐家的声音，并将其与数字化的音乐乐谱进行比较，以检测语调、节奏和动态方面的偏差。评估者还会分析捕获的视频，并将其与记录了正确姿势的表演者样本数据库进行比较，从而推荐更好的姿势，这既可以提高音乐表现，又可以降低受伤风险。第二个工具是Companion，旨在支持在集体表演排练中缺少一个或多个音乐家时的常见用例。伴奏可以扮演一种或几种乐器的角色，通过对音乐家表演的音频分析来取代缺席的音乐家，匹配他们的节奏和风格，同时还可以实时响应口头指令。这些工具将通过一系列用户研究、调查、焦点小组和纵向部署来开发和评估。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Nearly 300,000 people in the U.S are working in the music field. Recent progress in Artificial Intelligence (AI) has had profound impacts for music creators but has not yet had much impact on the practices of professional music performers. This project will investigate how AI technology could transform the work of future music performers for both individual practice (through developing coaching tools that analyze music performance) and collaborative practice (through developing systems that can substitute for missing performers in a group). The team will assess the tools in terms of two main questions: (1) When can AI technology provide measurable benefits to professional musicians' practice and performance? (2) What factors would affect future musicians' acceptance of AI technology in their work? The initial tools will focus on stringed instruments, but the underlying technologies are likely to be adaptable to a wide variety of performance contexts in the art and entertainment industry. The project team will also explore ways to use the tools to benefit students from disadvantaged groups; the tools may improve learning opportunities for students with limited access to music instruction. The team will recruit student researchers from groups underrepresented in STEM.This project will develop and integrate techniques from computer vision, natural language processing, and audio analysis to create two AI-enabled tools to support string music performers. The first tool, the Evaluator, aims to improve individual practice and performance. It analyzes a musician's sound and compares it to digitized music scores to detect deviations in intonation, rhythm, and dynamics. The Evaluator also analyzes captured video and compares it to a database of sample performers recorded with correct postures, allowing it to recommend better postures, which can both improve musical performance and reduce injury risks. The second tool, the Companion, aims to support common use cases when one or more musicians are missing from a group performance rehearsal. The Companion can play the part of one or several instruments to replace absent musicians, matching tempo, and style of the human musicians through audio analysis of their performance while also responding in real-time to verbal instructions. These tools will be developed and evaluated through a series of user studies, surveys, focus groups, and longitudinal deployments.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.