Collaborative Research: FW-HTF-RM: Artificial Intelligence Technology for Future Music Performers

美国有近30万人在音乐领域工作。人工智能（AI）的最新进展对音乐创作者产生了深远的影响，但尚未对专业音乐表演者的实践产生太大影响。该项目将研究人工智能技术如何改变未来音乐表演者的个人练习（通过开发分析音乐表演的指导工具）和合作练习（通过开发可以替代团队中缺失表演者的系统）。该团队将从两个主要问题来评估这些工具：(1)人工智能技术何时能为专业音乐家的实践和表演提供可衡量的好处？(2)哪些因素会影响未来音乐家在工作中接受人工智能技术？最初的工具将专注于弦乐器，但其基础技术可能适用于艺术和娱乐行业的各种表演环境。项目团队还将探索如何利用这些工具使弱势群体的学生受益；这些工具可以为那些接触音乐教学机会有限的学生提供学习机会。该团队将从STEM中代表性不足的群体中招募学生研究人员。该项目将开发和整合计算机视觉、自然语言处理和音频分析技术，以创建两个支持人工智能的工具来支持弦乐表演者。第一个工具，评估器，旨在提高个人的实践和表现。它分析音乐家的声音，并将其与数字化乐谱进行比较，以检测音调、节奏和动态的偏差。Evaluator还会分析捕捉到的视频，并将其与数据库中记录的正确姿势的表演者样本进行比较，从而推荐更好的姿势，这既可以提高音乐表现，又可以降低受伤风险。第二个工具，同伴，旨在支持一个或多个音乐家在团体演出排练中缺席的常见用例。“同伴”可以演奏一种或几种乐器来代替缺席的音乐家，通过音频分析他们的演奏，匹配人类音乐家的节奏和风格，同时还可以实时响应口头指令。这些工具将通过一系列用户研究、调查、焦点小组和纵向部署来开发和评估。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Nearly 300,000 people in the U.S are working in the music field. Recent progress in Artificial Intelligence (AI) has had profound impacts for music creators but has not yet had much impact on the practices of professional music performers. This project will investigate how AI technology could transform the work of future music performers for both individual practice (through developing coaching tools that analyze music performance) and collaborative practice (through developing systems that can substitute for missing performers in a group). The team will assess the tools in terms of two main questions: (1) When can AI technology provide measurable benefits to professional musicians' practice and performance? (2) What factors would affect future musicians' acceptance of AI technology in their work? The initial tools will focus on stringed instruments, but the underlying technologies are likely to be adaptable to a wide variety of performance contexts in the art and entertainment industry. The project team will also explore ways to use the tools to benefit students from disadvantaged groups; the tools may improve learning opportunities for students with limited access to music instruction. The team will recruit student researchers from groups underrepresented in STEM.This project will develop and integrate techniques from computer vision, natural language processing, and audio analysis to create two AI-enabled tools to support string music performers. The first tool, the Evaluator, aims to improve individual practice and performance. It analyzes a musician's sound and compares it to digitized music scores to detect deviations in intonation, rhythm, and dynamics. The Evaluator also analyzes captured video and compares it to a database of sample performers recorded with correct postures, allowing it to recommend better postures, which can both improve musical performance and reduce injury risks. The second tool, the Companion, aims to support common use cases when one or more musicians are missing from a group performance rehearsal. The Companion can play the part of one or several instruments to replace absent musicians, matching tempo, and style of the human musicians through audio analysis of their performance while also responding in real-time to verbal instructions. These tools will be developed and evaluated through a series of user studies, surveys, focus groups, and longitudinal deployments.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.