Collaborative Research: FW-HTF-RM: Artificial Intelligence Technology for Future Music Performers

合作研究：FW-HTF-RM：未来音乐表演者的人工智能技术

• 批准号: 2326198
• 负责人: Yeon Ji Yun
• 金额: $ 90万
• 依托单位: Purdue University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-10-01 至 2026-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2326198&HistoricalAwards=false
• 关键词: Collaborative; Research; FW; HTF; RM

Nearly 300,000 people in the U.S are working in the music field. Recent progress in Artificial Intelligence (AI) has had profound impacts for music creators but has not yet had much impact on the practices of professional music performers. This project will investigate how AI technology could transform the work of future music performers for both individual practice (through developing coaching tools that analyze music performance) and collaborative practice (through developing systems that can substitute for missing performers in a group). The team will assess the tools in terms of two main questions: (1) When can AI technology provide measurable benefits to professional musicians' practice and performance? (2) What factors would affect future musicians' acceptance of AI technology in their work? The initial tools will focus on stringed instruments, but the underlying technologies are likely to be adaptable to a wide variety of performance contexts in the art and entertainment industry. The project team will also explore ways to use the tools to benefit students from disadvantaged groups; the tools may improve learning opportunities for students with limited access to music instruction. The team will recruit student researchers from groups underrepresented in STEM.This project will develop and integrate techniques from computer vision, natural language processing, and audio analysis to create two AI-enabled tools to support string music performers. The first tool, the Evaluator, aims to improve individual practice and performance. It analyzes a musician's sound and compares it to digitized music scores to detect deviations in intonation, rhythm, and dynamics. The Evaluator also analyzes captured video and compares it to a database of sample performers recorded with correct postures, allowing it to recommend better postures, which can both improve musical performance and reduce injury risks. The second tool, the Companion, aims to support common use cases when one or more musicians are missing from a group performance rehearsal. The Companion can play the part of one or several instruments to replace absent musicians, matching tempo, and style of the human musicians through audio analysis of their performance while also responding in real-time to verbal instructions. These tools will be developed and evaluated through a series of user studies, surveys, focus groups, and longitudinal deployments.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

美国有近30万人在音乐领域工作。人工智能（AI）的最新进展对音乐创作者产生了深远的影响，但尚未对专业音乐表演者的实践产生太大影响。该项目将研究人工智能技术如何改变未来音乐表演者的工作，包括个人练习（通过开发分析音乐表演的教练工具）和协作练习（通过开发可以替代团体中缺失表演者的系统）。该团队将根据两个主要问题来评估这些工具：（1）人工智能技术何时能为专业音乐家的练习和表演提供可衡量的好处？(2)哪些因素会影响未来音乐家在工作中接受AI技术？最初的工具将集中在弦乐器上，但基础技术可能适用于艺术和娱乐行业的各种表演环境。项目小组还将探讨如何利用这些工具使弱势群体的学生受益;这些工具可以改善获得音乐教育机会有限的学生的学习机会。该团队将从STEM领域代表性不足的群体中招募学生研究人员。该项目将开发和整合计算机视觉、自然语言处理和音频分析等技术，以创建两个支持人工智能的工具来支持弦乐表演者。第一个工具是“评价员”，旨在改进个人做法和业绩。它分析音乐家的声音，并将其与数字化乐谱进行比较，以检测语调，节奏和力度的偏差。Evaluator还分析捕获的视频，并将其与记录有正确姿势的样本表演者数据库进行比较，从而能够推荐更好的姿势，这既可以提高音乐表现，又可以降低受伤风险。第二个工具是Companion，旨在支持一个或多个音乐家在团体表演排练中缺席时的常见用例。 同伴可以演奏一种或多种乐器，以取代缺席的音乐家，通过对他们的表演进行音频分析来匹配人类音乐家的克里思和风格，同时还实时响应口头指令。这些工具将通过一系列用户研究、调查、焦点小组和纵向部署进行开发和评估。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的知识价值和更广泛的影响审查标准进行评估来支持。