Emphasizing Explanation in AI Augmented String Instrumental Education

音乐是我们生活中不可或缺的一部分，有大量证据表明，给学生进行器乐教育的机会很重要，因为它有利于发展认知、社交和身体技能，并有利于心理健康。然而，学习演奏乐器是一个漫长而复杂的过程。私立音乐教育费用昂贵，只有世界上享有特权的一部分人才能负担得起。最近开发的人工智能和在线教育工具提供了一个机会，为音乐教育创造新技术，可以接触到非常多样化的学生群体。然而，关于如何进行乐器的智能在线教学，以及如何通过涉及运动指令的学科中的个性化教学来保持学生的参与度，人们知之甚少。在这个项目中，一个在小提琴教学和视觉和听觉信号分析方面拥有专业知识的团队旨在开发一个人工智能平台来分析学生在个人练习时间的演奏情况。通过手持设备上的摄像头和麦克风录制的视频和音频输入用于提供关于姿势、弓形运动、音质和课程材料选择的反馈。这个项目促进了我们对运动任务的学习和教学的理解，并为从多模式数据中计算感知人体运动提供了新的见解。它支持音乐领域的数字人文；从人文的角度来看，它使音乐教育民主化，并为更多和更多样化的人群提供器乐教学的好处。受教育反馈干预的认知研究的启发，该项目开发了一个原型AI系统，作为对小提琴学生、教师和监督家长的虚拟辅助，并像人类教师一样提供解释。这是通过两个独特的组成部分实现的：1)反馈系统，它基于视觉和听觉分析以及错误的因果关系提供建议，即为什么运动错误会导致不理想的声音；以及2)根据感知评估和记录材料语料库为学生分配特定的教育材料。技术工作涉及从小提琴演奏者那里收集多模式数据，开发用于分析演奏者表演的机器学习算法，从视频和创建的乐谱中录制和整理教育音乐作品语料库，设计游戏化的用户界面，以及反馈指令。开发的人工智能软件，通过观察小提琴学生收集的数据，以及将被数字化、记录和分类的音乐语料库的结合，构成了音乐教育学领域向21世纪迈出的重要一步，以及通过感知研究人类运动学习的工具的创新。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Music is an integral part of our lives and there is a lot of evidence that giving students opportunities to instrumental music education is important, because of its advantages for developing cognitive, social, and physical skills, and its benefits for mental health. Learning to play a musical instrument, however, constitutes a lengthy and complex process. Private music instruction is costly, afforded only by a privileged part of the world population. The recently developed tools in AI and online education provide an opportunity to create new technology for music education which can reach a very diverse student population. Little is known, however, on how to do intelligent online teaching of musical instruments, and how to keep students engaged through individualized instructions in a discipline that involves motor instructions. In this project, a team with expertise in violin pedagogy and visual and auditory signal analysis aims to develop an AI platform to analyze a student’s playing during individual practice time. Video and audio input recorded by cameras and microphones on handheld devices are used to provide feedback on posture, bow movements, sound quality and selection of curricular materials. This project advances our understanding of the learning and teaching of motor tasks, and provides new insights for computational perception of human movements from multi-modal data. It supports digital humanities in the field of music; from a humanity perspective, it democratizes music education, and provides the benefits of instrumental instruction to a large and more diverse population.Inspired by cognitive studies on educational feedback intervention, this project develops a prototype AI system that acts as a virtual assistance to violin students, teachers, and supervising parents, and like a human teacher it also provides explanations. This is accomplished through two unique components: 1) a feedback system that provides advice based on visual and auditory analysis and causal relationships of errors, i.e., why a movement error caused non-ideal sound; and 2) assignment of educational materials specific to the students based on the perceptual evaluation and a corpus of recorded materials. The technical work involves the collection of multi-modal data from violin players, the development of machine learning algorithms for analyzing the players’ performance, the recording and curation of a corpus of educational music pieces from videos and created sheet music, the design of a gamified user interface, and feedback instructions. The combination of developed AI software, data collected through observation of violin students, and the music corpuses to be digitized, recorded, and categorized constitute a major step forward into the 21st century for the field of music pedagogy and innovation of tools for studying human motor learning through perception.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.