Emphasizing Explanation in AI Augmented String Instrumental Education

音乐是我们生活中不可或缺的一部分，有很多证据表明，让学生有机会接受器乐教育是很重要的，因为它有利于发展认知，社交和身体技能，并有利于心理健康。然而，学习演奏乐器是一个漫长而复杂的过程。私人音乐教育费用昂贵，只有世界上一部分特权阶层才能负担得起。最近开发的人工智能和在线教育工具为音乐教育创造了新技术，可以覆盖非常多样化的学生群体。然而，关于如何进行乐器的智能在线教学，以及如何通过涉及运动指令的学科中的个性化指令让学生参与其中，人们知之甚少。在这个项目中，一个拥有小提琴教学法和视觉和听觉信号分析专业知识的团队旨在开发一个人工智能平台，以分析学生在个人练习时间的演奏。手持设备上的摄像头和麦克风记录的视频和音频输入用于提供有关姿势，弓运动，音质和课程材料选择的反馈。该项目推进了我们对运动任务的学习和教学的理解，并为多模态数据中人类运动的计算感知提供了新的见解。它支持音乐领域的数字人文;从人性化的角度来看，它使音乐教育民主化，并为更多样化的人群提供乐器教学的好处。受教育反馈干预的认知研究的启发，该项目开发了一个原型AI系统，作为小提琴学生，教师和监督家长的虚拟帮助，就像一个人类老师一样，它也提供解释。这是通过两个独特的组成部分来实现的：1）反馈系统，其基于视觉和听觉分析以及错误的因果关系提供建议，即，为什么运动误差导致非理想的声音;以及2）基于感知评估和记录材料的语料库来分配特定于学生的教育材料。这项技术工作涉及从小提琴演奏者那里收集多模态数据，开发用于分析演奏者表现的机器学习算法，录制和管理来自视频和创作的乐谱的教育音乐作品语料库，设计游戏化的用户界面和反馈说明。结合开发的人工智能软件，通过观察小提琴学生收集的数据，以及数字化，记录的音乐语料库，这一奖项反映了NSF的法定使命，并被认为值得通过使用基金会的知识价值和更广泛的影响审查标准。

Music is an integral part of our lives and there is a lot of evidence that giving students opportunities to instrumental music education is important, because of its advantages for developing cognitive, social, and physical skills, and its benefits for mental health. Learning to play a musical instrument, however, constitutes a lengthy and complex process. Private music instruction is costly, afforded only by a privileged part of the world population. The recently developed tools in AI and online education provide an opportunity to create new technology for music education which can reach a very diverse student population. Little is known, however, on how to do intelligent online teaching of musical instruments, and how to keep students engaged through individualized instructions in a discipline that involves motor instructions. In this project, a team with expertise in violin pedagogy and visual and auditory signal analysis aims to develop an AI platform to analyze a student’s playing during individual practice time. Video and audio input recorded by cameras and microphones on handheld devices are used to provide feedback on posture, bow movements, sound quality and selection of curricular materials. This project advances our understanding of the learning and teaching of motor tasks, and provides new insights for computational perception of human movements from multi-modal data. It supports digital humanities in the field of music; from a humanity perspective, it democratizes music education, and provides the benefits of instrumental instruction to a large and more diverse population.Inspired by cognitive studies on educational feedback intervention, this project develops a prototype AI system that acts as a virtual assistance to violin students, teachers, and supervising parents, and like a human teacher it also provides explanations. This is accomplished through two unique components: 1) a feedback system that provides advice based on visual and auditory analysis and causal relationships of errors, i.e., why a movement error caused non-ideal sound; and 2) assignment of educational materials specific to the students based on the perceptual evaluation and a corpus of recorded materials. The technical work involves the collection of multi-modal data from violin players, the development of machine learning algorithms for analyzing the players’ performance, the recording and curation of a corpus of educational music pieces from videos and created sheet music, the design of a gamified user interface, and feedback instructions. The combination of developed AI software, data collected through observation of violin students, and the music corpuses to be digitized, recorded, and categorized constitute a major step forward into the 21st century for the field of music pedagogy and innovation of tools for studying human motor learning through perception.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.