CAREER: Making music documents accessible in musical terms

在二十一世纪，我们面临的关键问题之一是信息检索和管理。寻找以有意义的方式自动索引、标记和访问多媒体内容（如音乐文档）的方法是一个开放的研究问题，随着多媒体数据库的激增和增长，这个问题的重要性也在增加。音乐收藏，如苹果电脑iTunes存储库中的350万份唱片，构成了最受欢迎的在线多媒体内容之一。对于学者、音乐家，甚至是不经意的听众来说，音乐文件只是一个开始，是启动手头任务的工具。音乐家可能会对混音感兴趣，即使他们所有可用的是最后的混音。学者们可能希望分析一首曲子中的和声。还有一些人可能希望卡拉ok能跟随歌手富有表现力的节奏，或者想办法消除女儿长笛演奏会录音中不想要的手机铃声。本研究的目的是开发两种关键的促进技术来实现这些类型的交互：分数对齐和源分离。乐谱对齐，包括将音频表演与机器可读乐谱中的事件对齐。当与乐谱对齐时，可以通过旋律和和声内容来处理演出。我们建议通过使机器能够跟随部分指定的分数（例如Jazz的导音表）来推进最先进的技术。这种排列需要从乐谱中更深层的结构描述（导音表中的和弦）中推断出可能的表面结构（即兴独奏中的音符序列）。这将使整个音乐类别，如爵士乐，流行音乐和摇滚，目前不能对齐到分数。第二种技术，源分离，是在给定的混合源信号中隔离单个源信号的过程。通过音源分离，单个乐器和声音可以被访问、识别和操纵，其方式超出了商业音频搜索和编辑软件的能力。我们将通过分数通知分离以及从声学混合中近似源模型的新迭代方法来推进该领域。这个想法是开发一个音乐信息检索和交互的协同系统，它使用多种文档模式（书面乐谱、音频文件、MIDI）来推断更多关于音乐结构的信息，而不是使用单一模式。这项研究将影响信号处理社区（源分离），音乐信息检索社区（音乐索引和搜索）和人工智能社区（现实世界数据的智能抽象工具）。为了广泛传播这项工作，将在互联网上提供示范工具，并将结果发表在相关期刊和会议上。PI致力于让本科生和历史上代表性不足的群体成员参与研究，并与SROP和UROP项目合作实现这一目标。PI还教授“音乐的机器感知”课程，研究成果将传播给各种各样的学生。

Making Music Documents Accessible in Musical TermsOne of the key problems facing us in the 21st century is information retrieval and management. Finding ways to automatically index, label, and access multimedia content (such as music documents) in meaningful ways is an open research question that increases in importance as multimedia databases proliferate and grow. Music collections, such as the 3.5 million recordings in Apple Computer's iTunes repository, comprise one of the most popular categories of on-line multimedia content.For scholars, musicians and even casual listeners, the music document is only the beginning, a tool to initiate the task at hand. Musicians may be interested in remixing a musical recording even though all they have available is the final mix. Scholars may wish to analyze the harmonies in a piece. Others may want karaoke that follows the singer's expressive timing, or a way to remove the sound of an unwanted cell phone ring from a recording of their daughter's flute recital.The objective of this research is to develop two key facilitating technologies to enable these kinds of interactions: score alignment and source separation. Score alignment, involves aligning an audio performance and to the events in a machine-readable music score. When aligned to a score, a performance can be addressed by melodic and harmonic content. We propose to advance the state-of-the-art by enabling a machine to follow partially specified scores (such as Jazz lead sheets). This alignment require significant inference about likely surface structures (the note sequence in an improvised solo) from deeper structural descriptions in the score (the chords in a lead sheet). This will enable alignment of entire classes of music, such as much Jazz, Pop and Rock, that cannot currently be aligned to scores.The second technology, source separation, is the process of isolating individual source signals, given mixtures of the source signals. With source separation, individual instruments and sounds can be accessed, identified and manipulated in ways beyond the power of commercial audio search and editing software. We will advance the field through score-informed separation, as well as new iterative methods for approximating source models from acoustic mixtures.The idea is to develop a synergistic system for music-information-retrieval and interaction that uses multiple document modalities (written scores, audio files, MIDI) to infer more about the music structure than is possible using a single modality.This research will impact the signal-processing community (source separation), the music information retrieval community (music indexing and search) and the artificial intelligence community (tools for intelligent abstraction of real-world data). To broadly disseminate the work, demonstration tools will be made available over the internet and results will be published in relevant journals and conferences. The PI is committed to involving undergraduates and members of historically underrepresented groups in research, working with the SROP and UROP programs to make this happen. The PI also teaches the course "Machine Perception of Music" where research results will be disseminated to a wide variety of students.