ITR: A User-centric Content-based Approach to Indexing, Query and Retrieval of Music through Signal Processing and Knowledge-based Methods

这一研究项目将开发基于内容的音乐数据库索引方法，使用信号处理和基于知识的方法相结合，设计统计算法以便能够使用歌唱或哼唱的旋律进行查询，并设计稳健的搜索技术来检索查询的信息，特别是在存在不确定性的情况下。该研究方法以统计模型为基础，以用户为中心，由三个主要部分组成：(1)音乐信息的表示和索引--利用人类音乐直觉和音乐感知与认知的理论和知识。(2)查询表达和交互模式-通过哼唱这一自然活动与音乐数据进行交互的算法设计。(3)搜索和检索-算法设计用于将用户查询与数据库进行匹配，该算法对查询生成中的不确定性和错误具有健壮性。该体系结构将包括一个前端识别器，该识别器使用统计模式识别方法将哼唱信号转换为音符，该方法与使用感知上可行的特征进行索引的后端音乐数据库接口。将查询与索引进行匹配的搜索过程表示为统计信息检索问题。该项目将采用逐步丰富的索引表示法，包括重复的主题、和弦、节拍和关键信息的重复模式。这种音乐理论知识的衍生将受益于对音乐中的调性进行数学建模的原则性方法。统计框架允许处理查询编制和检索中的可变性和不确定性。这项工作不仅将有助于音乐的特定经济应用领域，而且将有助于一般信息科学知识如何在复杂的感觉数据中索引和搜索具有人类意义的模式。该项目还将成为促进跨学科研究生和本科生研究和教育的工具。

This research project will develop methods for content-based indexing of music databases, using a combination of signal processing and knowledge-based methods, design of statistical algorithms for enabling queries using sung or hummed melodies, and design of robust search techniques for retrieving the queried information, especially in the presence of uncertainty. The research approach, which is based on statistical modeling, is user-centric and comprises three major components: (1) Representation and Indexing - musical information utilizing theories and knowledge about human musical intuition, and music perception and cognition. (2) Query Formulation and Interaction Modality - algorithm design for enabling interaction with music data through humming, a natural activity. (3) Search and Retrieval - algorithm design to match user query against the database that will be robust to uncertainties and errors in the query generation.The architecture will include a front-end recognizer, that converts the humming signal to notes using a statistical pattern recognition approach, which interfaces with a back-end music database that is indexed using perceptually viable features. The search process matching a query against indices is formulated as a statistical information retrieval problem. The project will employ progressively rich indexing representation including repeating patterns for recurring themes, chord, beat, and key information. Derivation of such music-theoretic knowledge will benefit from a principled approach of mathematically modeling tonality in music. The statistical framework allows for handling variability and uncertainty in query formulation and retrieval. It also enables providing for quality of solution in the query results.This work will contribute not only to the specific economically significant application area of music, but also to general information science knowledge on how to index and search humanly-meaningful patterns in complex sensory data. The project will also serve as a vehicle to foster cross-disciplinary graduate and undergraduate research and education.