ITR: A User-centric Content-based Approach to Indexing, Query and Retrieval of Music through Signal Processing and Knowledge-based Methods

该研究项目将开发基于内容的音乐数据库索引方法，使用信号处理和基于知识的方法相结合，设计用于使用歌唱或哼唱旋律进行查询的统计算法，以及设计用于检索查询信息的强大搜索技术，特别是在存在不确定性的情况下。研究方法以统计建模为基础，以用户为中心，包括三个主要部分：(1)表示和索引——利用人类音乐直觉和音乐感知与认知的理论和知识的音乐信息。(2)查询公式和交互方式——通过哼唱这种自然活动与音乐数据交互的算法设计。(3)搜索和检索——将用户查询与数据库匹配的算法，对查询生成中的不确定性和错误具有鲁棒性。该架构将包括一个前端识别器，该识别器使用统计模式识别方法将嗡嗡声信号转换为音符，该方法与后端音乐数据库接口，该数据库使用感知上可行的特征进行索引。根据索引匹配查询的搜索过程被表述为统计信息检索问题。该项目将逐步采用丰富的索引表示，包括重复出现的主题、和弦、节拍和关键信息的重复模式。这种音乐理论知识的推导将受益于对音乐调性进行数学建模的原则方法。统计框架允许处理查询公式和检索中的可变性和不确定性。它还支持在查询结果中提供解决方案的质量。这项工作将不仅有助于音乐的特定经济意义的应用领域，而且有助于如何在复杂的感官数据中索引和搜索人类有意义的模式的一般信息科学知识。该项目还将作为促进跨学科研究生和本科生研究和教育的工具。

This research project will develop methods for content-based indexing of music databases, using a combination of signal processing and knowledge-based methods, design of statistical algorithms for enabling queries using sung or hummed melodies, and design of robust search techniques for retrieving the queried information, especially in the presence of uncertainty. The research approach, which is based on statistical modeling, is user-centric and comprises three major components: (1) Representation and Indexing - musical information utilizing theories and knowledge about human musical intuition, and music perception and cognition. (2) Query Formulation and Interaction Modality - algorithm design for enabling interaction with music data through humming, a natural activity. (3) Search and Retrieval - algorithm design to match user query against the database that will be robust to uncertainties and errors in the query generation.The architecture will include a front-end recognizer, that converts the humming signal to notes using a statistical pattern recognition approach, which interfaces with a back-end music database that is indexed using perceptually viable features. The search process matching a query against indices is formulated as a statistical information retrieval problem. The project will employ progressively rich indexing representation including repeating patterns for recurring themes, chord, beat, and key information. Derivation of such music-theoretic knowledge will benefit from a principled approach of mathematically modeling tonality in music. The statistical framework allows for handling variability and uncertainty in query formulation and retrieval. It also enables providing for quality of solution in the query results.This work will contribute not only to the specific economically significant application area of music, but also to general information science knowledge on how to index and search humanly-meaningful patterns in complex sensory data. The project will also serve as a vehicle to foster cross-disciplinary graduate and undergraduate research and education.