III: Small: Collaborative Research: Algorithms for Query by Example of Audio Databases

随着多媒体存储库的激增和增长，寻找自动索引、标记和访问多媒体内容（如音频文档）的方法变得越来越重要。社区生成的SoundCloud存储库就是一个例子。它包含乐队、音效、播客等录音，贡献者每分钟上传12小时的音频。像SoundCloud这样的存储库通常在文件级别用短文本标签标记音频。使用这些标签对所需记录进行基于文本的搜索可能会有问题。在曲目中进行基于文本的搜索是不可能的，因为它们没有使用文件主体中的标记进行索引。在这个项目中，罗彻斯特大学和西北大学的研究人员旨在通过实例查询开发音频搜索的方法和系统，其中示例在某些关键方面与数据库中所需的音频相似，但不是完全匹配。这将允许在文件中搜索，而不需要基于文本的标记。这个项目将专注于使用声音模仿作为搜索关键字，因为它们对人类来说是自然的，并广泛用于互动。它将开发一种新的声音搜索引擎，将声音模仿作为查询（例如，模仿鸟叫来查找鸟叫的录音）。为这种搜索音频/视频收集的新方法而开发的技术也将以许多其他方式造福社会，例如犯罪监视（例如，政策监测站的自动枪击或尖叫检测），生物多样性测量（例如，在现场录音中自动识别声音“像这样”的鸟类物种），听力受损人士的环境意识（例如，当我的狗在叫时提醒我），电影音效设计师的制作辅助(例如，从包含数千种声音效果的数据库中查找摔门声)，以及基于声音的诊断（例如，“你的车需要一个新的启动马达”）。该项目将有利于科学技术工程和数学（STEM）教育，因为基于音频的研究已被证明是吸引多样化大学生进入STEM学科的成功方法。人声模仿传递了丰富的信息，包括许多声学方面：音高、响度、音色、它们的时间演变和节奏模式等。这使用户可以查询难以用文本标记搜索的精确声音。然而，出于同样的原因，声音模仿可能在许多方面与期望的目标不同。由于人类声音系统的物理限制，与要检索的声音相比，查询声音也可能位于非常有限的声音空间中。建立一个成功的语音模仿查询系统需要研究表示音频和检索基于查询的音频方法，这些查询仅在其可测量维度的子集上与目标声音相似。它还需要能够在非基于文本的上下文中方便地提供查询和精炼搜索结果的接口。对于前者，研究人员将研究使用深度神经网络学习特定方面音频表示的方法。研究人员还将开发适合这些表示的匹配算法。研究人员将设计新颖的搜索界面，让用户迭代地改进他们的搜索结果。系统将从相互作用中学习，并调整不同声学方面的权重，以搜索所需的声音。本研究的预期结果是：(1)突出语音查询与一般音频目标声音匹配的感知相关特征的音频表示；(2)将语音查询与一般音频匹配和对齐的算法；(3)利用声音模仿和声音实例迭代精炼搜索结果的交互方法；(4)大型人声模仿和声音数据集；(5)一个体现这些成果的开源声音检索系统。关于这个项目的更多信息可以在项目网站（http://www.ece.rochester.edu/projects/air/projects/audiosearch）上找到。

Finding ways to automatically index, label, and access multimedia content (such as audio documents) is increasing in importance as multimedia repositories proliferate and grow. The community-generated SoundCloud repository is one example. It contains recordings of bands, sound effects, podcasts, etc., and contributors upload 12 hours of audio every minute. Repositories like SoundCloud typically tag audio at the file level with short text labels. Text-based search for a desired recording using these labels can be problematic. Text-based search within a track is not possible, since they are not indexed with tags in the body of the file. In this project, investigators at the University of Rochester and Northwestern University aim to develop methods and a system for audio search via query-by-example, where the example is similar, in some key way, to the desired audio in the database, but is not an exact match. This will allow search within files, bypassing the need for text-based tagging. This project will be focusing on using vocal imitations as search keys because they are natural for humans and are widely used in interaction. It will develop a novel search engine for sounds that takes vocal imitations as queries (e.g., imitation of a bird call to find recordings of the bird call). The technology developed for this novel way to search through audio/video collections will also benefit society in numerous other ways, such as crime surveillance (e.g., automated gunshot or scream detection for policy monitoring stations), biodiversity measurement (e.g., automatic ID of bird species that sound "like this" in field recordings), environmental awareness for the hearing impaired (e.g., alert me when my dog is the one barking), a production aid for a movie sound designer (e.g., finding door slam sounds in a database of thousands of sound effects), and sound-based diagnosis (e.g., "your car needs a new starter motor"). The project will benefit science technology engineering and mathematics (STEM) education as audio-based research has been shown to be a successful way to attract diverse college students into STEM disciplines.Vocal imitation conveys rich information covering many acoustic aspects: pitch, loudness, timbre, their temporal evolutions, and rhythmic patterns, etc. This lets a user query for precise sounds that are difficult to search for with text tags. For the same reason, however, vocal imitations may vary from the desired target on many dimensions. The query sound can also lies in a very constrained sound space compared to the sounds to be retrieved, due to the physical constraints of the human vocal system. Building a successful query-by-vocal-imitation system will require research into methods for representing audio and retrieving audio based on queries that are similar to target sounds only on a subset of their measurable dimensions. It will also require interfaces that facilitate providing queries and refining search results in a non-text-based context. For the former, the investigators will research on methods for learning of aspect-specific audio representations using deep neural networks. The investigators will also develop matching algorithms suitable for these representations. The investigators will design novel search interfaces that let users iteratively refine their search results. The system will learn from the interactions and adjust the weightings of different acoustic aspects to search for the wanted sound. Expected outcomes of this research are: (1) audio representations that highlight perceptually relevant features of vocal queries for matching to general audio target sounds; (2) algorithms for matching and aligning vocal queries to general audio; (3) interaction methods for iteratively refining search results using vocal imitations and sound examples; (4) a large vocal imitation and sound dataset; and (5) an open-source sound retrieval system that embodies these outcomes. More information about this project can be found at the project web site (http://www.ece.rochester.edu/projects/air/projects/audiosearch).