SBIR Phase I: Low-latency polyphonic coding for interactive immersive applications

这个小企业创新研究（SBIR）第一阶段项目及其基础技术创新的更广泛的影响/商业潜力在于，它使涉及交互式和沉浸式媒体的各种应用成为可能，这些应用对未来几年有望大幅增长的几个行业至关重要。具体来说，该项目开发的低延迟音频技术是开发未来产品的关键推手，这些产品对技术行业的几个部门具有重要价值，最重要的是，为增强现实游戏和相关应用提供完全沉浸式互动媒体产品。在推动和支持互联网上的音乐合作以及实现远程音乐教育方面，预计会产生更多的影响，这两者都具有明确的文化和教育意义。另一个重大影响是使真正现实的电话会议体验成为可能，这对商业和社交网络都有重大影响，后者进一步为团体和家庭的充分互动社交聚会提供了一种现实的选择，而无需依靠昂贵的旅行。这个小型企业创新研究（SBIR）第一阶段项目开发了一种新的范式，通过有效的预测，以低延迟的方式对复调音频内容进行编码和联网，这对交互式沉浸式超现实多媒体新兴领域的众多应用至关重要。复调音频，或多个周期分量加上噪声的混合，长期以来一直无法有效预测，因此迫使最先进的编码器要么采用长时间的转换，这会导致大量延迟，并且与需要低延迟、低复杂性和低比特率的应用程序不兼容，要么接受显著降低的性能。该项目通过有效地利用复音音频信号所有周期分量的时间冗余，开发了在延迟、复杂性和比特率限制下接近最佳性能的技术。具体而言，编码范式建立在级联长期预测的新技术之上，该技术能够以低延迟对混合物中的所有周期分量进行联合预测。这种预测方法得到了强大的低复杂性参数估计技术的发展的补充，以最大限度地减少资源需求，有效地适应基频变化，侧信息优化以最大限度地减少比特率成本，对所有编码器模块的实际重新设计以充分利用预测能力，并增强了在损耗包网络上传输的容错性。

The broader impact/commercial potential of this Small Business Innovation Research (SBIR) Phase I project, and of its underlying technological innovation, is in enabling a diverse variety of applications involving interactive and immersive media, which are central to several sectors that are poised to grow substantially in the coming years. Specifically, the low-latency audio technology developed in the project is a critical enabler for the development of future products that are of significant value to several sectors of the technology industry including, most importantly, the enabling of fully immersive interactive media products for augmented reality games and related applications. Additional impact is expected in the advancement and support of musical collaboration over the internet and enabling remote music education, both with clear cultural and educational implications. Another significant impact is in enabling a truly realistic teleconferencing experience with considerable implications for both business and social networks, with the latter further providing a realistic alternative to fully interactive social gatherings of groups and families without recourse to costly travel.This Small Business Innovation Research (SBIR) Phase I project develops a novel paradigm for coding and networking of polyphonic audio content at low-latency via efficient prediction, which is critical to numerous applications in the emerging field of interactive immersive hyper-realistic multimedia. Polyphonic audio, or the mixture of multiple periodic components plus noise, has long resisted effective prediction, thus forcing state-of-the-art coders to either employ long transformation that incurs substantial delay and is incompatible with applications requiring low latency, low complexity and low bitrate, or accept significantly degraded performance. This project develops technologies that approach optimal performance despite constraints on latency, complexity and bit rate, by effectively exploiting temporal redundancies in all periodic components of polyphonic audio signals. Specifically, the coding paradigm builds on the novel technique of cascaded long term prediction, which enables joint prediction of all periodic components in the mixture, at low delay. This prediction approach is complemented by the development of powerful low-complexity parameter estimation techniques to minimize resource requirements, effective adaptation to fundamental frequency changes, side information optimization to minimize bitrate costs, practical redesign of all coder modules to fully exploit the prediction capabilities, and enhanced error-resilience for streaming over lossy packet networks.