EPSRC-SFI: SpheryStream

该方案解决了在快速发展的工程领域中虚拟现实和沉浸式技术的重要但尚未得到充分研究的领域。我们的最终愿景是赋予任何最终用户在任何应用场景和任何连接级别的虚拟环境中前所未有的完全沉浸感。这就要求VR系统以个性化的方式大规模运行，在满足质量和延迟标准的同时保持带宽容忍度。这只能通过编码/流/呈现链的根本变革来实现，即必须将交互用户置于系统的核心，而不是链的末端。具体来说，将我们的最终愿景变为现实需要VR流媒体技术的不断创新，以克服以下主要挑战：i)在媒体编解码器针对平面内容进行优化的场景中，视频内容的球形格式；Ii)用户的交互行为给视频内容的受欢迎程度带来了不确定性。为了克服前面的挑战，球形视频目前投影在二维平面域上，但这引入了明显的变形，因为球体不是一个可展开的表面。为了解决用户行为的不确定性，最近已经研究了预测模型，但没有一个模型实际上适用于球形（非变形）域。为了克服沉浸式技术中的上述挑战，需要开发一种有效的工具，用于在球面域中进行导航模式分析，并利用该工具来预测用户的行为，并将整个编码-交付-渲染管道构建为以用户和几何为中心的系统。本提案侧重于开发新的VR交付策略，该策略将以用户、内容和应用依赖的方式，依赖于对全向内容导航模式的分析和预测。通过预测用户的行为，我们可以只传递用户真正会消费的内容，从而最大限度地减少传播资源消耗，同时最大限度地提高用户体验质量。关键的新颖之处将是直接在球面上解决上述挑战，而不是在投影（因此变形）的二维域上。开发的VR系统将通过一个真正的全方位视频流测试平台进行测试，该测试平台将在项目范围内实施。该项目的成果将有三方面：i)依赖于用户内容和应用的VR流媒体平台，从源编解码器到交付平台进行优化；Ii)用户分析工具，能够以客观的方式评估用户行为（当与内容交互时），以识别常见的导航模式，并预测未来的行为；iii) VR流媒体测试平台，以测试所提出的技术。

This proposal addresses the important and not well-studied area in the fast-developing engineering space of the virtual reality and immersive technology. Our final vision is to endow any final user with an unprecedented sense of full immersion in virtual environments for any applicative scenario and with any connectivity level. This requires VR systems that operate at scale, in a personalized manner, remaining bandwidth-tolerant whilst meeting quality and latency criteria. This can be accomplished only by a fundamental revolution of the coding/streaming/rendering chain that has to put the interactive user at the heart of the system rather than at the end of the chain.Specifically, bringing our final vision to reality requires constant innovation in VR streaming technology to overcome the following main challenges: i) the spherical format of the video content in a scenario in which media codecs are optimized for planar content; ii) the interactive behaviour of users that introduces uncertainty on the popularity of the video content. To overcome the former challenges, the spherical video is currently projected on a bidimensional planar domain, but this introduces noticeable deformation since the sphere is not a developable surface. To address the uncertainty of users' behaviours, predicting models have been investigated recently, but none of them actually work on the spherical (not deformed) domain. To overcome the above challenges in immersive technology, there is the need to develop an efficient tool for navigation patterns analysis in the spherical domain and leverage on that to predict users' behaviour and build the entire coding-delivery-rendering pipeline as a user- and geometry-centric system. This proposal focuses on develop novel VR delivery strategies, which will rely on the analysis and prediction of the navigation patterns of the omnidirectional content navigation, in a user, content- and application-dependent fashion. Predicting users' behaviour will allow to delivery only the content that will be actually consumed by the users, minimising the transmission resources consumption while maximising the user's Quality of Experience. The key novelty will be to address the above challenges working directly on the spherical domain, rather than on the projected (thus deformed) bidimensional domain. The developed VR system will be tested via a real omnidirectional video streaming testbed that will be implemented within the scope of the project. The outcomes of the project will be three-folds: i) user- content- and application-dependent VR streaming platform, optimized from the source codec to the delivery platform; ii) users analysis tools able to assess users' behaviour (when interactive with the content) in an objective fashion to identify common navigation patterns, and to predict future behaviours; iii) VR streaming testbed to test the proposed technology.