CCSS: Collaborative Research: Ubiquitous Sensing for VR/AR Immersive Communication: A Machine Learning Perspective

CCSS：协作研究：VR/AR 沉浸式通信的无处不在的感知：机器学习的视角

• 批准号: 1711335
• 负责人: Nicholas Mastronarde
• 金额: $ 15万
• 依托单位: SUNY at Buffalo
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2017
• 资助国家: 美国
• 起止时间: 2017-07-01 至 2022-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1711335&HistoricalAwards=false
• 关键词: CCSS; Collaborative; Research; Ubiquitous; Sensing

Virtual and augmented reality systems comprise multi-view camera sensors that capture a scene from multiple perspectives. The captured data is then used to construct an immersive representation of the scene on the user's head mounted display. Such systems are poised to enable and enhance numerous important applications, e.g., inspection of large-scale infrastructure, archival of historical sites, search and rescue, disaster response, military reconnaissance, natural resource management, and immersive telepresence. However, due to its emerging nature, virtual/augmented reality immersive communication is presently limited to gaming or entertainment demonstrations featuring off-line captured/computer-generated content, studio-type settings, and high-end workstations to sustain its high data/computing workload. Moreover, there is little understanding of the fundamental trade-offs between the required signal acquisition density and sensor locations across space and time, the dynamics of the captured scene (motion, geometry, and textures), the available network and system resources, and the delivered immersion quality. This renders existing solutions impractical for deployment on bandwidth and energy constrained remote sensors. The project addresses these challenges via rigorous analysis and concerted algorithmic and application advances at the intersection of multi-view space-time sensing and signal representation, delay-sensitive communication, and machine learning. Education and outreach activities will immerse students in the exciting areas of visual sensing, wireless communications, and machine learning, and will engage underrepresented students spanning K-12 through undergraduate levels.The objective of this project is to efficiently capture a remote environment using multiple camera sensors with the highest possible reconstruction quality under limited sampling and communication resources. This is achieved through four interrelated research tasks: (i) analysis of optimal space-time sampling policies that determine the sensors' locations and sampling rates to minimize the remote scene's reconstruction error; (ii) design of optimal signal representation methods that embed the sampled data jointly across space and time according to the allocated sampling rates; (iii) design of online learning sampling policies based on spectral graph theory that take sampling actions while exploring new sensor locations in the absence of a priori scene viewpoint signal knowledge; and (vi) design of computationally efficient self-organizing reinforcement learning methods that allow the wireless sensors to compute optimal transmission scheduling policies that meet the low-latency requirements of the overlaying virtual/augmented reality application while conserving their available energy. Integration, experimentation, and prototyping activities will be conducted to asses and validate the research advances in real-world settings. These technical advances will enable diverse applications of transformative impact.

虚拟和增强现实系统包含多视图相机传感器，可以从多个角度捕获场景。然后使用捕获的数据在用户的头戴式显示器上构建场景的沉浸式表示。此类系统有望实现和增强许多重要的应用，例如大型基础设施的检查、历史遗址的归档、搜索和救援、灾难响应、军事侦察、自然资源管理和沉浸式远程呈现。然而，由于其新兴性质，虚拟/增强现实沉浸式通信目前仅限于以离线捕获/计算机生成内容、工作室类型设置和高端工作站为特色的游戏或娱乐演示，以维持其高数据/计算工作负载。此外，人们对所需信号采集密度和跨空间和时间的传感器位置、捕获场景的动态（运动、几何和纹理）、可用网络和系统资源以及提供的沉浸质量之间的基本权衡了解甚少。这使得现有的解决方案无法部署在带宽和能量受限的远程传感器上。该项目通过在多视图时空传感和信号表示、延迟敏感通信和机器学习的交叉点上进行严格的分析以及协调一致的算法和应用进步来应对这些挑战。教育和推广活动将使学生沉浸在视觉传感、无线通信和机器学习等令人兴奋的领域，并将吸引从 K-12 到本科阶段代表性不足的学生。该项目的目标是在有限的采样和通信资源下，使用多个摄像头传感器有效地捕获远程环境，并具有尽可能高的重建质量。这是通过四个相互关联的研究任务实现的：（i）分析最佳时空采样策略，确定传感器的位置和采样率，以最大限度地减少远程场景的重建误差； (ii) 设计最佳信号表示方法，根据分配的采样率跨空间和时间联合嵌入采样数据； (iii) 基于谱图理论设计在线学习采样策略，在缺乏先验场景视点信号知识的情况下，在探索新传感器位置的同时采取采样动作； (vi)设计计算高效的自组织强化学习方法，允许无线传感器计算最佳传输调度策略，满足覆盖虚拟/增强现实应用的低延迟要求，同时节省可用能量。将进行集成、实验和原型设计活动，以评估和验证现实环境中的研究进展。这些技术进步将使变革性影响的多样化应用成为可能。

项目成果

Delay-Sensitive Energy-Harvesting Wireless Sensors: Optimal Scheduling, Structural Properties, and Approximation Analysis

延迟敏感能量收集无线传感器：最优调度、结构特性和近似分析

• DOI: 10.1109/tcomm.2019.2956510
• 发表时间: 2020
• 期刊: IEEE Transactions on Communications
• 影响因子: 8.3
• 作者: Sharma, Nikhilesh;Mastronarde, Nicholas;Chakareski, Jacob
• 通讯作者: Chakareski, Jacob

Deep Reinforcement Learning for Delay-Sensitive LTE Downlink Scheduling

• DOI: 10.1109/pimrc48278.2020.9217110
• 发表时间: 2020-08
• 期刊: 2020 IEEE 31st Annual International Symposium on Personal, Indoor and Mobile Radio Communications
• 作者: Nikhilesh Sharma;Sen Zhang;Someshwar Rao Somayajula Venkata;Filippo Malandra;Nicholas Mastronarde;Jacob Chakareski

Improving Data-Driven Reinforcement Learning in Wireless IoT Systems Using Domain Knowledge

使用领域知识改进无线物联网系统中数据驱动的强化学习

• DOI: 10.1109/mcom.111.2000949
• 发表时间: 2021
• 期刊: IEEE Communications Magazine
• 影响因子: 11.2
• 作者: Mastronarde, Nicholas;Sharma, Nikhilesh;Chakareski, Jacob
• 通讯作者: Chakareski, Jacob

Action Evaluation Hardware Accelerator for Next-Generation Real-Time Reinforcement Learning in Emerging IoT Systems

用于新兴物联网系统中的下一代实时强化学习的动作评估硬件加速器

• DOI: 10.1109/isvlsi49217.2020.00084
• 发表时间: 2020
• 期刊: 2020 IEEE Computer Society Annual Symposium on VLSI (ISVLSI
• 作者: Sun, Jianchi;Sharma, Nikhilesh;Chakareski, Jacob;Mastronarde, Nicholas;Lao, Yingjie
• 通讯作者: Lao, Yingjie

Energy Efficiency Analysis of UAV-Assisted mmWave HetNets

• DOI: 10.1109/icc.2018.8422870
• 发表时间: 2018-07
• 期刊: 2018 IEEE International Conference on Communications (ICC)
• 作者: Syed Naqvi;Jacob Chakareski;Nicholas Mastronarde;J. Xu;F. Afghah;Abolfazl Razi