3D human tracking and scene reconstruction for audio-visual AR/VR

在这项研究中，学生将研究在AR/VR应用中使用摄像机的视听输入来识别和跟踪复杂的多人场景中的人。实时3D场景渲染和空间音频再现算法需要与显示器/扬声器和房间几何形状相关的听者位置和头部姿势(最佳位置)。实时场景跟踪和自适应3D场景绘制将是本项目的目标。将考虑两种类型的显示场景。第一个是个性化头盔显示(HMD)系统，例如带耳机的VR耳机。第二种类型是使用带有扬声器的公共显示器(例如，电视或汽车挡风玻璃)。

In this research, the student will investigate identification and tracking of people throughout complex multi-person scenes using audio-visual input from a camera for AR/VR applications. Real-time 3D scene rendering and spatial audio reproduction algorithms require listener location and head pose (sweet spot) related to the display/loudspeakers and room geometry. Real-time scene tracking and adaptive 3D scene rendering will be the goal of this project. Two types of display scenarios will be considered. First one is a personalised Head-Mounted Display (HMD) system such as a VR headset with headphones. The second type is to use a communal display (e.g., TV or car windscreen) with loudspeakers.