CHS: Small: TICE - Telematic Immersive Classroom Environments

CHS：小型：TICE - 远程信息处理沉浸式课堂环境

• 批准号: 1909229
• 负责人: Jonas Braasch
• 金额: $ 50万
• 依托单位: Rensselaer Polytechnic Institute
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-04-01 至 2024-03-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1909229&HistoricalAwards=false
• 关键词: CHS; Small; TICE; Telematic; Immersive

Since their inception, telecommunication systems have targeted the exchange between exactly two people. This holds true for the telegraph, the telephone, and typical video conference systems. Meetings for larger groups of people are usually conducted using the same personal interfaces – take, for example, a telephone conference call. This practice contrasts with a traditional group meeting, for example, or a classroom scenario. In the latter, students and teachers can communicate with each other while having an awareness of the spatial location of each participant and the ability to identify each speaker easily. The goal of this project is to develop technology that enables two distant classrooms to be joined by telecommunication devices without the need for personal communication interfaces. An electronically adjustable microphone array will be designed that can virtually focus on an active speaker. This way, echoes that would otherwise occur in a collaborative telecommunication scenario with loudspeakers will be avoided. The system will be designed such that the sounds of a speaker will emanate from the position at which they are seen on a live video feed. Project outcomes are expected to have broad impact by providing new educational opportunities for classroom students by virtually connecting them to other classrooms for collaborative learning experiences. This project will develop a spatially correct, long-distance audio connection between collaborative spaces, for example two classrooms with groups of people at each site. The missing link needed to enable such remote collaboration is an audio tracking system that can follow all participants on both sides of the connection. The approach will use two 16-channel spherical microphones as the main audio tracking devices. The key idea is that each spherical microphone will receive continuous information from additional body-worn microphones, which is utilized to ensure echo-free bidirectional communication. The body-worn microphone signals will inform the tracking system who among the participants is speaking at a given time interval and in a given frequency range. The spherical microphone is then used to track each sound source during the time intervals that the source is active and in those frequency bands that are not disturbed by other sources. Using machine-learning tools, including Bayesian methods, Kalman filters, and Deep Neural Networks, the tracking system will implement a joint bottom-up/top-down architecture. Two existing sites are equipped with immersive loudspeaker systems, each of which will enable accurate local reproduction of spatialized audio from the remote site. These sites are also equipped with collaborative virtual reality systems with immersive video capabilities, and they will serve as laboratories for this project, enabling a system to transmit bidirectional audio/video streams with spatially congruent audio/video images. As an extensions of the project, the audio-tracking system will be paired with an existing 6-camera tracking system that is mounted on the laboratory ceiling and technologies will be developed to replace the body-worn microphones with virtual microphones using the beam-forming capabilities of a spherical microphone array. In another extension, visual gestures of participants will be considered to further facilitate the telematic experience. An important component of this research will be the development of an affordable and deployable version of the collaborative virtual reality system that can be easily used in schools, for example as a ceiling-mounted system hosted in a school's gymnasium.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

从一开始，电信系统的目标就是两个人之间的交流。这适用于电报、电话和典型的视频会议系统。较大群体的会议通常使用相同的个人界面--例如，电话会议。例如，这种做法与传统的小组会议或课堂情景形成了对比。在后者中，学生和教师可以相互交流，同时了解每个参与者的空间位置，并能够轻松识别每个说话人。这个项目的目标是开发一种技术，使两个遥远的教室能够通过电信设备连接在一起，而不需要个人通信接口。将设计一种电子可调麦克风阵列，可以虚拟地聚焦在有源扬声器上。这样，将避免在带有扬声器的协作电信场景中原本会出现的回声。该系统的设计将使扬声器的声音从他们在视频直播中被看到的位置发出。预计项目成果将产生广泛影响，为课堂学生提供新的教育机会，将他们与其他教室联系起来，以获得协作学习体验。该项目将在协作空间之间开发空间正确的远距离音频连接，例如两个教室，每个地点都有一群人。实现这种远程协作所需的缺失环节是一个音频跟踪系统，该系统可以跟踪连接两端的所有参与者。该方法将使用两个16声道球形麦克风作为主要的音频跟踪设备。关键的想法是，每个球形麦克风将从额外的穿戴在身上的麦克风接收连续信息，这被用来确保无回声的双向通信。佩戴在身上的麦克风信号将通知跟踪系统，参与者中的谁在给定的时间间隔和给定的频率范围内发言。然后，在声源活动的时间间隔期间以及在不受其他声源干扰的频段内，使用球形麦克风来跟踪每个声源。使用机器学习工具，包括贝叶斯方法、卡尔曼过滤器和深度神经网络，跟踪系统将实现自下而上/自上而下的联合架构。现有的两个站点配备了身临其境的扬声器系统，每个系统都将能够从远程站点准确地在本地再现空间化的音频。这些网站还配备了具有身临其境视频能力的协作虚拟现实系统，它们将作为该项目的实验室，使系统能够传输具有空间一致的音频/视频图像的双向音频/视频流。作为该项目的延伸，音频跟踪系统将与安装在实验室天花板上的现有6摄像头跟踪系统配对，并将开发技术，利用球形麦克风阵列的波束形成能力，用虚拟麦克风取代身体佩戴的麦克风。在另一个扩展中，参与者的视觉手势将被考虑以进一步促进远程信息处理体验。这项研究的一个重要组成部分将是开发一个负担得起、可部署的协作式虚拟现实系统版本，该系统可以方便地在学校使用，例如作为学校体育馆的天花板安装系统。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Walkable auralizations for experiential learning in an immersive classroom

适合步行的听觉体验，在沉浸式课堂中进行体验式学习

• DOI: 10.1121/10.0012985
• 发表时间: 2022
• 期刊: The Journal of the Acoustical Society of America
• 作者: Chabot, Samuel;Braasch, Jonas
• 通讯作者: Braasch, Jonas

Auralizing concert venues over extended listening areas using wave field synthesis

使用波场合成在扩展的聆听区域中对音乐会场地进行可听化

• 发表时间: 2021
• 期刊: Audio Engineering Society Convention 150
• 作者: Braasch, Jonas;Chabot, Samuel;Chertok, Evan;Mathews, Jonathan;Scott, E.k. Ellington
• 通讯作者: Scott, E.k. Ellington

Interactive Application to Control and Rapid-prototype in a Collaborative Immersive Environment

协作沉浸式环境中控制和快速原型的交互式应用程序

• 发表时间: 2021
• 期刊: Audio Engineering Society Convention 151
• 作者: Chabot, Samuel;Braasch, Jonas
• 通讯作者: Braasch, Jonas

Predicting room acoustical parameters from running signals using a precedence effect model and deep neural networks

使用优先效应模型和深度神经网络根据运行信号预测房间声学参数

• 发表时间: 2022
• 期刊: Proceedings of the 24th International Congress on Acoustics (ICA
• 作者: Tyler, J.;Si, M.;Braasch, J.
• 通讯作者: Braasch, J.

Panoptic Reconstruction of Immersive Virtual Soundscapes Using Human-Scale Panoramic Imagery with Visual Recognition

• DOI: 10.21785/icad2021.043
• 发表时间: 2021-06
• 期刊: Proceedings of the 26th International Conference on Auditory Display (ICAD 2021)
• 作者: Mincong Huang;Samuel Chabot;J. Braasch