II-EN The Human Virtualization Studio: From Distributed Sensor to Interactive Audiovisual Environment

II-EN 人类虚拟化工作室：从分布式传感器到交互式视听环境

• 批准号: 0855163
• 负责人: Takeo Kanade
• 金额: $ 60万
• 依托单位: Carnegie-Mellon University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2009
• 资助国家: 美国
• 起止时间: 2009-07-15 至 2012-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0855163&HistoricalAwards=false
• 关键词: II; EN; Human; Virtualization; Studio

The Virtualization Studio spearheads research to reconstruct, record, and render dynamic events in 3D. The studio creates a ?full-body? interactive environment where multiple users are simultaneously given a visceral sense of three dimensional space, through vision and sound, and are able to interact, through action and speech, unencumbered by 3D glasses, head-mounted displays or special clothing. The studio pursues the thesis that robust sensors for hard problems, in this case audiovisual reconstruction of highly dynamic multiple actors/speakers, can be constructed by using a large number of sensors running simple, parallelized algorithms. High fidelity reconstructions are created using a grid of 1132 cameras, and a 128-node multi-speaker microphone array to localize and associate multiple sound sources in the event space. In addition, a multi-viewer lenticular display screen, consisting of 48 projectors, and a front surround sound speaker are used to render interactive environments. The reconstruction algorithms are parallelized and a cluster is used to process the data and respond to behaviors in the event space in realtime.Audiovisual reconstruction and rendering of scenes containing multiple users will revolutionize research into collaborative interfaces, and will allow digital preservation of culturally significant events, like theatrical performances, sports events, and key speeches. In addition to these core research objectives, the Virtualization Studio will act as a gathering place for multidisciplinary research, bringing together researchers from interactive art, human behavior analysis, computer graphics, computer vision, psychology, big data research, and speech processing. The infrastructure will be used to develop a new course on Human Virtualization, and will be used as a pedagogical tool in several existing courses and outreach projects, introducing the next generation of students to the power of interdisciplinary research in computer science.

Virtualization Studio是研究在3D中重建、记录和渲染动态事件的先锋。工作室创造了一个？全身？交互式环境，其中多个用户通过视觉和声音同时被给予三维空间的本能感觉，并且能够通过动作和语音进行交互，不受3D眼镜、头戴式显示器或特殊服装的阻碍。该工作室追求的论文，强大的传感器的硬问题，在这种情况下，高动态多个演员/扬声器的视听重建，可以通过使用大量的传感器运行简单的，并行化的算法。高保真重建使用1132个摄像机的网格和128节点多扬声器麦克风阵列来创建，以定位和关联事件空间中的多个声源。此外，一个由48台投影仪组成的多观众透镜状显示屏和一个前置环绕声扬声器用于呈现交互式环境。重建算法是并行的，一个集群被用来处理数据，并在事件空间中的实时行为作出反应。视听重建和渲染的场景包含多个用户将彻底改变研究到协作接口，并将允许数字保存的文化意义重大的事件，如戏剧表演，体育赛事，和关键演讲。除了这些核心研究目标外，虚拟化工作室还将作为多学科研究的聚集地，汇集来自交互艺术，人类行为分析，计算机图形学，计算机视觉，心理学，大数据研究和语音处理的研究人员。该基础设施将用于开发一门关于人类虚拟化的新课程，并将在几个现有课程和推广项目中用作教学工具，向下一代学生介绍计算机科学跨学科研究的力量。