II-EN The Human Virtualization Studio: From Distributed Sensor to Interactive Audiovisual Environment

虚拟化工作室率先研究在3D中重建、记录和渲染动态事件。工作室创造了一个全身？交互式环境，多个用户同时通过视觉和声音获得三维空间的内在感觉，并能够通过动作和语言进行交互，而不受3D眼镜，头戴式显示器或特殊服装的阻碍。该工作室致力于通过使用大量运行简单并行算法的传感器来构建用于解决难题的鲁棒传感器，在这种情况下，高度动态的多个演员/扬声器的视听重建。使用1132台摄像机组成的网格和128个节点的多扬声器麦克风阵列创建高保真重建，以定位和关联事件空间中的多个声源。此外，一个由48个投影仪组成的多观众透镜式显示屏和一个前置环绕立体声扬声器用于呈现交互式环境。重构算法并行化，采用聚类对数据进行处理，实时响应事件空间中的行为。包含多个用户的场景的视听重建和渲染将彻底改变对协作界面的研究，并将允许对文化上重要的事件进行数字化保存，如戏剧表演、体育赛事和重要演讲。除了这些核心研究目标之外，虚拟化工作室还将作为多学科研究的聚集地，汇集来自互动艺术、人类行为分析、计算机图形学、计算机视觉、心理学、大数据研究和语音处理等领域的研究人员。该基础设施将用于开发一门关于人类虚拟化的新课程，并将在几个现有课程和扩展项目中用作教学工具，向下一代学生介绍计算机科学跨学科研究的力量。

The Virtualization Studio spearheads research to reconstruct, record, and render dynamic events in 3D. The studio creates a ?full-body? interactive environment where multiple users are simultaneously given a visceral sense of three dimensional space, through vision and sound, and are able to interact, through action and speech, unencumbered by 3D glasses, head-mounted displays or special clothing. The studio pursues the thesis that robust sensors for hard problems, in this case audiovisual reconstruction of highly dynamic multiple actors/speakers, can be constructed by using a large number of sensors running simple, parallelized algorithms. High fidelity reconstructions are created using a grid of 1132 cameras, and a 128-node multi-speaker microphone array to localize and associate multiple sound sources in the event space. In addition, a multi-viewer lenticular display screen, consisting of 48 projectors, and a front surround sound speaker are used to render interactive environments. The reconstruction algorithms are parallelized and a cluster is used to process the data and respond to behaviors in the event space in realtime.Audiovisual reconstruction and rendering of scenes containing multiple users will revolutionize research into collaborative interfaces, and will allow digital preservation of culturally significant events, like theatrical performances, sports events, and key speeches. In addition to these core research objectives, the Virtualization Studio will act as a gathering place for multidisciplinary research, bringing together researchers from interactive art, human behavior analysis, computer graphics, computer vision, psychology, big data research, and speech processing. The infrastructure will be used to develop a new course on Human Virtualization, and will be used as a pedagogical tool in several existing courses and outreach projects, introducing the next generation of students to the power of interdisciplinary research in computer science.