Creating and Evaluating New Media Interfaces for Expression

我建议推动两种形式的互动视觉媒体的界限，以增加人类学习和合作的能力。首先，将研究与新型3D显示器交互的新形式。第二，将创建先进的互动、协作视频(IV)，以及三个关键的新负担，将视频提升为一流的学习媒体。 首先，使用我们的3D鱼缸虚拟现实(FTVR)球面显示器(3DD)的研究重点放在两种主要类型的交互上：1)需要实时3D捕获的协同定位和远程多人交互，以及2)人与3D代理的交互。为了支持共处一地的四人互动，将创建一个曲面偏振透镜，以匹配耦合到活动眼镜的球形显示表面的曲率，使四个不同的单视透视校正后的视图能够投影到显示器上，但不同的用户可以看到。这些显示器提供了固定在真实世界坐标上的虚拟内容，允许每个人共享一个共同的参照系。因此，指示语的互动，比如指着和说“那个东西在那里”。在每个人都能看到所指内容的地方得到支持。对于远程参与者，将创建一个使用深度摄像头支持眼神接触的实时3D捕获系统。我们将研究3D任务的感知效果和任务表现，并将其与使用平板屏幕的类似配置进行比较。其次，将研究球形显示器内的观众和房间与智能虚拟代理(IVA)的交互。与真实参与者一样，将研究语音和手势交互。同样，将在远程和协同定位的场景中探索协作指向、选择、翻译和旋转内容的交互语义。这项研究将创造先进的3D球面显示器，供一群人和代理人跨越空间和时间进行工作和娱乐。 其次，从过去的工作中，我们发现对教育至关重要的三个视频提供缺失：可变性、双向性和互操作性。对于易变性，将创建用于在管理每个状态变化和注释的同时进行协作视频制作的界面。在互操作性方面，将创建支持视频结构和语义捕获和复制的接口，以有效融入学生的信息生态，包括笔记。对于双向，将研究丰富的参考接口，以支持与视频元素进行复杂的通信方法。这些互动将在教育场景中进行研究，例如：笔记、演示和社交学习。归根结底，这项研究的目标是为视频提供纸和笔，这样人们就可以像通过文本一样轻松地阅读/观看和编写/合成视频。 这项研究直接涉及33名在世界级国际跨学科团队中工作的高素质人员，他们将成为高级媒体互动领域的领导者。

I propose to push the boundaries of two forms of interactive visual media to increase the capacity for humans to learn and work together. First, new forms of interaction with a novel 3D display will be investigated. Second, advanced interactive, collaborative video (IV) with three critical new affordances to elevate video to be a first-class' learning medium will be created. First, research using our 3D fish-tank virtual reality (FTVR) spherical display (3DD) focuses on two main types of interactions: 1) co-located and remote multi-person interaction requiring real-time 3D capture, and 2) human-3D agent interaction. To support co-located four-person interaction, a curved polarizing lens will be created to match the curvature of the spherical display surface coupled to active glasses to enable four different monoscopic perspective corrected views to be projected on the display but seen by different users. These displays provide virtual contents anchored to real-world coordinates allowing everyone to share a common reference frame. Thus, deictic interactions, like pointing and saying “That thing there.” are supported where everyone can see what is being pointed at. For remote participants, a real-time 3D capture system using depth cameras supporting eye-contact will be created. Perceptual effects as well as task performance on 3D tasks will be studied and compared to similar configurations using flat screens. Second, viewer and room interaction with an intelligent virtual agent (IVA) inside the spherical display will be studied. As with real-participants, voice and gesture interaction will investigated. Likewise, interaction semantics for collaborative pointing, selecting, translating, and rotating content will be explored in remote and co-located scenarios. This research will create advanced 3D spherical displays for groups of people and agents to work and play across space and time. Second, from past work, we discovered three video affordances critical for education are missing: mutability, bidirectionality and interoperability. For mutability, interfaces for collaborative video making while each state change and annotation is managed will be created. For interoperability, interfaces supporting video structure and semantic capture and copy for effective integration with students' information ecology, including notes will be created. For bidirectionality, rich referencing interfaces enabling complex methods of communicating with video elements will be investigated. These interactions will be studied in educational scenarios such as: note taking, presenting, and social learning. Ultimately, the goal of this research provides the pen and paper' for video so people can learn to read/watch and write/compose' video as easily as they can through text. This research directly involves 33 highly qualified personnel working in world-class international, interdisciplinary teams to become leaders in advanced media interaction.