An Immersive Programming Support System with Perceptual Interfaces that utilize Humans' Body

我们研究并开发了一个沉浸式编程学习支持系统，该系统有效地利用了3D可视化的立体视图，直接手动操作和感知反馈。我们创建了一个3D拼图般的图形表示，用于在沉浸式3D虚拟环境中可视化编程的语法规则或约束。它通过匹配凸形和凹形来表示约束。我们还进行了实验，使用类型的表示：文本表示通常用于编程，数字和符号的序列表示，2D字形和3D字形。实验结果表明，所提出的拼图式表示是有效的，三维拼图表示是容易为新手用户学习。此外，为了提高沉浸式虚拟环境中操作虚拟对象的可用性，研究了使用手和设备的交互技术。我们提出并研究了使用虚拟变速箱部件与直接手动操作相结合的操作，以及使用手动直接操作，无需部件即可自动调整。实验结果表明，本文提出的技术在沉浸式虚拟环境中具有上级的交互效果。此外，本文还研究了用于非侵入式和环境信息显示的芳香反馈技术，并开发了一个沉浸式环境中立体视频的多点传输系统，远程虚拟环境中的协作工作。这项研究使我们能够创建我们的有效的程序设计学习支持系统的新手用户在沉浸式虚拟环境。

We have investigated and developed an immersive programming learning support system that effectively uses a stereoscopic view of 3D visualization, direct manipulation by hand, and perceptual feedback.We have created a 3D jigsaw-puzzle-like glyph representation for visualizing grammatical rules or constraints of programming in an immersive 3D virtual environment. It represents constraints by matching convex and concave shapes. We also conducted experiments using types of representations: textual representation typically used in programming, a sequence representation of numbers and symbols, 2D glyphs, and 3D glyphs. The experiments showed that the proposed jigsaw-puzzle-like representation is effective and that 3D glyph representation is easy for novice users to learn. This result was also supported by subjective evaluation using questionnaires.Moreover, we studied interaction techniques using hands and devices to improve the usability of manipulating virtual objects in immersive virtual environments. We proposed and investigated manipulation using virtual gearbox widgets combined with direct manipulation by hand, and using direct manipulation by hand with automatic adjustments without widgets. Experimental evaluation showed that the proposed techniques are superior to conventional interaction techniques in immersive virtual environments.Furthermore, we studied aromatic feedback for non-intrusive and ambient information display, and developed a multipoint transmission system using IPv6 multicasting for stereoscopic video in immersive environments, which facilitates efficient, cooperative work among remote virtual environments.This study enabled us to create our effective programming learning support system for novice users in immersive virtual environments.