CHS: Small: Rich Surface Interaction for Augmented Environments

虚拟现实(VR)和增强现实(AR)头盔显示器正越来越多地被用于不同的计算相关活动，如数据可视化、教育和培训。目前，VR和AR设备缺乏有效和符合人体工程学的方式来执行常见的桌面交互，如点击和输入文本。这个项目的目标是将平坦的日常表面转化为丰富的交互表面。例如，一张桌子或一面墙可以被改造成虚拟键盘。平坦的表面不仅提供触觉反馈，还提供了符合人体工程学的优势，为您提供了一个休息手臂的地方。该项目将开发一种系统，将麦克风放置在表面上，以便能够感知何时何地发生了窃听。此外，该系统旨在区分不同类型的触摸交互，如用指甲轻敲、用指垫轻敲或做出简短的滑动手势。这个项目将研究不同的机器学习算法，以生成表面上丝锥的连续坐标以及相关的误差条。利用感知到的敲击的置信度，该项目将研究如何智能地告知用户界面的各个方面，例如指导虚拟键盘解码器的自动更正算法。最初，该项目将研究通过表面安装的麦克风阵列进行传感，并设计“表面算法”来确定和比较虚拟键盘上手指轻击的位置精度。这些算法将在不同的模型上进行实验，包括现有的飞行时间模型，基于高斯过程回归的新模型，以及使用支持向量机的分类基线。对于所有模型，该项目将调查来自其他用户的训练数据量的影响，并改变来自目标用户的适应数据量。该项目将比较表面麦克风与利用摄像头和手腕惯性传感器的方法。该项目将在空中与刚性表面交互的准确性、用户偏好和人体工程学方面产生人为因素的结果。通过检查不同的传感器、输入表面和接口设计，该项目将为未来的AR和VR交互系统绘制设计空间。该项目将传播软件和数据，允许其他人在桌子或墙壁上安装麦克风，以实现丰富的互动体验。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Virtual Reality (VR) and Augmented Reality (AR) head-mounted displays are increasingly being used in different computing related activities such as data visualization, education, and training. Currently, VR and AR devices lack efficient and ergonomic ways to perform common desktop interactions such as pointing-and-clicking and entering text. The goal of this project is to transform flat, everyday surfaces into a rich interactive surface. For example, a desk or a wall could be transformed into a virtual keyboard. Flat surfaces afford not only haptic feedback, but also provide ergonomic advantages by providing a place to rest your arms. This project will develop a system where microphones are placed on surfaces to enable the sensing of when and where a tap has occurred. Further, the system aims to differentiate different types of touch interactions such as tapping with a fingernail, tapping with a finger pad, or making short swipe gestures. This project will investigate different machine learning algorithms for producing a continuous coordinate for taps on a surface along with associated error bars. Using the confidence of sensed taps, the project will investigate ways to intelligently inform aspects of the user interface, e.g. guiding the autocorrection algorithm of a virtual keyboard decoder. Initially, the project will investigate sensing via an array of surface-mounted microphones and design "surface algorithms" to determine and compare the location accuracy of the finger taps on the virtual keyboard. These algorithms will experiment with different models including existing time-of-flight model, a new model based on Gaussian Process Regression, and a baseline of classification using support vector machines. For all models, the project will investigate the impact of the amount of training data from other users, and varying the amount of adaptation data from the target user. The project will compare surface microphones with approaches utilizing cameras and wrist-based inertial sensors. The project will generate human-factors results on the accuracy, user preference, and ergonomics of interacting midair versus on a rigid surface. By examining different sensors, input surfaces, and interface designs, the project will map the design space for future AR and VR interactive systems. The project will disseminate software and data allowing others to outfit tables or walls with microphones to enable rich interactive experiences.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.