ITR: Augmented Reality and Computer Vision for Enhancing Human Assembly Skills

这是三年持续奖励的第一年资助。该项目旨在解决使用计算机视觉实现增强现实界面的基本问题。增强现实的目标是增强一个人对周围世界的感知，为计算机集成到用户的活动中提供潜力，作为个性化的助手。启用这样的接口有两个关键挑战。第一个是“感知”，当用户与它交互时，它将允许增强与世界的状态相匹配。第二个挑战是开发系统的“增强方案”，从而产生以用户为中心的信息流。为了帮助问题的概念化和实验验证，项目的主要焦点是“组装域”。上下文是指一个人将其部件组装成一个机械物体。对“装配领域”的关注，使我们能够适当地制定和解决新型人机界面中的传感、增强和其他问题。同时，它允许我们使用增强现实来检查特定的交互式装配任务，例如装配期间的指导和培训以及原型装配序列的评估。为了解决跟踪“上下文”的关键问题，该项目旨在推进识别装配状态的计算机视觉技术的最新进展。基于外观和基于cad的方法的结合将用于解决同时跟踪大量已知装配部件的问题。提出了一种随装配任务进行而随时间变化的装配状态恢复的概率方法。另一个重点是寻找有效的方法来为具有大量零件的子组件构建模型空间。装配域的几何建模和分析将用于开发系统的增强流程，以辅助各种装配任务。将使用两个实验装置，包括一个透明的头戴式显示器和一个在实时视频上覆盖图形的计算机显示器。将设计特定的装配任务场景，以确定装配领域增强现实接口的实际可行性。另一个目标是通过实验评估增强现实界面作为训练工具的有效性，与其他指导手段相比。由此产生的增强现实界面预计将影响许多应用程序，这些应用程序受益于在线、场景相关的多模式信息呈现，例如装配原型、飞机维护、空间飞行器维修、工厂工人的成本效益培训，以及可能指导没有经验的用户完成复杂的机械维修。

This is the first year funding of a three-year continuing award. This project aims at addressing basic issues for enabling an augmented reality interface using computer vision. Augmented Reality has the goal of enhancing a person's perception of the surrounding world, offering the potential for the computer to be integrated into the activities of a user, serving as a personalized helper. There are two key challenges to enable such an interface. The first is "sensing", which would allow the augmentation to be matched to the state of the world as the user interacts with it. The second challenge is that of developing systematic "augmentation schemes" that result in user-centered information flow. To aid in the conceptualization of the problem and for experimental verification, the main focus of the project is on an "assembly domain". The context is that of a human engaged in assembling a mechanical object from its components. The focus on the "assembly domain", allows us to suitably formulate and address the sensing, augmentation, and other issues in the novel human-computer interface. At the same time, it allows us to examine specific interactive assembly tasks using augmented reality, such as guiding and training during assembly and for evaluation of prototype assembly sequences. To address the key problem of tracking the "context", the project seeks to advance the state of art of computer vision techniques for recognizing assembly states. A combination of appearance-based and CAD-based approach will be used for addressing the problem of simultaneously tracking a large number of known assembly parts. A probabilistic approach is proposed to improve the performance of assembly state recovery over time as the assembly task progresses. Another focus is to look for efficient approaches for building the model spaces for subassemblies with larger number of parts. Geometric modeling and analysis of the assembly domain will be utilized in the development of systematic flow of augmentation to aid various assembly tasks. Two experimental setups will be used, involving a see-through head-mounted display and a computer monitor with graphics overlaid on live video. Specific assembly task scenarios will be devised to determine the practical feasibility of augmented reality interface for the assembly domain. Another goal will be to experimentally evaluate the effectiveness of augmented reality interface as a training tool compared to other means of instructions. The resulting augmented reality interface is expected to impact many applications that benefit from the on-line, scene-dependent presentation of multimodal information, such as assembly prototyping, aircraft maintenance, repair of space vehicles, cost-effective training of factory workers, and perhaps guiding inexperienced users through complex repair of machinery.