HCC: Small: Robust Object Detection for Mobile Augmented Reality in the Wild

将虚拟对象与真实的环境相结合的移动的增强现实（AR）在包括零售、教育和医疗保健在内的许多领域都显示出了出色的潜力。过去几年在AR和机器学习方面取得的进展，通过利用基于机器学习的对象检测算法的输出，产生了与特定上下文很好匹配的AR体验的机会，该算法识别AR设备视野中的对象及其位置。然而，现有的对象检测方法是脆弱的，经常由于照明、对象位置、设备能力和用户动作的变化而出错。因此，该项目的目标是开发更强大的对象检测方法和使用它们的AR技术，以现实世界的用例为基础。激励场景是设施管理员希望用户在数月或数年的时间内受益于对象检测集成的AR体验的设置：例如，教师在教室中使用AR增强学习，博物馆馆长使用AR丰富游客的体验，或者建筑工地或工厂的经理为工人部署基于AR的安全指导。项目团队的目标是帮助管理员以最小的工作量开发各种AR体验，而不会在外观和内容方面限制设施的状态。该项目将实现广泛的情境感知AR应用，例如基于AR的安全指导，无障碍辅助以及对健康和福祉的支持。该研究将涉及多个不同的本科生和高中生群体，无论是在整个学年，并在激烈的综合夏季研究项目的经验。作为这项研究的一部分开发的互动演示将在面向K-12的活动中展示。该项目将通过开发新的AR特定对象检测训练方法、性能监控技术、输入和输出健全性检查方法以及应用程序接口，提高AR对象检测器在多个维度上的可靠性。这项工作分为三个重点。第一个推力将设计和开发一个强大的AR对象检测框架，通过使它们适应给定位置的条件并验证AR对象检测器的输入和输出的正确性，来提高AR对象检测器的可靠性。第二个重点将研究AR对象检测算法在大量不同用户群体和一组不同AR设备中的性能;该团队将设计和开发使对象检测器适应特定用户和具有有限标记数据的设备的机制。第三个重点将研究AR对象检测器在长期部署的自然变化环境中的性能。这项工作将涉及在12个月的时间内捕获一组环境，设计增强现实物体探测器性能监控策略，并开发方法，通过未来验证和再培训来保持增强现实物体探测器的性能。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Mobile augmented reality (AR), which integrates virtual objects with real environments, has shown outstanding potential in many areas including retail, education, and healthcare. The progress made in AR and machine learning over the last several years has given rise to opportunities to generate AR experiences that are well-matched with specific contexts, by leveraging the outputs of machine-learning-based object-detection algorithms that identify objects and their locations in the field of view of the AR device. However, existing object-detection methods are brittle, often making mistakes due to variations in lighting, object positions, device capabilities, and users’ actions. This project's goal, then, is to develop more robust object-detection methods and AR techniques that use them, grounded in real-world use cases. The motivating scenario is settings where a facility administrator would like users to benefit from object-detection-integrated AR experiences over the course of months or years: for example, teachers using AR-enhanced learning in a classroom, museum curators using AR to enrich visitors’ experience, or managers of a construction site or a factory deploying AR-based safety guidance for workers. The project team’s goal is to help administrators develop a variety of AR experiences with minimal workload, without placing restrictions on the state of the facility in terms of both its appearance and contents. This project will enable a wide range of context-aware AR applications, such as AR-based safety guidance, accessibility assistance, and support for health and well-being. The research will engage multiple diverse cohorts of undergraduate and high school students, both throughout the school year and on intense integrated summer research project experiences. Interactive demonstrations developed as part of this research will be showcased at K-12-oriented events. This project will enhance the reliability of AR object detectors on multiple dimensions, via the development of new AR-specific object-detection training approaches, performance monitoring techniques, input and output sanity-checking methods, and application interfaces. The work is divided into three thrusts. The first thrust will design and develop a robust AR object detection framework that will enhance the reliability of AR object detectors by adapting them to the conditions in a given location and by validating the correctness of AR object detectors’ inputs and outputs. The second thrust will examine the performance of AR object-detection algorithms across large and diverse groups of users and across a set of diverse AR devices; the team will design and develop mechanisms for adapting object detectors to specific users and devices with limited labeled data. The third thrust will examine the performance of AR object detectors in naturally changing environments across long-term deployments. This work will involve capturing a set of environments over a 12-month period, designing strategies for performance monitoring of AR object detectors, and developing approaches to maintain AR object detectors’ performance over time by future-proofing and retraining them.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.