HCC: Small: Robust Object Detection for Mobile Augmented Reality in the Wild

移动增强现实（AR）将虚拟对象与真实环境相结合，在零售、教育和医疗保健等许多领域显示出了巨大的潜力。在过去几年中，AR和机器学习领域取得的进展为生成与特定环境相匹配的AR体验提供了机会，通过利用基于机器学习的对象检测算法的输出来识别AR设备视野中的对象及其位置。然而，现有的物体检测方法是脆弱的，经常会由于光照、物体位置、设备功能和用户动作的变化而出错。因此，这个项目的目标是开发更强大的目标检测方法和AR技术，并以现实世界的用例为基础。激励场景是设施管理员希望用户在几个月或几年的时间里从对象检测集成的AR体验中受益的场景：例如，教师在课堂上使用AR增强学习，博物馆馆长使用AR丰富游客体验，或者建筑工地或工厂的管理人员为工人部署基于AR的安全指导。项目团队的目标是帮助管理员以最小的工作量开发各种AR体验，而不会在外观和内容方面对设施的状态施加限制。该项目将实现广泛的情境感知AR应用，例如基于AR的安全指导、无障碍辅助以及健康和福祉支持。这项研究将涉及多个不同的本科生和高中生群体，包括整个学年和密集的综合暑期研究项目经验。作为这项研究的一部分开发的交互式演示将在面向k -12的活动中展示。该项目将通过开发新的AR特定对象检测训练方法、性能监控技术、输入和输出安全检查方法以及应用程序接口，在多个维度上增强AR对象检测器的可靠性。这项工作分为三个重点。第一个重点是设计和开发一个强大的AR目标检测框架，通过使AR目标探测器适应给定位置的条件，并通过验证AR目标探测器输入和输出的正确性，来提高AR目标探测器的可靠性。第二个重点将检查AR对象检测算法在大型和不同用户群体以及一组不同AR设备中的性能；该团队将设计和开发使目标探测器适应特定用户和设备的机制，这些设备具有有限的标记数据。第三个重点将在长期部署中检查AR目标探测器在自然变化环境中的性能。这项工作将包括在12个月的时间内捕获一组环境，设计AR对象检测器的性能监控策略，并开发通过未来验证和再培训来保持AR对象检测器性能的方法。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Mobile augmented reality (AR), which integrates virtual objects with real environments, has shown outstanding potential in many areas including retail, education, and healthcare. The progress made in AR and machine learning over the last several years has given rise to opportunities to generate AR experiences that are well-matched with specific contexts, by leveraging the outputs of machine-learning-based object-detection algorithms that identify objects and their locations in the field of view of the AR device. However, existing object-detection methods are brittle, often making mistakes due to variations in lighting, object positions, device capabilities, and users’ actions. This project's goal, then, is to develop more robust object-detection methods and AR techniques that use them, grounded in real-world use cases. The motivating scenario is settings where a facility administrator would like users to benefit from object-detection-integrated AR experiences over the course of months or years: for example, teachers using AR-enhanced learning in a classroom, museum curators using AR to enrich visitors’ experience, or managers of a construction site or a factory deploying AR-based safety guidance for workers. The project team’s goal is to help administrators develop a variety of AR experiences with minimal workload, without placing restrictions on the state of the facility in terms of both its appearance and contents. This project will enable a wide range of context-aware AR applications, such as AR-based safety guidance, accessibility assistance, and support for health and well-being. The research will engage multiple diverse cohorts of undergraduate and high school students, both throughout the school year and on intense integrated summer research project experiences. Interactive demonstrations developed as part of this research will be showcased at K-12-oriented events. This project will enhance the reliability of AR object detectors on multiple dimensions, via the development of new AR-specific object-detection training approaches, performance monitoring techniques, input and output sanity-checking methods, and application interfaces. The work is divided into three thrusts. The first thrust will design and develop a robust AR object detection framework that will enhance the reliability of AR object detectors by adapting them to the conditions in a given location and by validating the correctness of AR object detectors’ inputs and outputs. The second thrust will examine the performance of AR object-detection algorithms across large and diverse groups of users and across a set of diverse AR devices; the team will design and develop mechanisms for adapting object detectors to specific users and devices with limited labeled data. The third thrust will examine the performance of AR object detectors in naturally changing environments across long-term deployments. This work will involve capturing a set of environments over a 12-month period, designing strategies for performance monitoring of AR object detectors, and developing approaches to maintain AR object detectors’ performance over time by future-proofing and retraining them.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.