CAREER: Enhancing Energy Efficiency in Mobile Augmented Reality Apps

移动增强现实（MAR）应用程序利用移动设备的摄像头通过一组虚拟对象来增强真实用户环境。因此，用户可以通过移动用户界面与增强的环境进行交互，从而实现高水平的参与。典型的MAR应用程序需要执行几个计算密集型任务来渲染虚拟对象，并使用人工智能（AI）来分析环境。然而，由于其相对较小的外形因素，移动设备只有有限的计算资源和能源可用于MAR应用程序的执行，这可能导致电池寿命短和糟糕的用户体验。因此，为了确保MAR未来的成功，不仅要对如何在MAR应用程序中实现高能效进行新颖的研究，还要向未来的劳动力传授节能移动软件设计的最佳实践，这一点非常重要。为了实现这些目标，该项目将为MAR应用程序提出一个边缘辅助和节能的框架。具体而言，该项目设计的运行时算法：(1)在每个移动设备上，通过操纵虚拟对象质量、人工智能模型复杂性和异构计算资源上的人工智能任务分配来控制用户感知的性能和能耗；(2)在支持的边缘服务器上，通过分组相似的虚拟对象抽取请求来确保可扩展性，从而在有限的边缘计算和存储资源下最小化跨用户性能成本。(3)将在物理测试台上实现和测试。通过提供一个使MAR应用程序更加节能的框架，该项目将有助于加快增强现实在现代社会许多不同关键领域（如教育和劳动力培训）的采用，并将释放增强远程会议和公共场所增强体验等新技术的全部潜力。该项目还将创建新的研究生课程，向未来的劳动力传授节能移动资源管理，为本科生提供新的研究机会，并为K12学生开设新的夏令营。课程材料将提供给其他感兴趣的大学，以引入类似的课程、项目和夏令营。该项目的成果将通过顶级会议和期刊的出版物进行传播。同样，所有提议算法的源代码都将作为开源提供。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Mobile Augmented Reality (MAR) apps leverage the camera of mobile devices to augment the real user environment with a set of virtual objects. Users can thus achieve a high level of engagement by interacting with the augmented environment through the mobile user interface. Typical MAR apps require executing several compute-intensive tasks to render virtual objects and, using Artificial Intelligence (AI), analyze the environment. However, due to their relatively small form factor, mobile devices only have limited computing resources and energy available for the MAR app execution, which may lead to short battery life and poor user experience. To ensure the future success of MAR, it is thus important not only to conduct novel research on how to enable high energy efficiency in MAR apps but also to educate the future workforce in the best practices for energy efficient mobile software design. To achieve these goals, this project will propose an edge-assisted and energy-efficient framework for MAR apps. Specifically, the project designs runtime algorithms that (1), on each mobile device, control user-perceived performance and energy consumption by manipulating virtual object quality, AI-model complexity, and AI-task allocation on the heterogeneous computing resources, (2), on the supporting edge servers, ensure scalability by grouping similar virtual object decimation requests for minimized cross-user performance cost under limited edge computing and storage resources, and (3) will be implemented and tested on a physical testbed.By providing a framework that makes MAR apps more energy-efficient, this project will help speed up the adoption of augmented reality in many different key sectors of modern society, such as education and workforce training, and will unlock the full potential of new technologies such as augmented remote meetings and augmented experiences in public places. The project will also result in the creation of new graduate-level courses to educate the future workforce on energy-efficient mobile resource management, new research opportunities for undergraduate students, and new summer camps for K12 students. The course material will be made available to other universities interested to introduce similar courses, projects, and summer camps. The results of this project will be disseminated through publications in top-tier conferences and journals. Similarly, all the source code of the proposed algorithms will be made available as open source.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.