CAREER: Enhancing Energy Efficiency in Mobile Augmented Reality Apps

移动增强现实(MAR)应用程序利用移动设备的摄像头来使用一组虚拟对象来增强真实的用户环境。因此，用户可以通过移动用户界面与增强环境交互来实现高水平的参与度。典型的MAR应用程序需要执行几个计算密集型任务来渲染虚拟对象，并使用人工智能(AI)分析环境。然而，由于移动设备的外形相对较小，可用于执行MAR应用程序的计算资源和能量有限，这可能会导致电池续航时间较短，用户体验较差。因此，为了确保MAR未来的成功，不仅要对如何在MAR应用程序中实现高能效进行新的研究，而且要教育未来的工作人员进行高能效移动软件设计的最佳实践。为了实现这些目标，该项目将为MAR应用程序提出一个边缘辅助和节能的框架。具体地说，该项目设计了运行时算法：(1)在每个移动设备上，通过操纵虚拟对象的质量、AI模型的复杂性以及异质计算资源上的AI任务分配来控制用户感知的性能和能量消耗；(2)在支持边缘服务器上，通过分组相似的虚拟对象抽取请求来确保可伸缩性，以在有限的边缘计算和存储资源下最小化跨用户性能成本；(3)将在物理测试床上实现和测试。通过提供一个使MAR应用程序更节能的框架，该项目将有助于加快增强现实在现代社会的许多不同关键领域的应用。例如教育和劳动力培训，并将充分释放新技术的潜力，如扩大远程会议和扩大公共场所的体验。该项目还将创建新的研究生课程，以教育未来的劳动力节能移动资源管理，为本科生提供新的研究机会，并为K12学生提供新的夏令营。课程材料将提供给其他有兴趣引入类似课程、项目和夏令营的大学。该项目的成果将通过在顶级会议和期刊上发表的出版物进行传播。同样，建议算法的所有源代码都将作为开源提供。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Mobile Augmented Reality (MAR) apps leverage the camera of mobile devices to augment the real user environment with a set of virtual objects. Users can thus achieve a high level of engagement by interacting with the augmented environment through the mobile user interface. Typical MAR apps require executing several compute-intensive tasks to render virtual objects and, using Artificial Intelligence (AI), analyze the environment. However, due to their relatively small form factor, mobile devices only have limited computing resources and energy available for the MAR app execution, which may lead to short battery life and poor user experience. To ensure the future success of MAR, it is thus important not only to conduct novel research on how to enable high energy efficiency in MAR apps but also to educate the future workforce in the best practices for energy efficient mobile software design. To achieve these goals, this project will propose an edge-assisted and energy-efficient framework for MAR apps. Specifically, the project designs runtime algorithms that (1), on each mobile device, control user-perceived performance and energy consumption by manipulating virtual object quality, AI-model complexity, and AI-task allocation on the heterogeneous computing resources, (2), on the supporting edge servers, ensure scalability by grouping similar virtual object decimation requests for minimized cross-user performance cost under limited edge computing and storage resources, and (3) will be implemented and tested on a physical testbed.By providing a framework that makes MAR apps more energy-efficient, this project will help speed up the adoption of augmented reality in many different key sectors of modern society, such as education and workforce training, and will unlock the full potential of new technologies such as augmented remote meetings and augmented experiences in public places. The project will also result in the creation of new graduate-level courses to educate the future workforce on energy-efficient mobile resource management, new research opportunities for undergraduate students, and new summer camps for K12 students. The course material will be made available to other universities interested to introduce similar courses, projects, and summer camps. The results of this project will be disseminated through publications in top-tier conferences and journals. Similarly, all the source code of the proposed algorithms will be made available as open source.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.