CRII: SHF: Graphical User Interface Test Code Adaptation for Developing Voice Assistant Features in Mobile Applications

近年来，语音助手(VA)的受欢迎程度急剧上升，用户可以通过简单的语音命令更快、更高效地执行任务。谷歌助手和亚马逊Alexa等领先产品的成功促使移动应用程序开发商将语音助手功能整合到他们的应用程序中。然而，添加这些功能对开发人员来说可能非常耗时，因为现有的代码设计主要侧重于图形交互，而不是语音交互。本项目旨在通过利用现有的测试代码来设计语音助手功能，从而为这一问题提供解决方案。测试代码是一系列模拟的用户与应用程序的交互，用于验证应用程序的功能。通过重用测试代码并将其映射到相应的语音查询，开发人员可以快速将语音助手功能整合到他们的应用程序中。作为这个项目的结果，开发人员将更有可能重新设计他们现有的代码，以适应这种新的语音交互。此外，该项目将为本科生和研究生带来许多培训和教育机会。它有可能支持数以千计的移动开发人员在他们的应用程序中添加语音助手选项。在这个项目中，研究人员旨在回答是否以及如何修改现有的图形用户界面(GUI)测试代码来开发移动应用程序中的语音助手功能的问题。调查员将首先从图形用户界面测试代码中的特定功能实例中提取一般功能。研究人员将研究如何区分图形用户界面测试代码中的通用代码和非通用代码，并开发自动化技术来检测它们。调查人员还将研究如何将通用功能转换为可重复使用的代码模板，以用于语音助手功能开发。其次，研究人员将研究如何使用人类可理解的语言来表示代码模板，并基于该语言创建潜在的语音查询模板。将创建一个映射来连接语音查询模板和代码模板。最后，调查员将开发技术，以促进移动应用程序响应特定语音查询执行代码模板。该项目旨在支持开发人员通过重用应用程序的测试构件来向现有移动应用程序添加语音助手功能。这项研究的结果将启发对语音助手或移动应用程序中其他新功能的其他形式的代码重用和迁移的进一步调查。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The popularity of Voice Assistant (VA) has skyrocketed in recent years, with users able to perform tasks faster and more efficiently through simple voice commands. The success of leading products like Google Assistant and Amazon Alexa has motivated mobile app developers to integrate voice assistant features into their applications. However, adding these features can be significantly time-consuming for developers, as the existing code design primarily focuses on graphical interactions rather than voice interactions. This project aims to provide a solution to this issue by utilizing existing test code to design the voice assistant features. Test code is a series of simulated user interactions with an application that is used to validate the functionality of the application. By reusing the test code and mapping it to the corresponding voice query, the developers can quickly incorporate voice assistant capabilities into their applications. As a result of this project, developers will be more likely to redesign their existing code to accommodate such new voice interactions. Furthermore, this project will bring many training and educational opportunities for both undergraduate and graduate students. It has the potential to support thousands of mobile developers in adding voice assistant options to their applications. In this project, the investigator aims to answer the question of whether and how existing graphical user interface (GUI) test code can be adapted to develop voice assistant features in mobile applications. The investigator will begin by extracting the general functionality from a specific functionality instance in a GUI test code. The investigator will study how to differentiate the general code and non-general code in GUI test code, and develop automated techniques to detect them. The investigator will also investigate how to convert the general functionality into a reusable code template for voice assistant feature development. Second, the investigator will study how to represent the code template using human-understandable language and create potential voice query templates based on this language. A map will be created to connect the voice query template with the code template. Finally, the investigator will develop techniques to facilitate mobile applications in executing the code template in response to a particular voice query. This project aims to support developers in adding voice assistant functionality to existing mobile apps by reusing the test artifacts of the application. The results of this research will inspire further investigation into other forms of code reuse and migration for voice assistants or other new features in mobile applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.