CRII: SHF: Graphical User Interface Test Code Adaptation for Developing Voice Assistant Features in Mobile Applications

近年来，语音助理（VA）的普及程度飙升，用户能够通过简单的语音命令更快，更有效地执行任务。Google Assistant和Amazon Alexa等领先产品的成功促使移动的应用程序开发人员将语音助手功能集成到他们的应用程序中。然而，添加这些功能对于开发人员来说可能非常耗时，因为现有的代码设计主要关注图形交互而不是语音交互。该项目旨在通过利用现有的测试代码来设计语音助手功能，从而为这个问题提供解决方案。测试代码是一系列模拟的用户与应用程序的交互，用于验证应用程序的功能。通过重用测试代码并将其映射到相应的语音查询，开发人员可以快速将语音助理功能集成到他们的应用程序中。由于这个项目，开发人员将更有可能重新设计他们现有的代码，以适应这种新的语音交互。 此外，该项目将为本科生和研究生带来许多培训和教育机会。它有可能支持成千上万的移动的开发人员在他们的应用程序中添加语音助理选项。在这个项目中，研究者的目的是回答现有的图形用户界面（GUI）测试代码是否以及如何适应移动的应用程序中开发语音助理功能的问题。研究者将开始从GUI测试代码中的特定功能实例中提取一般功能。研究者将研究如何区分GUI测试代码中的通用代码和非通用代码，并开发自动化技术来检测它们。研究人员还将研究如何将一般功能转换为可重复使用的代码模板，用于语音助手功能开发。其次，研究人员将研究如何使用人类可理解的语言表示代码模板，并基于该语言创建潜在的语音查询模板。将创建一个映射以连接语音查询模板和代码模板。最后，调查员将开发技术，以促进移动的应用程序执行代码模板，以响应特定的语音查询。该项目旨在通过重用应用程序的测试工件来支持开发人员向现有的移动的应用程序添加语音助理功能。这项研究的结果将激发进一步调查其他形式的代码重用和迁移的语音助理或其他新功能的移动的applications.This奖项反映了NSF的法定使命，并已被认为是值得的支持，通过评估使用基金会的智力价值和更广泛的影响审查标准。

The popularity of Voice Assistant (VA) has skyrocketed in recent years, with users able to perform tasks faster and more efficiently through simple voice commands. The success of leading products like Google Assistant and Amazon Alexa has motivated mobile app developers to integrate voice assistant features into their applications. However, adding these features can be significantly time-consuming for developers, as the existing code design primarily focuses on graphical interactions rather than voice interactions. This project aims to provide a solution to this issue by utilizing existing test code to design the voice assistant features. Test code is a series of simulated user interactions with an application that is used to validate the functionality of the application. By reusing the test code and mapping it to the corresponding voice query, the developers can quickly incorporate voice assistant capabilities into their applications. As a result of this project, developers will be more likely to redesign their existing code to accommodate such new voice interactions. Furthermore, this project will bring many training and educational opportunities for both undergraduate and graduate students. It has the potential to support thousands of mobile developers in adding voice assistant options to their applications. In this project, the investigator aims to answer the question of whether and how existing graphical user interface (GUI) test code can be adapted to develop voice assistant features in mobile applications. The investigator will begin by extracting the general functionality from a specific functionality instance in a GUI test code. The investigator will study how to differentiate the general code and non-general code in GUI test code, and develop automated techniques to detect them. The investigator will also investigate how to convert the general functionality into a reusable code template for voice assistant feature development. Second, the investigator will study how to represent the code template using human-understandable language and create potential voice query templates based on this language. A map will be created to connect the voice query template with the code template. Finally, the investigator will develop techniques to facilitate mobile applications in executing the code template in response to a particular voice query. This project aims to support developers in adding voice assistant functionality to existing mobile apps by reusing the test artifacts of the application. The results of this research will inspire further investigation into other forms of code reuse and migration for voice assistants or other new features in mobile applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.