CAREER: Tying Design to Outcomes: Open-sourced Analytics for Mobile App Testing

开发有效、可用的移动应用设计方法是一个重要的问题，影响到从医疗保健到金融的各个领域。目前的应用程序设计方法主要是基于个别设计师的专业知识和直觉，导致结果参差不齐，或者需要大量活跃用户和工程资源，而这些通常只对大公司可用。该项目的目标是解决在现有大量可供下载的移动应用程序中自动捕获和聚合设计功能和用户交互所涉及的研究挑战，并开发一个允许设计师学习其他设计师的选择和经验的平台。该平台将允许设计师找到具有相关设计要求和功能的相关第三方应用程序，定义涉及这些设计选择的实验，并通过利用这些应用程序的现有用户基础来运行实验。这将帮助个人设计师探索比开发自己的原型更便宜的选择，同时提供数据驱动的论点来支持设计决策，并与他们的项目团队的其他成员进行沟通。该平台还可以产生工具，以积累集体设计知识，确定新出现的趋势和最佳做法，对实践设计师有用，并作为现有网页和应用程序设计课程的教育资源。开发该平台将需要一些技术进步。第一个涉及开发可扩展的系统，用于从大量移动应用程序中捕获设计特征和交互数据，并将它们组合成对数据挖掘有用的表示形式。为此，该团队将扩展现有的Erica平台，在单个应用程序中捕获黑匣子设计，将其开发为后台监控应用程序，捕获用户允许的任何应用程序的交互和设计数据，同时检测和模糊个人身份识别信息。监控应用程序将主要通过长期的众包工作人员进行部署，这些工作人员负责安装该应用程序并参与设计师定义的实验。第二个主要进步是基于收集的数据为界面组件创建功能语义嵌入，这些数据定义了共同的特性、应用程序状态和功能。为此，团队将捕获有关每个界面元素的视觉、文本、结构和交互信息，然后使用数据的多模式嵌入首先对各个界面元素进行分类，然后使用这些标签来识别应用程序屏幕的语义以及元素和屏幕之间的交互流。第三个进步涉及开发一系列工具来使用数据和语义嵌入。这包括与设计人员合作开发应用程序集的查询技术和视图，以找到相关的应用程序和流程；基于Sankey流程图创建用户行为的可视化，使设计人员能够理解应用程序之间的用户行为；以及设计应用程序分析工具，以支持跨应用程序和实验的元分析。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Developing methods to do effective, usable design for mobile applications is an important problem, affecting domains ranging from healthcare to finance. Current methods for app design are largely based on either individual designer expertise and intuition, leading to variable results, or else require a large number of active users and engineering resources that are typically only available to large companies. This project's goal is to solve the research challenges involved in automatically capturing and aggregating design features and user interactions across the large number of existing mobile applications already available for download and develop a platform allowing designers to learn from the choices and experiences other designers have made. The platform will allow designers to find relevant third-party apps that have relevant design requirements and features, define experiments involving those design choices, and run the experiments by leveraging the existing user base of those apps. This will help individual designers explore options more cheaply than developing their own prototypes while providing data-driven arguments to support design decisions and communicate with other members of their project teams. The platform could also lead to tools to accumulate collective design knowledge, identifying emerging trends as well as best practices, useful for both practicing designers and as an educational resource for existing courses on web and app design. Developing the platform will require a number of technical advances. The first involves developing scalable systems for capturing design features and interaction data from large numbers of mobile apps and combining them into representations useful for data mining. To do this, the team will expand the existing ERICA platform for black-box design capture in a single application, developing it into a background monitoring app that captures interaction and design data for any app the user gives permission for, while detecting and obscuring personally identifying information. The monitoring app will be deployed primarily via long-term crowdworkers who are paid to install it and participate in designer-defined experiments. The second main advance is creating functional semantic embeddings for interface components based on the collected data that define common features, app states, and functions. To do this the team will capture visual, textual, structural, and interactive information about each interface element, then use multimodal embeddings of the data to first classify individual interface elements, then use those labels to identify semantics of app screens and interaction flows between elements and screens. The third advance involves developing a series of tools to use the data and semantic embeddings. This includes working with designers to develop query techniques and views over sets of applications to find relevant applications and flows; creating visualizations of user behavior based on Sankey flow diagrams that allow designers to make sense of user behavior across applications; and designing app analytics tools to support meta-analysis across applications and experiments.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.