CAREER: Tying Design to Outcomes: Open-sourced Analytics for Mobile App Testing

为移动的应用程序开发有效、可用的设计方法是一个重要的问题，影响着从医疗保健到金融的各个领域。 目前的应用程序设计方法主要基于个人设计师的专业知识和直觉，导致不同的结果，或者需要大量的活跃用户和工程资源，这些资源通常只适用于大公司。 该项目的目标是解决自动捕获和聚合设计功能和用户交互所涉及的研究挑战，这些功能和用户交互跨越大量现有的移动的应用程序已经可供下载，并开发一个平台，使设计师能够从其他设计师的选择和经验中学习。 该平台将允许设计人员找到具有相关设计要求和功能的相关第三方应用程序，定义涉及这些设计选择的实验，并通过利用这些应用程序的现有用户群来运行实验。 这将帮助个人设计师探索比开发自己的原型更便宜的选项，同时提供数据驱动的参数来支持设计决策并与项目团队的其他成员进行沟通。 该平台还可以带来积累集体设计知识的工具，识别新兴趋势以及最佳实践，对实践设计师有用，并作为现有网络和应用程序设计课程的教育资源。 开发该平台将需要一些技术进步。 第一个涉及开发可扩展的系统，用于从大量的移动的应用程序中捕获设计特征和交互数据，并将它们组合成对数据挖掘有用的表示。 为此，该团队将在单个应用程序中扩展现有的埃里卡黑盒设计捕获平台，将其开发成一个后台监控应用程序，可以捕获用户允许的任何应用程序的交互和设计数据，同时检测和隐藏个人身份信息。 监控应用程序将主要通过长期的众包工作者进行部署，这些工作者将付费安装它并参与设计师定义的实验。 第二个主要的进步是基于定义公共特性、应用状态和功能的收集数据为界面组件创建功能语义嵌入。 为此，该团队将捕获有关每个界面元素的视觉，文本，结构和交互信息，然后使用数据的多模态嵌入首先对各个界面元素进行分类，然后使用这些标签来识别应用程序屏幕的语义以及元素和屏幕之间的交互流。 第三个进步涉及开发一系列工具来使用数据和语义嵌入。 这包括与设计师合作，开发查询技术和应用程序集的视图，以找到相关的应用程序和流程;基于Sankey流程图创建用户行为的可视化，使设计师能够跨应用程序理解用户行为;并设计应用分析工具，以支持Meta该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值进行评估来支持和更广泛的影响审查标准。

Developing methods to do effective, usable design for mobile applications is an important problem, affecting domains ranging from healthcare to finance. Current methods for app design are largely based on either individual designer expertise and intuition, leading to variable results, or else require a large number of active users and engineering resources that are typically only available to large companies. This project's goal is to solve the research challenges involved in automatically capturing and aggregating design features and user interactions across the large number of existing mobile applications already available for download and develop a platform allowing designers to learn from the choices and experiences other designers have made. The platform will allow designers to find relevant third-party apps that have relevant design requirements and features, define experiments involving those design choices, and run the experiments by leveraging the existing user base of those apps. This will help individual designers explore options more cheaply than developing their own prototypes while providing data-driven arguments to support design decisions and communicate with other members of their project teams. The platform could also lead to tools to accumulate collective design knowledge, identifying emerging trends as well as best practices, useful for both practicing designers and as an educational resource for existing courses on web and app design. Developing the platform will require a number of technical advances. The first involves developing scalable systems for capturing design features and interaction data from large numbers of mobile apps and combining them into representations useful for data mining. To do this, the team will expand the existing ERICA platform for black-box design capture in a single application, developing it into a background monitoring app that captures interaction and design data for any app the user gives permission for, while detecting and obscuring personally identifying information. The monitoring app will be deployed primarily via long-term crowdworkers who are paid to install it and participate in designer-defined experiments. The second main advance is creating functional semantic embeddings for interface components based on the collected data that define common features, app states, and functions. To do this the team will capture visual, textual, structural, and interactive information about each interface element, then use multimodal embeddings of the data to first classify individual interface elements, then use those labels to identify semantics of app screens and interaction flows between elements and screens. The third advance involves developing a series of tools to use the data and semantic embeddings. This includes working with designers to develop query techniques and views over sets of applications to find relevant applications and flows; creating visualizations of user behavior based on Sankey flow diagrams that allow designers to make sense of user behavior across applications; and designing app analytics tools to support meta-analysis across applications and experiments.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.