I-Corps: Automating the Development Process of Mobile Applications

实际上，开发专业的移动的电话应用程序既麻烦又昂贵。为了为这样的应用程序创建优化的用户界面，专业图形艺术家（即，非程序员）设计用户界面的每个屏幕，例如，在Photoshop中基于像素的设计。然后程序员手动将这些屏幕设计转换为应用程序源代码。这种转换是劳动密集型的，通常需要在项目期间多次完成，因为在项目期间，用户界面设计通常是迭代优化的。这个项目探索了第一种自动执行从基于像素的屏幕设计图到工作用户界面源代码的转换的技术的商业化。该项目预计将有利于专业的移动的应用程序开发人员和学生。该工具可以节省专业开发人员的时间，并教学生如何在源代码中实现屏幕元素。为了满足消费者对定制用户界面（UI）设计的需求，移动的应用UI经常偏离标准UI组件，而提供它们自己的UI设计。虽然存在许多图形UI构建器工具，但它们并不适合这种自定义UI设计，并且专家图形设计师更喜欢基于像素的设计工具。在给定的屏幕设计图形上，项目团队的原型工具通过计算机视觉和光学字符识别（OCR）技术识别UI元素，如图像、文本、容器和列表。原型进一步推断合适的UI层次结构，并将结果导出为准备好在移动的计算设备上编译和执行的源代码。所生成的UI在逐像素的屏幕截图相似性方面以及在它们的运行时UI代码结构的相似性方面都紧密地模仿对应的真实的应用的UI。在I-Corps计划期间，该团队将专注于联系潜在客户，收集他们的反馈，并利用它来进一步增强拟议的原型。

Developing a professional mobile phone application is cumbersome and expensive in practice. To create for such an application an optimized user interface, professional graphic artists (i.e., non-programmers) design each screen of the user interface, for example, as pixel-based designs in Photoshop. Programmers then manually convert these screen designs to application source code. This conversion is labor-intensive and typically needs to be done several times during a project, as during a project user interface designs are often optimized iteratively. This project explores the commercialization of the first technique to automatically perform this conversion from pixel-based screen design drawing to working user interface source code. The project is expected to benefit both professional mobile application developers and students. The tool may save professional developers time and teach students how screen elements can be implemented in source code.To satisfy consumer demand for customized user interface (UI) designs, mobile application UIs often deviate from standard UI components and provide their own UI designs. While many graphical UI builder tools exist, they are not well suited for such custom UI designs and expert graphic designers prefer pixel-based design tools. On a given screen design graphic, the project team's prototype tool identifies UI elements such as images, text, containers, and lists, via computer vision and optical character recognition (OCR) techniques. The prototype further infers a suitable UI hierarchy and exports the results as source code that is ready for compilation and execution on mobile computing devices. The generated UI closely mimics the UI of a corresponding real application, both in terms of pixel-by-pixel screenshot similarity and by the similarity of their runtime UI code structures. During the I-Corps program, the team will focus on contacting potential customers, gathering their feedback, and using it to further enhance the proposed prototype.