CHS: Small: Multimodal Conversational Assistant that Learns from Demonstrations

苹果（Apple）的Siri、亚马逊（Amazon）的Alexa和微软（Microsoft）的Cortana等智能助手通过提供对话式自然语言界面，让用户访问各种在线服务和数字内容，迅速受到欢迎。它们允许在用户无法触摸手机的情况下（例如在驾驶时）以及可穿戴设备和物联网（IoT）设备（例如谷歌Home）上执行计算任务。然而，这种会话界面在处理任务的“长尾”能力方面受到限制，并且缺乏可定制性。本研究将探索一种新的多模式、交互式、基于演示的编程（PBD）方法，该方法使最终用户能够通过使用演示和口头指令的组合，为任何现有第三方Android移动应用程序中的任务编写自动化脚本，从而为智能助手添加新功能。该系统将利用最先进的机器学习和自然语言处理技术来理解用户的口头指令，提供演示中缺少的信息，如隐含条件、用户意图和个人偏好。用户在图形用户界面上的演示将用于对话的基础和加强自然语言理解模型。该系统将为公众更有效地使用智能手机、物联网设备和智能助手指明道路，提高这些技术的采用率、效率和使用的正确性。智能助手与PBD的集成将会产生广泛的影响，因为它以一种易于学习的方式向人们展示编程概念，从而增加计算思维。通过演示编程（PBD）和智能助手的进步，特别是在它们的集成方面，该项目将产生超越当前艺术水平的若干创新。这项工作将探索利用口头指示作为一种额外的方式来解决PBD研究中长期存在的挑战，包括概括数据描述和增加控制结构。如何协调这两种模式，帮助智能助手有效地从用户那里学习新任务，以及用户如何在不同情况下利用多模态PBD系统中的两种模式进行编程任务。将开发利用应用程序的显示图形用户界面（GUI）的新方法，通过使用智能手机上GUI的字符串和其他上下文来增强语音识别和语言理解。会话助手参与这一泛化过程的能力将得到增强，重点是让系统提出适当和有用的问题，以便任务自动化符合用户的需求和意图。将探索表示用户可以阅读、理解和编辑的PBD系统创建的脚本的新方法，同时还将增加脚本的信任和有用性，并支持错误处理、调试和维护。这项新技术还将能够从应用程序中提取数据并将数据输入应用程序，并通过演示和口头指导学习如何将数据转换为适当的格式。最后，还将研究如何支持PBD系统创建的脚本的共享，同时确保适当的隐私和安全级别。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Intelligent assistants such as Apple's Siri, Amazon's Alexa and Microsoft's Cortana are rapidly gaining popularity by providing a conversational natural language interface for users to access various online services and digital content. They allow computing tasks to be performed in contexts where users cannot touch their phones (such as while driving), and on wearable and Internet of Things (IoT) devices (such as Google Home). However, such conversational interfaces are limited in their ability to handle the "long-tail" of tasks and suffer from lack of customizability. This research will explore a new multi-modal, interactive, programming-by-demonstration (PBD) approach that enables end users to add new capabilities to an intelligent assistant by programming automation scripts for tasks in any existing third-party Android mobile app using a combination of demonstrations and verbal instructions. The system will leverage state-of-the-art machine learning and natural language processing techniques to comprehend the user's verbal instructions that supply information missing in the demonstration, such as implicit conditions, user intent and personal preferences. The user's demonstration on the graphical user interface will be used for grounding the conversation and reinforcing the natural language understanding model. The system will point the way to allowing the general public to more effectively use their smartphones, IoT devices and intelligent assistants, increasing the adoption, efficiency and correctness of uses of these technologies. The integration of intelligent assistants with PBD will have broad impact by exposing people to programming concepts in an easy-to-learn way, and thereby increasing computational thinking. This project will result in several innovations beyond the current state of the art through advances in programming by demonstration (PBD) and intelligent assistants, and especially in their integration. The work will explore leveraging verbal instructions as an additional modality to address long-standing challenges in PBD research including generalizing the data descriptions and adding control structures. How to coordinate the two modalities to help the intelligent assistant learn new tasks effectively and efficiently from users will be investigated, and how users utilize the two modalities in multi-modal PBD systems for programming tasks in different situations will also be studied. New ways to leverage the displayed graphical user interfaces (GUI) of apps to enhance the speech recognition and language understanding by using the strings and other context of the GUI on the smartphone will be developed. The ability of the conversational assistant to participate in this generalization process will be enhanced, with a focus on having the system ask appropriate and helpful questions so the task automation will fit the user's needs and intentions. New approaches to representing scripts created by PBD systems that users can read, understand and edit will be explored, as will increasing trust and usefulness of the scripts and supporting error handling, debugging and maintenance. The new technology will also be able to extract data from and enter data into apps, and to learn, through demonstration and verbal instruction, how to transform the data into appropriate formats. Finally, how to support sharing of scripts created by PBD systems while ensuring the appropriate levels of privacy and security will also be investigated.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.