CHS: Small: Multimodal Conversational Assistant that Learns from Demonstrations

苹果的Siri、亚马逊的Alexa和微软的Cortana等智能助手通过为用户提供对话式的自然语言界面来访问各种在线服务和数字内容，正迅速受到欢迎。它们允许在用户无法触摸手机的环境中(如开车时)以及在可穿戴设备和物联网(IoT)设备(如Google Home)上执行计算任务。然而，这样的对话界面在处理任务的“长尾”方面能力有限，并且缺乏可定制性。这项研究将探索一种新的多模式、交互式、按演示编程(PBD)方法，使最终用户能够通过使用演示和口头说明的组合，为任何现有第三方Android移动应用程序中的任务编写自动化脚本，从而为智能助手添加新功能。该系统将利用最先进的机器学习和自然语言处理技术来理解用户的口头指令，这些指令提供了演示中缺失的信息，如隐含条件、用户意图和个人偏好。用户在图形用户界面上的演示将用于为对话奠定基础并加强自然语言理解模型。该系统将为公众更有效地使用智能手机、物联网设备和智能助手指明方向，提高这些技术的采用率、效率和正确性。智能助手与PBD的集成将产生广泛的影响，使人们以一种易于学习的方式接触编程概念，从而增加计算思维。该项目将通过在演示编程(PBD)和智能助手方面的进步，特别是在它们的集成方面，带来几项超出当前技术水平的创新。这项工作将探索利用口头指令作为一种额外的方式来解决PBD研究中的长期挑战，包括概括数据描述和增加控制结构。如何协调这两种模式，以帮助智能助手有效和高效地从用户那里学习新任务，以及用户如何在不同情况下利用多模式PBD系统中的这两种模式来编程任务，也将进行研究。将开发新的方法，通过使用智能手机上的图形用户界面的字符串和其他上下文，利用应用程序显示的图形用户界面(GUI)来增强语音识别和语言理解。对话助理参与这一概括过程的能力将得到加强，重点是让系统提出适当和有用的问题，使任务自动化符合用户的需要和意图。将探索新的方法来表示由PBD系统创建的用户可以阅读、理解和编辑的脚本，并将增加脚本的可信度和有用性，并支持错误处理、调试和维护。这项新技术还将能够从应用程序中提取数据并将数据输入应用程序，并通过演示和口头指导学习如何将数据转换为适当的格式。最后，如何支持共享由PBD系统创建的脚本，同时确保适当的隐私和安全级别也将进行调查。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Intelligent assistants such as Apple's Siri, Amazon's Alexa and Microsoft's Cortana are rapidly gaining popularity by providing a conversational natural language interface for users to access various online services and digital content. They allow computing tasks to be performed in contexts where users cannot touch their phones (such as while driving), and on wearable and Internet of Things (IoT) devices (such as Google Home). However, such conversational interfaces are limited in their ability to handle the "long-tail" of tasks and suffer from lack of customizability. This research will explore a new multi-modal, interactive, programming-by-demonstration (PBD) approach that enables end users to add new capabilities to an intelligent assistant by programming automation scripts for tasks in any existing third-party Android mobile app using a combination of demonstrations and verbal instructions. The system will leverage state-of-the-art machine learning and natural language processing techniques to comprehend the user's verbal instructions that supply information missing in the demonstration, such as implicit conditions, user intent and personal preferences. The user's demonstration on the graphical user interface will be used for grounding the conversation and reinforcing the natural language understanding model. The system will point the way to allowing the general public to more effectively use their smartphones, IoT devices and intelligent assistants, increasing the adoption, efficiency and correctness of uses of these technologies. The integration of intelligent assistants with PBD will have broad impact by exposing people to programming concepts in an easy-to-learn way, and thereby increasing computational thinking. This project will result in several innovations beyond the current state of the art through advances in programming by demonstration (PBD) and intelligent assistants, and especially in their integration. The work will explore leveraging verbal instructions as an additional modality to address long-standing challenges in PBD research including generalizing the data descriptions and adding control structures. How to coordinate the two modalities to help the intelligent assistant learn new tasks effectively and efficiently from users will be investigated, and how users utilize the two modalities in multi-modal PBD systems for programming tasks in different situations will also be studied. New ways to leverage the displayed graphical user interfaces (GUI) of apps to enhance the speech recognition and language understanding by using the strings and other context of the GUI on the smartphone will be developed. The ability of the conversational assistant to participate in this generalization process will be enhanced, with a focus on having the system ask appropriate and helpful questions so the task automation will fit the user's needs and intentions. New approaches to representing scripts created by PBD systems that users can read, understand and edit will be explored, as will increasing trust and usefulness of the scripts and supporting error handling, debugging and maintenance. The new technology will also be able to extract data from and enter data into apps, and to learn, through demonstration and verbal instruction, how to transform the data into appropriate formats. Finally, how to support sharing of scripts created by PBD systems while ensuring the appropriate levels of privacy and security will also be investigated.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.