CHS: Small: Multimodal Conversational Assistant that Learns from Demonstrations

CHS：Small：从演示中学习的多模式对话助手

• 批准号: 1814472
• 负责人: Brad Myers
• 金额: $ 49.9万
• 依托单位: Carnegie-Mellon University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-08-15 至 2021-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1814472&HistoricalAwards=false
• 关键词: CHS; Small; Multimodal; Conversational; Assistant

Intelligent assistants such as Apple's Siri, Amazon's Alexa and Microsoft's Cortana are rapidly gaining popularity by providing a conversational natural language interface for users to access various online services and digital content. They allow computing tasks to be performed in contexts where users cannot touch their phones (such as while driving), and on wearable and Internet of Things (IoT) devices (such as Google Home). However, such conversational interfaces are limited in their ability to handle the "long-tail" of tasks and suffer from lack of customizability. This research will explore a new multi-modal, interactive, programming-by-demonstration (PBD) approach that enables end users to add new capabilities to an intelligent assistant by programming automation scripts for tasks in any existing third-party Android mobile app using a combination of demonstrations and verbal instructions. The system will leverage state-of-the-art machine learning and natural language processing techniques to comprehend the user's verbal instructions that supply information missing in the demonstration, such as implicit conditions, user intent and personal preferences. The user's demonstration on the graphical user interface will be used for grounding the conversation and reinforcing the natural language understanding model. The system will point the way to allowing the general public to more effectively use their smartphones, IoT devices and intelligent assistants, increasing the adoption, efficiency and correctness of uses of these technologies. The integration of intelligent assistants with PBD will have broad impact by exposing people to programming concepts in an easy-to-learn way, and thereby increasing computational thinking. This project will result in several innovations beyond the current state of the art through advances in programming by demonstration (PBD) and intelligent assistants, and especially in their integration. The work will explore leveraging verbal instructions as an additional modality to address long-standing challenges in PBD research including generalizing the data descriptions and adding control structures. How to coordinate the two modalities to help the intelligent assistant learn new tasks effectively and efficiently from users will be investigated, and how users utilize the two modalities in multi-modal PBD systems for programming tasks in different situations will also be studied. New ways to leverage the displayed graphical user interfaces (GUI) of apps to enhance the speech recognition and language understanding by using the strings and other context of the GUI on the smartphone will be developed. The ability of the conversational assistant to participate in this generalization process will be enhanced, with a focus on having the system ask appropriate and helpful questions so the task automation will fit the user's needs and intentions. New approaches to representing scripts created by PBD systems that users can read, understand and edit will be explored, as will increasing trust and usefulness of the scripts and supporting error handling, debugging and maintenance. The new technology will also be able to extract data from and enter data into apps, and to learn, through demonstration and verbal instruction, how to transform the data into appropriate formats. Finally, how to support sharing of scripts created by PBD systems while ensuring the appropriate levels of privacy and security will also be investigated.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

苹果的Siri、亚马逊的Alexa和微软的Cortana等智能助手通过为用户提供对话式自然语言界面来访问各种在线服务和数字内容，正在迅速普及。 它们允许在用户无法触摸手机的情况下（例如在驾驶时）以及在可穿戴和物联网（IoT）设备（例如Google Home）上执行计算任务。 然而，这样的会话接口在它们处理任务的“长尾”的能力方面是有限的，并且遭受缺乏可定制性的问题。 这项研究将探索一种新的多模态、交互式、演示编程（PBD）方法，使最终用户能够通过使用演示和口头指令的组合，为任何现有的第三方Android移动的应用程序中的任务编程自动化脚本，为智能助手添加新功能。 该系统将利用最先进的机器学习和自然语言处理技术来理解用户的口头指令，这些指令提供了演示中缺少的信息，例如隐式条件，用户意图和个人偏好。 用户在图形用户界面上的演示将用于为对话奠定基础并加强自然语言理解模型。 该系统将为公众更有效地使用智能手机、物联网设备和智能助手指明方向，提高这些技术的采用率、效率和正确性。 智能助手与PBD的集成将通过以易于学习的方式向人们展示编程概念，从而增加计算思维，从而产生广泛的影响。 该项目将通过演示编程（PBD）和智能助手的进步，特别是在它们的集成方面，带来超越当前技术水平的几项创新。 这项工作将探索利用口头指示作为一种额外的方式来解决PBD研究中长期存在的挑战，包括概括数据描述和添加控制结构。 如何协调这两种模式，以帮助智能助理学习新的任务有效和高效地从用户将进行调查，以及用户如何利用这两种模式在多模态PBD系统在不同的情况下编程任务也将进行研究。 将开发利用应用程序显示的图形用户界面（GUI）的新方法，通过使用智能手机上GUI的字符串和其他上下文来增强语音识别和语言理解。 会话助理参与这一概括过程的能力将得到增强，重点是让系统提出适当和有用的问题，以便任务自动化符合用户的需求和意图。 将探索表示PBD系统创建的用户可以阅读、理解和编辑的脚本的新方法，以及增加脚本的信任和有用性并支持错误处理、调试和维护。 新技术还将能够从应用程序中提取数据并将数据输入应用程序，并通过演示和口头指导学习如何将数据转换为适当的格式。 最后，如何支持共享的脚本创建的PBD系统，同时确保适当的隐私和安全水平也将被调查。这个奖项反映了NSF的法定使命，并已被认为是值得支持的评估使用基金会的智力价值和更广泛的影响审查标准。

项目成果

Screen2Vec: Semantic Embedding of GUI Screens and GUI Components

• DOI: 10.1145/3411764.3445049
• 发表时间: 2021-01
• 期刊: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
• 作者: Toby Jia-Jun Li;Lindsay Popowski;Tom Michael Mitchell;B. Myers

Interactive Task and Concept Learning from Natural Language Instructions and GUI Demonstrations

通过自然语言指令和 GUI 演示进行交互式任务和概念学习

• 发表时间: 2020
• 期刊: The AAAI-20 Workshop on Intelligent Process Automation (IPA-20
• 作者: Li, Toby Jia-Jun;Radensky, Marissa;Jia, Justin;Singarajah, Kirielle;Mitchell, Tom M.;Myers, Brad A.
• 通讯作者: Myers, Brad A.

A Multi-modal Approach to Concept Learning in Task Oriented Conversational Agents

面向任务的会话代理中概念学习的多模态方法

• 发表时间: 2019
• 期刊: CHI 2019 Workshop on Conversational Agents: Acting on the Wave of Research and Development (CHI19convai
• 作者: Li, Toby Jia-Jun;Radensky, Marissa;Mitchell, Tom;Myers, Brad
• 通讯作者: Myers, Brad

APPINITE: A Multi-Modal Interface for Specifying Data Descriptions in Programming by Demonstration Using Natural Language Instructions

APPINITE：使用自然语言指令通过演示指定编程中的数据描述的多模式接口

• DOI: 10.1109/vlhcc.2018.8506506
• 发表时间: 2018
• 期刊: 2018 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC'18
• 作者: Li, Toby Jia-Jun;Labutov, Igor;Li, Xiaohan Nancy;Zhang, Xiaoyi;Shi, Wenze;Ding, Wanling;Mitchell, Tom M.;Myers, Brad A.
• 通讯作者: Myers, Brad A.

End user programing of intelligent agents using demonstrations and natural language instructions

使用演示和自然语言指令对智能代理进行最终用户编程

• DOI: 10.1145/3308557.3308724
• 发表时间: 2019
• 期刊: IUI '19: Proceedings of the 24th International Conference on Intelligent User Interfaces: Companion (Student Consortium
• 作者: Li, Toby Jia-Jun