III: Small: Towards Speech-Driven Multimodal Querying

现代自动语音识别（ASR）工具在许多情况下提供接近人类的准确性。这使得语音驱动输入在现代设备环境（如平板电脑和智能手机）的许多应用程序中越来越受欢迎，同时也使个人会话助手成为可能。在此背景下，本项目将研究一个看似简单但重要的基本问题：如何设计一个语音驱动系统来查询结构化数据？结构化数据查询在企业、医疗保健和其他领域中无处不在。使用结构化查询语言（SQL）输入查询是此类查询的黄金标准。但是，在上述环境中键入SQL是痛苦的，甚至是不可能的，这限制了用户何时以及如何使用他们的数据。SQL也有一个学习曲线。现有的替代方案，如类型化的自然语言接口，有助于提高可用性，但在很大程度上牺牲了查询的复杂性。例如，今天的会话助手主要支持针对特定于供应商的数据集的查询，而不是任意的数据库模式，而且它们经常无法理解查询意图。这扩大了与SQL的高查询复杂性和无歧义性之间的差距。该项目将通过允许用户使用任意数据库模式上的语音查询与结构化数据进行交互来弥合这一差距。它将为流行的平板电脑、智能手机和会话助手环境带来原型系统。这可以帮助许多数据专业人员，如数据分析师、业务记者和数据库管理员，以及非技术数据爱好者。例如，护士信息学家可以更容易、更明确地检索病人的详细信息，以帮助医生，而分析师甚至可以在移动中对数据进行切片。这项研究将作为出版物在数据库和自然语言处理会议上散发。所做的研究和成果将被纳入研究生和本科生关于数据库系统的课程。作为该项目的一部分，pi将继续支持来自弱势群体的学生。这个项目将为三个“自然度”层次的语音查询创建三个新系统。第一层的目标是可处理且有意义的SQL子集。本研究将利用常规英语语音所缺乏的SQL的三个强大属性——明确的上下文无关语法、查询的数据库模式知识和查询的数据库实例中的令牌知识——来支持ASR词汇表中不存在的任意数据库模式和令牌。pi将对信息检索、自然语言处理和数据库索引等方面的思想进行综合和创新，并将其与人在环查询校正相结合，以提高准确性和效率。第二个版本将通过更改语法使SQL查询更加自然和有状态。这将产生第一个面向语音的SQL方言。第三个版本将把前几个版本的经验应用于两个最先进的数据库类型自然语言接口。这将导致重新设计这样的接口，利用语音的属性和查询的数据库实例。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Modern automatic speech recognition (ASR) tools offer near-human accuracy in many scenarios. This has increased the popularity of speech-driven input in many applications on modern device environments such as tablets and smartphones, while also enabling personal conversational assistants. In this context, this project will study a seemingly simple but important fundamental question: how should one design a speech-driven system to query structured data? Structured data querying is ubiquitous in the enterprise, healthcare, and other domains. Typing queries in the Structured Query Language (SQL) is the gold standard for such querying. But typing SQL is painful or impossible in the above environments, which restricts when and how users can consume their data. SQL also has a learning curve. Existing alternatives such as typed natural language interfaces help improve usability but sacrifice query sophistication substantially. For instance, conversational assistants today support queries mainly over curated vendor-specific datasets, not arbitrary database schemas, and they often fail to understand query intent. This has widened the gap with SQL's high query sophistication and unambiguity. This project will bridge this gap by enabling users to interact with structured data using spoken queries over arbitrary database schemas. It will lead to prototype systems on popular tablet, smartphone, and conversational assistant environments. This could help many data professionals such as data analysts, business reporters, and database administrators, as well as non-technical data enthusiasts. For instance, nurse informaticists can retrieve patient details more easily and unambiguously to assist doctors, while analysts can slice and dice their data even on the move. The research will be disseminated as publications in database and natural language processing conferences. The research and artifacts produced will be integrated into graduate and undergraduate courses on database systems. The PIs will continue supporting students from under-represented groups as part of this project.This project will create three new systems for spoken querying at three levels of "naturalness." The first level targets a tractable and meaningful subset of SQL. This research will exploit three powerful properties of SQL that regular English speech lacks--unambiguous context-free grammar, knowledge of the database schema queried, and knowledge of tokens from the database instance queried--to support arbitrary database schemas and tokens not present in the ASR vocabulary. The PIs will synthesize and innovate upon ideas from information retrieval, natural language processing, and database indexing and combine them with human-in-the-loop query correction to improve accuracy and efficiency. The second version will make SQL querying even more natural and stateful by changing its grammar. This will lead to the first speech-oriented dialect of SQL. The third version will apply the lessons from the previous versions to two state-of-the-art typed natural language interfaces for databases. This will lead to a redesign of such interfaces that exploits both the properties of speech and the database instance queried.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.