III: Small: Towards Speech-Driven Multimodal Querying

III：小型：迈向语音驱动的多模式查询

• 批准号: 1816701
• 负责人: Arun Kumar
• 金额: $ 50万
• 依托单位: University of California-San Diego
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-10-01 至 2022-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1816701&HistoricalAwards=false
• 关键词: III; Small; Towards; Speech; Driven

Modern automatic speech recognition (ASR) tools offer near-human accuracy in many scenarios. This has increased the popularity of speech-driven input in many applications on modern device environments such as tablets and smartphones, while also enabling personal conversational assistants. In this context, this project will study a seemingly simple but important fundamental question: how should one design a speech-driven system to query structured data? Structured data querying is ubiquitous in the enterprise, healthcare, and other domains. Typing queries in the Structured Query Language (SQL) is the gold standard for such querying. But typing SQL is painful or impossible in the above environments, which restricts when and how users can consume their data. SQL also has a learning curve. Existing alternatives such as typed natural language interfaces help improve usability but sacrifice query sophistication substantially. For instance, conversational assistants today support queries mainly over curated vendor-specific datasets, not arbitrary database schemas, and they often fail to understand query intent. This has widened the gap with SQL's high query sophistication and unambiguity. This project will bridge this gap by enabling users to interact with structured data using spoken queries over arbitrary database schemas. It will lead to prototype systems on popular tablet, smartphone, and conversational assistant environments. This could help many data professionals such as data analysts, business reporters, and database administrators, as well as non-technical data enthusiasts. For instance, nurse informaticists can retrieve patient details more easily and unambiguously to assist doctors, while analysts can slice and dice their data even on the move. The research will be disseminated as publications in database and natural language processing conferences. The research and artifacts produced will be integrated into graduate and undergraduate courses on database systems. The PIs will continue supporting students from under-represented groups as part of this project.This project will create three new systems for spoken querying at three levels of "naturalness." The first level targets a tractable and meaningful subset of SQL. This research will exploit three powerful properties of SQL that regular English speech lacks--unambiguous context-free grammar, knowledge of the database schema queried, and knowledge of tokens from the database instance queried--to support arbitrary database schemas and tokens not present in the ASR vocabulary. The PIs will synthesize and innovate upon ideas from information retrieval, natural language processing, and database indexing and combine them with human-in-the-loop query correction to improve accuracy and efficiency. The second version will make SQL querying even more natural and stateful by changing its grammar. This will lead to the first speech-oriented dialect of SQL. The third version will apply the lessons from the previous versions to two state-of-the-art typed natural language interfaces for databases. This will lead to a redesign of such interfaces that exploits both the properties of speech and the database instance queried.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

现代自动语音识别（ASR）工具在许多情况下提供接近人类的准确性。这增加了语音驱动输入在现代设备环境（例如平板电脑和智能手机）上的许多应用中的普及，同时还实现了个人会话助理。在这种情况下，这个项目将研究一个看似简单但重要的基本问题：应该如何设计一个语音驱动的系统来查询结构化数据？结构化数据查询在企业、医疗保健和其他领域中无处不在。使用结构化查询语言（SQL）输入查询是此类查询的黄金标准。但是在上述环境中输入SQL是痛苦的或不可能的，这限制了用户何时以及如何使用他们的数据。SQL也有一个学习曲线。现有的替代方案，如类型化的自然语言接口，有助于提高可用性，但牺牲查询的复杂性大大。例如，今天的会话助理主要支持针对特定于供应商的数据集的查询，而不是任意的数据库模式，并且它们通常无法理解查询意图。这就扩大了与SQL的高查询复杂性和明确性之间的差距。该项目将通过使用户能够使用任意数据库模式上的口头查询与结构化数据进行交互来弥合这一差距。它将导致流行的平板电脑，智能手机和会话助理环境的原型系统。这可以帮助许多数据专业人士，如数据分析师，商业记者和数据库管理员，以及非技术数据爱好者。例如，护士信息学家可以更轻松、更明确地检索患者的详细信息，以帮助医生，而分析师甚至可以在移动中对数据进行切片和切割。该研究将作为数据库和自然语言处理会议的出版物传播。研究和工件生产将被整合到研究生和本科生课程的数据库系统。作为这个项目的一部分，PI将继续支持来自代表性不足群体的学生。这个项目将创建三个新的系统，用于三个“自然”级别的口语查询。“第一级目标是一个易于处理和有意义的SQL子集。这项研究将利用SQL的三个强大的属性，定期英语演讲缺乏-明确的上下文无关语法，查询的数据库模式的知识，并从查询的数据库实例的令牌的知识-支持任意的数据库模式和令牌不存在于ASR词汇。PI将综合和创新信息检索、自然语言处理和数据库索引的思想，并将它们与人工在环查询校正联合收割机相结合，以提高准确性和效率。第二个版本将通过更改语法使SQL查询更加自然和有状态。这将导致第一个面向语音的SQL方言。第三个版本将把以前版本的经验应用到两个最先进的数据库类型化自然语言接口上。这将导致重新设计这样的接口，利用语音和查询的数据库实例的属性。这个奖项反映了NSF的法定使命，并已被认为是值得的支持，通过使用基金会的智力价值和更广泛的影响审查标准进行评估。

项目成果

Demonstration of SpeakQL: Speech-driven Multimodal Querying of Structured Data

SpeakQL 演示：语音驱动的结构化数据多模态查询

• DOI: 10.1145/3299869.3320224
• 发表时间: 2019
• 期刊: Proceedings of the 2019 International Conference on Management of Data
• 作者: Shah, Vraj;Li, Side;Yang, Kevin;Kumar, Arun;Saul, Lawrence
• 通讯作者: Saul, Lawrence

Structured Data Representation in Natural Language Interfaces

• 发表时间: 2022
• 期刊: IEEE Data Eng. Bull.
• 作者: Yutong Shao;Arun Kumar;Ndapandula Nakashole

SpeakQL: Towards Speech-driven Multimodal Querying of Structured Data

• DOI: 10.1145/3318464.3389777
• 发表时间: 2020-05
• 期刊: Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data
• 作者: Vraj Shah;Side Li;Arun Kumar;L. Saul