CCRI: ENS: Collaborative Research: Developing the Dialog Ecosystem to Support and Enhance Research in Spoken Dialog Systems

CCRI：ENS：协作研究：开发对话生态系统以支持和加强口语对话系统的研究

• 批准号: 1924855
• 负责人: Maxine Eskenazi
• 金额: $ 117.57万
• 依托单位: Carnegie-Mellon University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-09-01 至 2024-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1924855&HistoricalAwards=false
• 关键词: CCRI; ENS; Collaborative; Research; Developing

People are talking to dialog systems in everyday life. Siri, Alexa and others have become household names. But as any user of these systems knows, they are far from perfect. They are also currently limited in terms of the types of things they can do. Usage data is essential to improve these systems using machine learning and artificial intelligence techniques, but companies that create dialogue systems often keep this data to themselves, making it harder for researchers to create, improve, and evaluate such systems properly. DialPort was created with the goal of gathering data from real users for dialog systems around the world. Researchers can connect their systems to the Portal or request the data that others have collected. Beyond this, for researchers who need help creating a dialog system that they could use to run studies and collect data, the DialPort DialogEcosystem provides them with access to tools for the creation of their systems and tutorials on how to use them. For researchers who already have systems and want to test them using human computation (often called crowdsourcing), the DialPort DialogEcosystem provides easy task creation and connection to major crowdsourcing sites. And to lower the barrier to entry to the field, the DialPort DialogEcosystem helps train young students with its REAL Challenge in which students can imagine ideal dialog systems and learn how to create them. The results from this project will ultimately impact every person who uses dialog systems in daily life.The DialPort project, previously funded by the Computer and Information Science and Engineering (CISE) Research Infrastructure Program, has given the Spoken Dialog Community access to tools, data and users. Researchers who want to create a new dialog system consult DialPort's website for access to the tools they need. When a dialog system is up and running, they connect it to the Portal to get real users to communicate with their systems. The DialPort DialogEcosystem will keep up with the evolving needs of a growing community in several ways. More tools and tutorials will be available. Real user data has started to flow to the systems connected to the Portal and that flow will increase substantially in order to produce the large amounts of data that are needed by state-of-the-art systems. Many researchers test early versions of their systems with crowdworkers, but they are not familiar with how to set up tasks and run quality control and they need help. DialPort DialogEcosystem will respond to these evolving needs with its DialCrowd tools. The field of researchers who are creating dialog systems is expanding. The field of Question Answering is now using chatbots as a means of testing their retrieval capabilities. In Machine Learning, researchers are working on natural language generation and finding that dialog systems are vehicles that can test their work. The DialogEcosystem will reach out to everyone who works on dialog systems and offer a complete framework that can serve their needs from creation to assessment. Beyond present researchers, the DialPort DialogEcosystem will reach out to young students to teach them about our field. The DialPort DialogEcosystem will: create a handheld version of the Portal to address how real users and workers access the Portal; greatly extend DialTools to include wrappers and tutorials for popular system creation tools, extend DialCrowd to help researchers create, assess and analyze results from Crowdwork tasks, create a new REAL Challenge to help high school and undergraduate students become familiar with our field.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

人们在日常生活中与对话系统交谈。Siri、Alexa和其他人已经成为家喻户晓的名字。但正如任何使用这些系统的人所知道的，它们远非完美。他们目前也受到限制，他们可以做的事情的类型。使用数据对于使用机器学习和人工智能技术改进这些系统至关重要，但创建对话系统的公司通常会保留这些数据，这使得研究人员更难正确创建，改进和评估此类系统。创建DialPort的目的是为世界各地的对话系统收集来自真实的用户的数据。研究人员可以将他们的系统连接到门户网站或请求其他人收集的数据。除此之外，对于需要帮助创建对话系统以用于运行研究和收集数据的研究人员，DialPort DialogEcosystem为他们提供了创建系统的工具和如何使用它们的教程。对于已经拥有系统并希望使用人类计算（通常称为众包）对其进行测试的研究人员，DialPort DialogEcosystem提供了轻松的任务创建和与主要众包网站的连接。为了降低进入该领域的门槛，DialPort DialogEcosystem通过其真实的Challenge帮助培训年轻学生，学生可以想象理想的对话系统并学习如何创建它们。这个项目的结果将最终影响到每一个在日常生活中使用对话系统的人。DialPort项目，以前由计算机和信息科学与工程（CISE）研究基础设施计划资助，使口语对话社区能够获得工具，数据和用户。想要创建一个新的对话系统的研究人员可以访问DialPort的网站以获得他们需要的工具。当对话系统启动并运行时，他们将其连接到Portal，以使真实的用户与他们的系统进行通信。DialPort DialogEcosystem将以多种方式跟上不断增长的社区不断发展的需求。将提供更多工具和教程。真实的用户数据已开始流向与门户网站相连的系统，而且流量将大幅增加，以便产生最先进系统所需的大量数据。许多研究人员使用众包工作者测试他们系统的早期版本，但他们不熟悉如何设置任务和运行质量控制，他们需要帮助。DialPort DialogEcosystem将通过其DialCrowd工具来响应这些不断变化的需求。创建对话系统的研究人员的领域正在扩大。问题检索领域现在使用聊天机器人作为测试其检索能力的一种手段。在机器学习中，研究人员正在研究自然语言生成，并发现对话系统是可以测试其工作的工具。DialogEcosystem将覆盖所有从事对话系统工作的人，并提供一个完整的框架，可以满足他们从创建到评估的需求。除了目前的研究人员，DialPort对话生态系统将接触到年轻学生，教他们关于我们的领域。DialPort对话生态系统将：创建门户的手持版本，以解决真实的用户和工作人员如何访问门户的问题;大大扩展了DialTools，使其包含流行的系统创建工具的包装器和教程，扩展了DialCrowd以帮助研究人员创建、评估和分析Crowdwork任务的结果，创建一个新的真实的挑战，以帮助高中和本科生熟悉我们的领域。这个奖项反映了NSF的法定使命，并已被视为通过使用基金会的知识价值和更广泛的影响审查标准进行评估，

项目成果

Interactive Evaluation of Dialog Track at DSTC9

• DOI: 10.48550/arxiv.2207.14403
• 发表时间: 2022-07
• 期刊: ArXiv
• 作者: Shikib Mehri;Yulan Feng;Carla Gordon;S. Alavi;D. Traum;M. Eskénazi

Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challenges

• DOI: 10.48550/arxiv.2203.10012
• 发表时间: 2022-03
• 期刊: ArXiv
• 作者: Shikib Mehri;Jinho Choi;L. F. D’Haro;Jan Deriu;M. Eskénazi;Milica Gasic;Kallirroi Georgila;Dilek Z. Hakkani-Tür;Zekang Li;Verena Rieser;Samira Shaikh;D. Traum;Yi-Ting Yeh;Zhou Yu;Yizhe Zhang;Chen Zhang

Overview of the Ninth Dialog System Technology Challenge: DSTC9

• 发表时间: 2020-11
• 期刊: ArXiv
• 作者: Chulaka Gunasekara;Seokhwan Kim;L. F. D’Haro;Abhinav Rastogi;Yun-Nung Chen;Mihail Eric;Behnam Hedayatnia;Karthik Gopalakrishnan;Yang Liu;Chao-Wei Huang;Dilek Z. Hakkani-Tür;Jinchao Li;Qi Zhu;Lingxiao Luo;Lars Lidén;Kaili Huang;Shahin Shayandeh;Runze Liang;Baolin Peng;Zheng Zhang;Swadheen Shukla;Minlie Huang;Jianfeng Gao;Shikib Mehri;Yulan Feng;Carla Gordon;S. Alavi;D. Traum;M. Eskénazi;Ahmad Beirami;Eunjoon Cho;Paul A. Crook;Ankita De;A. Geramifard;Satwik Kottur;Seungwhan Moon;Shivani Poddar;R. Subba

A Comprehensive Assessment of Dialog Evaluation Metrics

• DOI: 10.18653/v1/2021.eancs-1.3
• 发表时间: 2021-06
• 期刊: ArXiv
• 作者: Yi-Ting Yeh;M. Eskénazi;Shikib Mehri

DialCrowd 2.0: A Quality-Focused Dialog System Crowdsourcing Toolkit

• DOI: 10.48550/arxiv.2207.12551
• 发表时间: 2022-07
• 作者: Jessica Huynh;Ting-Rui Chiang;Jeffrey P. Bigham;M. Eskénazi