MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering
MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering
复制标题
DOI：
10.1162/tacl_a_00433
复制
发表时间：
2020-07
影响因子：
10.9
通讯作者：
S. Longpre;Yi Lu;Joachim Daiber
中科院分区：
文献类型：
--
作者：
S. Longpre;Yi Lu;Joachim Daiber
跨语言建模的进展取决于具有挑战性的、现实的和多样化的评估集。我们介绍了多语言知识问答（MKQA），这是一个开放领域的问答评估集，包含26种不同类型语言的10k对问答（总共260k对问答）。答案是基于精心策划的、独立于语言的数据表示，使结果在不同语言之间具有可比性，并且独立于特定语言的段落。这个数据集有26种语言，为评估问题回答提供了迄今为止最广泛的语言。我们对各种最先进的方法和基线进行基准测试，用于生成和抽取问题回答，在零射击和翻译设置中训练自然问题。结果表明，即使在英语中，这个数据集也具有挑战性，尤其是在资源匮乏的语言中
Abstract Progress in cross-lingual modeling depends on challenging, realistic, and diverse evaluation sets. We introduce Multilingual Knowledge Questions and Answers (MKQA), an open- domain question answering evaluation set comprising 10k question-answer pairs aligned across 26 typologically diverse languages (260k question-answer pairs in total). Answers are based on heavily curated, language- independent data representation, making results comparable across languages and independent of language-specific passages. With 26 languages, this dataset supplies the widest range of languages to-date for evaluating question answering. We benchmark a variety of state- of-the-art methods and baselines for generative and extractive question answering, trained on Natural Questions, in zero shot and translation settings. Results indicate this dataset is challenging even in English, but especially in low-resource languages.1