TYDI QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages
TYDI QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages
复制标题
DOI：
10.1162/tacl_a_00317
复制
发表时间：
2020-01-01
影响因子：
10.9
通讯作者：
Palomaki, Jennimaria
中科院分区：
文献类型：
--
作者：
Clark, Jonathan H.;Choi, Eunsol;Palomaki, Jennimaria
自信地在多语言建模方面取得进展需要具有挑战性的、值得信赖的评估。我们提出了TYDI问答数据集，涵盖11种不同类型的语言，有204K的问答对。TYDI QA的语言在类型学（每种语言表达的语言特征集）方面是多种多样的，因此我们期望在这个集上表现良好的模型可以泛化到世界上的大量语言。我们对数据质量进行了定量分析，并对所观察到的语言现象进行了实例级定性语言分析，这些现象在纯英语语料库中是找不到的。为了提供一个真实的信息搜索任务，避免启动效应，问题是由想知道答案，但还不知道答案的人写的，数据是直接用每种语言收集的，不使用翻译。
Confidently making progress on multilingual modeling requires challenging, trustworthy evaluations. We present TYDI QA-a question answering dataset covering 11 typologically diverse languages with 204K question-answer pairs. The languages of TYDI QA are diverse with regard to their typology-the set of linguistic features each language expresses-such that we expect models performing well on this set to generalize across a large number of the world's languages. We present a quantitative analysis of the data quality and example-level qualitative linguistic analyses of observed language phenomena that would not be found in English-only corpora. To provide a realistic information-seeking task and avoid priming effects, questions are written by people who want to know the answer, but don't know the answer yet, and the data is collected directly in each language without the use of translation.