ReadMe++: Benchmarking Multilingual Language Models for Multi-Domain Readability Assessment
ReadMe++: Benchmarking Multilingual Language Models for Multi-Domain Readability Assessment
复制标题
DOI：
--
复制
发表时间：
2023-05
期刊：
影响因子：
2.2
通讯作者：
Tarek Naous;Michael Joseph Ryan;Mohit Chandra;Wei Xu
中科院分区：
文献类型：
--
作者：
Tarek Naous;Michael Joseph Ryan;Mohit Chandra;Wei Xu
我们提出了一种用于多语言可读性评估的大型语言模型的综合评估。现有评价资源缺乏领域和语言多样性，限制了跨领域和跨语言分析的能力。本文介绍了ReadMe++，这是一个多语言多领域数据集，包含9757个阿拉伯语、英语、法语、印地语和俄语句子的人工标注，从112个不同的数据源收集。这一基准将鼓励研究开发强有力的多语种可读性评估方法。使用ReadMe++，我们在有监督、无监督和少提示的情况下对多语言和单语言模型进行了基准测试。ReadMe++的领域和语言多样性使我们能够测试更有效的少量提示，并找出最先进的非监督方法的缺陷。我们的实验还显示了令人兴奋的结果，通过在ReadMe++上训练的模型，具有更好的领域泛化能力和增强的跨语言迁移能力。我们将公开我们的数据，并发布一个用于使用我们训练的模型进行多语言句子可读性预测的Python包工具：https://github.com/tareknaous/readme
We present a comprehensive evaluation of large language models for multilingual readability assessment. Existing evaluation resources lack domain and language diversity, limiting the ability for cross-domain and cross-lingual analyses. This paper introduces ReadMe++, a multilingual multi-domain dataset with human annotations of 9757 sentences in Arabic, English, French, Hindi, and Russian, collected from 112 different data sources. This benchmark will encourage research on developing robust multilingual readability assessment methods. Using ReadMe++, we benchmark multilingual and monolingual language models in the supervised, unsupervised, and few-shot prompting settings. The domain and language diversity in ReadMe++ enable us to test more effective few-shot prompting, and identify shortcomings in state-of-the-art unsupervised methods. Our experiments also reveal exciting results of superior domain generalization and enhanced cross-lingual transfer capabilities by models trained on ReadMe++. We will make our data publicly available and release a python package tool for multilingual sentence readability prediction using our trained models at: https://github.com/tareknaous/readme