SyntaxGym: An Online Platform for Targeted Evaluation of Language Models
SyntaxGym: An Online Platform for Targeted Evaluation of Language Models
复制标题
SyntaxGym：语言模型定向评估在线平台
DOI：
--
复制
发表时间：
2020
期刊：
影响因子：
--
通讯作者：
R. Levy
中科院分区：
文献类型：
--
作者：
Jon Gauthier;Jennifer Hu;Ethan Gotlieb Wilcox;Peng Qian;R. Levy
有针对性的句法评估已经深入了解了神经网络语言模型所学习的概括。然而，这一研究领域需要多种技能的罕见融合：设计受控心理语言学实验所需的理论知识，以及训练和部署大规模语言模型所需的技术熟练程度。我们提出了SyntaxGym，一个在线平台，旨在使有针对性的评估访问NLP和语言学的专家，跨计算环境的可重复性，并按照心理语言学实验设计的规范标准化。本文为计算语言学社区发布了两个具有独立价值的工具：1。一个网站，syntaxgym.org，它集中了有针对性的句法评估的过程，并提供了简单的分析和可视化工具; 2。两个命令行工具，'syntaxgym'和'lm-zoo'，允许任何用户在自己的机器上重现有针对性的语法评估和通用语言模型推理。
Targeted syntactic evaluations have yielded insights into the generalizations learned by neural network language models. However, this line of research requires an uncommon confluence of skills: both the theoretical knowledge needed to design controlled psycholinguistic experiments, and the technical proficiency needed to train and deploy large-scale language models. We present SyntaxGym, an online platform designed to make targeted evaluations accessible to both experts in NLP and linguistics, reproducible across computing environments, and standardized following the norms of psycholinguistic experimental design. This paper releases two tools of independent value for the computational linguistics community: 1. A website, syntaxgym.org, which centralizes the process of targeted syntactic evaluation and provides easy tools for analysis and visualization; 2. Two command-line tools, ‘syntaxgym‘ and ‘lm-zoo‘, which allow any user to reproduce targeted syntactic evaluations and general language model inference on their own machine.
BLiMP：英语语言最小对的基准
DOI：
10.1162/tacl_a_00321
发表时间：
2020
影响因子：
10.9
作者：
Warstadt, Alex;Parrish, Alicia;Liu, Haokun;Mohananey, Anhad;Peng, Wei;Wang, Sheng-Fu;Bowman, Samuel R.
通讯作者：
Bowman, Samuel R.