Towards Framework-Independent Evaluation of Deep Linguistic Parsers
Towards Framework-Independent Evaluation of Deep Linguistic Parsers
复制标题
DOI：
--
复制
发表时间：
2007
期刊：
影响因子：
--
通讯作者：
Yusuke Miyao;Kenji Sagae
中科院分区：
文献类型：
--
作者：
Yusuke Miyao;Kenji Sagae
本文描述了在深度和浅层解析器的框架独立评估的实际问题。我们重点讨论了两种基于依存关系的句法表示格式在句法分析器评估中的应用，即卡罗尔等人（1998）的语法关系和de Marneffe等人（2006）的斯坦福大学依存关系方案。我们的方法是将解析器的输出转换为这两种格式，并测量转换后输出的准确性。通过对HPSG解析器和Penn Treebank短语结构解析器的评估，我们发现不同表示方案之间的映射是一项重要的任务，它会导致有损转换，从而可能掩盖不同解析方法之间的重要差异。我们讨论了两个依赖性为基础的格式表示的句法结构的分歧的来源，表明可能的方向改进框架独立的分析器评估。
This paper describes practical issues in the framework-independent evaluation of deep and shallow parsers. We focus on the use of two dependencybased syntactic representation formats in parser evaluation, namely, Carroll et al. (1998)’s Grammatical Relations and de Marneffe et al. (2006)’s Stanford Dependency scheme. Our approach is to convert the output of parsers into these two formats, and measure the accuracy of the resulting converted output. Through the evaluation of an HPSG parser and Penn Treebank phrase structure parsers, we found that mapping between different representation schemes is a non-trivial task that results in lossy conversions that may obscure important differences between different parsing approaches. We discuss sources of disagreements in the representation of syntactic structures in the two dependency-based formats, indicating possible directions for improved framework-independent parser evaluation.