Design, Compilation, and Preliminary Analyses of Balanced Corpus of Contemporary Written Japanese
Design, Compilation, and Preliminary Analyses of Balanced Corpus of Contemporary Written Japanese
复制标题
当代书面日语平衡语料库的设计、编制与初步分析
DOI：
--
复制
发表时间：
2010
期刊：
影响因子：
--
通讯作者：
Yasuharu Den
中科院分区：
文献类型：
--
作者：
K. Maekawa;Makoto Yamazaki;Takehiko Maruyama;Masaya Yamaguchi;Hideki Ogura;Wakako Kashino;Toshinobu Ogiso;H. Koiso;Yasuharu Den
国立日本语研究所正在编制名为“当代日语平衡语料库”（BCCWJ）的 1 亿字平衡语料库。语料库涵盖书籍、杂志、报纸、政府白皮书、教科书、国会会议纪要、网络文本（布告栏和博客）等多种文本体裁，在可能的情况下，通过随机抽样的方式从严格定义的统计总体中抽取样本。所有文本均基于两种不同但相互关联的“词”定义进行双重词性分析。目前，已对超过 9000 万个词进行了采样，并对文本结构、词汇和字符信息进行了 XML 注释。利用词性频率和句子长度数据对文本体裁进行初步线性判别分析，结果表明，对于书籍、报纸、白皮书、网络公告板等样本，可以对文本体裁进行分类，正确识别率为88%。然而，当博客样本包含在该数据集中时，识别率下降至 68%，这表明博客文本在文本语域和风格方面存在相当大的差异。
Compilation of a 100 million words balanced corpus called the Balanced Corpus of Contemporary Written Japanese (or BCCWJ) is underway at the National Institute for Japanese Language and Linguistics. The corpus covers a wide range of text genres including books, magazines, newspapers, governmental white papers, textbooks, minutes of the National Diet, internet text (bulletin board and blogs) and so forth, and when possible, samples are drawn from the rigidly defined statistical populations by means of random sampling. All texts are dually POS-analyzed based upon two different, but mutually related, definitions of word. Currently, more than 90 million words have been sampled and XML annotated with respect to text-structure and lexical and character information. A preliminary linear discriminant analysis of text genres using the data of POS frequencies and sentence length revealed it was possible to classify the text genres with a correct identification rate of 88% as far as the samples of books, newspapers, whitepapers, and internet bulletin boards are concerned. When the samples of blogs were included in this data set, however, the identification rate went down to 68%, suggesting the considerable variance of the blog texts in terms of the textual register and style.