Analysis of all-digitized textbook data and the construction of a large-scale Japanese corpus

研究成果2002年，我们使用形态分析技术处理了一个日语语料库，该语料库由日语（65个文件）、社会研究（209个文件）和家政学（11个文件）等学科组成，共285个文件的数字化教科书数据。由于平假名字在小学教科书中的出现率较高，计算形态分析产生了大量错误结果。需要采取一种语言学的，尽管是手工的方法来分析错误的结果并改进方法。结果发现，这些错误不仅是由于平假名在中小学教科书中出现较多，而且在词形分析的某些部分缺乏语言学上的考虑。错误的类型被分类和组织。2005年的计划是参照这种类型分类，将更多的教科书数据添加到语料库中进行处理。在形态学分析方面，使用了一款名为“追逐者”的软件（由奈良高等科学技术研究所松本实验室开发）。另外还开发了软件来帮助分析经过形态学处理的语料库。随着时间的推移，日语分析软件得到了改进。这些软件程序在互联网上作为免费软件提供。2003年，以日本教科书（内容包括日语（65档）、社会（209档）、家庭经济学（11档），共285档）为基础，建立了3a日语语料库。对语料库进行形态学处理并对结果进行分析。2004年开发了一些软件来进行分析。这个日语文本分析软件在2004年作为免费软件提供。开发的软件使有效地分析日语和从大规模语料库中收集信息成为可能，这在很大程度上是前所未有的。它被研究生用来做他们自己的日语研究。出版了介绍日语分析方法的书，并附带了软件光盘。利用教材语料库，对汉字使用属性、词为单位、词的语境用法以及汉字在语境中的使用比例进行了分析。我们发现汉字的使用很大程度上取决于它们出现在什么类型的单词中。在此研究之前，对汉字的使用质量、分布及其在书面文本中的使用频率没有明确的研究。二零零四年，我们在方法上有进一步的改进。以段落为单位，进一步分析了汉字和词语的使用属性。收集英语教材，对英语语料库进行并行分析。少

Research resultsIn 2002In 2002, we used morphological analysis techniques to process a Japanese corpus that was made up of academic subjects such as Japanese language (65 files), social studies (209 files), and home economics (11 files) totaling 285 files of digitized textbook data. Because of the relatively high occurrence of hiragana characters in primary school textbooks, the computational morphological analysis produced a great deal of erroneous results. It was required to take a linguistic, albeit manual, approach to analyze the erroneous results and improve the methodology. It was found that the errors were not only caused by the high occurrence of hiragana in primary and secondary school textbooks, but also a lack of linguistic consideration in some parts of the morphological analysis. The types of errors were classified and organized. In 2005, the plan is to refer to this type classification and add more textbook data to the corpus for processing.For the morphological analysis, … More one piece of software that was used was called "Chaser" (developed by the Matsumoto Lab at Nara Advanced Institute of Science and Technology). Additional software was developed to aid in the analysis of the morphologically processed corpus. Improvements were made to the Japanese language analytical software over time as it was used. These software programs were made available as freeware on the Internet.In 2003A Japanese language corpus was created from Japanese textbooks (the contents included Japanese language (65 files), social studies (209 files), and home economics (11 files) for a total of 285 files). The corpus was morphologically processed and the results were analyzed. Some software was developed for this analysis in 2004. This software for Japanese text analysis was made available as freeware in 2004.The software that was developed made it possible to efficiently do analysis of the Japanese language and to collect information from a large-scale corpus to a degree that was not available before. It was used by graduate students to do their own Japanese language research. A book introducing methodology for analysis of Japanese was published, and the software was attached to it as a CD-ROM.Using the textbook corpus, our research also included the analyses of kanji usage attributes, the word as a unit, the contextual usage of words, and the usage ratios of kanji characters in context. We found that the use of kanji characters largely depends on the type of word they appear in. Prior to this research, a definitive study of the usage qualities of kanji, their distribution, and their usage frequencies in written text did not exist.In 2004Further incremental improvements were made on our methodologies. The usage attributes of kanji and words were further analyzed within the paragraph as a unit. English textbooks were collected for the concurrent analysis of an English corpus. Less