Analysis of all-digitized textbook data and the construction of a large-scale Japanese corpus

研究结果2002年，我们使用形态分析技术对日语语料库进行了处理，语料库由日语(65个文件)、社会研究(209个文件)和家政学(11个文件)等学科组成，共计285个数字化教科书数据。由于小学教科书中平假名字符出现的频率较高，计算形态分析产生了大量的错误结果。它被要求采取语言学的方法，尽管是手工的，来分析错误的结果并改进方法。研究发现，这些偏误不仅是由于中小学教材中平假名的出现频率较高，而且在词法分析的某些部分也缺乏语言学上的考虑。对错误类型进行了分类和整理。2005年，计划参照这一类型分类，在语料库中增加更多的教科书数据进行处理。对于词法分析，…有一款以上的软件被称为“Chaser”(由奈良高等科学技术学院松本实验室开发)。另外还开发了辅助分析经形态处理的语料库的软件。日语分析软件在使用过程中随着时间的推移得到了改进。这些软件程序在互联网上以免费软件的形式提供。2003年，根据日语教科书创建了日语语料库(内容包括日语(65个文件)、社会研究(209个文件)和家政学(11个文件)，共285个文件)。对语料进行了形态处理，并对结果进行了分析。2004年为这一分析开发了一些软件。这个日语文本分析软件于2004年以免费软件的形式提供。开发的软件使人们能够有效地对日语进行分析，并从大规模语料库中收集信息，达到了以前没有的程度。它被研究生用来做他们自己的日语研究。出版了一本介绍日语分析方法的书，并将该软件附在光盘上。利用教科书语料库，我们的研究还包括汉字用法属性、词作为一个单元、词的语境用法以及汉字在语境中的使用率的分析。我们发现，汉字字符的使用在很大程度上取决于它们出现的单词类型。在此之前，还没有关于汉字在书面文本中的使用质量、分布和使用频率的明确研究。2004年，我们的方法得到了进一步的改进。作为一个单位，进一步分析了汉字和词语的用法属性。收集的英语教材用于对英语语料库的同时分析。较少

Research resultsIn 2002In 2002, we used morphological analysis techniques to process a Japanese corpus that was made up of academic subjects such as Japanese language (65 files), social studies (209 files), and home economics (11 files) totaling 285 files of digitized textbook data. Because of the relatively high occurrence of hiragana characters in primary school textbooks, the computational morphological analysis produced a great deal of erroneous results. It was required to take a linguistic, albeit manual, approach to analyze the erroneous results and improve the methodology. It was found that the errors were not only caused by the high occurrence of hiragana in primary and secondary school textbooks, but also a lack of linguistic consideration in some parts of the morphological analysis. The types of errors were classified and organized. In 2005, the plan is to refer to this type classification and add more textbook data to the corpus for processing.For the morphological analysis, … More one piece of software that was used was called "Chaser" (developed by the Matsumoto Lab at Nara Advanced Institute of Science and Technology). Additional software was developed to aid in the analysis of the morphologically processed corpus. Improvements were made to the Japanese language analytical software over time as it was used. These software programs were made available as freeware on the Internet.In 2003A Japanese language corpus was created from Japanese textbooks (the contents included Japanese language (65 files), social studies (209 files), and home economics (11 files) for a total of 285 files). The corpus was morphologically processed and the results were analyzed. Some software was developed for this analysis in 2004. This software for Japanese text analysis was made available as freeware in 2004.The software that was developed made it possible to efficiently do analysis of the Japanese language and to collect information from a large-scale corpus to a degree that was not available before. It was used by graduate students to do their own Japanese language research. A book introducing methodology for analysis of Japanese was published, and the software was attached to it as a CD-ROM.Using the textbook corpus, our research also included the analyses of kanji usage attributes, the word as a unit, the contextual usage of words, and the usage ratios of kanji characters in context. We found that the use of kanji characters largely depends on the type of word they appear in. Prior to this research, a definitive study of the usage qualities of kanji, their distribution, and their usage frequencies in written text did not exist.In 2004Further incremental improvements were made on our methodologies. The usage attributes of kanji and words were further analyzed within the paragraph as a unit. English textbooks were collected for the concurrent analysis of an English corpus. Less