Analysis of all-digitized textbook data and the construction of a large-scale Japanese corpus

点击翻译按钮获取中文摘要

Research resultsIn 2002In 2002, we used morphological analysis techniques to process a Japanese corpus that was made up of academic subjects such as Japanese language (65 files), social studies (209 files), and home economics (11 files) totaling 285 files of digitized textbook data. Because of the relatively high occurrence of hiragana characters in primary school textbooks, the computational morphological analysis produced a great deal of erroneous results. It was required to take a linguistic, albeit manual, approach to analyze the erroneous results and improve the methodology. It was found that the errors were not only caused by the high occurrence of hiragana in primary and secondary school textbooks, but also a lack of linguistic consideration in some parts of the morphological analysis. The types of errors were classified and organized. In 2005, the plan is to refer to this type classification and add more textbook data to the corpus for processing.For the morphological analysis, … More one piece of software that was used was called "Chaser" (developed by the Matsumoto Lab at Nara Advanced Institute of Science and Technology). Additional software was developed to aid in the analysis of the morphologically processed corpus. Improvements were made to the Japanese language analytical software over time as it was used. These software programs were made available as freeware on the Internet.In 2003A Japanese language corpus was created from Japanese textbooks (the contents included Japanese language (65 files), social studies (209 files), and home economics (11 files) for a total of 285 files). The corpus was morphologically processed and the results were analyzed. Some software was developed for this analysis in 2004. This software for Japanese text analysis was made available as freeware in 2004.The software that was developed made it possible to efficiently do analysis of the Japanese language and to collect information from a large-scale corpus to a degree that was not available before. It was used by graduate students to do their own Japanese language research. A book introducing methodology for analysis of Japanese was published, and the software was attached to it as a CD-ROM.Using the textbook corpus, our research also included the analyses of kanji usage attributes, the word as a unit, the contextual usage of words, and the usage ratios of kanji characters in context. We found that the use of kanji characters largely depends on the type of word they appear in. Prior to this research, a definitive study of the usage qualities of kanji, their distribution, and their usage frequencies in written text did not exist.In 2004Further incremental improvements were made on our methodologies. The usage attributes of kanji and words were further analyzed within the paragraph as a unit. English textbooks were collected for the concurrent analysis of an English corpus. Less