Construction of Programming Support Environment for Japanese Text Processing

1.日语文本由两种字符集的段落自由混合组成：传统的字母集编码为一个字节单位，日语字符集编码为两个字节单位。目前，还没有标准的形式来表示这两种字符集的混合。首先，我们提出了一个规范化的日语文本，其中每个单字节字符通过附加前导字节编码为两个字节。这段文字允许我们对待这两个字符集统一。2.我们应用字符串匹配算法，这是众所周知的罗马字母世界规范化的日语文本，并比较其性能的实验。因此，我们指出，Boyer-Moore算法，这是非常有效的罗马字母文本显示最差的性能与大字符集的日文文本。这是因为算法所需的表的初始化时间与字符集的大小成比例。我们提出了一种有效的方法，以减少初始化时间，通过表示层次结构的表。我们还开发了一种文本扫描的方法，认为规范化的日语文本只有一系列的一个字节的代码。在该方法中，可能发生误检测。但是，通过使用规范化的日文文本，可以很容易地发现误检，用这种方法实现的Boyer-Moore算法表现出最好的性能.我们构建了包，使其能够在Ada中处理日语文本，而Ada最初不能将日语字符视为预定义字符。使用这些软件包，我们可以很容易地编写处理日语文本的Ada程序。我们开发了一个用于日语文档的写作工具原型，称为SUIKtek，它可以分析机器可读的日语文档，并为作者提供有用的信息来润色它们。

1. The Japanese text is composed of a free mixture of passages in two kinds of character sets: traditional alphabetic set coded into one byte units and Japanese character set coded into two byte units. At present, there is no standard form to express the mixture of these two character sets. To start with, we proposed a normalized Japanese text in which each one-byte character is coded into two bytes by appending a leading byte. This text allows us to treat both character sets uniformly.2. We applied the string matching algorithms which are well known in the Roman alphabetic world to the normalized Japanese texts and compared their performances experimentally. As a result, we pointed out that the Boyer-Moore algorithm which is extremely efficient on Roman alphabetic texts shows the worst performance on Japanese texts with a large character set. This is because the initialization time of the table necessary for the algorithm is proportional to the size of the character set. We presented an efficient method to reduce the initialization time by representing the structure of the table hierarchically. We also developed a text scanning method which regards the normalized Japanese texts as only a series of one byte codes. In this method, mis-detections may occur. But, by using the normalized Japanese text, the mis-detections can be easily found. The Boyer-Moore algorithm implemented by this method shows the best performance.3. We constructed packages to make it possible to handle Japanese texts in Ada which can not originally treat Japanese characters as predefined characters. Using these packages we can write easily Ada programs processing Japanese texts.4. We developed a prototype of writing tools for Japanese documents, called SUIKOU, that analyzes machine-readable Japanese documents textually and provides writers with the useful information for polishing them.