Construction of Programming Support Environment for Japanese Text Processing

1. 日文文本由两种字符集的自由混合组成：编码为一个字节单位的传统字母集和编码为两个字节单位的日文字符集。目前还没有一个标准的形式来表示这两种字符集的混合。首先，我们提出了一种规范化的日语文本，其中每个一字节字符通过附加一个前导字节被编码为两个字节。这个文本允许我们统一地对待这两个字符集。我们将罗马字母世界中著名的字符串匹配算法应用于规范化的日语文本，并对其性能进行了实验比较。结果，我们指出，Boyer-Moore算法在罗马字母文本上非常有效，但在具有大字符集的日语文本上表现最差。这是因为算法所需的表初始化时间与字符集的大小成正比。提出了一种通过分层表示表结构来减少初始化时间的有效方法。我们还开发了一种文本扫描方法，该方法将规范化的日文文本仅视为一系列一个字节的代码。在这种方法中，可能会出现误检测。但是，通过使用规范化的日语文本，很容易发现错误的检测。用该方法实现的Boyer-Moore算法表现出最好的性能。我们构建了一些包，使得在Ada中处理原本不能将日文字符视为预定义字符的日文文本成为可能。使用这些软件包，我们可以很容易地编写处理日文文本的Ada程序。我们开发了一个日语文档写作工具的原型，称为SUIKOU，它可以从文本上分析机器可读的日语文档，并为作者提供有用的信息来润色它们。

1. The Japanese text is composed of a free mixture of passages in two kinds of character sets: traditional alphabetic set coded into one byte units and Japanese character set coded into two byte units. At present, there is no standard form to express the mixture of these two character sets. To start with, we proposed a normalized Japanese text in which each one-byte character is coded into two bytes by appending a leading byte. This text allows us to treat both character sets uniformly.2. We applied the string matching algorithms which are well known in the Roman alphabetic world to the normalized Japanese texts and compared their performances experimentally. As a result, we pointed out that the Boyer-Moore algorithm which is extremely efficient on Roman alphabetic texts shows the worst performance on Japanese texts with a large character set. This is because the initialization time of the table necessary for the algorithm is proportional to the size of the character set. We presented an efficient method to reduce the initialization time by representing the structure of the table hierarchically. We also developed a text scanning method which regards the normalized Japanese texts as only a series of one byte codes. In this method, mis-detections may occur. But, by using the normalized Japanese text, the mis-detections can be easily found. The Boyer-Moore algorithm implemented by this method shows the best performance.3. We constructed packages to make it possible to handle Japanese texts in Ada which can not originally treat Japanese characters as predefined characters. Using these packages we can write easily Ada programs processing Japanese texts.4. We developed a prototype of writing tools for Japanese documents, called SUIKOU, that analyzes machine-readable Japanese documents textually and provides writers with the useful information for polishing them.