Novel statistical models for text mining with applications to Chinese history and texts

在本项目中，研究人员研究了一系列具有挑战性的中文文本信息提取问题，包括：（1）词/短语发现，（2）文本分割，（3）技术术语识别，（4）技术术语之间的关联发现。与英语等字母语言不同，汉语有许多特殊的性质：没有词的边界，没有明确的词的定义，传统上没有标点符号，以及独特的语法。因此，将大多数为字母语言开发的方法直接应用于中文是有问题的。此外，现有的中文文本分析方法也存在诸多局限性。相反，研究人员提出了一种先进的单词词典模型（AWDM），可以同时实现单词发现，文本分割和技术术语识别，这些传统上是分开研究的。其思想是首先通过从文本中枚举满足一定标准的所有候选词来建立一个词词典，并为每个候选词分配一个潜在的词类型标签，该标签表示不同类型的技术术语（如姓名、地址、办公室头衔、时间标签以及背景文本）和相应的词使用频率。然后，一个马尔可夫依赖模型之间的不同的词和词类型的模型的潜在的语法和语义结构的文本。在训练数据的帮助下（即，已知技术术语列表），AWDM可以自动从候选词的巨大空间中选择最有意义的词，不仅基于词的内容而且基于词周围的上下文来确定每个词的词类型，并且基于语法和语义信息来分割文本。与现有的方法相比，AWDM由于对语法和语义信息的联合建模，以及对词发现、文本切分和术语识别的综合分析，具有更高的效率。该方法结合主题模型、主题词典模型等文本挖掘工具，将形成一个强大的多层次（汉字级、词/短语级、主题级、主题级）中文文本分析平台。例如，许多以繁体中文撰写的中国历史文献现在都以数字形式提供;而公共媒体，如报纸、论坛、博客和微博，每天都在产生大量的中文文本。因此，开发文本挖掘工具来自动从这些数据中提取信息并创建新知识具有很大的吸引力。这个项目的思想和方法可能会对中国历史的研究产生重大影响。从不断增长的数字化历史文件数据库中提取信息的有效和可靠的方法将使研究人员能够根据大量的分类数据点分析随时间的变化，这在过去是不切实际的。此外，尽管这些方法最初是为中文设计的，但它们有可能应用于与中文类似的其他亚洲语言，如日语和韩语，从而为亚洲历史研究提供一个强大的多语言平台。此外，该项目中研究的识别命名实体的挑战的新方法也有可能扩展到英语等字母语言。最后，本研究的思想和方法有可能被推广为一个系统化的工具，它可以识别任何中文文本的数据流，并输出一个结构化的数据库，其中包含输入数据所描述的个人和组织的关键信息，从而使研究人员更容易发现我们社会生活中各种“单元”的社会网络。我们的算法发现的各种项目关联模式对公共媒体和社会学的研究也是非常宝贵的，并可能有助于及时揭示新的重要流行病学事件和社会趋势。这些类型的信息可能对商业决策和政府政策制定产生重要影响。

In this project, the investigators study a series of challenging problems of extracting information from Chinese text, including: (1) word/phrase discovery, (2) text segmentation, (3) technical term recognition, and (4) association discovery among technical terms. Different from alphabetical languages such as English, Chinese has many special properties: no word boundaries, no clear definition of words, traditionally no punctuation, and a unique grammar. Thus, it is problematic to apply most methods developed for alphabetical languages directly to Chinese. Moreover, the available methods for analyzing Chinese text in the literature have many limitations. Instead the investigators propose an advanced word dictionary model (AWDM) that can simultaneously achieve word discovery, text segmentation and technical term recognition, which are traditionally studied separately. The idea is to build up a word dictionary first by enumerating all word candidates satisfying a certain criterion from the texts and assign to each word candidate a latent word type label representing different types of technical terms (such as names, addresses, office titles, time labels, as well as background texts) and corresponding word usage frequencies. Then, a Markov dependence model among different words and word types is given to model the potential grammatical and semantic structure of the texts. With the help of the training data (i.e., lists of known technical terms), the AWDM can automatically select the most meaningful words from the huge space of word candidates, determine the word type for each word based on not only the content of the word but also the context around the word, and segment the texts based on both grammatical and semantic information. Compared to the existing methods in the literature, the AWDM enjoys a better efficiency due to the joint modeling of the grammatical and semantic information and the integrated analysis of word discovery, text segmentation and technical term recognition. Combined with other text mining tools, such as topic models and theme dictionary models, the proposed method will lead to a powerful multi-level (Chinese character level, word/phrase level, theme level, topic level) analysis platform for Chinese texts.With the explosive growth of the internet and digital technologies, large quantities of digitalized Chinese texts can be easily collected. For example, lots of Chinese historical documents written in traditional Chinese are now available in digital form; and, public media such as new papers, forums, blogs and microblogs, are producing huge amounts of Chinese text every day. Thus there is great appeal in developing text mining tools to automatically extract information from these data and create new knowledge. The ideas and approaches in this project may have significant impacts on how Chinese history will be studied. An efficient and reliable method for extracting information from the ever growing databases of digitized historical documents will enable researchers to analyze change over time based on large numbers of disaggregated data points, something impractical in the past. Furthermore, although originally designed for Chinese, these approaches have the potential to be applied to other Asian languages similar to Chinese, such as Japanese and Korean, and thus provide a powerful multi-language platform for the study of Asian history. In addition, the novel way of combatting the challenges in recognizing named entities studied in this project also has the potential to be extended to alphabetical languages such as English. Finally, the ideas and approaches studied in this project have the potential to be generalized into a systematic tool that digests any data flow of Chinese texts, and outputs a structured database that contains key information about the individuals and organizations described by the input data, thus making it easier for researchers to discover social network of all kinds of "units" in our social life. Various item association patterns discovered by our algorithms are also invaluable to the study of public media and sociology, and may help reveal new important epidemiological events and societal trends in a timely fashion. These types of information can have important implications in business decision making and governmental policy making.