ITR: Mining Text for General World Knowledge

尽管近年来在语音识别生成技术和统计语言建模方面取得了重大进展，但现有的自然语言系统仍然局限于非常具体、狭窄的领域，完全缺乏常识——在与用户交互时“看到显而易见的东西”的能力。造成这种情况的一个主要原因是，在当前的人工智能系统中，缺乏广泛的通用世界知识基础——比如三明治是食物的知识。人类)，而餐具不是；住宅通常有门和墙；或者，当一个人被另一个人杀害时，通常是用枪；等。本项目将利用先前从文本中挖掘语言知识的工作作为跳板，解决从文本中挖掘一般世界知识的问题。该方法既不依赖于“深度”文本理解，也不依赖于目标语料库中所期望的一般事实的明确出现。相反，PI的方法详细阐述了这样一种观点，即在文本的预测模式中观察到的规律通常反映了世界的规律，特别是某些类型的实体共同参与各种事件和关系的方式的规律。虽然这种模式的绝对统计频率可能会产生严重的误导（人们犯罪、发生事故或担任公职的频率几乎不像浏览报纸所暗示的那样频繁），但将采用的技术依赖于条件频率来获得事实可靠的假设。所提取的知识将以一种正式的可解释的命题形式投射，使其适合于确定和不确定的推理。反过来，这将有助于“净化”提取的知识，通过揭示和帮助纠正明显的矛盾。适合这项工作的语料库不仅包括报纸和其他事实来源，还包括现实主义小说和儿童作品——事实上，几乎所有电子可访问的文本都可能有用，不需要注释。虽然不是所有的常识都可以通过这种方式获得，但是可以获得的知识是非常广泛的，对于语言理解和常识推理是必不可少的，而且相对来说是近在咫尺的。从文本语料库中挖掘的那种一般知识不仅有用，而且从长远来看，对于具有一些一般语言能力和一点点常识的智能系统来说是必不可少的。因此，这项工作将使计算机真正理解其用户的前景更近一步。

Despite significant advances in recent years in speech recognition generation technology and statistical language modeling, existing natural language systems are still limited to very specific, narrow domains, and totally lack common sense - the ability to "see the obvious" when interacting with a user. A major reason for this is the lack of a broad base of general world knowledge in current AI systems - knowledge such as that a sandwich is food (for. humans), while dinnerware is not; that dwellings usually have doors and walls; or, that when one person is killed by another, it is often with a gun; etc. This project will use previous work on mining linguistic knowledge from text as a springboard for tackling the problem of mining general world knowledge from texts. The methodology depends neither on "deep" text understanding nor on explicit occurrence of the desired general facts in the targeted corpora. Rather, the PI's approach elaborates on the idea that regularities observed in patterns of predication in texts generally reflect regularities in the world, particularly regularities in the way certain types of entities jointly participate in various events and relationships. While absolute statistical frequencies of such patterns can be severely misleading (people do not commit crimes, or have accidents or hold public office nearly as often as scanning of newspapers might suggest), the techniques that will be employed rely on conditional frequencies to obtain factually reliable hypotheses. The knowledge extracted will be cast in a formally interpretable propositional form, lending itself to certain and uncertain inference. This in turn will help "sanitize" the extracted knowledge, by revealing and helping to remedy apparent contradictions. Suitable corpora for this work include not only newspapers and other factual sources, but also realistic novels and writings for children - in fact, almost all electronically accessible texts are potentially useful, and no annotation will be required. While not all kinds of common-sense knowledge can be acquired in this way, the knowledge that can be acquired is very extensive, is essential to language understanding and common-sense reasoning, and is relatively close at hand. The kind of general knowledge to be mined from text corpora is not only useful, but essential in the long run for intelligent systems with some general linguistic competence and a modicum of common sense. Thus the work will bring a step closer the prospect of computers that genuinely understand their users.