SBIR Phase II:Statistical Inference for Advanced Entity Resolution

该小企业创新研究（SBIR）第二阶段项目旨在更好地集成从异构数据源中提取的有关实体（如人员、公司和产品）的信息。当使用不同的格式和术语来描述同一实体时，这个集成问题可能会很有挑战性。这个问题可以通过统计学习方法来解决，该方法允许系统估计实体引用之间匹配的概率，而不是基于特定规则或权重计算分数。研究的重点是改进这种统计学习方法，使系统能够处理不同类型的现实世界数据。由于该方法基于可靠的统计原则，并使用从大型数据集收集的证据，因此它可以产生比现有商业方法更准确的结果。此外，当处理具有高度可变、缺失或嘈杂属性的数据（例如从网站提取的数据）时，这些优势会被放大。该项目更广泛的影响在于使企业能够执行更准确、更可靠的数据集成。目前，企业往往难以利用从非结构化或半结构化来源提取的数据，因为提取的数据有噪声且难以集成。这种能力对美国一些最大的公司和机构来说至关重要。例如，该项目正在开发的技术将降低整合来自医院和卫生信息提供者的数据的成本。它还可以帮助情报机构在调查公司和个人时更好地将这些点联系起来，并帮助人力资源经理更好地寻找；招聘求职者。最终，这个项目产生的技术将帮助许多类型的企业更好地利用通过Web和专用网络可访问的不断增长的信息量。

This Small Business Innovation Research (SBIR) Phase II project aims to make it possible to do a better job of integrating information about entities, such as people, companies, and products, extracted from heterogeneous data sources. This integration problem can be challenging when different formats and terminology are used to describe the same entity. This problem can be addressed by a statistical learning approach that allows a system to estimate the probability of a match between entity references, rather than computing a score based on ad-hoc rules or weights. The research focuses on refinements to this statistical learning approach that will enable a system to handle diverse types of real-world data. Because the approach is based on sound statistical principles and uses evidence compiled from large datasets, it can produce more accurate results than existing commercial methods. Moreover, these advantages are amplified when handling data that that has highly variable, missing or noisy attributes, such as data extracted from websites.The broader impact of this project lies in enabling enterprises to perform more accurate and reliable data integration. Today, enterprises often have difficulty utilizing data extracted from unstructured or semi-structured source because the extracted data is noisy and difficult to integrate. This capability is critical for some of the nation's largest companies and institutions. For instance, the technology being developed in this project will reduce the cost of integrating data from hospitals and health information providers. It also can help intelligence agencies do a better job of connecting the dots, when investigating companies and individuals, and help human resource managers do a better job of finding;and recruiting job candidates. Ultimately the technology resulting from this project will help many types of enterprises make better use of the growing amount of information accessible through the Web and private networks.