SBIR Phase I: Statistical Inference for Advanced Entity Resolution

该小型企业创新研究(SBIR)第一阶段项目解决了从大量数据源集成有关命名实体的信息的问题，例如人员、公司和产品。整合来自多个来源的有关实体的信息可能很困难，因为来源可能使用不同的格式和术语来描述同一实体，这一问题称为“实体解析”。大多数现有的商业企业系统依赖于基于规则的匹配技术来进行实体解析。这个项目研究的是统计学习技术，它允许系统估计匹配的概率，而不是基于特别规则或权重计算分数。由于该方法基于可靠的统计原理，并使用了从大数据集中汇编的证据，因此它可以比现有方法产生更准确的结果。此外，当处理具有高度可变、缺失或噪声属性的数据时，这些优势被放大，例如从网站提取的数据。该项目更广泛的影响/商业潜力在于使企业能够进行更准确、更可靠的数据集成。有许多潜在的目标市场需要更好的技术来集成有关企业、产品、人员、地点和其他实体的信息。这一能力对美国一些最大的公司和机构至关重要，从搜索引擎到美国情报和执法部门，再到金融机构。特别是，大型企业往往难以利用从新闻、外语数据源和社交媒体提取的数据，因为提取的数据噪声大，结构不好。该项目开发的技术将帮助企业利用Web上日益增长的信息量，以便它们能够利用将人员、公司和其他实体联系在一起的关系网络，以更好地为客户服务。

This Small Business Innovation Research (SBIR) Phase I project addresses the problem of integrating information about named entities, such as people, companies, and products, from numerous data sources. Integrating information about entities from multiple sources can be difficult because sources may use different formats and terminology to describe the same entity, a problem referred to as "entity resolution". Most existing commercial enterprise systems rely on rule-based matching techniques for entity resolution. This project investigates statistical learning techniques that allow a system to estimate the probability of a match, rather than computing a score based on ad-hoc rules or weights. Because the approach is based on sound statistical principles and uses evidence compiled from large datasets, it can produce more accurate results than existing methods. Moreover, these advantages are amplified when handling data that that has highly variable, missing or noisy attributes, such as data extracted from Web sites. The broader impact/commercial potential of this project lies in enabling enterprises to perform more accurate and reliable data integration. The are many potential target markets that need better technology for integrating information about businesses, products, people, locations, and other entities. This capability is critical for some of the nation's largest companies and institutions, from search engines, to the U.S. Intelligence and law enforcement community, to financial institutions. In particular, large enterprises often have difficulty utilizing data extracted from news, foreign language data sources, and social media, because the extracted data is noisy and not-well structured. The technology developed in this project will help enterprises make use of the growing amount of information on the Web, so that they can take advantage of the network of relationships that link people, companies, and other entities to serve their customers better.