Collaborative Research: Large-scale Database Construction of Firms' Organizational Form, Competition, and Industry Change

本项目拟通过建立一个由100多万家私营和公营企业的互联网网页汇编而成的大规模数据库，以及一个基于网络的分析工具，以提供对该数据库的有效访问，从而分析组织和产业如何随着时间的推移而变化。这个大型数据库将包含过去20年来公共和私营公司直接向客户提供的产品和服务的信息，以及过去20年来美国专利商标局专利数据的链接。由此产生的数据库和分析工具将使研究人员能够回答许多问题，包括：产业、产业竞争及其产品如何随着时间的推移而变化？在最近的金融危机期间，私营和公共企业的产品推出率动态如何？私营企业及其产品系列如何变化？随着一个行业内专利活动的增加，公共和私营公司推出了哪些产品？专利公司还是非专利公司推出新产品？政府的哪些政策变化在刺激创业型公司的增长方面最为有效，这些政策在什么样的市场中效果最好？什么样的当地产品市场条件最有利于创业公司成功进入市场，创新浪潮如何影响产品市场竞争？什么样的经济力量促使企业跨越公私地位的界限？除了影响研究创新和行业如何随着时间的推移而变化的学者外，商业决策者，消费者和监管机构将从新的行业指定中受益。企业可以使用数据库和基于网络的工具来评估围绕新产品的现有市场结构。这将有助于更明智地决定在何处和何时投入稀缺资源进入新市场。企业家还可以通过审查公营和私营公司，无论大小，现有竞争的性质和市场结构，更好地评估其新企业可能取得的成功。包括证券交易委员会（SEC）和司法部（DOJ）在内的监管机构也可以从行业结构和产品市场边界的精确知识中受益。该项目将产生一个高度可扩展的方法来挖掘历史Web数据，以创建全面的基于产品的数据库和工具来查询和分析这些数据。这一综合数据库将使用互联网档案Wayback Machine项目的固定网页来建立。使用这些网页的文本，所产生的数据库将把公司分类为竞争对手，这将用于建立新的行业定义。这些新的行业定义将直接基于公司用于与客户交互的产品和服务描述。该项目将在“Elasticsearch”（www.elasticsearch.com）的基础上制作一个公开提供的网上分析工具。该工具将允许用户查询公共和私营公司、其竞争对手及其提供的产品的数据库。使用该工具，用户将能够输入公司或公司列表，并且该工具将提供该公司的产品描述和其生产类似产品的竞争者的列表，连同相应的相似性分数沿着，指示竞争链接的强度。该分析工具还将使公司的当地产品市场成为一个网络。除了托管的在线搜索工具外，用户还可以下载批量数据集以执行自己的本地处理。

This project proposes to analyze how organizations and industries change over time by building a large-scale database compiled from Internet web pages of over 1,000,000 private and public firms, and an analytical web-based tool to provide efficient access to this database. This large-scale database will contain information on products and services directly offered by public and private firms to their customers over the last 20 years, as well as links to the US Patent and Trademark Office's patent data for the last 20 years. The resulting database and analysis tool will enable researchers to answer many questions, including: How do industries, competition in industries and their products change over time? What are the dynamics of product introduction rates for private and public firms and how did private firms and their array of product offerings change during the recent financial crisis? What products are introduced by public and private firms following increases in patenting activity within an industry? Do the patenting firms introduce the new products or do non-patenting firms? Which government policy changes were most effective in stimulating the growth of entrepreneurial firms, and in what kind of markets did these policies work best? What local product market conditions are most conducive to successful entry by entrepreneurial firms and how do waves of innovation impact product market competition? What economic forces trigger firms to cross the boundary between public and private status? In addition to impacting academics that study innovation and how industries change over time, business decision makers, consumers, and regulators will benefit from the new industry designations. Businesses can use the database and web based tool to assess existing market structure around new products. This will facilitate more informed decisions about where and when to commit scarce resources to enter new markets. By examining the nature of existing competition and market structure by both public and private firms, large and small, entrepreneurs can also better assess the likely success of their new ventures. Regulators including the Securities and Exchange Commission (SEC) and the Department of Justice (DOJ) can also benefit from refined knowledge of industry structure and product market boundaries. This project will produce a highly scalable approach for mining historical Web data to create comprehensive product-based databases and tools to query and analyze this data. This integrated database will be built using firm web pages from the Internet Archive Wayback Machine project. Using the text from these web pages, the resulting database will classify firms as competitors, which will be used to build new industry definitions. These new industry definitions will be based directly on the product and service descriptions firms use to interface with customers. The project will produce a publicly available web-based analytical tool based on "Elasticsearch" (www.elasticsearch.com). This tool will allow users to query the database of public and private firms, their competitors, and the products they offer. Using this tool, a user will be able to input a company, or list of companies, and the tool will provide that company's product descriptions and a list of its competitors that produce similar products, along with the corresponding similarity scores, indicating the strength of competitive links. The analytical tool will also allow a firm's local product market to be visualized as a network. In addition to the hosted, online search tool, users will be able to download bulk datasets to perform their own local processing.