Collaborative Research: Large-scale Database Construction of Firms' Organizational Form, Competition, and Industry Change

该项目建议通过建立一个大型数据库来分析组织和行业如何随时间变化，该数据库由100多万家私营和上市公司的互联网网页汇编而成，并建立一个基于网络的分析工具来提供对该数据库的有效访问。这个大型数据库将包含过去20年中由公共和私营公司直接向其客户提供的产品和服务的信息，以及与美国专利和商标局过去20年的专利数据的链接。由此产生的数据库和分析工具将使研究人员能够回答许多问题，包括：工业、工业中的竞争及其产品如何随时间变化？在最近的金融危机中，私营和上市公司的产品引进率的动态是什么？私营公司及其产品供应的变化是怎样的？随着行业内专利活动的增加，公共和私营公司推出了哪些产品？是专利公司引进新产品还是非专利公司引进新产品？哪些政府政策变化在刺激创业公司的成长方面最有效，这些政策在什么样的市场中效果最好？什么样的本地产品市场条件最有利于创业公司的成功进入？创新浪潮如何影响产品市场竞争？是什么经济力量促使企业跨越公私地位的界限？除了影响研究创新和行业如何随时间变化的学者外，商业决策者、消费者和监管机构也将从新的行业名称中受益。企业可以使用数据库和基于网络的工具来评估围绕新产品的现有市场结构。这将有助于就何时何地投入稀缺资源进入新市场作出更明智的决定。通过审查大小公共和私营公司现有竞争的性质和市场结构，企业家还可以更好地评估其新企业可能取得的成功。包括证券交易委员会（SEC）和司法部（DOJ）在内的监管机构也可以从行业结构和产品市场边界的精确知识中受益。该项目将产生一种高度可扩展的方法，用于挖掘历史Web数据，以创建全面的基于产品的数据库和工具来查询和分析这些数据。这个集成的数据库将使用来自Internet Archive Wayback Machine项目的公司网页来构建。利用这些网页上的文本，生成的数据库将把企业归类为竞争对手，这将用于建立新的行业定义。这些新的行业定义将直接基于公司用于与客户交互的产品和服务描述。该项目将基于“Elasticsearch”（www.elasticsearch.com）生成一个公开可用的基于web的分析工具。这个工具将允许用户查询数据库的公共和私营企业，他们的竞争对手，以及他们提供的产品。使用此工具，用户将能够输入一个公司或公司列表，该工具将提供该公司的产品描述及其生产类似产品的竞争对手列表，以及相应的相似度分数，表明竞争链接的强度。该分析工具还可以将公司的本地产品市场可视化为一个网络。除了托管的在线搜索工具之外，用户还可以下载批量数据集来执行自己的本地处理。

This project proposes to analyze how organizations and industries change over time by building a large-scale database compiled from Internet web pages of over 1,000,000 private and public firms, and an analytical web-based tool to provide efficient access to this database. This large-scale database will contain information on products and services directly offered by public and private firms to their customers over the last 20 years, as well as links to the US Patent and Trademark Office's patent data for the last 20 years. The resulting database and analysis tool will enable researchers to answer many questions, including: How do industries, competition in industries and their products change over time? What are the dynamics of product introduction rates for private and public firms and how did private firms and their array of product offerings change during the recent financial crisis? What products are introduced by public and private firms following increases in patenting activity within an industry? Do the patenting firms introduce the new products or do non-patenting firms? Which government policy changes were most effective in stimulating the growth of entrepreneurial firms, and in what kind of markets did these policies work best? What local product market conditions are most conducive to successful entry by entrepreneurial firms and how do waves of innovation impact product market competition? What economic forces trigger firms to cross the boundary between public and private status? In addition to impacting academics that study innovation and how industries change over time, business decision makers, consumers, and regulators will benefit from the new industry designations. Businesses can use the database and web based tool to assess existing market structure around new products. This will facilitate more informed decisions about where and when to commit scarce resources to enter new markets. By examining the nature of existing competition and market structure by both public and private firms, large and small, entrepreneurs can also better assess the likely success of their new ventures. Regulators including the Securities and Exchange Commission (SEC) and the Department of Justice (DOJ) can also benefit from refined knowledge of industry structure and product market boundaries. This project will produce a highly scalable approach for mining historical Web data to create comprehensive product-based databases and tools to query and analyze this data. This integrated database will be built using firm web pages from the Internet Archive Wayback Machine project. Using the text from these web pages, the resulting database will classify firms as competitors, which will be used to build new industry definitions. These new industry definitions will be based directly on the product and service descriptions firms use to interface with customers. The project will produce a publicly available web-based analytical tool based on "Elasticsearch" (www.elasticsearch.com). This tool will allow users to query the database of public and private firms, their competitors, and the products they offer. Using this tool, a user will be able to input a company, or list of companies, and the tool will provide that company's product descriptions and a list of its competitors that produce similar products, along with the corresponding similarity scores, indicating the strength of competitive links. The analytical tool will also allow a firm's local product market to be visualized as a network. In addition to the hosted, online search tool, users will be able to download bulk datasets to perform their own local processing.