ICES: Small: Using Web Crawling and Complex Network Analysis to Understand Institutional Activity and Connectivity

机构是如何组织起来履行它们试图履行的许多职能的，如教育、卫生、基础设施、市场监管或治安？ 他们如何在内部以及与社会上的其他组织进行互动，以执行这些职能并获得对其绩效的反馈，这些活动和连接模式的含义是什么？ 本课题提出，制度是一个复杂的系统，它通过自我组织，为经济和社会提供一系列必要的产品、服务、规则和规章。 它将利用因特网上越来越多的机构来收集和分析第一个全面、客观的数据集，说明各机构所履行的职能以及各机构之间以及与社会不同部分之间的联系。 该项目将开发社区发现，图形挖掘和分析网络爬行的新方法，以转换美国机构网站中的信息，以及这些网站彼此之间以及与社会组织的联系，成为美国机构活动和连通性的试点数据集。 然后，这些数据将用于调查美国机构的活动和连通性的性质和演变，以及它们对公共政策、与企业和社会的关系以及经济生产力和绩效的影响。 该项目将为分析机构活动和连通性的变化开辟新的途径，包括一系列机构联系和连通性的新指标，以及网络爬行、数据挖掘和复杂网络数据分析的新技术。 该项目还将为计算机科学家和经济学家之间的新技术和合作奠定基础。

How do institutions organize to perform the many functions that they seek to perform, such as education, health, infrastructure, market regulation, or policing? How do they interact internally and with other organizations in the society at large to execute these functions and gain feedback on their performance, and what are the implications of these patterns of activity and connectivity? This project proposes that institutions are complex systems that self-organize so as to provide a range of essential products, services, rules, and regulations for the economy and society. It will use the growing presence of institutions on the Internet to gather and analyze the first comprehensive, objective dataset on the functions institutions perform and the connectivity of individual institutions with each other and with different parts of society. The project will develop novel methods for community discovery, graph mining, and analytical web crawling to transform the information in the websites of U.S. institutions, and the linkages of these websites with each other and with societal organizations, into a pilot dataset on the activity and connectivity of institutions in the United States. These data will then be used to investigate the nature and evolution of the activity and connectivity of U.S. institutions, and their implications for public policy, relations with business and society, and economic productivity and performance. The project will open new avenues for analysis of variation in the activity and connectivity of institutions, including a range of new indicators of institutional linkages and connectivity, as well as new techniques for web crawling, data mining, and data analysis of complex networks. The project will also form the basis for new techniques and collaborations between computer scientists and economists.