Very Large Semi-Structured Datasets for Social Science Research

在美国国家科学基金会（National Science Foundation）的资助下，康奈尔项目将建立一个社会科学研究实验室，该实验室的基础是一个尚未开发的数据集：互联网档案馆（Internet Archive）的400亿页网页集。近十年来，这些Web快照每两个月被捕获并存档一次。该项目将使这一庞大收藏的很大一部分广泛用于社会科学研究。从企业网页到新闻组和博客，大量可用的在线信息有可能开辟社会科学研究的新领域。康奈尔大学的研究小组计划建立一个智能前端，使社会科学家可以广泛地访问互联网档案，并通过一个特定的研究应用——创新的扩散——来开发、测试和完善这些工具。这些工具的开发需要应用自然语言处理和机器学习算法方面的前沿研究。项目团队还包括在数据隐私保护分析方面具有专业知识的计算机科学家——这是使在线数据更容易用于社会科学研究和政策应用的基本挑战。Web Archive的重要性超越了纯粹的研究，扩展到了商业和政府的实际应用。这些工具可以用来识别市场趋势，需求的上升和下降，以及消费者意见的传播。社区监督组织将能够追踪“仇恨网站”的传播，政府机构将能够追踪过去和现在利用网络组织和协调恐怖袭击的情况。

With funding from the National Science Foundation, The Cornell project will create a research laboratory for social science research based on a largely untapped dataset: the Internet Archive's 40-billion page collection of Web pages. These snapshots of the Web have been captured and archived every two months for nearly ten years. The project will make very large portions of this massive collection widely accessible for social science research. The flood of available on-line information - from corporate web pages to news groups and blogs - has the potential to open up new frontiers in social science research. The Cornell team plans to build an intelligent front-end that will make the Internet Archive broadly accessible to social scientists, and to develop, test, and refine these tools through a specific research application - the diffusion of innovation. The development of such tools requires the application of cutting edge research in natural language processing and machine learning algorithms. The project team also includes computer scientists with expertise in the privacy-preserving analysis of data -- a basic challenge in making on-line data more readily accessible for research and policy applications in the social sciences.The importance of the Web Archive extends beyond pure research to practical applications for business and government. These tools can be used to identify market trends, the rise and fall of demand, and the spread of consumer opinion. Community watchdog groups will be able to track the spread of "hate sites" and government agencies will be able to trace past and current uses of the Web for organizing and coordinating terrorist attacks.