BIGDATA: F: Collaborative Research: Collective Mining of Vertical Social Communities

互联网社交媒体内容的很大一部分是在新闻媒体托管的数千个专业社区中发现的，这些社区通常以读者论坛或新闻文章评论的形式出现。这种网站的用户被认为形成了一个垂直的社会社区（VSC），因为他们深度参与单一的媒体来源。虽然与Facebook等广泛的社区相比，每个VSC都很小，但它们很重要，因为它们揭示了社会不同阶层对各种世界事件的看法。这对于下游情报和预测分析来说是非常有用的资源。然而，目前的网络爬虫不能有效地访问vsc。因此，他们的数据对搜索引擎来说是不可见的，对分析工具来说也是隐藏的。该项目的目标是使人们能够有效地进入在线新闻报道的垂直社会社区，并挖掘他们的评论和辩论。该项目将为研究人员提供从这些社区收集数据并进行分析的工具。该项目的教育部分包括研究生和本科生的培训和研究，以及将研究项目和结果纳入课程。研究人员将开发算法，以挖掘数千个垂直社交社区产生的内容，并使其内容对数据管理和分析工具透明地访问。研究人员将开发用于内容检测的新型深度学习技术，并为这些社区的实时访问和集体挖掘构建一种新型可扩展的端到端系统，能够处理基于转变思想的大型并行数据流。具体算法将包括用户人口估计、用于自动抓取内容的自引导通信模式，以及用于智能和预测分析的细粒度情感分析。软件工具将提供给学术界和工业界的研究人员。为实现所开发的技术而分发免费的开源软件将加强现有的研究基础设施。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

A large fraction of internet social media content is found in thousands of specialized communities that are hosted by news outlets, typically in the form of reader forums or comments on news articles. The users of the such a site are said to form a vertical social community (VSC), because they deeply engage with a single media source. While each VSC is tiny compared to broad communities such as Facebook, they are important because they expose how different segments of society feel about various world events. This can be a very useful resource for downstream intelligence and predictive analytics. However, current web crawlers cannot effectively access VSCs. Thus their data is invisible to search engines, and remains hidden from analytics tools. The goals of this project are to enable effective access to vertical social communities coalesced at news reports online, and to mine their comments and debates. This project will provide researchers with tools to collect data from these communities and analyze them. The educational component of the project includes the involvement of graduate and undergraduate student training and research and the incorporation of research projects and results in courses.The researchers will develop algorithms to unearth the content generated at thousands of vertical social communities and make their content transparently accessible to data management and analytics tools. The researchers will develop novel deep learning techniques for content detection, and build a novel scalable end-to-end system for real-time access and collective mining of these communities, capable of handling large parallel data streams based on shifting ideas. The specific algorithms will include user population estimation, bootstrap communication patterns for automatic crawling of content, and fine-grained sentiment analysis for intelligence and predictive analytics. Software tools will be made available to researchers in academe and industry. Distribution of free, open-source software for implementing the techniques developed will enhance existing research infrastructure.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.