BIGDATA: F: Collaborative Research: Collective Mining of Vertical Social Communities

很大一部分互联网社交媒体内容是在新闻机构托管的数千个专业社区中找到的，通常是以读者论坛或对新闻文章的评论的形式。这样一个网站的用户被认为是一个垂直社交社区(VSC)，因为他们深度接触单一的媒体来源。虽然与Facebook等广泛的社区相比，每个VSC都很小，但它们很重要，因为它们揭示了社会不同阶层对各种世界事件的感受。对于下游情报和预测性分析来说，这可能是非常有用的资源。然而，当前的网络爬虫不能有效地访问VSC。因此，他们的数据对搜索引擎是不可见的，对分析工具也是隐藏的。该项目的目标是使人们能够有效地接触到通过在线新闻报道联合起来的垂直社会社区，并挖掘他们的评论和辩论。该项目将为研究人员提供从这些社区收集数据并对其进行分析的工具。该项目的教育部分包括参与研究生和本科生的培训和研究，并将研究项目和结果纳入课程。研究人员将开发算法来挖掘数千个垂直社交社区产生的内容，并使其内容对数据管理和分析工具透明。研究人员将开发用于内容检测的新的深度学习技术，并构建一个新的可扩展的端到端系统，用于实时访问和集体挖掘这些社区，能够基于不断变化的想法处理大型并行数据流。具体的算法将包括用户群体估计，用于自动爬行内容的引导通信模式，以及用于智能和预测分析的细粒度情感分析。将向学术界和工业界的研究人员提供软件工具。分发用于实施开发的技术的免费、开源软件将增强现有的研究基础设施。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

A large fraction of internet social media content is found in thousands of specialized communities that are hosted by news outlets, typically in the form of reader forums or comments on news articles. The users of the such a site are said to form a vertical social community (VSC), because they deeply engage with a single media source. While each VSC is tiny compared to broad communities such as Facebook, they are important because they expose how different segments of society feel about various world events. This can be a very useful resource for downstream intelligence and predictive analytics. However, current web crawlers cannot effectively access VSCs. Thus their data is invisible to search engines, and remains hidden from analytics tools. The goals of this project are to enable effective access to vertical social communities coalesced at news reports online, and to mine their comments and debates. This project will provide researchers with tools to collect data from these communities and analyze them. The educational component of the project includes the involvement of graduate and undergraduate student training and research and the incorporation of research projects and results in courses.The researchers will develop algorithms to unearth the content generated at thousands of vertical social communities and make their content transparently accessible to data management and analytics tools. The researchers will develop novel deep learning techniques for content detection, and build a novel scalable end-to-end system for real-time access and collective mining of these communities, capable of handling large parallel data streams based on shifting ideas. The specific algorithms will include user population estimation, bootstrap communication patterns for automatic crawling of content, and fine-grained sentiment analysis for intelligence and predictive analytics. Software tools will be made available to researchers in academe and industry. Distribution of free, open-source software for implementing the techniques developed will enhance existing research infrastructure.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.