II-NEW: Collaborative Research: Spam Processing, Archiving, and Monitoring Community Facility (SPAM Commons)

在这个项目中，PI建议构建和开发一个共享的基础设施，以支持收集和维护现实的、大规模的垃圾数据集，称为垃圾公有。垃圾邮件是许多重要的通信媒体，如电子邮件和网络中的一个问题。垃圾邮件的一个子问题，网络钓鱼(一种在线借口)，在2007年造成了大约32亿美元的损失。有效的垃圾邮件过滤方法在电子邮件和网络等几种通信媒体中的广泛影响可以估计为数十亿美元。垃圾邮件还入侵了其他媒体，具体攻击例子包括社交网络、博客圈、互联网电话(VoIP)、即时消息和点击欺诈。不幸的是，由于对隐私和公司知识产权的担忧，缺乏已公布的真实世界数据集，阻碍了垃圾邮件的研究。该项目组开发了一个共享基础设施，以支持收集和维护实际的大规模垃圾邮件数据集，称为垃圾邮件处理、归档和监控社区设施(Spam Commons)。垃圾邮件共享的主要目标是：(1)促进补救研究，以阻止垃圾邮件造成的浪费和损失；(2)使旨在完全阻止某些类型的垃圾邮件攻击的革命性研究成为可能。公共分区是用于语音和图像识别研究的标准语料库的直接模拟，包括对各种通信媒体中的垃圾邮件和合法数据的系统和定期收集，从电子邮件和网络垃圾开始，随着垃圾邮件在每个领域成为严重威胁和数据变得可用，扩展到其他通信媒体。受保护分区包括一个组合的数据和处理设施，使私有数据或接近实时的垃圾数据可用于在受保护的试验台中对垃圾邮件防御机制进行试验性评估。对这些受保护数据的访问将使对实时演变的垃圾邮件和现实世界数据集进行新的垃圾邮件研究成为可能，这在今天是不可行的。垃圾邮件共享项目的智力挑战超出了对上述各种垃圾邮件领域的新研究，这些领域是由可用的数据集实现的。垃圾邮件共享的两个分区的构建都包括它们自己的重大智力挑战。首先，隔离受保护分区在一定程度上解决了隐私问题，这仍然是一个普遍的研究问题。其次，有用的垃圾邮件和合法数据集需要自动确定地区分垃圾邮件和合法文档，这在电子邮件、网络和其他媒体中仍然是一个悬而未决的研究问题。第三，垃圾邮件生产者和防御者之间的对抗性和相互进化需要不断收集新的数据以供进一步研究。最后，收集和传输近乎实时的垃圾邮件数据是垃圾邮件研究人员目前无法获得的研究资源。这些领域的进展将刺激垃圾公有数据的增长和演变，从而使新的研究能够对不断演变和不断增长的垃圾邮件问题进行新的研究。垃圾公有数据集对实验性垃圾邮件研究的影响可能类似于语音/图像识别和自然语言处理等学科的大型语料库的影响，在使用此类语料库成为标准要求后，这些学科实现了一定程度的科学结果重复性和可比性。9个大学合作伙伴(克莱顿州立大学、埃默里大学、佐治亚理工学院、北卡罗来纳大学、西北大学、德克萨斯州农工大学、加州大学戴维斯分校、乔治亚州大学、北卡罗来纳州大学夏洛特分校)以及几个行业合作伙伴(IBM、Purewire、安全计算)将支持和使用拟议的数据存储库。

In this project, the PIs propose to construct and develop a shared infrastructure to support the collection and maintenance of realistic, large scale spam data sets, referred as SPAM Commons.Spam is a problem in many important communications media such as email and web. A sub-problem of spam, phishing (a form of online pretexting), caused an estimated $3.2B in damages in 2007. The broad impact of effective spam filtering methods can be estimated in billions of dollars in several communications media such as email and web.Spam has also invaded other media, with concrete attack examples in social networks, blogosphere, Internet telephony (VoIP), instant messaging, and click fraud. Unfortunately, spam research has been hampered by the lack of published real world data sets due to concerns with privacy and company intellectual property. This project team develops a shared infrastructure to support the collection and maintenance of realistic, large scale spam data sets, called Spam Processing, Archiving, and Monitoring Community Facility (SPAM Commons). The main goals of SPAM Commons are: (1) to facilitate remedial research that will stem the wastes and losses caused by spam, and (2) enable revolutionary research that aim for stopping certain kinds of spam attacks altogether. SPAM Commons is divided into a Public Partition and a Protected Partition.The Public Partition is a direct analog of standard corpora for speech and image recognition research, consisting of a systematic and regular collection of both spam and legitimate data in the various communications media, starting from email and web spam, and expanding into other communications media as spam becomes a serious threat in each area and data become available. The Protected Partition consists of a combined data and processing facility that makes private data or near real-time spam data available for experimental evaluation of spam defense mechanisms in a protected testbed. Access to such protected data will enable new spam research on real-time evolving spam and real world data sets that is infeasible today. The intellectual challenges of the SPAM Commons project extend beyond the new research on various abovementioned spam areas enabled by the availability of data sets. The construction of both partitions of SPAM Commons includes significant intellectual challenges of their own. First, the isolation of Protected Partition addresses partially the concerns of privacy, which remains a general research problem. Second, useful spam and legitimate data sets require automated distinction of spam from legitimate documents with certainty, which remains an open research question in email, web, and other media. Third, the adversarial and mutual evolution of spam producers and defenders require continuous collection of fresh data for further study. Finally, the collection and streaming of near-real-time spam data represent research resources currently unavailable to spam researchers. Advances in these areas will spur the growth and evolution of SPAM Commons that will enable new research on the evolving and growing spam problem.The impact of SPAM Commons data sets on experimental spam research may be similar to the impact of large corpora in disciplines such as speech/image recognition and natural language processing, which achieved a level of scientific result reproducibility and comparativeness after the use of such corpora became standard requirements. The proposed data repository will be supported and used by 9 university partners (Clayton State, Emory, Georgia Tech, NC A&T, Northwestern, Texas A&M, UC Davis, U. Georgia, UNC Charlotte), and several industry partners (IBM, PureWire, Secure Computing).