CSR: Small: Large-Scale Web Crawling and Spam Avoidance in Search-Engine Applications

搜索引擎和各种数据挖掘应用程序通常依赖于网络爬虫来导航网络，发现有价值的内容并保持新鲜。然而，大量的可用信息和复杂的垃圾邮件技术通常用于欺骗搜索引擎，在网络爬行，特别是在非商业应用，如研究提出了重大挑战。该项目的第一部分设计了高效的实时图形操作算法，并建立了一个高性能的分布式网络爬虫架构，无缝耦合互联网规模的网络，信息检索和图论的各个组件。第二部分创建了快速估计域名信誉的概率技术，并探讨了各种排名技术，以实现更好的鲁棒性，对垃圾邮件。第三部分设计了先进的预算机制，在多个粒度级别上控制Web不同部分的抓取率。该项目预计将使得克萨斯州农工大学的学生&参与跨学科领域的研究密集型教育（如数据密集型计算，网络，图论，分布式系统，并行体系结构和建模），扩大网络研究融入课堂教学，吸引本科生REU，扩大少数群体在工程中的参与，激发学生之间的合作和思想共享，并通过公开分享我们的工作成果，允许其他机构进行与网络相关的研究。

Search engines and various data-mining applications commonly rely on web crawlers to navigate the web, discover valuable content, and keep it fresh. However, the enormous volume of available information and sophisticated spam techniques commonly used to deceive search engines present significant challenges in web crawling, especially in non-commercial applications such as research. The first part of this project designs efficient real-time graph-manipulation algorithms and builds a high-performance distributed web-crawler architecture that seamlessly couples the various components of Internet-scale networking, information retrieval, and graph theory. The second part creates probabilistic techniques for quick estimation of domain reputation and explores various ranking techniques to achieve better robustness against spam. The third part designs advanced budgeting mechanisms to control the crawl rate of different parts of the web at multiple levels of granularity. The project is expected to engage students at Texas A&M in research-intensive education in cross-disciplinary fields (such as data-intensive computing, networking, graph theory, distributed systems, parallel architectures, and modeling), broaden integration of web research into classroom teaching, attract undergraduate students to REU, extend participation of minority groups in engineering, stimulate collaboration among students and sharing of ideas, and permit web-related research at other institutions through publicly shared outcomes of our work.