SaTC: CORE: Small: Understanding, Measuring, and Defending against Malicious Web Crawlers

鉴于网络的不断扩展，搜索引擎依靠自动化网络爬虫来自动发现新网页并为其建立索引。除了搜索引擎之外，许多不同的行业都依赖网络爬虫，从查找滥用页面的安全相关爬虫到拍摄内容快照以在社交网络上显示页面预览的爬虫。与此同时，攻击者利用恶意爬虫自动查找和利用网站上的漏洞，抓取内容和电子邮件地址，并暴力破解登录表单。该项目的重点是更好地了解恶意网络爬虫，收集其在线活动数据，并开发能够区分良性和恶意网络爬虫的防御系统。该项目旨在通过多管齐下的方法来了解、衡量和防御恶意网络爬虫。首先，该项目建议开发类似蜜罐的基础设施，用于收集现有良性和恶意爬虫的信息。此信息用于跟踪最滥用的爬虫并提供有关网络爬虫活动的统计数据。其次，该项目包括设计用于区分真实浏览用户和冒充真实用户的恶意爬虫的工具和技术。第三，该项目提出了网络爬虫实时检测和防御技术的设计、开发和评估。最后，该项目包括设计新的爬虫协议，允许合法爬虫不受阻碍地工作，同时严格限制恶意爬虫的爬行能力。这项研究工作的成果预计将提高对网络上恶意爬虫活动的理解，并在保护良性网站免受恶意爬虫攻击方面产生重大的实际影响。此外，通过改进对危害网站和窃取用户数据的恶意爬虫的检测，该项目提高了所有网络用户的安全性。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Given the constant expansion of the web, search engines rely on automated web crawlers to automatically discover new web pages and index them. Next to search engines, many different industries rely on web crawlers, ranging from security-related crawlers that find abusive pages, to crawlers that take snapshots of content in order to show previews of pages on social networks. At the same time, attackers are utilizing malicious crawlers to automatically find and exploit vulnerabilities on websites, to scrape content and email addresses, and to brute-force login forms. This project focuses on better understanding malicious web crawlers, gathering data about their activity online, and developing defensive systems that can differentiate between benign and malicious web crawlers.The project seeks to understand, measure, and defend against malicious web crawlers through a multi-pronged approach. First, the project proposes the development of honeypot-like infrastructure for collecting information on existing benign and malicious crawlers. This information is used to track the most abusive crawlers and offer statistics about crawler activity on the web. Second, the project includes the design of tools and techniques for differentiating between real browsing users and malicious crawlers that pretend to be real users. Third, the project proposes the design, development, and evaluation of technologies for real-time detection of web crawlers and for defending against them. Last, the project includes the design of new crawling protocols that allow legitimate crawlers to work unhindered while severely restricting the crawling abilities of malicious crawlers. The outcomes of this research effort are expected to improve the understanding of malicious crawler activity on the web and to achieve substantial practical impact in protecting benign websites against malicious crawlers. Moreover, by improving the detection of malicious crawlers that compromise websites and exfiltrate user data, the project improves the security of all web users.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.