SaTC: CORE: Small: Understanding, Measuring, and Defending against Malicious Web Crawlers

鉴于网络的不断扩展，搜索引擎依赖于自动网络爬虫来自动发现新的网页并将其编入索引。除了搜索引擎之外，许多不同的行业都依赖网络爬虫，从查找滥用网页的安全相关爬虫，到抓取内容快照以便在社交网络上显示网页预览的爬虫。与此同时，攻击者正在利用恶意爬虫程序自动查找和利用网站上的漏洞，抓取内容和电子邮件地址，以及暴力登录表单。该项目侧重于更好地理解恶意网络爬虫，收集有关其在线活动的数据，并开发能够区分良性和恶意网络爬虫的防御系统。该项目旨在通过多管齐下的方法来理解、衡量和防御恶意的网络爬虫。首先，该项目建议开发类似蜜罐的基础设施，用于收集现有良性和恶意爬虫的信息。此信息用于跟踪最滥用的爬虫，并提供有关爬虫活动的统计数据在网络上。其次，该项目包括工具和技术的设计，以区分真正的浏览用户和恶意爬虫假装是真正的用户。第三，该项目提出了网络爬虫实时检测和防御技术的设计、开发和评估。最后，该项目包括设计新的爬行协议，允许合法的爬行程序不受阻碍地工作，同时严格限制恶意爬行程序的爬行能力。这项研究的成果有望提高对网络上恶意爬虫活动的理解，并在保护良性网站免受恶意爬虫攻击方面取得实质性的实际影响。此外，通过改进对危害网站和泄露用户数据的恶意爬虫的检测，该项目提高了所有web用户的安全性。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Given the constant expansion of the web, search engines rely on automated web crawlers to automatically discover new web pages and index them. Next to search engines, many different industries rely on web crawlers, ranging from security-related crawlers that find abusive pages, to crawlers that take snapshots of content in order to show previews of pages on social networks. At the same time, attackers are utilizing malicious crawlers to automatically find and exploit vulnerabilities on websites, to scrape content and email addresses, and to brute-force login forms. This project focuses on better understanding malicious web crawlers, gathering data about their activity online, and developing defensive systems that can differentiate between benign and malicious web crawlers.The project seeks to understand, measure, and defend against malicious web crawlers through a multi-pronged approach. First, the project proposes the development of honeypot-like infrastructure for collecting information on existing benign and malicious crawlers. This information is used to track the most abusive crawlers and offer statistics about crawler activity on the web. Second, the project includes the design of tools and techniques for differentiating between real browsing users and malicious crawlers that pretend to be real users. Third, the project proposes the design, development, and evaluation of technologies for real-time detection of web crawlers and for defending against them. Last, the project includes the design of new crawling protocols that allow legitimate crawlers to work unhindered while severely restricting the crawling abilities of malicious crawlers. The outcomes of this research effort are expected to improve the understanding of malicious crawler activity on the web and to achieve substantial practical impact in protecting benign websites against malicious crawlers. Moreover, by improving the detection of malicious crawlers that compromise websites and exfiltrate user data, the project improves the security of all web users.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.