SaTC: CORE: Small: Understanding, Measuring, and Defending against Malicious Web Crawlers

SaTC：核心：小：理解、衡量和防御恶意网络爬虫

• 批准号: 1813974
• 负责人: Nick Nikiforakis
• 金额: $ 49.96万
• 依托单位: SUNY at Stony Brook
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-09-01 至 2023-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1813974&HistoricalAwards=false
• 关键词: SaTC; CORE; Small; Understanding; Measuring

Given the constant expansion of the web, search engines rely on automated web crawlers to automatically discover new web pages and index them. Next to search engines, many different industries rely on web crawlers, ranging from security-related crawlers that find abusive pages, to crawlers that take snapshots of content in order to show previews of pages on social networks. At the same time, attackers are utilizing malicious crawlers to automatically find and exploit vulnerabilities on websites, to scrape content and email addresses, and to brute-force login forms. This project focuses on better understanding malicious web crawlers, gathering data about their activity online, and developing defensive systems that can differentiate between benign and malicious web crawlers.The project seeks to understand, measure, and defend against malicious web crawlers through a multi-pronged approach. First, the project proposes the development of honeypot-like infrastructure for collecting information on existing benign and malicious crawlers. This information is used to track the most abusive crawlers and offer statistics about crawler activity on the web. Second, the project includes the design of tools and techniques for differentiating between real browsing users and malicious crawlers that pretend to be real users. Third, the project proposes the design, development, and evaluation of technologies for real-time detection of web crawlers and for defending against them. Last, the project includes the design of new crawling protocols that allow legitimate crawlers to work unhindered while severely restricting the crawling abilities of malicious crawlers. The outcomes of this research effort are expected to improve the understanding of malicious crawler activity on the web and to achieve substantial practical impact in protecting benign websites against malicious crawlers. Moreover, by improving the detection of malicious crawlers that compromise websites and exfiltrate user data, the project improves the security of all web users.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

鉴于网络的不断扩张，搜索引擎依赖于自动网络爬虫来自动发现新的网页并对其进行索引。除了搜索引擎，许多不同的行业都依赖网络爬虫，从发现滥用页面的安全相关爬虫到获取内容快照以显示社交网络上页面预览的爬虫。与此同时，攻击者正在利用恶意爬虫程序自动查找和利用网站上的漏洞，抓取内容和电子邮件地址，并强行登录表单。该项目致力于更好地了解恶意网络爬虫，收集有关其在线活动的数据，并开发能够区分良性和恶意网络爬虫的防御系统。该项目旨在通过多管齐下的方法来了解，测量和防御恶意网络爬虫。首先，该项目建议开发类似蜜罐的基础设施，用于收集现有良性和恶意爬虫的信息。这些信息用于跟踪最滥用的爬虫，并提供有关网络上爬虫活动的统计数据。其次，该项目包括设计区分真实的浏览用户和伪装成真实的用户的恶意爬虫的工具和技术。第三，该项目提出了设计，开发和评估技术的实时检测网络爬虫和防御他们。最后，该项目包括设计新的爬行协议，允许合法的爬虫不受阻碍地工作，同时严格限制恶意爬虫的爬行能力。这项研究工作的成果，预计将提高对网络上的恶意爬虫活动的理解，并在保护良性网站免受恶意爬虫攻击方面取得实质性的实际影响。此外，该项目通过改进对危害网站和泄露用户数据的恶意爬虫程序的检测，提高了所有网络用户的安全性。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Morellian Analysis for Browsers: Making Web Authentication Stronger With Canvas Fingerprinting

浏览器的 Morellian 分析：通过 Canvas 指纹识别增强 Web 身份验证

• 发表时间: 2019
• 期刊: Proceedings of the 16th Conference on Detection of Intrusions and Malware & Vulnerability Assessment (DIMVA
• 作者: Laperdrix, Pierre;Avoine, Gildas;Baudry, Benoit;Nikiforakis, Nick
• 通讯作者: Nikiforakis, Nick

Less is More: Quantifying the Security Benefits of Debloating Web Applications

• 发表时间: 2019
• 作者: Babak Amin Azad;Pierre Laperdrix;Nick Nikiforakis

Click This, Not That: Extending Web Authentication with Deception

点击这个，而不是那个：通过欺骗扩展 Web 身份验证

• DOI: 10.1145/3433210.3453088
• 发表时间: 2021
• 期刊: Proceedings of the 2021 ACM Asia Conference on Computer and Communications Security (ASIACCS
• 作者: Barron, Timothy;So, Johnny;Nikiforakis, Nick
• 通讯作者: Nikiforakis, Nick

To Err.Is Human: Characterizing the Threat of Unintended URLs in Social Media

• DOI: 10.14722/ndss.2021.24322
• 发表时间: 2021
• 期刊: Proceedings 2021 Network and Distributed System Security Symposium
• 作者: Beliz Kaleli;Brian Kondracki;Manuel Egele;Nick Nikiforakis;G. Stringhini

Unnecessarily Identifiable: Quantifying the fingerprintability of browser extensions due to bloat

不必要的可识别：量化由于膨胀而导致的浏览器扩展的指纹识别能力

• 发表时间: 2019
• 期刊: Proceedings of the Web Conference (WWW
• 作者: Starov, Oleksii;Laperdrix, Pierre;Kapravelos, Alexandros;Nikiforakis, Nick
• 通讯作者: Nikiforakis, Nick