CAREER: Machine Learning Assisted Crowdsourcing for Phishing Defense

该项目旨在解决日益增长的网络钓鱼攻击威胁，通过结合人类和机器智能，试图欺骗人们泄露敏感信息的消息。现有的基于机器学习和黑名单的检测方法对新的攻击都很脆弱，而且为了避免阻止合法消息，有些宽容;因此，广泛使用的电子邮件系统很容易受到精心制作的网络钓鱼电子邮件的攻击。为了解决这个问题，项目团队将开发一种系统，可以自动阻止明显的诈骗，同时将不太确定的案例转发给经过培训的人群工作人员，以检测网络钓鱼邮件。为了支持这些工作人员的决策，该团队将开发对系统决策的新解释，这些解释将突出信息及其算法的各个方面，这些方面触发了对人类判断的需求。该系统还将汇总这些群体决策，以生成可以共享给个人用户和电子邮件系统的实时网络钓鱼警报。该项目将导致可解释机器学习的进步，这是一个重要的主题，因为人工智能和机器学习系统在社会中发挥的作用越来越大，并且还提高了我们描述网络钓鱼攻击演变以及互联网平台和用户随着时间的推移对这些攻击的脆弱性的能力。项目团队还将把这项工作作为高中教师和学生可用的安全和推广计划的新课程的重要组成部分，以教育他们并增加他们对网络安全研究的参与。这项工作围绕三个主要目标组织：网络钓鱼风险的经验表征，开发用于网络钓鱼检测的准确和可解释的机器学习模型，并开发可靠的网络钓鱼警报众包系统。该团队将通过开发分析工具来评估网络钓鱼风险，这些工具可以有效地采用和配置电子邮件系统中的反欺骗协议，使用对抗机器学习方法对现有网络钓鱼检测器进行黑盒测试，创建反应式蜜罐，引诱和响应网络钓鱼攻击，不仅收集最初的网络钓鱼电子邮件数据，还收集攻击者在整个网络钓鱼过程中的行为数据。成功的网络钓鱼攻击从网络钓鱼电子邮件中收集的数据将用于开发机器学习模型，使用卷积神经网络和基于长短期记忆的深度学习技术来生成可疑特征和个人决策的置信度估计。可疑特征将用于生成可解释的安全提示，例如文本注释或图标，首先创建更简单、更可解释的机器学习模型，例如模仿特征空间中目标电子邮件附近的本地检测边界的决策树。决策树中的规则将被映射回界面元素和电子邮件内容以提供警告，这些将与一系列用户研究中的通用电子邮件安全警告进行比较，这些用户研究还模拟了人们使用各种线索，功能和媒体检测网络钓鱼的能力。这些单独的模型，沿着来自网络钓鱼检测模型的置信度估计，将被用来推动众包基于系统的个人用户的质量模型将被聚合，以围绕模型判断为太可疑而不能通过但又不足以自动过滤的电子邮件做出可靠的判断。该奖项反映了NSF的法定使命，并通过使用基金会的学术价值和更广泛的影响审查标准。

This project aims to address the growing threat of phishing attacks, messages that try to trick people into revealing sensitive information, by combining human and machine intelligence. Existing detection methods based on machine learning and blacklists are both brittle to new attacks and somewhat lenient, in order to avoid blocking legitimate messages; as a result, widely used email systems are vulnerable to carefully crafted phishing emails. To address this, the project team will develop systems that automatically block obvious scams while forwarding less certain cases to groups of crowd workers trained to detect phishing mails. To support these workers' decision-making, the team will develop novel explanations of the system's decision making that will highlight the aspects of both the message and its algorithm that triggered the need for human judgment. The system will also aggregate these crowd decisions to generate real-time phishing alerts that can be shared to both individual users and to email systems. The project will lead to advances in interpretable machine learning, an important topic given the increasing role that artificial intelligence and machine learning systems play in society, and also increase our ability to characterize the evolution of phishing attacks and the vulnerability of internet platforms and users to those attacks over time. The project team will also use the work as an important component of new courses on usable security and outreach programs to high school teachers and students to both educate them about and increase their participation in cybersecurity research.The work is organized around three main objectives: empirical characterization of phishing risks, developing accurate and interpretable machine learning models for phishing detection, and developing reliable crowdsourcing systems for phishing alerts. The team will assess phishing risks through developing analytics tools on the effective adoption and configuration of anti-spoofing protocols in email systems, using adversarial machine learning methods to conduct black box testing on existing phishing detectors, and creating reactive honeypots that entice and respond to phishing attacks in order to collect data on not just the initial phishing emails but on attackers' behaviors throughout the course of a successful phishing attack. The data collected on phishing emails will be used to develop the machine learning models, using Convolutional Neural Network and Long Short-Term Memory based deep learning techniques to generate both suspicious features and confidence estimates of individual decisions. The suspicious features will be used to generate interpretable security cues such as text annotations or icons by first creating simpler and more interpretable machine learning models such as decision trees that mimic the local detection boundary near the target emails in the feature space. Rules in the decision tree will be mapped back to interface elements and email content to provide the warnings, and these will be compared to generic email security warnings in a series of user studies that also model people's ability to detect phishing using a variety of cues, features, and media. Those individual models, along with the confidence estimates from the phishing detection model, will then be used to drive a crowdsourcing-based system where the models of individual users' quality will be aggregated to make reliable judgments around emails the models judge as too suspicious to pass but not suspicious enough to automatically filter.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.