CAREER: Machine Learning Assisted Crowdsourcing for Phishing Defense

该项目旨在通过结合人类和机器智能来解决日益增长的网络钓鱼攻击威胁，即试图诱骗人们泄露敏感信息的信息。现有的基于机器学习和黑名单的检测方法既容易受到新的攻击，又有些宽松，以避免阻止合法消息；因此，广泛使用的电子邮件系统很容易受到精心制作的网络钓鱼邮件的攻击。为了解决这个问题，项目团队将开发系统，自动阻止明显的诈骗，同时将不太确定的案件转发给经过培训的检测网络钓鱼邮件的工作人员小组。为了支持这些工作人员的决策，该团队将对系统的决策做出新的解释，强调信息及其算法的各个方面，这些方面触发了对人类判断的需求。该系统还将汇总这些人群的决策，以生成实时的网络钓鱼警报，这些警报可以共享给个人用户和电子邮件系统。该项目将推动可解释机器学习的发展，这是一个重要的主题，因为人工智能和机器学习系统在社会中扮演着越来越重要的角色，同时也提高了我们描述网络钓鱼攻击演变的能力，以及随着时间的推移，互联网平台和用户对这些攻击的脆弱性。项目团队还将把这项工作作为可用安全性新课程的重要组成部分，并向高中教师和学生提供推广计划，以教育他们并增加他们对网络安全研究的参与。这项工作围绕三个主要目标进行：网络钓鱼风险的经验表征，为网络钓鱼检测开发准确且可解释的机器学习模型，以及为网络钓鱼警报开发可靠的众包系统。该团队将通过开发电子邮件系统中有效采用和配置反欺骗协议的分析工具来评估网络钓鱼风险，使用对抗性机器学习方法对现有的网络钓鱼检测器进行黑盒测试，并创建诱骗和响应网络钓鱼攻击的反应蜜罐，以便不仅收集初始网络钓鱼电子邮件的数据，还收集攻击者在整个成功的网络钓鱼攻击过程中的行为。从钓鱼邮件中收集的数据将用于开发机器学习模型，使用卷积神经网络和基于长短期记忆的深度学习技术来生成可疑特征和个人决策的置信度估计。可疑特征将用于生成可解释的安全线索，如文本注释或图标，首先创建更简单，更可解释的机器学习模型，如决策树，模拟特征空间中目标电子邮件附近的本地检测边界。决策树中的规则将被映射回界面元素和电子邮件内容，以提供警告，这些将与一系列用户研究中的通用电子邮件安全警告进行比较，这些研究还模拟了人们使用各种线索、特征和媒体检测网络钓鱼的能力。这些个体模型，连同来自网络钓鱼检测模型的置信度估计，将被用于驱动一个基于众包的系统，在这个系统中，个体用户的质量模型将被汇总起来，对模型认为过于可疑而无法通过但又不够可疑而无法自动过滤的电子邮件做出可靠的判断。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project aims to address the growing threat of phishing attacks, messages that try to trick people into revealing sensitive information, by combining human and machine intelligence. Existing detection methods based on machine learning and blacklists are both brittle to new attacks and somewhat lenient, in order to avoid blocking legitimate messages; as a result, widely used email systems are vulnerable to carefully crafted phishing emails. To address this, the project team will develop systems that automatically block obvious scams while forwarding less certain cases to groups of crowd workers trained to detect phishing mails. To support these workers' decision-making, the team will develop novel explanations of the system's decision making that will highlight the aspects of both the message and its algorithm that triggered the need for human judgment. The system will also aggregate these crowd decisions to generate real-time phishing alerts that can be shared to both individual users and to email systems. The project will lead to advances in interpretable machine learning, an important topic given the increasing role that artificial intelligence and machine learning systems play in society, and also increase our ability to characterize the evolution of phishing attacks and the vulnerability of internet platforms and users to those attacks over time. The project team will also use the work as an important component of new courses on usable security and outreach programs to high school teachers and students to both educate them about and increase their participation in cybersecurity research.The work is organized around three main objectives: empirical characterization of phishing risks, developing accurate and interpretable machine learning models for phishing detection, and developing reliable crowdsourcing systems for phishing alerts. The team will assess phishing risks through developing analytics tools on the effective adoption and configuration of anti-spoofing protocols in email systems, using adversarial machine learning methods to conduct black box testing on existing phishing detectors, and creating reactive honeypots that entice and respond to phishing attacks in order to collect data on not just the initial phishing emails but on attackers' behaviors throughout the course of a successful phishing attack. The data collected on phishing emails will be used to develop the machine learning models, using Convolutional Neural Network and Long Short-Term Memory based deep learning techniques to generate both suspicious features and confidence estimates of individual decisions. The suspicious features will be used to generate interpretable security cues such as text annotations or icons by first creating simpler and more interpretable machine learning models such as decision trees that mimic the local detection boundary near the target emails in the feature space. Rules in the decision tree will be mapped back to interface elements and email content to provide the warnings, and these will be compared to generic email security warnings in a series of user studies that also model people's ability to detect phishing using a variety of cues, features, and media. Those individual models, along with the confidence estimates from the phishing detection model, will then be used to drive a crowdsourcing-based system where the models of individual users' quality will be aggregated to make reliable judgments around emails the models judge as too suspicious to pass but not suspicious enough to automatically filter.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.