Advanced Malware Detection Techniques based on Artificial Intelligence and Distributed Machine Learning

现有的防病毒系统依赖于基于签名、行为或基于沙箱的解决方案，这在当今快速变化的互联网中是不够的。基于签名和启发式的方法对目标威胁和新的恶意软件无效。沙盒可以检测到以前未知的威胁，但在预防方面并不有效，因为可疑程序必须在沙盒中执行，这通常需要几分钟甚至几小时。在这个项目中，我们的目标是研究机器学习和自动化大数据处理在恶意软件检测和分析中的应用，重点关注四个子项目：1）通过广泛的特征工程和因子分解机器进行Android恶意软件检测; 2）通过图卷积网络进行Android恶意软件检测，这是一种新兴的在图上执行的深度学习技术; 3）基于深度神经网络和遗传算法的PC恶意软件检测; 4）用于多方模型训练的分布式算法和架构的开发。第一个子项目侧重于通过反编译APK文件将Android应用功能提取到稀疏数组中，然后通过使用因子分解机检测恶意软件。第二个子项目的目的是通过使用调用图生成应用程序的内部结构来检测Android恶意软件。然后将这些图输入到图卷积神经网络中。第三个子项目将首先从PC文件中提取特征到通用格式，然后通过使用遗传算法训练和优化深度神经网络。最后一个子项目的核心是允许不同的反病毒客户端为训练机器学习模型做出贡献，而不会共享原始数据或遇到隐私泄露风险。该项目将强调开发用于恶意软件检测的可操作情报，并基于分散的数据集在现实中大规模实施。

Existing antivirus systems rely on signature-based, behavioural or sandbox-based solutions, which are insufficient in today's fast-changing Internet. Signature and heuristic based approaches are ineffective against targeted threats and new malware. Sandboxes can detect previously unknown threats, but are not effective at prevention, since the suspicious program must be executed in a sandbox, which often requires minutes or even hours. In this project, we aim to investigate the application of machine learning and automated big data processing to malware detection and analysis, focusing on four sub-projects: 1) Android Malware detection via extensive feature engineering and Factorization Machines; 2) Android Malware detection via Graph Convolutional Networks, a emerging deep learning technique executed on graphs; 3) PC Malware detection based on Deep Neural Networks and Genetic Algorithms; 4) the development of distributed algorithms and architectures for multiparty model training. The first subproject focuses on extracting Android app features into sparse arrays through the decompilation of APK files and then detecting malware via the use of a Factorization Machine. The the aim of the second subproject is to detect Android malware by generating an app's internal structure using call graphs. These graphs are then fed into Graph Convolutional Neural Networks. The third subproject will first aim to extract features from PC files into a common format and then both train and optimize Deep Neural Networks via the use of Genetic Algorithms. At its core, the final subproject is meant to allow different anti-virus clients to contribute to training machine learning models without sharing raw data or running into privacy leakage risks. The project will emphasize the development of actionable intelligence for malware detection and its large-scale implementation based on decentralized datasets in reality.