TWC: Medium: Collaborative: HIMALAYAS: Hierarchical Machine Learning Stack for Fine-Grained Analysis of Malware Domain Groups

DNS （domain name system）协议通过实现域名与IP地址的双向关联，在互联网的运行中发挥着重要作用。它也越来越多地被恶意软件，特别是僵尸网络滥用，通过使用：(1)与命令和控制（C&amp；C）服务器集合的自动域生成算法，(2)DNS快速通量作为隐藏恶意服务器位置的方式，以及(3)DNS作为C&amp；C通信的载波通道。该项目探索了一种可扩展的、分层的机器学习堆栈的开发，称为喜马拉雅山，它专门研究自动挖掘DNS数据的恶意软件活动的算法。特别是，我们感兴趣的是隔离有序和无序的恶意软件域组集合，它们的访问模式在时间上和逻辑上是相关的。喜玛拉雅山在每个层面上执行的任务都越来越复杂。从较低层次的可扩展聚类和特征选择开始，到较高层次的更高级的恶意软件域子序列识别算法。它具有多种优点，包括速度、准确性、可解释性和使用领域知识的能力，这使得它非常适合恶意软件分析和相关任务。喜马拉雅的分析将加速互联网上恶意域名的识别和删除，并改善谷歌safearch等服务。作为喜马拉雅项目的一部分开发的机器学习堆栈在许多重要的数据挖掘问题上具有更广泛的应用，例如，在金融数据分析中，以及从web访问日志中挖掘用户模式。该项目为学生提供了参与技术发展和转型的机会。

The domain name system (DNS) protocol plays a significant role in operation of the Internet by enabling the bi-directional association of domain names with IP addresses. It is also increasingly abused by malware, particularly botnets, by use of: (1) automated domain generation algorithms for rendezvous with a command-and-control (C&C) server, (2) DNS fast flux as a way to hide the location of malicious servers, and (3) DNS as a carrier channel for C&C communications.This project explores the development of a scalable, hierarchical machine-learning stack, called HIMALAYAS, which specializes in algorithms for automatically mining DNS data for malware activity. In particular, we are interested in isolating both ordered and unordered sets of malware domain groups whose access patterns are temporally and logically correlated. HIMALAYAS performs a task of increasing complexity at each level ? starting from scalable clustering and feature selection at lower levels, to more advanced malware domain subsequence identification algorithms at higher levels. It has multiple benefits, including speed, accuracy, interpretability, and ability to use domain knowledge, which makes it very well suited for malware analysis and related tasks. The analysis by HIMALAYAS should accelerate the identification and takedown of malware domains on the Internet and improve services such as Google SafeSearch. The machine-learning stack developed as part of the HIMALAYAS project has broader application to many important data mining problems, e.g., in financial data analysis, and mining user patterns from web access logs. The project provides opportunities for students to participate in the development and transition of the technology.