TWC: Medium: Collaborative: HIMALAYAS: Hierarchical Machine Learning Stack for Fine-Grained Analysis of Malware Domain Groups

域名系统（DNS）协议通过实现域名与IP地址的双向关联而在互联网的操作中起着重要作用。 它也越来越多地被恶意软件滥用，特别是僵尸网络，通过使用：（1）自动域生成算法与命令和控制（C C）服务器会合，（2）DNS快速流量作为隐藏恶意服务器位置的方式，以及（3）DNS作为C C通信的载波信道。&&该项目探索了一个可扩展的分层机器学习堆栈的开发，称为HIMALAYAS，它专门研究自动挖掘恶意软件活动的DNS数据的算法。特别是，我们有兴趣在隔离有序和无序集的恶意软件域组的访问模式是时间和逻辑相关。 HIMALAYAS在每个级别执行增加复杂性的任务-从较低级别的可扩展聚类和特征选择开始，到更高级别的更高级恶意软件域子序列识别算法。它具有多种优势，包括速度，准确性，可解释性和使用领域知识的能力，这使得它非常适合恶意软件分析和相关任务。HIMALAYAS的分析应该会加速互联网上恶意软件域名的识别和删除，并改善谷歌安全搜索等服务。作为HIMALAYAS项目的一部分开发的机器学习堆栈在许多重要的数据挖掘问题中具有更广泛的应用，例如，金融数据分析，以及从Web访问日志中挖掘用户模式。 该项目为学生提供了参与技术开发和过渡的机会。

The domain name system (DNS) protocol plays a significant role in operation of the Internet by enabling the bi-directional association of domain names with IP addresses. It is also increasingly abused by malware, particularly botnets, by use of: (1) automated domain generation algorithms for rendezvous with a command-and-control (C&C) server, (2) DNS fast flux as a way to hide the location of malicious servers, and (3) DNS as a carrier channel for C&C communications. This project explores the development of a scalable, hierarchical machine-learning stack, called HIMALAYAS, which specializes in algorithms for automatically mining DNS data for malware activity. In particular, we are interested in isolating both ordered and unordered sets of malware domain groups whose access patterns are temporally and logically correlated. HIMALAYAS performs a task of increasing complexity at each level - starting from scalable clustering and feature selection at lower levels, to more advanced malware domain subsequence identification algorithms at higher levels. It has multiple benefits, including speed, accuracy, interpretability, and ability to use domain knowledge, which makes it very well suited for malware analysis and related tasks. The analysis by HIMALAYAS should accelerate the identification and takedown of malware domains on the Internet and improve services such as Google SafeSearch. The machine-learning stack developed as part of the HIMALAYAS project has broader application to many important data mining problems, e.g., in financial data analysis, and mining user patterns from web access logs. The project provides opportunities for students to participate in the development and transition of the technology.