Fingerprinting and Big Data Security Analytics for the Scalable Generation of Cyber Threat Intelligence

每天都有大量针对企业、政府机构和个人的网络基础设施的网络攻击，其复杂性、速度、强度、数量、破坏性和胆大程度都是前所未有的。此外，威胁形势正在向更加隐蔽、多变和有针对性的高级持续威胁转变：(a)工业控制系统、(b)物联网设备、(c)社交网络、(d) SDN和云基础设施以及(e)移动设备，这进一步加剧了安全挑战。这些攻击来自广泛的犯罪者，如罪犯、网络恐怖分子、恐怖分子和外国情报/军事机构。当目标涉及到关键基础设施时，破坏可能会更严重。组织部署了一系列安全设备，如防火墙、入侵检测和防御系统以及网络安全监控，这些设备会生成各种警报、事件、代码和日志，这些警报、事件、代码和日志通常是海量的、实时不可用的、未充分利用的。在这种情况下，迫切需要利用大数据技术，以便对上述安全日志、数据馈送和数据流进行实时聚合、分析、挖掘和关联，以获得及时和相关的网络威胁情报，从而能够检测、预防、缓解和归因网络威胁。在短期内，我们将专注于最突出的操作系统平台，即基于Android操作系统的平台。事实上，Android在移动领域占有近87.6%的市场份额。此外，通过谷歌的Brillo平台，它正在迅速扩展到各种消费电子产品和物联网（IoT）设备。在这方面，本研究提案的长期目标是制定一个实用框架，以生成及时、相关和可操作的情报，以应对网络威胁。在短期内，我们将重点分析Android威胁。在这方面，我们的短期和中期目标如下：(i)精心设计一套高度可扩展的技术，用于自动分析大量涌入的Android恶意软件和目标应用程序。典型的分析包括：恶意目标的分类和聚类、新恶意软件家族的检测和恶意行为的隔离；（ii）设计可扩展算法，通过分析各种网络信息（如被动DNS流、通过动态分析收集的恶意软件网络流以及暗网流量流）来表征、跟踪和聚合Android威胁的网络足迹；（iii）设计并实施用于生成网络威胁情报的框架，该框架利用上述创新、近实时、高度可扩展和简化的技术来分析恶意软件馈送、应用程序和相关网络信息流。

Everyday, a deluge of cyber attacks is launched against the cyber infrastructure of corporations, governmental agencies and individuals, with unprecedented sophistication, speed, intensity, volume, damage and audacity. Besides, the threat landscape is shifting towards more stealthy, mercurial and targeted advanced persistent threats against: (a) industrial control systems, (b) IoT devices, (c) social networks, (d) SDN and cloud infrastructure, and (e) mobile devices, which exacerbates even more the security challenge. These attacks emanate from a wide spectrum of perpetrators such as criminals, cyber-terrorists, terrorists, and foreign intelligence/military services. The damage can be even more potent when the target involves critical infrastructure. Organizations deploy an arsenal of security apparatus such as firewalls, intrusion detection and prevention systems, and network security monitoring, which generates various alerts, events, code and logs that are generally voluminous, unavailable in real-time, and underused. In this context, there is an acute desideratum that consists of harnessing big data technologies in order to subject the aforementioned security logs, data feeds and streams to real-time aggregation, analysis, mining and correlation to derive timely and relevant cyber threat intelligence that will enable detection, prevention, mitigation and attribution of cyber threats. In the short term, we will focus on the most prominent OS platforms, namely those based on Android operating system. Indeed, Android holds nearly 87.6% of the market share in the mobile world. Moreover, it is rapidly expanding to various consumer electronics and Internet of Things (IoT) devices through the Google's Brillo platform. In this regard, the long-term goal of this research proposal is to elaborate a practical framework for the generation of timely, relevant, and actionable intelligence to counter cyber threats. In the short term, we will focus on the analysis of Android threats. In this respect, our short and mid-term goals are as follows: (i) elaborate a suite of highly scalable techniques for the automatic analysis of large influx of Android malware and target applications. Typical analyses include: classification and clustering of malicious targets, new malware family detection and isolation of malicious behaviours; (ii) devise scalable algorithms to characterize, track and aggregate network footprints of Android threats by analyzing various network information such as passive DNS streams, malware network flows collected via dynamic analysis, as well as darknet traffic streams; (iii) design and implement a framework for the generation of cyber threat intelligence that leverages the aforementioned innovative, near-real-time, highly-scalable and streamlined techniques for the analysis of the malware feeds, applications and related network information streams.