Fingerprinting and Big Data Security Analytics for the Scalable Generation of Cyber Threat Intelligence

每天都有大量的网络攻击针对企业、政府机构和个人的网络基础设施，其复杂性、速度、强度、数量、破坏力和胆量都是前所未有的。此外，威胁格局正在转向更隐蔽、更多变和更有针对性的高级持续性威胁：（a）工业控制系统，（B）物联网设备，（c）社交网络，（d）SDN和云基础设施，以及（e）移动的设备，这进一步加剧了安全挑战。这些攻击来自各种各样的犯罪者，如犯罪分子、网络恐怖分子、恐怖分子和外国情报/军事部门。当目标涉及关键基础设施时，损害可能更大。组织部署了大量的安全设备，例如防火墙、入侵检测和防御系统以及网络安全监控，这些安全设备生成各种警报、事件、代码和日志，这些警报、事件、代码和日志通常是大量的、实时不可用的和未充分使用的。在这种情况下，迫切需要利用大数据技术，对上述安全日志、数据馈送和数据流进行实时聚合、分析、挖掘和关联，以获得及时和相关的网络威胁情报，从而能够检测、预防、缓解和确定网络威胁的归属。在短期内，我们将专注于最突出的操作系统平台，即基于Android操作系统的平台。事实上，Android占据了移动的世界近87.6%的市场份额。此外，它正在通过谷歌的Brillo平台迅速扩展到各种消费电子和物联网（IoT）设备。在这方面，这项研究提案的长期目标是制定一个实用的框架，以生成及时，相关和可操作的情报，以应对网络威胁。在短期内，我们将专注于Android威胁的分析。在这方面，我们的短期和中期目标如下：（i）制定一套高度可扩展的技术，用于自动分析大量涌入的Android恶意软件和目标应用程序。典型分析包括：恶意目标的分类和聚类、新的恶意软件家族检测和恶意行为的隔离;（ii）设计可扩展的算法，通过分析各种网络信息（如被动DNS流、通过动态分析收集的恶意软件网络流以及暗网流量流）来表征、跟踪和聚合Android威胁的网络足迹;（iii）设计和实施一个框架，以生成网络威胁情报，利用上述创新、近实时、高度可扩展和简化的技术来分析恶意软件源、应用程序和相关网络信息流。

Everyday, a deluge of cyber attacks is launched against the cyber infrastructure of corporations, governmental agencies and individuals, with unprecedented sophistication, speed, intensity, volume, damage and audacity. Besides, the threat landscape is shifting towards more stealthy, mercurial and targeted advanced persistent threats against: (a) industrial control systems, (b) IoT devices, (c) social networks, (d) SDN and cloud infrastructure, and (e) mobile devices, which exacerbates even more the security challenge. These attacks emanate from a wide spectrum of perpetrators such as criminals, cyber-terrorists, terrorists, and foreign intelligence/military services. The damage can be even more potent when the target involves critical infrastructure. Organizations deploy an arsenal of security apparatus such as firewalls, intrusion detection and prevention systems, and network security monitoring, which generates various alerts, events, code and logs that are generally voluminous, unavailable in real-time, and underused. In this context, there is an acute desideratum that consists of harnessing big data technologies in order to subject the aforementioned security logs, data feeds and streams to real-time aggregation, analysis, mining and correlation to derive timely and relevant cyber threat intelligence that will enable detection, prevention, mitigation and attribution of cyber threats. In the short term, we will focus on the most prominent OS platforms, namely those based on Android operating system. Indeed, Android holds nearly 87.6% of the market share in the mobile world. Moreover, it is rapidly expanding to various consumer electronics and Internet of Things (IoT) devices through the Google's Brillo platform. In this regard, the long-term goal of this research proposal is to elaborate a practical framework for the generation of timely, relevant, and actionable intelligence to counter cyber threats. In the short term, we will focus on the analysis of Android threats. In this respect, our short and mid-term goals are as follows: (i) elaborate a suite of highly scalable techniques for the automatic analysis of large influx of Android malware and target applications. Typical analyses include: classification and clustering of malicious targets, new malware family detection and isolation of malicious behaviours; (ii) devise scalable algorithms to characterize, track and aggregate network footprints of Android threats by analyzing various network information such as passive DNS streams, malware network flows collected via dynamic analysis, as well as darknet traffic streams; (iii) design and implement a framework for the generation of cyber threat intelligence that leverages the aforementioned innovative, near-real-time, highly-scalable and streamlined techniques for the analysis of the malware feeds, applications and related network information streams.