Hadoop based Deep Packet Inspection system for traffic analysis of e-business websites
Hadoop based Deep Packet Inspection system for traffic analysis of e-business websites
复制标题
基于 Hadoop 的深度包检测系统，用于电子商务网站流量分析
DOI：
--
复制
发表时间：
2014
期刊：
影响因子：
--
通讯作者：
Junchao Yang
中科院分区：
文献类型：
--
作者：
Jiangtao Luo;Yan Liang;Wei Gao;Junchao Yang
互联网流量正在经历爆炸式增长，网上购物是其中一个重要的驱动力。然而，警觉的网络运营商，不愿意成为哑巴管道，正在尽一切努力挖掘海量流量的帮助下，深度包检测（DPI）被认为是一个巨大的挑战，特别是对于海量数据时，传统的方法和编程模型使用。Hadoop以其在分布式存储和并行计算方面的优势提供了一种替代方法。本文介绍了一个基于Hadoop的DPI系统，该系统集成了网络爬虫。给出了系统的体系结构和数据包分析、URL恢复的MapReduce模型。作为一个例子，访问天猫，在中国领先的电子购物巨头的实时网络流量，专门使用该系统进行了调查。从产品的页面浏览量来评估产品、类别和品牌在一定时期内的受欢迎程度。网络爬虫建立的产品信息库提供了产品的详细信息。这项工作探讨了在DPI中使用Hadoop的方法，并为开发这样一个系统提供了有价值的指导方针，该系统可以进一步用于分析其他服务和挖掘网络运营商的网络流量价值。
Internet traffic is experiencing an explosive growth, and online shopping is one of the significant drivers. However, alert network operators, unwilling to be dumb pipes, are making every effort to mine mass traffic with the help of Deep Packet Inspection (DPI) which is regarded as a big challenge especially for massive data when traditional methods and programming model are utilized. Hadoop provides an alternative approach with its strength in distributed storage and parallel computing. In this paper, a Hadoop based DPI system was reported, which was integrated with a web crawler. The system architecture and MapReduce models of packet analysis, web URL restoration were presented. As an example, live web traffic visiting the Tmall, the leading e-shopping giant in China, was specifically investigated using this system. Popularity of product, category and brand for a certain period was evaluated from page views of product. The detailed information of products was provided by the product information base built by the web crawler. This work explored the methodology of using Hadoop in DPI and presented valuable guidelines to develop such a system, which can be further used in analyzing other services and mining the value of network traffic by network operators.