A web log real-time analysis platform based on stream computing
A web log real-time analysis platform based on stream computing
复制标题
一种基于流计算的Web日志实时分析平台
DOI：
10.1117/12.2660112
复制
发表时间：
2023
期刊：
影响因子：
--
通讯作者：
Taizhi Lv
中科院分区：
文献类型：
--
作者：
Jun Zhang;Li Zhang;P. Tang;Taizhi Lv
随着Internet技术的飞速发展，Web应用得到了广泛的应用。如何保证Web应用程序的可靠性和高性能已成为网站管理的重点。当Web应用程序提供服务时，会生成大量的Web日志。这些日志包含大量有关用户访问此Web应用程序的信息。对Web日志进行实时分析，可以获得系统的性能指标和瓶颈。为了提高Web应用的可靠性和性能，设计并实现了一个基于流计算的Web日志实时分析平台。平台通过Flume采集Web日志数据，通过Kafka消息队列实现数据流，通过Flink流计算平台分析Web日志，将计算结果存储在Redis中进行实时查询，存储在Doris中进行历史数据查询。通过在真实的环境中运行，证明了该平台的可用性，能够提高Web应用的性能和可靠性。
With the rapid development of Internet technology, web applications have been widely used. How to ensure the reliability and high performance of web applications has become the focus of web site management. When a web application provides services, huge web logs are generated. These logs contain a great deal of information about users access to this web application. Real-time analysis of web logs can obtain system performance indicators and bottlenecks. In order to improve the reliability and performance of web applications, a real-time web log analysis platform based on stream computing is designed and implemented. The platform collects web log data by Flume, realizes data flow by Kafka message queue, analyzes web log by the Flink stream computing platform, stores the computing results in Redis for real-time query, and in Doris for historical data query. The availability of the platform is proved by running in a real environment, and it can improve the performance and reliability of web applications.