An incremental clustering method for anomaly detection in flight data
An incremental clustering method for anomaly detection in flight data
复制标题
飞行数据异常检测的增量聚类方法
DOI：
10.1016/j.trc.2021.103406
复制
发表时间：
2020-05
期刊：
影响因子：
--
通讯作者：
Wang Yanjun
中科院分区：
文献类型：
--
作者：
Zhao Weizun;Li Lishuai;Alam Sameer;Wang Yanjun
安全是民航的重中之重。对数字飞行数据记录器(FDR)或快速访问记录器(QAR)数据进行数据挖掘，通常被称为飞机上的黑匣子数据，已经引起了人们对主动安全管理的兴趣。已经开发了新的异常检测方法，主要是集群方法，以监测飞行员的操作并从此类飞行数据中检测任何风险。然而，所有现有的异常检测方法都是离线学习的-使用历史数据对模型进行一次训练，并用于未来的所有预测。实际上，航空公司每月都会不断积累和分析新的飞行数据。对于离线方法来说，对这种动态增长的数据进行集群是具有挑战性的，因为每次有新数据进入时重新训练模型是内存和时间密集型的。如果不对模型进行重新训练，由于模型不能反映数据模式的变化，错误警报或漏检可能会增加。针对这一问题，我们提出了一种基于高斯混合模型(GMM)的增量式异常检测方法，用于从数字飞行数据中识别常见模式和检测飞行操作中的离群点。它是一种航班运营的概率集群模型，可以基于新数据增量更新集群，而不是从头开始重新集群所有数据。它基于历史离线数据训练初始GMM模型。然后，它通过期望最大化(EM)算法不断适应新进入的数据点。要跟踪飞行运行模式的变化，只需保存模型参数，而不需要保存原始飞行数据。该方法在三组仿真数据和两组真实飞行数据上进行了测试。与传统的离线GMM方法相比，该方法在处理时间(测试集时间减少57%-99%)和内存使用量(测试集减少91%-95%)的情况下，可以产生相似的聚类结果。初步结果表明，该增量学习方案能够有效地处理飞行数据分析中动态增长的数据。
Safety is a top priority for civil aviation. Data mining in digital Flight Data Recorder (FDR) or Quick Access Recorder (QAR) data, commonly referred to as black box data on aircraft, has gained interest for proactive safety management. New anomaly detection methods, primarily clustering methods, have been developed to monitor pilot operations and detect any risks from such flight data. However, all existing anomaly detection methods are offline learning — the models are trained once using historical data and used for all future predictions. In practice, new flight data are accumulated continuously and analyzed every month at airlines. Clustering such dynamically growing data is challenging for an offline method because it is memory and time intensive to re-train the model every time new data come in. If the model is not re-trained, false alarms or missed detections may increase since the model cannot reflect changes in data patterns. To address this problem, we propose a novel incremental anomaly detection method based on Gaussian Mixture Model (GMM) to identify common patterns and detect outliers in flight operations from digital flight data. It is a probabilistic clustering model of flight operations that can incrementally update its clusters based on new data rather than to re-cluster all data from scratch. It trains an initial GMM model based on historical offline data. Then, it continuously adapts to new incoming data points via an expectation–maximization (EM) algorithm. To track changes in flight operation patterns, only model parameters need to be saved, not the raw flight data. The proposed method was tested on three sets of simulation data and two sets of real-world flight data. Compared with the traditional offline GMM method, the proposed method can generate similar clustering results with significantly reduced processing time (57 %–99 % time reduction in testing sets) and memory usage (91 %–95 % memory usage reduction in testing sets). Preliminary results indicate that the incremental learning scheme is effective in dealing with dynamically growing data in flight data analytics.
登录
查看更多内容
影响因子：
3.7
作者：
Bao J;Wang W;Yang T;Wu G
通讯作者：
Wu G
DOI：
10.1016/j.trc.2017.05.008
发表时间：
2017-08
影响因子：
8.3
作者：
Xiongwen Qian;Jianfeng Mao;Chun-Hsien Chen;Songlin Chen;Changpeng Yang
通讯作者：
Xiongwen Qian;Jianfeng Mao;Chun-Hsien Chen;Songlin Chen;Changpeng Yang
DOI：
10.1016/j.trc.2018.10.002
发表时间：
2018-12
期刊：
Transportation Research Part C: Emerging Technologies
影响因子：
--
作者：
Lei Kang;M. Hansen
通讯作者：
Lei Kang;M. Hansen
DOI：
--
发表时间：
2009
期刊：
--
影响因子：
--
作者：
Elsevier Sdol
通讯作者：
Elsevier Sdol
影响因子：
3.3
作者：
Chang, LC;Jones, DK;Pierpaoli, C
通讯作者：
Pierpaoli, C