Statistical Foundations for Detecting Anomalous Structure in Stream Settings (DASS)

检测流设置中的异常结构的统计基础 (DASS)

• 批准号: EP/Z531327/1
• 负责人: Idris Eckley
• 金额: $ 515.13万
• 依托单位: Lancaster University
• 依托单位国家: 英国
• 项目类别: Research Grant
• 财政年份: 2024
• 资助国家: 英国
• 起止时间: 2024 至 无数据
• 项目状态: 未结题
• 来源: https://gtr.ukri.org/projects?ref=EP%2FZ531327%2F1
• 关键词: Statistical; Foundations; Detecting; Anomalous; Structure

With the exponentially increasing prevalence of networked sensors and other devices for collecting data in real-time, automated data analysis methods with theoretically justified performance guarantees are in constant demand. Often a key question with such streaming data is whether they show evidence of anomalous behaviour. This could, e.g., be due to malignant bot activity on a website; early warning of potential equipment failure or detection of methane leakages. These and other motivating examples share a common feature which is not accommodated by classical point anomaly models in statistics: the anomaly may not simply be an 'outlying' observation, but rather a distinctive pattern observed over consecutive observations. The strategic vision for this programme grant is to establish the statistical foundations for Detecting Anomalous Structure in Streaming data settings (DASS).Discussions with a wide-range of industrial partners from different sectors have identified important, generic challenges that cut across distinct DASS applications, and are relevant for analysing streaming data more broadly:Contemporary Constrained Environments: Anomaly detection is often performed under various constraints due, for example, to the restrictions on measurement frequency, the volume of data transferable between sensors and a central processor, or battery usage limits. Additionally, certain scenarios may impose privacy restrictions when handling sensitive data. Consequently, it has become imperative to establish the mathematical underpinning for rigorously examining the trade-offs between, e.g., statistical accuracy, communication efficiency, privacy preservation and computational demands.Handling Data Realities: A substantial portion of research in statistical anomaly detection operates under the assumption of clean data. Nevertheless, real-world data typically exhibit various imperfections, such as missing values, labelling errors in data streams, synchronisation discrepancies, sensor malfunctions and heterogeneous sensor performance. Consequently, there is a pressing need for the development of principled, model-based procedures that can effectively address the features of real data and enhance the resilience of anomaly detection methods.Identifying, Accounting for and Tracking Dependence: Not only are data streams often interdependent, but also anomalous patterns may be dependent across those streams. Taking into account both types of dependence is crucial in enhancing the statistical efficiency of anomaly detection algorithms, and also in controlling the errors arising from handling a large number of data streams in a principled way. Other challenges include tracking the path of an anomaly across multiple data sources with a view to learning causal indicators allowing for precautionary intervention.Our ambitious goal of comprehensively addressing these challenges is only achievable via the programme grant scheme. Our philosophy is to tackle the methodological, theoretical and computational aspects of these statistical problems together. This integrated approach is essential to achieving the substantive fundamental advances in statistics envisaged, and to ensuring that our new methods are sufficiently robust and efficient to be widely adopted by academics, industry and society more generally.

随着网络传感器和其他设备的呈指数增长，以实时收集数据，具有理论上合理的性能保证的自动数据分析方法始终存在。这种流数据通常是一个关键问题是它们是否显示出异常行为的证据。例如，这可能是由于网站上的恶性机器人活动所致；潜在设备故障或检测甲烷泄漏的预警。这些和其他激励的例子共享一个共同的特征，这是统计中经典异常模型所无法适应的：异常可能不仅是“偏远”观察结果，而是对连续观察结果观察到的独特模式。 The strategic vision for this programme grant is to establish the statistical foundations for Detecting Anomalous Structure in Streaming data settings (DASS).Discussions with a wide-range of industrial partners from different sectors have identified important, generic challenges that cut across distinct DASS applications, and are relevant for analysing streaming data more broadly:Contemporary Constrained Environments: Anomaly detection is often performed under various constraints due, for example, to the测量频率的限制，传感器和中央处理器之间可传输的数据量或电池使用限制。此外，在处理敏感数据时，某些方案可能会施加隐私限制。因此，必须建立严格研究统计准确性，沟通效率，隐私保护和计算需求之间的权衡的数学基础，以建立数学上的基础：统计数据现实：统计异常检测的大部分研究在清洁数据的假设下运行。然而，现实世界中的数据通常表现出各种缺陷，例如缺失值，标记数据流中的错误，同步差异，传感器故障和异质传感器性能。因此，迫切需要开发基于模型的基于模型的程序，这些程序可以有效地解决真实数据的特征并增强异常检测方法的弹性。识别，核算和跟踪依赖性：不仅数据流通常相互依存，而且异常模式也可能跨这些流。考虑到两种类型的依赖性对于增强异常检测算法的统计效率以及控制大量数据流而产生的误差至关重要。其他挑战包括跟踪多个数据源的异常路径，以期学习因果指标允许预防性干预。我们雄心勃勃的目标是通过计划赠款计划全面解决这些挑战。我们的哲学是解决这些统计问题的方法论，理论和计算方面。这种综合方法对于实现所设想的统计数据的实质性基本进步至关重要，并确保我们的新方法足够强大且有效，可以更广泛地被学术界，工业和社会广泛采用。