Statistical Foundations for Detecting Anomalous Structure in Stream Settings (DASS)

检测流设置中的异常结构的统计基础 (DASS)

• 批准号: EP/Z531327/1
• 负责人: Idris Eckley
• 金额: $ 515.13万
• 依托单位: Lancaster University
• 依托单位国家: 英国
• 项目类别: Research Grant
• 财政年份: 2024
• 资助国家: 英国
• 起止时间: 2024 至 无数据
• 项目状态: 未结题
• 来源: https://gtr.ukri.org/projects?ref=EP%2FZ531327%2F1
• 关键词: Statistical; Foundations; Detecting; Anomalous; Structure

With the exponentially increasing prevalence of networked sensors and other devices for collecting data in real-time, automated data analysis methods with theoretically justified performance guarantees are in constant demand. Often a key question with such streaming data is whether they show evidence of anomalous behaviour. This could, e.g., be due to malignant bot activity on a website; early warning of potential equipment failure or detection of methane leakages. These and other motivating examples share a common feature which is not accommodated by classical point anomaly models in statistics: the anomaly may not simply be an 'outlying' observation, but rather a distinctive pattern observed over consecutive observations. The strategic vision for this programme grant is to establish the statistical foundations for Detecting Anomalous Structure in Streaming data settings (DASS).Discussions with a wide-range of industrial partners from different sectors have identified important, generic challenges that cut across distinct DASS applications, and are relevant for analysing streaming data more broadly:Contemporary Constrained Environments: Anomaly detection is often performed under various constraints due, for example, to the restrictions on measurement frequency, the volume of data transferable between sensors and a central processor, or battery usage limits. Additionally, certain scenarios may impose privacy restrictions when handling sensitive data. Consequently, it has become imperative to establish the mathematical underpinning for rigorously examining the trade-offs between, e.g., statistical accuracy, communication efficiency, privacy preservation and computational demands.Handling Data Realities: A substantial portion of research in statistical anomaly detection operates under the assumption of clean data. Nevertheless, real-world data typically exhibit various imperfections, such as missing values, labelling errors in data streams, synchronisation discrepancies, sensor malfunctions and heterogeneous sensor performance. Consequently, there is a pressing need for the development of principled, model-based procedures that can effectively address the features of real data and enhance the resilience of anomaly detection methods.Identifying, Accounting for and Tracking Dependence: Not only are data streams often interdependent, but also anomalous patterns may be dependent across those streams. Taking into account both types of dependence is crucial in enhancing the statistical efficiency of anomaly detection algorithms, and also in controlling the errors arising from handling a large number of data streams in a principled way. Other challenges include tracking the path of an anomaly across multiple data sources with a view to learning causal indicators allowing for precautionary intervention.Our ambitious goal of comprehensively addressing these challenges is only achievable via the programme grant scheme. Our philosophy is to tackle the methodological, theoretical and computational aspects of these statistical problems together. This integrated approach is essential to achieving the substantive fundamental advances in statistics envisaged, and to ensuring that our new methods are sufficiently robust and efficient to be widely adopted by academics, industry and society more generally.

随着网络传感器和其他实时收集数据的设备呈指数级增长，对具有理论上合理的性能保证的自动数据分析方法的需求不断增加。通常，这种流数据的一个关键问题是它们是否显示出异常行为的证据。这可以，例如，由于网站上的恶意机器人活动;潜在设备故障的早期预警或甲烷泄漏的检测。这些和其他激励的例子有一个共同的特点，这是不适应经典的点异常模型在统计：异常可能不仅仅是一个“外围”的观察，而是一个独特的模式观察到连续的观察。该计划赠款的战略愿景是为检测流数据设置中的异常结构（DASS）建立统计基础。与来自不同行业的广泛工业合作伙伴进行的讨论确定了跨越不同DASS应用的重要通用挑战，并与更广泛地分析流数据相关：当代受限环境：异常检测通常在各种约束下执行，例如，由于对测量频率、传感器和中央处理器之间可传输的数据量或电池使用限制的限制。此外，在处理敏感数据时，某些场景可能会施加隐私限制。因此，必须建立数学基础，以严格审查以下方面之间的权衡：统计准确性、通信效率、隐私保护和计算需求。处理数据现实：统计异常检测中的相当一部分研究都是在干净数据的假设下进行的。然而，现实世界的数据通常表现出各种缺陷，如缺失值，数据流中的标签错误，同步差异，传感器故障和异构传感器性能。因此，有一个迫切需要的原则，基于模型的程序，可以有效地解决真实的数据的功能，并提高异常检测方法的弹性。识别，会计和跟踪依赖性：不仅是数据流往往是相互依赖的，而且异常模式可能是依赖于这些流。考虑到这两种类型的依赖性是至关重要的，在提高异常检测算法的统计效率，并在控制所产生的错误处理大量的数据流中的原则性的方式。其他挑战包括跟踪多个数据源的异常路径，以了解允许预防性干预的因果指标。我们全面应对这些挑战的宏伟目标只能通过方案赠款计划实现。我们的理念是一起解决这些统计问题的方法，理论和计算方面。这一综合办法对于实现所设想的统计方面的实质性根本进展，以及确保我们的新方法足够强大和有效，以便被学术界、工业界和更广泛的社会广泛采用至关重要。