Statistical Foundations for Detecting Anomalous Structure in Stream Settings (DASS)

随着网络传感器和其他用于实时采集数据的设备的普及，具有理论上合理的性能保证的自动化数据分析方法的需求不断增加。通常，此类流媒体数据的一个关键问题是，它们是否显示出异常行为的证据。例如，这可能是由于网站上的恶意机器人活动；潜在设备故障的早期预警或甲烷泄漏的检测。这些和其他激励性的例子有一个共同的特征，这是统计学中的经典点异常模型所不具备的：异常可能不是简单的‘离群式’观测，而是在连续观测中观察到的一种独特的模式。这项计划拨款的战略愿景是为检测流数据设置(DASS)中的异常结构奠定统计学基础。与来自不同行业的广泛行业合作伙伴的讨论确定了跨越不同DASS应用的重要的一般性挑战，并与更广泛的流数据分析相关：当代受限环境：异常检测通常在各种限制下执行，例如，由于测量频率、传感器和中央处理器之间可传输的数据量或电池使用限制的限制。此外，在处理敏感数据时，某些情况可能会施加隐私限制。因此，必须建立数学基础，以严格检查统计准确性、通信效率、隐私保护和计算需求之间的权衡。处理数据现实：统计异常检测的大部分研究都是在数据干净的假设下进行的。然而，真实世界的数据通常表现出各种缺陷，例如缺失值、数据流中的标记错误、同步不一致、传感器故障和传感器性能不一致。因此，迫切需要开发有原则的、基于模型的程序，能够有效地处理真实数据的特征，并增强异常检测方法的弹性。识别、解释和跟踪相关性：不仅数据流经常相互依赖，而且异常模式可能跨这些流依赖。考虑这两种依赖关系对于提高异常检测算法的统计效率以及有原则地控制因处理大量数据流而产生的错误是至关重要的。其他挑战包括跟踪多个数据来源的异常路径，以了解允许预防性干预的因果指标。我们全面解决这些挑战的宏伟目标只有通过方案赠款计划才能实现。我们的理念是共同解决这些统计问题的方法、理论和计算方面的问题。这一综合办法对于实现所设想的统计方面的实质性基本进展，以及确保我们的新方法足够稳健和有效，以便被学术界、工业界和整个社会广泛采用，是至关重要的。

With the exponentially increasing prevalence of networked sensors and other devices for collecting data in real-time, automated data analysis methods with theoretically justified performance guarantees are in constant demand. Often a key question with such streaming data is whether they show evidence of anomalous behaviour. This could, e.g., be due to malignant bot activity on a website; early warning of potential equipment failure or detection of methane leakages. These and other motivating examples share a common feature which is not accommodated by classical point anomaly models in statistics: the anomaly may not simply be an 'outlying' observation, but rather a distinctive pattern observed over consecutive observations. The strategic vision for this programme grant is to establish the statistical foundations for Detecting Anomalous Structure in Streaming data settings (DASS).Discussions with a wide-range of industrial partners from different sectors have identified important, generic challenges that cut across distinct DASS applications, and are relevant for analysing streaming data more broadly:Contemporary Constrained Environments: Anomaly detection is often performed under various constraints due, for example, to the restrictions on measurement frequency, the volume of data transferable between sensors and a central processor, or battery usage limits. Additionally, certain scenarios may impose privacy restrictions when handling sensitive data. Consequently, it has become imperative to establish the mathematical underpinning for rigorously examining the trade-offs between, e.g., statistical accuracy, communication efficiency, privacy preservation and computational demands.Handling Data Realities: A substantial portion of research in statistical anomaly detection operates under the assumption of clean data. Nevertheless, real-world data typically exhibit various imperfections, such as missing values, labelling errors in data streams, synchronisation discrepancies, sensor malfunctions and heterogeneous sensor performance. Consequently, there is a pressing need for the development of principled, model-based procedures that can effectively address the features of real data and enhance the resilience of anomaly detection methods.Identifying, Accounting for and Tracking Dependence: Not only are data streams often interdependent, but also anomalous patterns may be dependent across those streams. Taking into account both types of dependence is crucial in enhancing the statistical efficiency of anomaly detection algorithms, and also in controlling the errors arising from handling a large number of data streams in a principled way. Other challenges include tracking the path of an anomaly across multiple data sources with a view to learning causal indicators allowing for precautionary intervention.Our ambitious goal of comprehensively addressing these challenges is only achievable via the programme grant scheme. Our philosophy is to tackle the methodological, theoretical and computational aspects of these statistical problems together. This integrated approach is essential to achieving the substantive fundamental advances in statistics envisaged, and to ensuring that our new methods are sufficiently robust and efficient to be widely adopted by academics, industry and society more generally.