StatScale: Statistical Scalability for Streaming Data

我们生活在数据时代。科技正在以前所未有的规模改变我们收集和存储数据的能力。从使用牡蛎卡数据来改善伦敦的交通网络，到有可能改变我们对宇宙的理解的平方公里阵列天体物理项目，大数据可以为我们生活的许多方面提供信息和丰富。由于在日常生活中广泛使用基于传感器的系统，甚至智能手机都有可以监控位置和活动水平的传感器，因此数据爆炸的大部分都是以数据流的形式出现的：来自一个或多个相关来源的数据随着时间的推移而到达。据估计，到2020年，将有超过300亿台设备收集数据流。统计在“大数据”和数据流中的重要作用已经明确了一段时间。然而，目前的趋势是纯粹关注算法的可扩展性，例如如何开发现有统计算法的版本，使其更好地随数据量扩展。然而，这种方法忽略了这样一个事实，即在处理这种规模的数据集时，往往会出现根本性的新问题，需要高度创新的解决方案。模型误差就是这样一个问题。许多统计方法都是基于对数据的数学模型的使用。这些模型只是真实数据生成机制的近似值。在传统应用中，与数据固有的采样变异性相比，这种模型误差通常很小，可以忽略不计。然而，越来越多的人意识到，模型误差可能在大数据应用中占主导地位。理解模型误差的影响，以及开发即使在存在模型误差的情况下也具有出色统计特性的健壮方法，是主要的挑战。第二个问题是，许多当前的统计方法在计算上并不适用于大数据。在实践中，我们经常需要使用效率较低的统计方法，这些方法的计算速度更快，或者需要更少的计算机内存。这引入了大数据特有的统计计算权衡，导致许多开放的理论问题和重要的实际问题。该方案赠款的战略愿景是调查和制定一种综合方法，以解决这些和其他基本统计挑战。为了做到这一点，我们将特别关注分析数据流。这类数据的一个重要问题是检测数据结构随时间的变化。这将是该计划的早期重点领域，因为它已被确定为大数据的七个关键问题领域之一。此外，这是一个我们的研究将导致实际重要突破的领域。我们的理念是一起解决这些统计问题的方法、理论和计算方面的问题，这种方法只有通过项目资助计划才能实现。这种广阔的视野对于实现所设想的统计方面的实质性根本进展至关重要，并确保我们的新方法足够强大和有效，以便被学术界、工业界和社会广泛采用。

We live in the age of data. Technology is transforming our ability to collect and store data on unprecedented scales. From the use of Oyster card data to improve London's transport network, to the Square Kilometre Array astrophysics project that has the potential to transform our understanding of the universe, Big Data can inform and enrich many aspects of our lives. Due to the widespread use of sensor-based systems in everyday life, with even smartphones having sensors that can monitor location and activity level, much of the explosion of data is in the form of data streams: data from one or more related sources that arrive over time. It has even been estimates that there will be over 30 billion devices collecting data streams by 2020. The important role of Statistics within "Big Data" and data streams has been clear for some time. However the current tendency has been to focus purely on algorithmic scalability, such as how to develop versions of existing statistical algorithms that scale better with the amount of data. Such an approach, however, ignores the fact that fundamentally new issues often arise when dealing with data sets of this magnitude, and highly innovative solutions are required. Model error is one such issue. Many statistical approaches are based on the use of mathematical models for data. These models are only approximations of the real data-generating mechanisms. In traditional applications, this model error is usually small compared with the inherent sampling variability of the data, and can be overlooked. However, there is an increasing realisation that model error can dominate in Big Data applications. Understanding the impact of model error, and developing robust methods that have excellent statistical properties even in the presence of model error, are major challenges. A second issue is that many current statistical approaches are not computationally feasible for Big Data. In practice we will often need to use less efficient statistical methods that are computationally faster, or require less computer memory. This introduces a statistical-computational trade-off that is unique to Big Data, leading to many open theoretical questions, and important practical problems.The strategic vision for this programme grant is to investigate and develop an integrated approach to tackling these and other fundamental statistical challenges. In order to do this we will focus in particular on analysing data streams. An important issue with this type of data is detecting changes in the structure of the data over time. This will be an early area of focus for the programme, as it has been identified as one of seven key problem areas for Big Data. Moreover it is an area in which our research will lead to practically important breakthroughs. Our philosophy is to tackle methodological, theoretical and computational aspects of these statistical problems together, an approach that is only possible through the programme grant scheme. Such a broad perspective is essential to achieve the substantive fundamental advances in statistics envisaged, and to ensure our new methods are sufficiently robust and efficient to be widely adopted by academics, industry and society more generally.