Nonparametric Confidence Sequences and their Applications

大规模的顺序测试和评估现在是科技行业的日常任务，大型互联网公司每周都会进行数百或数千次实验(有时称为A/B测试)，以了解他们的客户群偏好，以改善产品性能和用户体验。这样的实验本质上是按顺序进行的：参观者成群结队地到达，结果通常会相对于测试的持续时间迅速观察到。这些实验计划松散：启动一项新测试几乎没有障碍，对如何运行几乎没有监督，不像临床试验那样，由于从一开始就有统计学家的参与，临床试验受到严格监管，有明确的正式计划。这些实验也被持续监测，并做出适应性选择，决定是提前停止并得出结论，还是收集更多数据。换句话说，样本的大小和预算很少是事先确定的，运行实验的数据科学家手中有很大的灵活性。这种情况在科学界也很常见，要么是望远镜按顺序收集天文数据(也许是测试黑洞的存在或估计星系的大小)，要么是心理学家按顺序收集人类受试者的数据(并在此过程中分析效应大小)。然而，这种灵活的、计划松散的、具有流动性决策的顺序实验的一个主要缺点是，无论是在实验过程中还是在实验结束时，提供正确的推理保证都是非常重要的。传统的可信区间和p值是经典统计学的基础，它们是为固定样本量而设计的，在那个预定的时间只能使用一次。以不同的样本大小重复使用标准CI，或者在自适应地停止之后，不进行任何校正来说明所构造的多个区间，完全使它们的保证无效，导致错误结论的增加。PI建议重温Darling和Robbins(1967)提出的一个经典概念--“置信度序列”，它是一个(潜在无限的)置信度区间序列，它具有很高的概率，在所有时间都是同时有效的。由于同时保证，分析师可以继续窥视数据和构建的置信度序列，自适应地选择停止收集数据或收集更多数据，并且通过该过程仍然具有正确的推理保证，包括当该过程停止时。这些值可以转换为始终有效的p值，在任意停止时间也是有效的。利用现代的鞅技巧，我们最近已经能够将置信度序列的先前构造推广到几个新的非参数设置，从而产生了已知的最紧的闭式CS表达式以及实践中最精确的数值方法。本项目力求从理论上和实践上扩大上述进展的范围。PI希望进行的扩展的几个例子包括为向量值平均向量设计新的置信度序列，以及不依赖于未知参数的完全经验界。我们还将探索这些界限在顺序测试和评估任务中的应用。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Large scale sequential testing and estimation is now a daily task in the tech industry, with large internet companies running hundreds or thousands of experiments (sometimes called A/B tests) per week to understand their customer base preferences, in order to improve product performance and user experience. Such experiments are inherently sequential: visitors arrive in a stream and outcomes are typically observed quickly relative to the duration of the test. These experiments are loosely planned: there are few hurdles to starting a new test, and little oversight on how they are run, unlike clinical trials which are heavily regulated with clear formal planning due to the involvement of statisticians from the start. The experiments are also continuously monitored, and adaptive choices are made of whether to stop early and make conclusions, or to collect more data. In other words, sample sizes and budgets are rarely fixed in stone in advance and there is plenty of flexibility at the hands of the data scientist who is running the experiment. Such situations are common in the sciences as well, with either telescopes collecting astronomical data sequentially (and perhaps testing for presence of black holes or estimating sizes of galaxies), or psychologists collecting human subject data sequentially (and analyzing effect sizes along the way). However, a major drawback of such flexible, loosely planned, sequential experimentation with fluid decision making, is that it is very nontrivial to provide correct inferential guarantees, either along the way or when the experiment is terminated. Traditional confidence intervals and p-values, the bread and butter of classical statistics, are designed for fixed sample sizes, and can only be used once at that predetermined time. Using the standard CIs repeatedly at different sample sizes, or after adaptively stopping, without any correction to account for the multiple intervals constructed, completely invalidates their guarantees, leading to an increase in erroneous conclusions. The graduate student support will be used for research on sequential analysis and concentration inequalities.The PI proposes to revisit a classical notion called a "confidence sequence" by Darling and Robbins (1967), which is a (potentially infinite) sequence of confidence intervals that is, with high probability, simultaneously valid over all times. Due to the simultaneous guarantee, an analyst may keep peeking at the data and the constructed confidence sequence, adaptively choosing to stop collecting data or to collect more, and still have correct inferential guarantees through the process including when it stops. These can be converted to always-valid p-values, that are also valid at arbitrary stopping times. Using modern martingale techniques, we have recently been able to generalize prior constructions of confidence sequences to several novel nonparametric settings, yielding both the tightest known closed-form CS expressions as well as the sharpest numerical methods in practice. This project seeks to extend the scope of the above advances both theoretically and practically. A few examples of extensions that the PI wishes to pursue include designing new confidence sequences for vector-valued mean vectors, and fully empirical bounds that do not depend on unknown parameters. We will also explore applications of these bounds to sequential testing and estimation tasks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.