Designing sensitivity analyses for weakly identified or non-identified models

许多现代统计分析需要对参数或假设进行选择，而这些参数或假设无法从观测数据中很好地识别或验证。例如，任何使用贝叶斯方法的数据分析都需要选择先验分布和这些先验分布的超参数。粗化数据的分析，例如截尾的事件发生时间数据或缺失值数据，可能无法假设数据是随机粗化的，因此必须选择指定数据粗化如何取决于未观察值本身的模型。因果分析需要假设观察到的变量与未观察到的混杂因素和中介因素之间的关系。随机效应荟萃分析对随机效应的分布和研究人群之间的潜在相似性做出假设，通常是针对少数研究。** 在上述所有情况下，进行分析的研究人员通常只是对超参数或输入的决策或选择做出一个选择，如果这些选择不好，可能会导致灾难性的后果。同行评议界越来越多地要求研究人员在这些情况下进行敏感性分析。然而，敏感性分析往往是随意进行的，依赖于对模型输入和假设的“合理选择”，而这些输入和假设并不系统相关或合理。拟议拨款的目的是建立一个原则性的计算方法来进行敏感性分析，重点是从统计分析中做出的潜在决策对不同假设和模型输入的敏感性，而不是专注于参数估计本身的敏感性。** 设计原则性敏感性分析需要克服几个障碍。首先，任何解决方案都必须可扩展到大量参数。其次，不同的统计分析将要求用户能够指定不同种类的决策阈值。第三，必须开发公开提供的软件，使用户能够进行敏感性分析。** 贝叶斯优化方法背后的基本思想是使用贝叶斯非参数建模（通常使用高斯过程）来模拟昂贵的目标函数的输出。例如，它已用于选择参数设置（层数，隐藏节点数等）。用于神经网络和深度贝叶斯学习模型以优化预测。贝叶斯优化也被用于估计决策过程问题的后悔界限。** 拟议项目将开发使用贝叶斯优化的方法，以确定最佳区域，而不是最佳值，这些区域对应于由于最佳参数的不同选择而导致的决策中的重要变化。

Many modern statistical analyses require choices regarding parameters or assumptions that can not be well-identified or verified from the observed data. For example, any data analysis using Bayesian methods requires selecting prior distributions and hyperparameters for those prior distributions. Analyses of coarsened data, such as censored time-to-event data or data with missing values, may not be able to assume that the data are coarsened at random and thus must choose models which specify how the coarsening of the data depends on the unobserved values themselves. Causal analyses require assumptions regarding the relationships of observed variables to unobserved confounders and mediators. Random effect meta-analyses make assumptions about the distribution of the random effects and the underlying similarity between study populations, often for small numbers of studies. ******In all of the above cases, researchers conducting analyses often simply make one choice for decisions or choices of hyperparameters or inputs, which can be lead to disastrous consequences if those choices are poor. The peer review community has increasingly demanded sensitivity analyses from researchers in these situations. However, the sensitivity analyses are often conducted haphazardly, relying on “reasonable choices” for model inputs and assumptions that are not systematically related or justified. The purpose of the proposed grant is to create a principled computational approach to sensitivity analyses that focuses on the sensitivity of potential decisions made from statistical analyses to different assumptions and model inputs, rather than focusing on the sensitivity of the parameter estimates themselves. ******Designing principled sensitivity analyses requires overcoming several obstacles. First, any solution must be scalable to large numbers of parameters. Second, different statistical analyses will require users to be able to specify different kinds of decision thresholds. Third, there must be the development of publicly available software for users to be able to implement the sensitivity analyses. ******The basic idea behind Bayesian optimization approach is to use Bayeisan non-parametric modelling (typically using Gaussian processes) to model the output of an expensive to evaluate objective function. For example, it has been used to choose parameter settings (numbers of levels, numbers of hidden nodes, etc.) for neural network and deep Bayes learning models to optimize prediction. Bayesian optimization has also been used for estimating regret bounds for decision process problems. ******The proposed project will develop methods for using Bayesian optimization to identify optimal regions, rather than optimal values, that correspond to important changes in decision making due to different choices of optimal parameters.