Designing sensitivity analyses for weakly identified or non-identified models

点击翻译按钮获取中文摘要

Many modern statistical analyses require choices regarding parameters or assumptions that can not be well-identified or verified from the observed data. For example, any data analysis using Bayesian methods requires selecting prior distributions and hyperparameters for those prior distributions. Analyses of coarsened data, such as censored time-to-event data or data with missing values, may not be able to assume that the data are coarsened at random and thus must choose models which specify how the coarsening of the data depends on the unobserved values themselves. Causal analyses require assumptions regarding the relationships of observed variables to unobserved confounders and mediators. Random effect meta-analyses make assumptions about the distribution of the random effects and the underlying similarity between study populations, often for small numbers of studies. ******In all of the above cases, researchers conducting analyses often simply make one choice for decisions or choices of hyperparameters or inputs, which can be lead to disastrous consequences if those choices are poor. The peer review community has increasingly demanded sensitivity analyses from researchers in these situations. However, the sensitivity analyses are often conducted haphazardly, relying on “reasonable choices” for model inputs and assumptions that are not systematically related or justified. The purpose of the proposed grant is to create a principled computational approach to sensitivity analyses that focuses on the sensitivity of potential decisions made from statistical analyses to different assumptions and model inputs, rather than focusing on the sensitivity of the parameter estimates themselves. ******Designing principled sensitivity analyses requires overcoming several obstacles. First, any solution must be scalable to large numbers of parameters. Second, different statistical analyses will require users to be able to specify different kinds of decision thresholds. Third, there must be the development of publicly available software for users to be able to implement the sensitivity analyses. ******The basic idea behind Bayesian optimization approach is to use Bayeisan non-parametric modelling (typically using Gaussian processes) to model the output of an expensive to evaluate objective function. For example, it has been used to choose parameter settings (numbers of levels, numbers of hidden nodes, etc.) for neural network and deep Bayes learning models to optimize prediction. Bayesian optimization has also been used for estimating regret bounds for decision process problems. ******The proposed project will develop methods for using Bayesian optimization to identify optimal regions, rather than optimal values, that correspond to important changes in decision making due to different choices of optimal parameters.