Optimal Shrinkage and Empirical Bayes Prediction under Asymmetry, Censoring and Nonexchangeability

点击翻译按钮获取中文摘要

In every branch of big-data analytics, it is now commonplace to use notions of shrinkage in the construction of robust algorithms and predictors. The concept of shrinkage is important because it provides an elegant framework for combining information from related populations and often leads to substantial improvements in the performances of algorithms used for simultaneous inference. Driven by applications in a wide range of scientific problems over the last decade, the traditional roles of statistical shrinkage have rapidly evolved as new perspectives have been introduced to address and exploit complex, latent structural properties of modern datasets. These new applications often involve non-standard inferential attributes such as asymmetric predictive objectives as well as intricate modeling caveats, such as nonexchangeable prior structures and censored observations. These new age statistical problems pose challenges not only in developing flexible shrinkage algorithms but also in optimally tuning them to obtain efficient shrinkage properties. This project will develop new empirical Bayes predictive methods that possess optimal shrinkage properties and can produce significant enhancements over existing algorithms built on the mathematical convenience of symmetric loss functions and exchangeable prior structures.The common theme underlying this project is that of using optimal shrinkage properties to develop efficient predictive methods. Existing shrinkage algorithms rely heavily on decision theoretic identities that break down under asymmetry and nonexchangeability, and so, there is an urgent need to develop new statistical methodologies, theories, and algorithms. The PI will develop new conditionally linear decision rules for prediction under asymmetry in nonexchangeable Gaussian hierarchical models and will extend the proposed methodologies to non-Gaussian models as well as to settings with non-linear structural constraints. Additionally, optimal empirical Bayes rules will be developed that will work with censored data and can be used for prediction in multi-stage decision making scenarios with asymmetric objectives. The results developed in this project will provide practitioners with an improved understanding of where existing prediction approaches fail under asymmetry, censoring, and nonexchangeability and why algorithms specifically developed to operate under these conditions should be used.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.