High-dimensional Data Analysis: Modeling Unobserved Heterogeneity in Data, and Studying Imbalanced Classification Problems

点击翻译按钮获取中文摘要

Data science has become the center of attention in a wide range of scientific disciplines, thanks to ever-expanding means of data collection in today's world. Unprecedented size and structural complexity of current data in many applications call for computationally efficient and statistically sound methodologies for extracting useful information from such data. Toward this goal, the general theme of my research program focuses on analyzing high-dimensional data. More specifically, over the five years of this proposal, my short-term objectives are: I) Statistical modeling of heterogeneous high-dimensional data: In applications such as health sciences, engineering and environment, social sciences, and financial econometrics, high-dimensional data often arise from heterogeneous populations consisting of multiple hidden homogeneous sub-populations. Finite mixture of regressions (FMR) and Markov regime-switching autoregressive (MSAR) models provide flexible tools for capturing unobserved heterogeneity in data. The later models are used for modeling time series data. In practice, when fitting such models to a dataset, one faces three inferential problems: order selection or estimation of the number of hidden sub-populations or regimes, variable selection, and so-called post-selection statistical inference such as hypothesis testing or confidence intervals for parameters of a data-driven selected model. Despite their wide applications, rigorous methodological developments addressing the aforementioned problems in the growing literature on high-dimensional statistics have been very limited. In my short-term objectives, I will investigate new likelihood-based regularization techniques for: order selection in FMR and MSAR, and variable selection in sparse dynamic FMR and vector MSAR with fixed order and in high-dimensional settings. Establishment of such results will pave the way toward post-selection inference problems which are the subjects of my long-term objectives. II) High-dimensional imbalanced classification problems: In applications such as fraud detection, medical diagnosis, or equipment malfunction detection, classification tasks often suffer from both high-dimensionality and imbalance in the observed frequency of some classes in the training data. The latter is due to either data collection process or because some classes are indeed rare in the population. Due to data scarcity in minority class(es), conventional discriminative methods are often biased toward the majority class(es) resulting in much higher misclassification rates for the minority class(es). Imbalanced classification problems are generally hard, so I begin by studying imbalanced linear binary cases. I will investigate the utility of divide-and-conquer techniques coupled with hard-thresholding variable selection methods for bias correction in the standard linear discriminant analysis toward the minority class in high-dimensions. I will also study multi-class problems.