Statistical Challenges and Methods in the Analysis of High Dimensional and Complex Structured Data

随着现代技术的进步，复杂的结构化数据，如相关数据和具有测量误差和高维数据出现在各种应用领域，包括医学和癌症研究，遗传研究和工业。在从现有数据中提取信息时出现了新的挑战。因此，统计方法发挥着至关重要的作用，迫切需要更有效地利用丰富的数据来源。最具挑战性的统计问题之一是解决数据中的复杂结构，以有效地识别显着的协变量，并利用获得的协变量来构建强大的预测模型。广泛的研究注意力已被导向这一领域，近年来在文献中有了快速增长。然而，现有方法的假设往往过于严格，缺乏处理数据各种具体特征的方法。本研究的主要目标是开发连贯和新颖的方法来解决数据的复杂结构，包括协变量的超高维度，不平衡的观测，协变量的测量误差，响应的测量误差，从而更好地理解底层统计结构并有效地提取有用的信息。具体而言，我计划在以下方向开发方法。*我计划构建预测模型的结果与住宿的具体功能，如不平衡的观察和测量误差。广泛使用的预测模型包括各种分类方法，如果观察是平衡的，则表现良好，但在真实的生活中，经常会出现观察不平衡并且观察到的数据包含测量误差的情况，特别是当数据通过复杂的实验（例如基因组研究中的那些实验）获得时。这项工作旨在扩大关于这一专题的现有方法的范围，并提供有用的补充工具。我计划开发的另一个领域是通过结合随机、确定性或混合设计矩阵来进行超高维变量筛选的新方法。 常见的是，大多数高维变量与响应不相关，但是现有的变量选择方法通常对于处理超高维变量不是有效的或高效的。在调用变量选择程序以确定最终统计模型之前，通过应用筛选方法来减少变量的数量至关重要。此外，调查测量误差对变量筛选的影响也很重要，我计划深入探讨这一点。拟议的方法将使人们对统计研究的许多方面产生宝贵的新见解，并将对包括医学、计算机和国防科学在内的各个领域的发展具有重大意义。

With the advancement of modern technologies, complex structured data such as dependent data and data with measurement error and high dimensionality arise in various application areas, including medical and cancer research, genetic studies, and industries. New challenges emerge when extracting information from the available data. Statistical methods therefore play a vital role, and are urgently in demand for enabling more efficient use of the rich source of data. One of the most challenging statistical issues is to address the complex structures in data to effectively identify significant covariates and to utilize the obtained covariates to construct powerful predictive models. Extensive research attention has been directed to this area and there has been a rapid growth in the literature in recent years. However, the assumptions of existing methods are often too stringent and there is a lack of methodology to deal with various specific features of the data. The primary objective of this research is to develop coherent and novel methodology to address the complex structures of data, including ultra-high dimensionality of covariates, imbalanced observations, measurement error in covariates, measurement error in response and hence, to better understand the underlying statistical structure and to efficiently extract helpful information. Specifically, I plan to develop methodologies in the following directions.******I plan to construct predictive models for outcomes with the accommodation of specific features, such as imbalanced observations and measurement errors. Widely used predictive models includes a variety of classification methods perform well if the observations are balanced, but in real life it is often the case that observations are imbalanced and the observed data contain measurement error, especially when data are obtained through complex experiments such as those in genomic studies. This work aims to broaden the scope of existing methods on this topic and offer useful complement tools.******Another area I plan to develop concerns new methods for ultra-high dimensional variable screening by incorporating random, deterministic or a mix of design matrices. It is common that the majority of the high dimensional variables do not have relevance to the response, but the existing variable selection methods are not usually valid or efficient for handling ultra high dimensional variables. It is crucial to reduce the number of variables by applying screening methods before invoking variable selection procedures to identify the final statistical models. Furthermore, it is important to investigate the measurement error effects on variable screening which I plan to explore in depth.******The proposed methodology will lead to valuable new insights into many aspects of statistical research and will be of great importance for the development of areas including medical, computer and defence sciences.**