Statistical Challenges and Methods in the Analysis of High Dimensional and Complex Structured Data

随着现代技术的进步，在医学和癌症研究、基因研究和工业等各个应用领域出现了依赖数据和具有测量误差和高维数的数据等复杂结构化数据。当从可用数据中提取信息时，会出现新的挑战。因此，统计方法发挥着至关重要的作用，迫切需要统计方法，以便更有效地利用丰富的数据来源。最具挑战性的统计问题之一是处理数据中的复杂结构，以有效地识别重要的协变量，并利用获得的协变量构建强大的预测模型。近年来，这方面的研究得到了广泛的关注，文献也有了快速的增长。然而，现有方法的假设往往过于严格，缺乏处理数据的各种具体特征的方法。本研究的主要目的是发展连贯和新颖的方法来解决复杂的数据结构，包括协变量的超高维数、不平衡的观测、协变量的测量误差、响应的测量误差，从而更好地理解潜在的统计结构，并有效地提取有用的信息。具体而言，我计划在以下方向开发方法。******我计划为结果构建预测模型，以适应特定的特征，如不平衡的观测和测量误差。广泛使用的预测模型包括多种分类方法，如果观察结果是平衡的，则表现良好，但在现实生活中，观察结果往往是不平衡的，观察到的数据包含测量误差，特别是当数据是通过复杂的实验获得的，如基因组研究。这项工作的目的是扩大现有的方法在这个主题的范围，并提供有用的补充工具。******我计划开发的另一个领域涉及通过结合随机、确定性或混合设计矩阵进行超高维变量筛选的新方法。大多数高维变量与响应不相关是很常见的，但现有的变量选择方法对于处理超高维变量通常不有效或效率不高。在调用变量选择程序来确定最终的统计模型之前，通过应用筛选方法来减少变量的数量是至关重要的。此外，重要的是研究测量误差对变量筛选的影响，这是我打算深入探讨的。******提出的方法将导致对统计研究的许多方面有价值的新见解，并将对包括医学、计算机和国防科学在内的领域的发展具有重要意义

With the advancement of modern technologies, complex structured data such as dependent data and data with measurement error and high dimensionality arise in various application areas, including medical and cancer research, genetic studies, and industries. New challenges emerge when extracting information from the available data. Statistical methods therefore play a vital role, and are urgently in demand for enabling more efficient use of the rich source of data. One of the most challenging statistical issues is to address the complex structures in data to effectively identify significant covariates and to utilize the obtained covariates to construct powerful predictive models. Extensive research attention has been directed to this area and there has been a rapid growth in the literature in recent years. However, the assumptions of existing methods are often too stringent and there is a lack of methodology to deal with various specific features of the data. The primary objective of this research is to develop coherent and novel methodology to address the complex structures of data, including ultra-high dimensionality of covariates, imbalanced observations, measurement error in covariates, measurement error in response and hence, to better understand the underlying statistical structure and to efficiently extract helpful information. Specifically, I plan to develop methodologies in the following directions.******I plan to construct predictive models for outcomes with the accommodation of specific features, such as imbalanced observations and measurement errors. Widely used predictive models includes a variety of classification methods perform well if the observations are balanced, but in real life it is often the case that observations are imbalanced and the observed data contain measurement error, especially when data are obtained through complex experiments such as those in genomic studies. This work aims to broaden the scope of existing methods on this topic and offer useful complement tools.******Another area I plan to develop concerns new methods for ultra-high dimensional variable screening by incorporating random, deterministic or a mix of design matrices. It is common that the majority of the high dimensional variables do not have relevance to the response, but the existing variable selection methods are not usually valid or efficient for handling ultra high dimensional variables. It is crucial to reduce the number of variables by applying screening methods before invoking variable selection procedures to identify the final statistical models. Furthermore, it is important to investigate the measurement error effects on variable screening which I plan to explore in depth.******The proposed methodology will lead to valuable new insights into many aspects of statistical research and will be of great importance for the development of areas including medical, computer and defence sciences.**