High dimensional data clustering and pattern recognition

探索性统计学在生物信息学、医学成像和制药学等领域发挥着关键作用。从这些领域得到的数据是复杂的、大的和高维的。因此，大多数经典的统计方法对它们的分析是不够的。基于内核的方法通常会产生良好的结果。然而，它们有两个缺点：缺乏推论，以及概括性太强。前者的缺点与常见的统计问题有关，如模型选择、离群值检测和多重测试。后者与它们是通用方法有关，因此它们不能适应特定的数据点或变量关系。我们的长期目标是开发合理的统计方法，用于对高维大型数据集的探索性分析，同时考虑到上述两个缺点。我们的目标是开发模型：(A)用于发现内在数据结构(聚类)、模式(例如变量选择)和异常(多个异常值)；同时(B)适合于统计推断，以及(C)考虑数据中的结构约束。目标短期任务是：(1)使用类似Potts模型的方法对高维数据进行聚类；(2)通过共识估计聚类和聚类的数量；(3)在聚类中选择变量；(4)双聚类；以及(5)揭示变量之间的强烈关联。为了解释(B)，并从基于核的方法的良好性能中获益，我们寻求贝叶斯模型，或基于概率的模型，该模型需要非参数核密度估计器和有效使用采样机制(例如MCMC)来生成感兴趣的量的样本。指导这项研究的一些应用包括发现与某些疾病或肿瘤相关的基因组，通过与已知功能的基因关联来发现蛋白质的作用，检测药物和不良反应之间的相关关联，以及发现大脑反应的模式。

Exploratory statistics plays a key role in fields such as bioinformatics, medical imaging and pharmaceutics. Data derived from these fields are complex, large and high-dimensional. Thus most classical statistical methods are inadequate for their analysis. Kernel-based methods often yield good results. However, they have two drawbacks: lack of inference, and too much generality. The former drawback is related to common statistical issues such as model selection, outlier detection, and multiple testing. The latter is related to their being general-purpose methods, and thus they cannot accommodate for specific data point or variable relations. Our long term goal is to develop sound statistical methodology for the exploratory analysis of high-dimensional large size data sets that take into account the two drawbacks mentioned above. We aim at developing models: (A) for finding intrinsic data structure (clustering), patterns (e.g. variable selection), and anomalies (multiple outliers); that at the same time (B) are amenable  for statistical inference, and (C) take into account structural constraints in the data. Short term tasks targeted are: (i) clustering of high-dimensional data using Potts model-like methods; (ii) estimating the clusters and the number of clusters by consensus; (iii) variable selection within clustering; (iv) biclustering; and (v) uncovering strong associations between variables. To account for (B), and to profit from the good performance of kernel-based methods, we seek Bayesian models, or probability-based models that entail non-parametric kernel density estimators and efficient use of sampling mechanisms (e.g. MCMC) to generate samples of the quantities of interest. Some applications guiding this research are discovering groups of genes associated to certain diseases or tumors, finding the role of proteins through association with those whose functions are known, detecting relevant association between drugs and adverse reactions, and discovering patterns in brain responses.