CAREER: Distilling information structure from big and dirty data: Efficient learning of clusters and graphs in modern datasets

这个CAREER项目旨在推进从现代应用领域中出现的大而脏的数据集中提取聚类和图的理论和方法的最新进展。聚类和图提供了数据中包含的信息结构的有意义的表示，例如在神经科学和医疗保健领域，聚类具有相似表型和基因型的患者有助于确定药物设计的目标群体，由高分辨率数字表面成像（DSI）大脑扫描生成的聚类纤维轨迹有助于识别重要的神经通路，图结构可以反映大脑区域之间的连接。这项工作的结果将显著提高利用这些现代数据集的能力，通过新的方法从大规模、高维、低采样、损坏的数据中学习聚类和图，并且通常只能以压缩或流表示的形式提供。具体而言，该项目将开发计算效率高且有原则的方法来学习聚类和图，这些方法可以(i)执行无监督特征选择，以放弃高维中的无关特征，（ii）利用基于智能自适应查询的反馈，将资源集中在大多数信息变量和特征上，（iii）使用适应信息结构的压缩测量设计，以提高测量和计算效率。（iv）能够处理有噪声的流数据。这些算法将以精确描述错误聚类率和图恢复误差的形式提供性能保证。此外，该项目将研究这些问题中测量数量、计算复杂性和鲁棒性之间的权衡。与这些领域的从业者合作，将通过模拟以及它们对神经科学和医疗保健领域真实数据集的适用性来评估所开发的方法和理论。这项研究的结果可能会潜在地改变许多应用领域，这些领域涉及对相似变量进行分组，并基于大数据集和脏数据集学习它们之间的复杂交互。特别是，神经科学和医疗保健应用可能对社会产生非常直接和重要的影响。准确绘制神经通路将有助于在早期阶段诊断和治疗脑病，并有助于了解大脑功能。基于对相关遗传特征或指标的少量测量，将患者聚类并发现疾病传播途径有助于预防和治疗疾病，并最大限度地降低医疗成本。研究活动将与教育工作紧密结合，旨在培养多样化的劳动力，更好地配备跨学科工具，以应对现代数据集的挑战。该教育计划包括开发两门跨学科课程，并加强卡内基梅隆大学（CMU）统计学和机器学习联合博士课程。外展活动包括通过OurCS（计算机科学本科生研究机会）促进本科生研究，扩大女性和代表性不足群体在STEM领域的参与。美国的Leap（面向地区高中生和中学生的暑期充实项目）和针对卡内基梅隆大学高中和K-8教师的CS4HS项目。该项目的结果（包括出版物、数据集和软件）将在http://www.cs.cmu.edu/~aarti/research_projects/网站上在线传播。

This CAREER project aims to advance the state-of-the-art in theory and methods for extracting clusters and graphs from big and dirty datasets arising in modern application domains. Clusters and graphs provide a meaningful representation of the structure of information contained in data, e.g. in neuroscience and health care domains, clustering patients with similar phenotypes and genotypes helps identify target groups for drug design, clustering fiber tracks generated by high-resolution Digital Surface Imaging (DSI) scans of brains help identify significant neural pathways, and graph structures can reflect connectivity between brain regions. The results of this work will significantly enhance the ability to exploit such modern datasets through new methods for learning clusters and graphs from data that is large-scale, high-dimensional, under-sampled, corrupted, and often only available in a compressed or streaming representation. Specifically, this project will develop computationally efficient and principled methods for learning clusters and graphs that can (i) perform unsupervised feature selection to discard irrelevant features in high dimensions, (ii) leverage feedback based on intelligent adaptive queries that focus resources on most informative variables and features, (iii) use compressive measurement design that adapts to the information structure for measurement and computation efficiency, and (iv) be able to handle noisy streaming data. The algorithms will be accompanied with performance guarantees in the form of a precise characterization of the mis-clustering rate and graph recovery error. Additionally, the project will investigate the tradeoffs between number of measurements, computational complexity and robustness in these problems. The methods and theory developed will be evaluated through simulations as well as their applicability to real datasets in neuroscience and healthcare domain, in collaboration with practitioners from these fields. The results of this research could potentially transform many application domains that involve grouping similar variables and learning complex interactions between them, based on big and dirty datasets. In particular, the neuroscience and healthcare applications are likely have very direct and significant implications for society. Accurately mapping neural pathways will help diagnose and treat brain pathologies at an early stage, and help understand brain functioning. Clustering patients and discovering disease spreading pathways based on few measurements of relevant genetic features or indicators could help prevent and cure diseases, and also minimize healthcare costs. The research activities will be tightly integrated with education efforts that aim to develop a diverse workforce that is better equipped with cross-disciplinary tools to address the challenges of modern datasets. The education plan includes development of two inter-disciplinary courses, and enhancement of the joint Statistics & Machine Learning PhD program at Carnegie Mellon University (CMU). Outreach activities include promoting undergraduate research, broadening participation of women and underrepresented groups in STEM fields through OurCS (Opportunities for Undergraduate Research in Computer Science), Andrew?s Leap (a summer enrichment program for area high school and middle school students) and CS4HS program aimed at High School and K-8 teachers at Carnegie Mellon University. The results of this project (including publications, data sets, and software) will be disseminated online at http://www.cs.cmu.edu/~aarti/research_projects/.