Optimization and Statistical Procedures for Big Data and Applications

在大数据时代，数据集的规模可能非常庞大，并且具有很高的维度。例子包括社交媒体数据、高分辨率图像数据和基因组数据。高维和大量的样本构成了巨大的计算和统计挑战。本项目旨在为重要应用中收集的大数据研究新的优化技术和统计分析工具。该项目将显著提升大数据分析的优化和统计能力。预计该项目的成果将惠及广泛的领域，包括公共卫生、医学研究和财务组合管理。该项目包括三个子项目，以推进现代优化技术和大数据统计程序的知识。(1)研究了高维约束正则化统一框架，并进一步研究了折叠凹罚和固定约束下高维数据通用框架的统计性能。(2)探索约束统计学习的高效分布式算法，研究具有通信效率的样本分割算法，以处理具有折叠凹罚的高维约束正则化问题中大量样本和高通信成本的问题。该项目将研究非凸学习新算法的收敛性以及统计收敛性。研究表明统计分析和优化分析同时考虑的必要性。(3)研究人员计划将该方法应用于分析大数据集，以解决与帕金森病相关的临床和科学问题。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

In the era of big data, data sets may be massive in size and have high dimensionality. Examples include social media data, high-resolution image data, and genomic data. High dimensionality and massive sample size pose great computational and statistical challenges. This project aims to investigate novel optimization techniques and statistical analytic tools for big data collected in important applications. The project will significantly enhance the capabilities of optimization and statistics in analyzing big data. Results of the project are anticipated to benefit a broad range of areas including public health, medical studies, and financial portfolio management.This project consists of three sub-projects to advance knowledge in modern optimization techniques and statistical procedures for big data. (1) The project studies a unified framework for high-dimensional constrained regularization, and further investigates the statistical performance of a general framework for high-dimensional data under folded concave penalty and fixed constraints. (2) The project explores efficient distributed algorithms for constrained statistical learning, investigating communication-efficient sample-splitting algorithms to handle the vast number of samples and high communication cost in high-dimensional constrained regularization problems with folded concave penalty. The project will study the convergence of the new algorithms for non-convex learning as well as statistical convergence. The work demonstrates the necessity to consider statistical analysis and optimization analysis simultaneously. (3) The investigators plan to apply the methodology in analyzing big data sets to address clinical and scientific questions related to Parkinson's disease.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.