BIGDATA: Small: DA: Statistical Machine Learning Methods for Scalable Data Analysis

大数据在现代工业和科学应用中无处不在，数据的大小和维度变得如此之大，以至于需要新的统计工具来进行有效的数据分析。这个合作项目涉及罗格斯大学和微软研究院的研究人员，重点是大数据分析高级计算方法的理论和算法开发。虽然要调查的问题是由各种互联网应用的动机，由此产生的解决方案，预计将广泛适用于其他领域。该项目考虑了大数据分析中三个相互关联的主题：（a）对大数据集进行有效采样，以过滤掉不可靠的数据源并改进统计分析;（B）降维技术，可以通过哈希和稀疏随机投影技术最好地保留信息;（c）用于机器学习的大规模优化技术，可以直接处理大数据量。这项工作的预期成果包括新的理论成果、新的数据分析算法及其开源软件实现。这项研究的更广泛影响包括广泛传播的可扩展数据分析算法的开源实现、基于研究的培训和对研究生和本科生的教育、以及学术与工业的合作，导致机器学习基础研究与工业应用之间的相互作用。

Big Data has become ubiquitous in modern industrial and scientific applications where the size and dimensionality of data are becoming so large as to require new statistical tools for efficient data analysis. This collaborative project involving researchers at Rutgers University and Microsoft Research focuses on the theoretical and algorithmic development of advanced computational methods for big data analytics. While the problems to be investigated are motivated by various Internet applications, the resulting solutions are expected to be broadly applicable to other domains. The project considers three interrelated main themes in big data analytics: (a) effective sampling of big datasets to filter out unreliable data source and improve statistical analysis; (b) dimensionality reduction techniques that can best preserve information via hashing and sparse random projection techniques; and (c) large scale optimization techniques for machine learning that can directly handle large datasize. Anticipated results of this work include new theoretical results, new data analytics algorithms, and their open source software implementations.Broader impacts of the research include broadly disseminated open source implementations of scalable data analytics algorithms, research-based training and education of graduate and undergraduate students, and academic-industrial collaborations resulting in an interplay between fundamental research in machine learning and industrial applications.