CIF: SMALL: Information Theoretic Foundations of Data Science

现代数据的生成、收集、交流、保存和应用带来了一门重要的新学科——数据科学。它借鉴了统计学、机器学习、计算机科学和信息论等多种技术来分析、理解，最重要的是利用数据。因此，它对从医药到营销、制造、金融和安全等各个领域产生了重大影响。这个领域的大部分初始工作都集中在简单的模型上，以促进分析。这个项目的重点是使这些结果更接近于现实世界的应用，比如自然语言处理。这项工作的结果有望带来更好的预测未来事件、检测异常和分类数据的能力。该项目的研究集中在三个技术重点上：(i)将数据科学概念扩展到实际制度，（ii）利用结构，以及（iii）理解数据源中内存的影响。关于(i)，目标是开发最优分布估计量，不仅适用于传统的Kullback-Leibler散度，而且适用于其他指标，并且适用于任意参数值，而不仅仅是有限的渐近状态。关于（ii），目标是利用结构来改进分布估计，并解决理论家和实践者使用相反分布估计技术的难题。关于（iii），总的目标是研究用内存估计分布的问题。初步结果揭示了记忆存在时压缩和学习之间有趣的二分法。通过这些技术重点，该项目将研究复杂和现实的数据模型，并得出具有重要理论意义的结果，以及在各种领域的应用。

The advent of modern data generation, collection, communication, retention, and application has brought about an important new discipline termed data science. It draws upon diverse techniques from statistics, machine learning, computer science, and information theory to analyze, understand, and most importantly, utilize, data. It therefore significantly impacts diverse fields ranging from medicine to marketing, manufacturing, finance, and security. Much of the initial work in this area has centered on simple models to to facilitate analysis. This project is focused on bringing these results closer to real-world applications such as natural language processing. The results of this work are expected to lead to a better ability to predict future events, detect anomalies, and classify data.The research in this project is focused on three technical thrusts: (i) extending data science concepts to practical regimes, (ii) utilizing structure, and (iii) understanding the effect of memory in data sources. With regards to (i), the objective is to develop optimal distribution estimators not just for the traditional Kullback-Leibler divergence, but also for other metrics, and to do so for arbitrary parameter values, not just restricted asymptotic regimes. With regards to (ii), the objective is to utilize structure to improve distribution estimates and to resolve a conundrum where theorists and practitioners employ opposing distribution estimation techniques. With regards to (iii), the general objective is to study the problem of estimating distributions with memory. Preliminary results have uncovered an interesting dichotomy between compression and learning when memory is present. Through these technical thrusts, this project will study complex and realistic data models and derive results with important theoretical implications as well as applications in a variety of fields.