CIF: Small: Collaborative Research: A Unifying Approach for Identification of Sparse Interactions in Large Datasets

每天以传感器测量、网络帖子和点击、监控视频、购买交易和医疗记录的形式创建超过2.5万亿字节的数据。然而，并不是所有收集的数据都是信息性的，也不是所有的特征都与感兴趣的结果相关。虽然一些研究人员将注意力集中在压缩采样的最小误差数据重建上，以改善数据的存储和获取，但本研究的目标更广泛，主要集中在显著特征发现上。关键的洞察力是稀疏性，即在一小部分相关的观察和感兴趣的结果之间存在紧密耦合。这项研究的重点是稀疏地识别对预测结果至关重要的最相关的观测。调查人员将开发一个新的信息理论框架和算法工具，以了解感兴趣的结果和一组特征/观测之间的内在关系和稀疏交互作用，以便在这种数据泛滥的背景下提高推理能力，以增强决策。我们的目标是发现对预测结果至关重要的最相关的稀疏特征子集，并揭示相关稀疏模型的基本限制。该方法基于统一的香农信息论框架，将显著特征识别问题映射为等效信道模型的容量分析问题。这项研究以统一的方式解决了具有相关特征的模型、具有缺失特征的模型、具有潜在变量的模型以及测量过程的非线性所带来的挑战。此外，这项研究涉及开发数据驱动的稀疏恢复算法，当基本统计模型部分或完全未知时，该算法可以增强信息的价值。调查人员将使用开发的方法来加强对带有荧光传感器阵列的稀疏爆炸物混合物的检测，并使用网络层析成像来识别计算机网络中的高度枢纽。

More than 2.5 quintillion bytes of data are created daily in the form of sensor measurements, web posts and clicks, surveillance videos, purchase transactions, and health-care records. However, not all data collected is informative and not all features are relevant to the outcomes of interest. While several researchers have focused attention on compressive sampling for minimum error data reconstruction to improve data storage and acquisition, the objective of this research is broader and is focused on salient feature discovery. The key insight is sparsity, namely, that there is a tight coupling between a small relevant set of observations and the outcomes of interest. This research is focused on the sparse identification of the most relevant observations that are essential to predicting the outcomes.The investigators will develop a new information-theoretic framework and algorithmic tools for understanding the intrinsic relationships and sparse interactions between outcomes of interest and a set of features/observations in order to improve inferencing capabilities for enhanced decision-making in the context of this data-deluge. The goal is to discover the most relevant sparse subset of features that are essential to predicting the outcomes, and to uncover the fundamental limits of the associated sparse models. The approach is based on a unifying Shannon information-theoretic framework, whereby the problem of salient feature identification is mapped to a problem of capacity analysis for an equivalent channel model. This research addresses challenges posed by models with correlated features, models with missing features, models with latent variables, and the non-linearities of the measurement processes, in a unified way. Furthermore, this research involves the development of data-driven sparse recovery algorithms that reinforce the value of information when the underlying statistical models are partially or completely unknown. The investigators will use the developed methods to enhance the detection of sparse mixtures of explosives with fluorescence sensor arrays, and to identify high-degree hubs in computer networks using network tomography.