BIGDATA: Small: DCM: JetStream: A Flexible Distributed System for Online and In-Place Data Analysis

通过使用大数据，许多研究和商业活动正在经历戏剧性的转变，其中收集和分析大型数据存储库，以揭示在较小样本中可能不明显的趋势、相关性和信息。当前的方法假定集中存储库，这可能不适合数据生成速率超过网络能力的环境。在这个项目中，pi研究了地理分布数据的实时和历史分析的系统架构，并结合了自适应减少数据量以优化带宽能力的研究。这种组合允许更好地使用与更智能的终端设备相关的计算和存储，包括但不限于分布式传感器、智能电表，甚至是完整的服务器，而不需要网络升级。考虑到计算和存储增长的历史趋势与广域网的容量限制，本研究能够以更低的总体系统成本执行更多的数据收集和分析。此外，根据可用网络带宽动态调整数据精度和保真度的能力允许系统在更高容量的网络中优雅地、自动地提高性能。这项研究能够收集和分析目前由于网络限制而无法分析的数据。这些数据可以包括更细粒度的使用数据，这可以指示减少家庭能源消耗的可操作步骤，或者它可以包括更大量的调试和监控数据，这可以更好地预测系统故障或提供比当前方法更深入的见解。

Many research and commercial endeavors are experiencing dramatic transformations through the use of Big Data, wherein large data repositories are collected and analyzed to reveal trends, correlation, and information that may not be apparent in smaller samples. Current approaches assume centralizing the repository, which may be a poor fit in environments where the data generation rate exceeds the network capabilities. In this project, the PIs investigate system architectures for both real-time and historical analysis of geographically distributed data, combined with research in adaptively reducing data volumes to optimize bandwidth capabilities. This combination allows better use of the computation and storage associated with smarter end devices, including, but not limited to, distributed sensors, smart meters, and even full servers, without requiring network upgrades. Given the historical trends of the growth of computation and storage versus the capacity limits of wide-area networks, this research enables more data collection and analysis to be performed at a lower overall system cost. Further, the ability to dynamically adapt data precision and fidelity to available network bandwidth allows systems to gracefully and automatically improve performance in the presence of higher-capacity networks. The research enables the collection and analysis of data that is currently left unanalyzed because of network constraints. Such data can include finer-granularity usage data, which could indicate actionable steps to reducing household energy consumption, or it could include a greater olume of debugging and monitoring data, which could better predict system failures or provide greater insight than with current methods.