BIGDATA: Small: DCM: JetStream: A Flexible Distributed System for Online and In-Place Data Analysis

通过使用大数据，许多研究和商业努力正在经历巨大的转变，其中收集和分析大型数据存储库以揭示在较小样本中可能不明显的趋势，相关性和信息。当前的方法假设集中存储库，这可能不适合数据生成速率超过网络能力的环境。在这个项目中，PI研究系统架构，用于地理分布数据的实时和历史分析，并结合自适应减少数据量以优化带宽能力的研究。这种组合允许更好地使用与智能终端设备相关的计算和存储，包括但不限于分布式传感器、智能电表甚至完整的服务器，而无需网络升级。考虑到计算和存储增长的历史趋势与广域网的容量限制，这项研究使更多的数据收集和分析，以较低的整体系统成本进行。 此外，动态地使数据精度和保真度适应可用网络带宽的能力允许系统在存在更高容量网络的情况下优雅地且自动地提高性能。这项研究使收集和分析数据，目前是由于网络的限制，未经分析。这样的数据可以包括更细粒度的使用数据，这可以指示减少家庭能耗的可行步骤，或者它可以包括更大范围的调试和监控数据，这可以更好地预测系统故障或提供比当前方法更好的洞察力。

Many research and commercial endeavors are experiencing dramatic transformations through the use of Big Data, wherein large data repositories are collected and analyzed to reveal trends, correlation, and information that may not be apparent in smaller samples. Current approaches assume centralizing the repository, which may be a poor fit in environments where the data generation rate exceeds the network capabilities. In this project, the PIs investigate system architectures for both real-time and historical analysis of geographically distributed data, combined with research in adaptively reducing data volumes to optimize bandwidth capabilities. This combination allows better use of the computation and storage associated with smarter end devices, including, but not limited to, distributed sensors, smart meters, and even full servers, without requiring network upgrades. Given the historical trends of the growth of computation and storage versus the capacity limits of wide-area networks, this research enables more data collection and analysis to be performed at a lower overall system cost. Further, the ability to dynamically adapt data precision and fidelity to available network bandwidth allows systems to gracefully and automatically improve performance in the presence of higher-capacity networks. The research enables the collection and analysis of data that is currently left unanalyzed because of network constraints. Such data can include finer-granularity usage data, which could indicate actionable steps to reducing household energy consumption, or it could include a greater olume of debugging and monitoring data, which could better predict system failures or provide greater insight than with current methods.