SPX: Collaborative Research: Multicore to Wide Area Analytics on Streaming Data

在当今的大数据时代，迫切需要能够从不断生成的大量数据中快速获得分析见解的方法。这些流数据包括视频、音频、活动日志和传感器数据，并在世界各地大规模生成。实时流分析的需求只有在适当设计的并行和分布式算法的帮助下才能实现。然而，并行和分布式计算系统有各种各样的形状和大小，并且应该设计算法来匹配底层系统的特征。该项目开发了分析计算系统上大量流数据的方法，从具有多核共享内存的机器到通过广域网通信的地理分布式数据中心。本研究的结果有望提高流分析的效率、延迟和吞吐量。由于所考虑的分析任务的基础性质，该项目的结果将影响使用大规模机器学习和图形分析的学科，包括网络安全，社交网络分析和交通。由此产生的软件将作为流处理平台上的工具包发布，并部署在智能城市摄像机基础设施中。pi的研究目标和教学目标之间的协同作用将导致现有课程中新的教学材料以及数据分析新课程的开发。来自代表性不足群体的个人将被纳入项目的一部分。该项目将受益于并加强学术界、工业界和国家实验室在流媒体分析方面的合作。该项目的第一个技术重点是设计用于数据流计算的共享内存并行算法，该算法可以实现高吞吐量和快速收敛，以完成复杂的分析任务。第二个重点是设计分布式流算法，通过确定结果的新鲜度和通信量之间的良好权衡，可以容忍可变的通信延迟并适应广域网中的可用带宽。这些进展将在基本图分析和机器学习任务的背景下进行研究，如子图计数、图连通性和聚类、矩阵分解和深度网络。该项目将利用并行计算领域中开发的大量理论和技术来设计处理流数据的方法，从而形成一个可以跨应用程序重用的技术工具包。该项目还将导致某些问题的顺序流和增量算法的进步；例如，机器学习中使用迭代收敛方法的问题。基于所设计的技术，该项目将设计和构建一个分层参数服务器，该服务器可以从多核机器到数据中心再到广域数据源有效地运行。

In today's big data era, there is an urgent need for methods that can quickly derive analytical insights from large volumes of data that are continuously generated. Such streaming data include video, audio, activity logs, and sensor data, and are generated on a massive scale all over the world. The need for real-time streaming analytics can only be fulfilled with the help of appropriately designed parallel and distributed algorithms. However, parallel and distributed computing systems come in a variety of shapes and sizes, and algorithms should be designed to match the characteristics of the underlying system. This project develops methods for analyzing massive streaming data on computing systems ranging from machines with multiple cores sharing memory to geo-distributed data centers communicating over wide-area networks. The results of this research are expected to improve the efficiency, latency, and throughput of streaming analytics. Due to the foundational nature of the analytical tasks considered, results of this project will impact disciplines that use large-scale machine learning and graph analytics, including cybersecurity, social network analysis, and transportation. Resulting software will be released as toolkits on stream processing platforms, and deployed in a smart-city camera infrastructure. Synergy between the research goals and the teaching goals of the PIs will lead to new instructional material in existing courses as well as development of new courses in data analytics. Individuals from underrepresented groups will be included as a part of the project. The project will benefit from and strengthen collaborations between academia, industry, and national labs on streaming analytics. The first technical thrust of the project is on designing shared memory parallel algorithms for computation on data streams, that can achieve a high throughput and fast convergence for complex analytics tasks. The second thrust is on designing distributed streaming algorithms that can tolerate variable communication delays and adapt to available bandwidth in a wide-area network, through identifying good tradeoffs between freshness of results and volume of communication. These advances will be studied in the context of fundamental graph analytics and machine learning tasks such as subgraph counting, graph connectivity and clustering, matrix factorization, and deep networks. The project will utilize the vast body of theory and techniques developed in the realm of parallel computing in the design of methods for processing streaming data, leading to a toolkit of techniques that can be reused across applications. The project will also lead to advances in sequential streaming and incremental algorithms for certain problems; for instance, problems in machine learning that use iterative convergent methods. Based on the techniques designed, the project will design and build a hierarchical parameter server that operates effectively across the spectrum from multicore machines to data centers to wide-area data sources.