CSR: Small: Collaborative Research: Dispersed Real-time Data Analytics

近年来，由分布在多个地理位置的各种各样的移动的应用、传感器和物联网（IoT）设备产生的数据呈爆炸式增长。必须对这些数据进行汇总和分析，以获得跨多个领域的实时见解。计算平台的异构性也急剧增加：计算和存储资源在互联网边缘的最终用户设备和数据中心以及集中式云上都可用。该项目提出了利用这种分散的计算平台来执行大量数据的实时处理的算法和架构。从大量数据中获得实时见解是数字经济的关键，因为必须快速处理来自用户和传感器的信息，以获得即时的可操作见解。该项目旨在使这种见解可行，从而提高生产力和用户满意度。数据科学和分析是美国劳动力需求的关键领域。拟议的教育和推广活动包括新的课程材料和本科生研究经验（REU）计划，旨在提供更多的曝光在这一领域。该项目还包括强有力的外联举措，以吸引女性和代表性不足的少数群体参与数据分析研究。该项目提出了一个两级架构，用于在大量自治资源提供者（ARP）之间建立分散的实时分析系统：一个全球子系统，在多个资源提供者之间端到端分配资源;一个本地子系统，在每个阿普内调度资源。全局资源分配过程引入了聚合树的概念，聚合树用于跨多个ARP发现、分配和协调资源。本地资源分配过程协调阿普内的计算和数据移动。在这两个层次上，用于处理资源的动态变化和故障恢复能力的技术进行了研究。作为该项目的结果开发的所有软件将被明确记录和开源。对于从公共来源收集的任何数据集，将公布足够的元数据，使其他人能够为自己的目的重新使用数据。该项目编写的技术报告和论文将出版并与公众分享。项目生成的信息将在NSF要求的期限内得到维护、保存和提供。一个综合网站将提供这些信息。网址是：http://geo-anal.cs.umn.edu。

Recent years have seen an explosion of data produced by a wide variety of mobile applications, sensors, and Internet of Things (IoT) devices spread across multiple geographic locations. This data must be aggregated and analyzed to gain real-time insights across several domains. There has also been a dramatic increase in the heterogeneity of the computing platform: compute and storage resources are available at end-user devices and data centers at the edge of the Internet, as well as at centralized clouds. This project proposes algorithms and architectures for leveraging such a dispersed computing platform to perform real-time processing of large volumes of data. Gaining real-time insights from large volumes of data is at the crux of the digital economy, as information from users and sensors must be processed quickly to gain immediate actionable insights. This project aims to make such insights feasible leading to greater productivity and user satisfaction. Data science and analytics is a key area of workforce demand in the US. The proposed educational and outreach activities include new course material and Research Experiences for Undergraduates (REU) programs targeted to provide greater exposure in this area. This project also includes strong outreach initiatives to attract both women and under-represented minorities to data analytics research.This project proposes a two-level architecture for a dispersed real-time analytics system across a large number of autonomous resource providers (ARPs): a global subsystem that allocates resources end-to-end across multiple resource providers, and a local subsystem that schedules resources within each ARP. The global resource allocation process introduces the notion of an aggregation tree that is used for discovering, allocating, and coordinating resources across multiple ARPs. The local resource allocation process orchestrates the movement of computation and data within an ARP. Across both levels, techniques for dealing with dynamic variability in resources and resilience to failure are investigated. All software developed as a result of this project will be clearly documented and open-sourced. For any datasets collected from public sources, sufficient meta-data will be published to enable others to reuse the data for their own purposes. The technical reports and papers produced by the project will be published and shared with the general public. The information generated by the project will be maintained, preserved, and made available for the duration required by NSF. A comprehensive website will provide access to this information. The website URL is: http://geo-anal.cs.umn.edu.