CSR: Small: Collaborative Research: Dispersed Real-time Analytics

近年来，各种移动应用、传感器和物联网(IoT)设备产生的数据呈爆炸式增长，分布在多个地理位置。必须对这些数据进行汇总和分析，以获得跨多个领域的实时洞察。计算平台的异构性也大幅增加：计算和存储资源可在互联网边缘的最终用户设备和数据中心以及集中式云中使用。该项目提出了利用这种分散的计算平台来执行大量数据的实时处理的算法和体系结构。从海量数据中获得实时洞察是数字经济的关键，因为来自用户和传感器的信息必须得到快速处理，才能立即获得可操作的洞察。该项目旨在使这些见解变得可行，从而提高生产率和用户满意度。数据科学和分析是美国劳动力需求的一个关键领域。拟议的教育和外展活动包括新的课程材料和本科生研究经验(REU)计划，旨在提供更多的这一领域的接触。该项目还包括强有力的外展举措，以吸引女性和代表性不足的少数群体参与数据分析研究。该项目为跨大量自主资源提供商(ARP)的分散实时分析系统提出了一个两级体系结构：一个全球子系统在多个资源提供商之间端到端分配资源，以及一个本地子系统在每个ARP内调度资源。全局资源分配过程引入了聚合树的概念，该聚合树用于跨多个ARP发现、分配和协调资源。本地资源分配过程协调ARP内计算和数据的移动。在这两个级别上，研究了处理资源的动态可变性和故障恢复能力的技术。作为这个项目的结果而开发的所有软件都将被清楚地记录下来并开放源码。对于从公共来源收集的任何数据集，将发布足够的元数据，以使其他人能够将数据重复用于自己的目的。该项目制作的技术报告和论文将出版并与公众分享。该项目产生的信息将被维护、保存，并在NSF要求的持续时间内可用。一个全面的网站将提供对这些信息的访问。网站网址为：http://geo-anal.cs.umn.edu.

Recent years have seen an explosion of data produced by a wide variety of mobile applications, sensors, and Internet of Things (IoT) devices spread across multiple geographic locations. This data must be aggregated and analyzed to gain real-time insights across several domains. There has also been a dramatic increase in the heterogeneity of the computing platform: compute and storage resources are available at end-user devices and data centers at the edge of the Internet, as well as at centralized clouds. This project proposes algorithms and architectures for leveraging such a dispersed computing platform to perform real-time processing of large volumes of data. Gaining real-time insights from large volumes of data is at the crux of the digital economy, as information from users and sensors must be processed quickly to gain immediate actionable insights. This project aims to make such insights feasible leading to greater productivity and user satisfaction. Data science and analytics is a key area of workforce demand in the US. The proposed educational and outreach activities include new course material and Research Experiences for Undergraduates (REU) programs targeted to provide greater exposure in this area. This project also includes strong outreach initiatives to attract both women and under-represented minorities to data analytics research.This project proposes a two-level architecture for a dispersed real-time analytics system across a large number of autonomous resource providers (ARPs): a global subsystem that allocates resources end-to-end across multiple resource providers, and a local subsystem that schedules resources within each ARP. The global resource allocation process introduces the notion of an aggregation tree that is used for discovering, allocating, and coordinating resources across multiple ARPs. The local resource allocation process orchestrates the movement of computation and data within an ARP. Across both levels, techniques for dealing with dynamic variability in resources and resilience to failure are investigated. All software developed as a result of this project will be clearly documented and open-sourced. For any datasets collected from public sources, sufficient meta-data will be published to enable others to reuse the data for their own purposes. The technical reports and papers produced by the project will be published and shared with the general public. The information generated by the project will be maintained, preserved, and made available for the duration required by NSF. A comprehensive website will provide access to this information. The website URL is: http://geo-anal.cs.umn.edu.