CAREER: Scalable and Adaptive Edge Stream Processing

自动驾驶汽车、增强现实、互动游戏和事件监控等物联网(IoT)应用程序具有改善我们生活的巨大潜力。这些应用程序产生了大量大规模的传感器数据。在许多时间关键型场景下，必须在非常短的时间内处理这些海量数据流，以获取可操作的情报。该职业项目旨在通过将流处理范例应用于边缘计算架构来支持时间关键型物联网应用。这项研究的成功将使工厂自动化、触觉互联网、自动驾驶汽车和过程自动化等领域的许多时间关键型物联网应用受益。它还将大大改善各种数据处理系统的性能，如广域数据分析系统、移动数据访问系统、事件跟踪系统和流数据库。作为其研究计划的组成部分，该职业项目涉及与当地公立学校系统合作的K-12、本科生和研究生水平的教育。具体地说，该职业项目将构建一个可扩展和自适应的边缘流处理引擎，该引擎能够在动态、异构的边缘环境中快速处理大量并发的物联网查询。本工作包括三个主要研究方向。首先，实现了一种新的动态数据流图抽象，它自动链接、并行和复制流运算符，以适应Edge的动态变化并以可扩展的方式处理故障。其次，将实现新的可定制的数据洗牌服务抽象，其针对给定的网络拓扑和工作负载在运行时定制数据洗牌路径(例如，环形洗牌、分层树洗牌、蝴蝶包裹洗牌)。第三，将实现具有多个分布式调度器的完全去中心化的体系结构，其中每个调度器自主操作来处理物联网查询。该项目的所有三个部分都将在真实世界的流处理系统上进行原型和实施，并通过执行真实世界的实验进行验证。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Internet-of-Things (IoT) applications such as self-driving cars, augmented reality, interactive gaming, and event monitoring have a tremendous potential to improve our lives. These applications generate a large influx of sensor data at massive scales. Under many time-critical scenarios, these massive data streams must be processed in a very short time to derive actionable intelligence. This CAREER project aims to support time-critical IoT applications by applying the stream processing paradigm to the Edge computing architecture. The success of this research will benefit many time-critical IoT applications in the areas such as factory automation, the tactile internet, autonomous vehicles, and process automation. It will also substantially improve the performance profiles of a variety of data processing systems, such as wide-area data analytics systems, mobile data access systems, event tracking systems, and streaming databases. As an integral part of its research program, this CAREER project involves K-12, undergraduate and graduate level education in partnership with the local Public School system.Specifically, this CAREER project will build a scalable and adaptive Edge stream processing engine, which enables fast stream processing of a large number of concurrent IoT queries in the dynamic, heterogeneous Edge environment. This work includes three primary research directions. First, a new dynamic dataflow graph abstraction will be implemented, which automatically chains, parallelizes and replicates stream operators to adapt to the Edge dynamics and handle failures in a scalable way. Second, a new customizable data shuffling service abstraction will be implemented, which customizes the data shuffling path (e.g., ring shuffle, hierarchical tree shuffle, butterfly wrap shuffle) at runtime for the given network topology and workload. Third, a fully decentralized architecture with many distributed schedulers will be implemented, in which each scheduler operates autonomously to process IoT queries. All three parts of the project will be prototyped and implemented on real-world stream processing systems and validated by performing real-world experiments.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.