NEARDATA - Extreme Near-Data Processing Platform

主要目标是设计一个Extreme近数据平台，以支持分布式和联合数据的消费、挖掘和处理，而无需掌握跨异构数据位置和池的数据访问物流。我们超越了从存储系统中摄取的传统被动或批量数据，转向了云计算和边缘计算中的下一代近数据处理平台。在我们的平台中，Extreme Data In包括元数据和可信数据连接器，支持高级数据管理操作，如数据发现、挖掘和从异构数据源过滤。O-1为极端数据类型提供高性能的近数据处理：第一个目标是创建一个新的中间数据服务（XtremeDataHub），提供无服务器数据连接器，优化数据管理操作（分区、过滤、转换、聚合）和交互式查询（搜索、发现、匹配、多对象查询），从而有效地将数据呈现给分析平台。我们的数据连接器促进了一个快速的数据驱动的过程-然后计算范式，这大大减少了数据互连上的数据通信，最终导致更高的整体数据吞吐量。O-2支持实时视频流，但也支持事件流，这些事件流必须被快速地吸收和处理到对象存储中：第二个目标是无缝地结合流和批处理数据处理以进行分析。为此，我们将开发作为流操作符部署的流数据连接器，在低延迟事件和视频流上提供非常快速的状态计算。第三个目标是创建一个数据代理服务，支持可信的数据共享和跨计算连续体的数据管道的机密编排。我们将通过可信执行环境（tee）和联邦学习架构提供安全的数据编排、传输、处理和访问

The main goal is to design an Extreme near-data platform to enable consumption, mining and processing of dis- tributed and federated data without needing to master the logistics of data access across heterogeneous data locations and pools. We go beyond traditional passive or bulk data ingested from storage systems towards next generation near-data processing platforms both in the Cloud and in the Edge. In our platform, Extreme Data in- cludes both metadata and trustworthy data connectors enabling advanced data management operations like data discovery, mining, and filtering from heterogeneous data sources. The three core objectives are: O-1 Provide high-performance near-data processing for Extreme Data Types: The first objective is to create a novel intermediary data service (XtremeDataHub) providing serverless data connectors that optimize data management operations (partitioning, filtering, transformation, aggregation) and interactive queries (search, discovery, matching, multi-object queries) to efficiently present data to analytics platforms. Our data connectors facilitate a elas- tic data-driven process-then-compute paradigm which significantly reduces data communication on the data interconnect, ultimately resulting in higher overall data throughput. O-2 Support real-time video streams but also event streams that must be ingested and processed very fast to Object Storage: The second objective is to seamlessly combine streaming and batch data processing for analytics. To this end, we will develop stream data connectors deployed as stream operators offering very fast stateful computations over low-latency event and video streams. O-3 The third objective is to create a Data Broker service enabling trustworthy data sharing and confidential orchestration of data pipelines across the Compute Continuum. We will provide secure data orchestration, transfer, processing and access thanks to Trusted Execution Environments (TEEs) and federated learning architectures