NEARDATA - Extreme Near-Data Processing Platform

NEARDATA - 极限近数据处理平台

• 批准号: 10048448
• 金额: $ 39.53万
• 依托单位: UK HEALTH SECURITY AGENCY
• 依托单位国家: 英国
• 项目类别: EU-Funded
• 财政年份: 2023
• 资助国家: 英国
• 起止时间: 2023 至 无数据
• 项目状态: 未结题
• 来源: https://gtr.ukri.org/projects?ref=10048448
• 关键词: NEARDATA; Extreme; Near; Data; Processing

The main goal is to design an Extreme near-data platform to enable consumption, mining and processing of dis- tributed and federated data without needing to master the logistics of data access across heterogeneous data locations and pools. We go beyond traditional passive or bulk data ingested from storage systems towards next generation near-data processing platforms both in the Cloud and in the Edge. In our platform, Extreme Data in- cludes both metadata and trustworthy data connectors enabling advanced data management operations like data discovery, mining, and filtering from heterogeneous data sources. The three core objectives are: O-1 Provide high-performance near-data processing for Extreme Data Types: The first objective is to create a novel intermediary data service (XtremeDataHub) providing serverless data connectors that optimize data management operations (partitioning, filtering, transformation, aggregation) and interactive queries (search, discovery, matching, multi-object queries) to efficiently present data to analytics platforms. Our data connectors facilitate a elas- tic data-driven process-then-compute paradigm which significantly reduces data communication on the data interconnect, ultimately resulting in higher overall data throughput. O-2 Support real-time video streams but also event streams that must be ingested and processed very fast to Object Storage: The second objective is to seamlessly combine streaming and batch data processing for analytics. To this end, we will develop stream data connectors deployed as stream operators offering very fast stateful computations over low-latency event and video streams. O-3 The third objective is to create a Data Broker service enabling trustworthy data sharing and confidential orchestration of data pipelines across the Compute Continuum. We will provide secure data orchestration, transfer, processing and access thanks to Trusted Execution Environments (TEEs) and federated learning architectures

其主要目标是设计一个Extreme近数据平台，以实现对分布式和联合数据的消费、挖掘和处理，而无需掌握跨异构数据位置和池的数据访问的逻辑。我们超越了从存储系统中摄取的传统被动或批量数据，转向云和边缘中的下一代近数据处理平台。在我们的平台中，Extreme Data包含元数据和可靠的数据连接器，可实现高级数据管理操作，如数据发现、挖掘和从异构数据源中筛选。三个核心目标是：O-1为极端数据类型提供高性能的近数据处理：第一个目标是创建一个新的中间数据服务（XtremeDataHub），提供无服务器数据连接器，优化数据管理操作（分区，过滤，转换，聚合）和交互式查询（搜索，发现，匹配，多对象查询），以有效地将数据呈现给分析平台。我们的数据连接器促进了弹性数据驱动的先处理后计算的范例，显著减少了数据互连上的数据通信，最终提高了整体数据吞吐量。O-2支持实时视频流，但也支持必须非常快速地摄取和处理到对象存储的事件流：第二个目标是无缝地将联合收割机流和批处理数据处理结合起来进行分析。为此，我们将开发流数据连接器，部署为流操作器，在低延迟事件和视频流上提供非常快速的有状态计算。O-3第三个目标是创建一个Data Broker服务，使跨Compute Continuum的数据管道的可信数据共享和机密编排成为可能。我们将通过可信执行环境（TEE）和联合学习架构提供安全的数据编排、传输、处理和访问