III: Small: EnrichDB - Supporting Enrichment in Database Systems

点击翻译按钮获取中文摘要

Emerging application domains such as sensor-driven smart spaces and social media platforms require incoming data to be appropriately enriched prior to being consumed by data analysts. Enrichment often requires the use of complex compiled code, declarative queries, and/or expensive machine learning/signal processing modules. Traditionally, enrichment is performed as an offline process prior to making the data available for analysis. The recent trend towards real-time analytics has prompted industrial and research systems to explore enrichment during online data processing. These efforts have focused on optimizing enrichment at the time of data ingestion. This project will develop a new type of data management technology, to support real-time data analytics, entitled EnrichDB, that represents a significant departure from the above ingestion-based enrichment approaches. EnrichDB is based on the premise that enriching data in its entirety at ingestion can be (a) wasteful -- since applications may not require all data to be enriched; (b) result in unacceptable latencies -- if data arrival rates are high, or (c) not be feasible -- if enrichment functions are learned and incorporated into the system at a later time after ingestion. EnrichDB will explore seamlessly integrating data enrichment through the entire data processing pipeline - from ingestion to event-based intermittent enrichment, and progressively during query processing. EnrichDB will benefit real-time data analytics in multiple domains including IoT-enabled smart spaces, text and social media analytics, cybersecurity, network surveillance, etc. EnrichDB will address a variety of challenges that arise in enabling enrichment through different stages of the data processing pipeline. One such challenge is to explicitly represent the state of enrichment of the objects (i.e., which enrichment functions have been applied to which objects); such a state will drive additional enrichments required downstream in data processing. Another challenge is to develop mechanisms to support enrichment during query processing efficiently since query-time enrichment could result in unacceptable latencies. Techniques need to be designed to enrich data progressively while processing queries to provide answers at acceptable levels of quality and latency. Such progressive processing logic could either be layered on top of existing databases or could be incorporated natively into database engines by rethinking storage, indexing, and query processing to support enrichment. The project will explore challenges that arise for both these cases. Finally, self-driving strategies to decide which objects should be enriched to what degree at what stage of the data processing pipelines need to be designed. Such an approach would complement a strategy wherein such decisions are made by the system administrator. EnrichDB will be used in advanced data management classes and will be integrated into a campus-level smart space testbed at UCI entitled TIPPERS that supports a variety of services including real-time occupancy counts and other building usage analysis, at the UCI campus.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.