III: Small: EnrichDB - Supporting Enrichment in Database Systems

传感器驱动的智能空间和社交媒体平台等新兴应用领域需要在数据分析师使用之前对传入数据进行适当的丰富。丰富通常需要使用复杂的编译代码、声明性查询和/或昂贵的机器学习/信号处理模块。传统上，富集在使数据可用于分析之前作为离线过程执行。最近的实时分析趋势促使工业和研究系统在在线数据处理过程中探索丰富。这些努力集中在数据摄取时优化富集。该项目将开发一种新型的数据管理技术，以支持名为EnrichDB的实时数据分析，该技术与上述基于摄取的富集方法有很大不同。EnrichDB基于这样的前提，即在摄取时完整地丰富数据可能是（a）浪费的--因为应用程序可能不需要丰富所有数据;（B）导致不可接受的延迟--如果数据到达率很高，或者（c）不可行--如果在摄取后的稍后时间学习丰富功能并将其并入系统。EnrichDB将探索通过整个数据处理管道无缝集成数据丰富-从摄取到基于事件的间歇性丰富，并在查询处理过程中逐步进行。EnrichDB将有利于多个领域的实时数据分析，包括物联网智能空间、文本和社交媒体分析、网络安全、网络监控等，EnrichDB将解决在数据处理管道的不同阶段实现丰富时出现的各种挑战。一个这样的挑战是明确地表示对象的富集状态（即，哪些富集函数已经被应用于哪些对象）;这样的状态将驱动数据处理中下游所需的附加富集。另一个挑战是开发在查询处理期间有效地支持充实的机制，因为查询时充实可能导致不可接受的延迟。技术需要被设计成在处理查询的同时逐步丰富数据，以提供质量和延迟可接受的答案。这种渐进式处理逻辑可以分层在现有数据库之上，也可以通过重新思考存储、索引和查询处理来本地合并到数据库引擎中，以支持丰富。该项目将探讨这两种情况下出现的挑战。最后，需要设计自驱动策略来决定哪些对象应该在数据处理管道的哪个阶段被丰富到什么程度。这种方法将补充由系统管理员作出此类决定的战略。EnrichDB将用于高级数据管理课程，并将集成到UCI的校园级智能空间测试平台TIPPERS中，该平台支持UCI校园的各种服务，包括实时占用计数和其他建筑使用分析。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Emerging application domains such as sensor-driven smart spaces and social media platforms require incoming data to be appropriately enriched prior to being consumed by data analysts. Enrichment often requires the use of complex compiled code, declarative queries, and/or expensive machine learning/signal processing modules. Traditionally, enrichment is performed as an offline process prior to making the data available for analysis. The recent trend towards real-time analytics has prompted industrial and research systems to explore enrichment during online data processing. These efforts have focused on optimizing enrichment at the time of data ingestion. This project will develop a new type of data management technology, to support real-time data analytics, entitled EnrichDB, that represents a significant departure from the above ingestion-based enrichment approaches. EnrichDB is based on the premise that enriching data in its entirety at ingestion can be (a) wasteful -- since applications may not require all data to be enriched; (b) result in unacceptable latencies -- if data arrival rates are high, or (c) not be feasible -- if enrichment functions are learned and incorporated into the system at a later time after ingestion. EnrichDB will explore seamlessly integrating data enrichment through the entire data processing pipeline - from ingestion to event-based intermittent enrichment, and progressively during query processing. EnrichDB will benefit real-time data analytics in multiple domains including IoT-enabled smart spaces, text and social media analytics, cybersecurity, network surveillance, etc. EnrichDB will address a variety of challenges that arise in enabling enrichment through different stages of the data processing pipeline. One such challenge is to explicitly represent the state of enrichment of the objects (i.e., which enrichment functions have been applied to which objects); such a state will drive additional enrichments required downstream in data processing. Another challenge is to develop mechanisms to support enrichment during query processing efficiently since query-time enrichment could result in unacceptable latencies. Techniques need to be designed to enrich data progressively while processing queries to provide answers at acceptable levels of quality and latency. Such progressive processing logic could either be layered on top of existing databases or could be incorporated natively into database engines by rethinking storage, indexing, and query processing to support enrichment. The project will explore challenges that arise for both these cases. Finally, self-driving strategies to decide which objects should be enriched to what degree at what stage of the data processing pipelines need to be designed. Such an approach would complement a strategy wherein such decisions are made by the system administrator. EnrichDB will be used in advanced data management classes and will be integrated into a campus-level smart space testbed at UCI entitled TIPPERS that supports a variety of services including real-time occupancy counts and other building usage analysis, at the UCI campus.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.