III-COR: Scalable Publish/Subscribe: Unifying Data Processing and Dissemination

数字时代带来了前所未有的增长，数据生成量，数据消费者的数量以及他们的兴趣和位置的多样性。 传统上，用户轮询信息源，但对于许多应用程序，轮询是难以扩展的，可能会错过重要的事件。 发布/订阅系统提供的替代方案是将通知推送给具有匹配兴趣的用户。 这种方法适用于许多应用，从个人、商业、医疗到环境、军事和安全。 然而，传统的发布/订阅系统对于高级应用程序来说是不足够的，用户需要在满足一定条件的情况下接收经过过滤、连接和汇总的信息，本项目旨在构建下一代发布/订阅系统以应对新的挑战。 PI提出了一个端到端的解决方案，包括从订阅处理和索引到分发网络设计的技术，这些技术共同支持高效和强大的订阅功能，允许用户精确地控制他们想要什么以及何时想要它。一个主要的特点是将所提出的方法与以前的工作区分开来，这是订阅处理和通知分发的联合考虑。 传统上，这些问题被数据库和网络社区分开考虑。然而，存在着广泛的有趣的替代方案，用于将处理与传播连接起来。 PI提出了一种有前途的方法，允许复杂的，有状态的订阅由简单的，无状态的传播机制处理，具有易于实现和扩展的清洁系统设计。 一个基于成本的优化器，受数据库查询优化的启发，联合动态地选择最佳的处理和传播策略。除了系统构建，该项目还解决了许多新的算法挑战，包括，例如，可扩展地处理大量复杂的订阅;利用事件和订阅特性来对抗最坏情况的复杂性;在传播网络设计中平衡语义相似性和网络邻近性;以及有效地维护高维事件和订阅的统计数据。更广泛的影响：将他们在数据库和算法方面的专业知识结合在一起，PI拥有相互合作以及与计算机科学之外的研究人员合作的良好记录。 该计划的一个应用是帮助生态学家进行环境监测。PI致力于整合研究和教育，特别是本科生研究，遵循他们的传统，通过REU和部门奖学金资助本科生。 PI正在教育部的支持下合作努力，以增加妇女，少数民族和残疾人的劳动力多样性。 他们还参加了一个针对弱势群体的实习项目，该项目为少数族裔学生提供了与杜克大学教职员工一起进行带薪暑期研究的机会。http://www.cs.duke.edu/dbgroup/prosem/

The Digital Age has brought about unprecedented growth in the amountof data being generated, the number of data consumers, and thediversity of their interests and locations. Traditionally, users pollsources for information, but for many applications, polling is hardlyscalable and may miss important events. The alternative offered bypublish/subscribe systems is to push notifications to users withmatching interests. This approach suits many applications, rangingfrom personal, commercial, medical, to environmental, military, andsecurity. However, traditional publish/subscribe systems are becominginadequate for advanced applications, where users want to receiveinformation that has been filtered, joined, and summarized, and onlywhen certain conditions are met.This project aims at building a next-generation publish/subscribesystem to face the new challenges. The PIs propose an end-to-endsolution consisting of techniques from subscription processing andindexing to dissemination network design, which work together tosupport efficient and powerful subscription functionalities, allowingusers to control precisely what they want and when they want it.One main feature distinguishing the proposed approach from previouswork is joint consideration of subscription processing andnotification dissemination. Traditionally, these problems areconsidered separately by database and networking communities.However, there exists a wide spectrum of interesting alternatives forinterfacing processing with dissemination. The PIs propose apromising approach that allows complex, stateful subscriptions to behandled by simple, stateless dissemination mechanisms, with a cleansystem design that is easy to implement and scale. A cost-basedoptimizer, inspired by database query optimization, chooses the bestprocessing and dissemination strategies jointly and dynamically.Besides system building, this project tackles many new algorithmicchallenges, including, e.g., scalably processing a large number ofcomplex subscriptions; exploiting event and subscriptioncharacteristics to combat worst-case complexity; balancing semanticsimilarity and network proximity in dissemination network design; andefficiently maintaining statistics for high-dimensional events andsubscriptions.Broader Impact:Bringing together their expertise in databases and algorithms, the PIshave a track record of collaborating with each other and withresearchers outside computer science. A planned application of thesystem is to help ecologists with environmental monitoring.The PIs are committed to integrating research and education, and inparticular undergraduate research, by following their tradition ofinvolving undergraduates through REU and department fellowships. ThePIs are collaborating on an effort supported by the Department ofEducation, to increase workforce diversity for women, minorities, andpersons with disabilities. They also participate in an internshipprogram for underrepresented groups, which provides minority studentsopportunities to perform paid summer research with Duke facultymembers.Project URL: http://www.cs.duke.edu/dbgroup/prosem/