III-COR: Scalable Publish/Subscribe: Unifying Data Processing and Dissemination

数字时代带来了前所未有的增长，产生的数据量、数据消费者的数量以及他们兴趣和位置的多样性。传统上，用户轮询信息源，但对于许多应用程序来说，轮询很难扩展，可能会错过重要事件。发布/订阅系统提供的替代方案是向具有匹配兴趣的用户推送通知。这种方法适合许多应用，从个人、商业、医疗到环境、军事和安全。然而，传统的发布/订阅系统已经不能满足高级应用的需要，在高级应用中，用户希望接收经过过滤、连接和汇总的信息，并且只有在满足一定条件的情况下才能接收。PIS提出了一种端到端的解决方案，包括从订阅处理和索引到分发网络设计的技术，这些技术协同工作支持高效和强大的订阅功能，允许用户精确地控制他们想要什么以及何时想要。传统上，这些问题是由数据库和网络社区分别考虑的。然而，存在着将处理与分发相结合的广泛的有趣的替代方案。PI提出了一种很有希望的方法，允许通过简单的无状态分发机制处理复杂的有状态订阅，并具有易于实现和扩展的干净系统设计。一个基于成本的优化器，受数据库查询优化的启发，联合和动态地选择最佳的处理和传播策略。除了系统建设，这个项目还解决了许多新的算法挑战，包括：可伸缩地处理大量复杂的订阅；利用事件和订阅特征来应对最坏情况的复杂性；在传播网络设计中平衡语义相似性和网络邻近性；以及高效地维护高维事件和订阅的统计数据。广泛的影响：PIS将他们在数据库和算法方面的专业知识结合在一起，在相互合作和与计算机科学以外的研究人员合作方面有着良好的记录。该系统的计划应用是帮助生态学家进行环境监测。PIs致力于整合研究和教育，特别是本科生研究，遵循他们通过REU和系奖学金培养本科生的传统。私人投资机构正在合作开展一项由教育部支持的努力，以增加妇女、少数群体和残疾人的劳动力多样性。他们还参加了一个针对代表不足的群体的实习计划，该计划为少数族裔学生提供机会，与杜克大学的教职员工一起进行有偿暑期研究。项目网址：http://www.cs.duke.edu/dbgroup/prosem/

The Digital Age has brought about unprecedented growth in the amountof data being generated, the number of data consumers, and thediversity of their interests and locations. Traditionally, users pollsources for information, but for many applications, polling is hardlyscalable and may miss important events. The alternative offered bypublish/subscribe systems is to push notifications to users withmatching interests. This approach suits many applications, rangingfrom personal, commercial, medical, to environmental, military, andsecurity. However, traditional publish/subscribe systems are becominginadequate for advanced applications, where users want to receiveinformation that has been filtered, joined, and summarized, and onlywhen certain conditions are met.This project aims at building a next-generation publish/subscribesystem to face the new challenges. The PIs propose an end-to-endsolution consisting of techniques from subscription processing andindexing to dissemination network design, which work together tosupport efficient and powerful subscription functionalities, allowingusers to control precisely what they want and when they want it.One main feature distinguishing the proposed approach from previouswork is joint consideration of subscription processing andnotification dissemination. Traditionally, these problems areconsidered separately by database and networking communities.However, there exists a wide spectrum of interesting alternatives forinterfacing processing with dissemination. The PIs propose apromising approach that allows complex, stateful subscriptions to behandled by simple, stateless dissemination mechanisms, with a cleansystem design that is easy to implement and scale. A cost-basedoptimizer, inspired by database query optimization, chooses the bestprocessing and dissemination strategies jointly and dynamically.Besides system building, this project tackles many new algorithmicchallenges, including, e.g., scalably processing a large number ofcomplex subscriptions; exploiting event and subscriptioncharacteristics to combat worst-case complexity; balancing semanticsimilarity and network proximity in dissemination network design; andefficiently maintaining statistics for high-dimensional events andsubscriptions.Broader Impact:Bringing together their expertise in databases and algorithms, the PIshave a track record of collaborating with each other and withresearchers outside computer science. A planned application of thesystem is to help ecologists with environmental monitoring.The PIs are committed to integrating research and education, and inparticular undergraduate research, by following their tradition ofinvolving undergraduates through REU and department fellowships. ThePIs are collaborating on an effort supported by the Department ofEducation, to increase workforce diversity for women, minorities, andpersons with disabilities. They also participate in an internshipprogram for underrepresented groups, which provides minority studentsopportunities to perform paid summer research with Duke facultymembers.Project URL: http://www.cs.duke.edu/dbgroup/prosem/