DISSP: Dependable Internet-Scale Stream Processing

DISSP：可靠的互联网规模流处理

• 批准号: EP/F035217/1
• 负责人: Peter Pietzuch
• 金额: $ 37.41万
• 依托单位: Imperial College London
• 依托单位国家: 英国
• 项目类别: Research Grant
• 财政年份: 2008
• 资助国家: 英国
• 起止时间: 2008 至 无数据
• 项目状态: 已结题
• 来源: https://gtr.ukri.org/projects?ref=EP%2FF035217%2F1
• 关键词: DISSP; Dependable; Internet; Scale; Stream

Real-time stream data has begun to play an increasingly important role on the Internet. One of the causes for this is the proliferation of geographically-distributed stream data sources such as sensor networks, scientific instruments, pervasive computing environments and web feeds connected to the Internet. Potentially millions of users world-wide want to take advantage of the availability of this data. Therefore they require a convenient way to process real-time stream data at a global scale through applications that perform Internet-scale stream processing (ISSP). Analogous to how search engines make static web data useful to users, an ISSP system reliably collects, filters and processes stream data from potentially thousands of data sources on behalf of many users.The research focus of this proposal is to address a major challenge facing all ISSP applications: achieving robustness in the presence of failures. Failures are a fact of life in such systems because of their scale, heterogeneity and reliance on best-effort Internet infrastructure. This conflicts with the requirements of many users that demand a dependable ISSP (DISSP) service --- a service that should continue functioning even during Internet path outages, processing host failures and resource shortages. For example, consider a scientific study that detects and analyses transient events in the sky by correlating high-bandwidth, real-time image streams from dozens of radio telescopes world-wide. After the failure of the network connection to one of the telescopes, a scientist would expect the system to continue operating, perhaps with reduced resolution. Similarly, the failure of a host that processes the sky images should not cause a service interruption, although it may lead to a decrease in detection confidence of anomalies. Unfortunately, mechanisms for building DISSP systems are lacking. Conventional techniques for reliable data processing cannot be applied to a DISSP system because of its requirements of global scalability, of short recovery times in a real-time setting and of resource efficiency due to a shared infrastructure.This proposal addresses these challenges so that ISSP systems needed for interconnecting tomorrow's pervasive sensor systems and global scientific experiments can become a reality. We intend to develop new techniques for building dependable ISSP systems, taking their unique features in terms of scale, failure model and data quality into account. In particular, we will devise approaches that gracefully degrade result quality in response to resource shortage after failure. While result quality is reduced, the system will provide constant feedback to users on the achieved level of service. Feedback will be expressed in a domain-specific way, e.g., by notifying a scientific user about the reduction in detection confidence of events of interest. This feedback will also drive an adaptive fault-tolerance mechanism, allowing the DISSP system to strategise about resource allocation in order to minimise the reduction in service quality of a maximum number of users.

实时流数据已经开始在互联网上发挥越来越重要的作用。其中一个原因是地理上分布的流数据源，如传感器网络，科学仪器，普及计算环境和连接到互联网的网络源的扩散。全球可能有数百万用户希望利用这些数据的可用性。因此，他们需要一种方便的方式来处理实时流数据在全球范围内通过应用程序，执行互联网规模的流处理（ISSP）。类似于搜索引擎如何使静态Web数据对用户有用，ISSP系统代表许多用户可靠地收集、过滤和处理来自潜在的数千个数据源的流数据。本提案的研究重点是解决所有ISSP应用程序面临的一个主要挑战：在出现故障时实现鲁棒性。由于这些系统的规模、异质性和对尽力而为的互联网基础设施的依赖，故障是这些系统中的一个生活事实。这与许多用户的要求相冲突，这些用户需要可靠的ISSP（DISSP）服务--即使在Internet路径中断、处理主机故障和资源短缺时，该服务也应该继续运行。例如，考虑一项科学研究，该研究通过关联来自世界各地数十个射电望远镜的高带宽实时图像流来检测和分析天空中的瞬态事件。在与其中一台望远镜的网络连接失败后，科学家预计该系统将继续运行，可能会降低分辨率。类似地，处理天空图像的主机的故障不应导致服务中断，尽管其可能导致异常的检测置信度的降低。不幸的是，缺乏建立DISSP系统的机制。传统的可靠的数据处理技术不能应用于DISSP系统，因为它的要求的全球可扩展性，在实时设置的短恢复时间和资源效率，由于共享infrastructure.This建议解决了这些挑战，使ISSP系统需要互连明天的普适传感器系统和全球科学实验可以成为现实。我们打算开发新技术，以建立可靠的ISSP系统，考虑到其在规模，故障模式和数据质量方面的独特功能。特别是，我们将设计的方法，优雅地降低结果质量，以响应失败后的资源短缺。虽然结果质量下降，但该系统将不断向用户提供关于所达到的服务水平的反馈。反馈将以特定领域的方式表达，例如，通过通知科学用户关于感兴趣的事件的检测置信度的降低。这种反馈还将驱动自适应容错机制，使DISSP系统能够制定资源分配策略，以尽量减少最大数量用户的服务质量下降。

项目成果

Scalable stateful stream processing for smart grids

智能电网的可扩展状态流处理

• DOI: 10.1145/2611286.2611326
• 发表时间: 2014
• 作者: Fernandez R

Integrating scale out and fault tolerance in stream processing using operator state management

• DOI: 10.1145/2463676.2465282
• 发表时间: 2013-06
• 作者: R. Fernandez;Matteo Migliavacca;Evangelia Kalyvianaki;P. Pietzuch

Living in the present

活在当下

• DOI: 10.1145/2168697.2168703
• 发表时间: 2012
• 作者: Eyers D

Making State Explicit for Imperative Big Data Processing

• DOI: 10.17863/cam.41706
• 发表时间: 2014-06
• 作者: R. Fernandez;Matteo Migliavacca;Evangelia Kalyvianaki;P. Pietzuch

Overload Management in Data Stream Processing Systems with Latency Guarantees

• 发表时间: 2012
• 作者: Evangelia Kalyviannaki;Themistoklis Charalambous;Marco Fiscato;P. Pietzuch