DISSP: Dependable Internet-Scale Stream Processing

实时流数据已经开始在互联网上发挥越来越重要的作用。造成这种情况的原因之一是地理分布的流数据源的激增，如传感器网络、科学仪器、普适计算环境和连接到互联网的web提要。全世界可能有数以百万计的用户希望利用这些数据的可用性。因此，他们需要一种方便的方法，通过执行互联网规模流处理（ISSP）的应用程序在全球范围内处理实时流数据。类似于搜索引擎如何使静态网络数据对用户有用，ISSP系统可靠地收集、过滤和处理来自潜在数千个数据源的流数据，代表许多用户。本提案的研究重点是解决所有ISSP应用程序面临的主要挑战：在存在故障的情况下实现鲁棒性。由于此类系统的规模、异构性和对尽力而为的互联网基础设施的依赖，故障是它们生活中的一个事实。这与许多需要可靠的ISSP （DISSP）服务的用户的需求相冲突——即使在Internet路径中断、处理主机故障和资源短缺的情况下，该服务也应该继续运行。例如，考虑一项科学研究，该研究通过关联来自世界各地数十个射电望远镜的高带宽实时图像流来检测和分析天空中的瞬变事件。在其中一架望远镜的网络连接出现故障后，科学家预计该系统会继续运行，也许分辨率会降低。同样，处理天空图像的主机故障不应导致业务中断，尽管它可能导致异常检测置信度降低。不幸的是，缺乏构建DISSP系统的机制。用于可靠数据处理的传统技术不能应用于DISSP系统，因为它需要全球可伸缩性、实时设置中的短恢复时间以及由于共享基础设施而需要的资源效率。该提案解决了这些挑战，以便将未来的普适传感器系统和全球科学实验互连所需的ISSP系统成为现实。考虑到ISSP系统在规模、故障模型和数据质量方面的独特特点，我们打算开发新的技术来构建可靠的ISSP系统。特别是，我们将设计一些方法，在失败后响应资源短缺而优雅地降低结果质量。虽然结果质量下降，但系统将不断向用户提供已达到的服务水平的反馈。反馈将以特定领域的方式表达，例如，通过通知科学用户有关感兴趣事件的检测置信度降低的情况。这种反馈还将推动自适应容错机制，使DISSP系统能够制定资源分配策略，以尽量减少最大数量用户的服务质量下降。

Real-time stream data has begun to play an increasingly important role on the Internet. One of the causes for this is the proliferation of geographically-distributed stream data sources such as sensor networks, scientific instruments, pervasive computing environments and web feeds connected to the Internet. Potentially millions of users world-wide want to take advantage of the availability of this data. Therefore they require a convenient way to process real-time stream data at a global scale through applications that perform Internet-scale stream processing (ISSP). Analogous to how search engines make static web data useful to users, an ISSP system reliably collects, filters and processes stream data from potentially thousands of data sources on behalf of many users.The research focus of this proposal is to address a major challenge facing all ISSP applications: achieving robustness in the presence of failures. Failures are a fact of life in such systems because of their scale, heterogeneity and reliance on best-effort Internet infrastructure. This conflicts with the requirements of many users that demand a dependable ISSP (DISSP) service --- a service that should continue functioning even during Internet path outages, processing host failures and resource shortages. For example, consider a scientific study that detects and analyses transient events in the sky by correlating high-bandwidth, real-time image streams from dozens of radio telescopes world-wide. After the failure of the network connection to one of the telescopes, a scientist would expect the system to continue operating, perhaps with reduced resolution. Similarly, the failure of a host that processes the sky images should not cause a service interruption, although it may lead to a decrease in detection confidence of anomalies. Unfortunately, mechanisms for building DISSP systems are lacking. Conventional techniques for reliable data processing cannot be applied to a DISSP system because of its requirements of global scalability, of short recovery times in a real-time setting and of resource efficiency due to a shared infrastructure.This proposal addresses these challenges so that ISSP systems needed for interconnecting tomorrow's pervasive sensor systems and global scientific experiments can become a reality. We intend to develop new techniques for building dependable ISSP systems, taking their unique features in terms of scale, failure model and data quality into account. In particular, we will devise approaches that gracefully degrade result quality in response to resource shortage after failure. While result quality is reduced, the system will provide constant feedback to users on the achieved level of service. Feedback will be expressed in a domain-specific way, e.g., by notifying a scientific user about the reduction in detection confidence of events of interest. This feedback will also drive an adaptive fault-tolerance mechanism, allowing the DISSP system to strategise about resource allocation in order to minimise the reduction in service quality of a maximum number of users.