The Columbia Hot Spot Rescue Service

虽然互联网流量通常相当大，但通常有足够的存储、处理和带宽容量来提供可接受的性能。然而，众所周知，对网络资源的需求越来越频繁地在本地迅速发展为热点或数据风暴，在这种情况下，受影响的网站和子网络几乎完全崩溃，造成大规模的收入损失和客户不满。我们提出了一种新的协作技术来缓解这些热点的影响，我们将通过设计和原型热点救援服务(HRS)来应用该技术。这项工作将建立在过去PI在网络、操作系统和分布式缓存方面的研究基础上。该技术建立的一个关键前提是观察到现有的互联网带宽资源足以有效地应对热点。换句话说，对严重超载地点的救援可以利用其他地方未得到充分利用的资源。因此，不存在为此目的唯一保留的资源的内在需要，即不需要过度配置、未充分利用的资源(例如分布式高速缓存)来保护免受热点的影响。相反，我们提出了一种范式转变，在这种转变中，我们提供的有效机制将使参与网站的社区能够共享他们的资源，以抑制热点。运行中的服务将是透明的，在一定程度上是自我调节的，并将采取自动流量重定向的形式，指向具有可用带宽的站点。拟议的哥伦比亚人力资源计划将是积极主动的，也是被动的。我们将收集热点数据，这些数据将被建模和分析，目的是设计热点守护程序或插件、用于监控流量行为的软件设备，以及通过热点监视和建议通知新热点的软件设备，以及相关统计数据。我们将实施这项技术的两种互补方法，一种是基于服务器的方法，另一种是基于客户端的对等(P2P)方法。在基于服务器的方法中，服务器监控自己的负载，即在其他服务器过载的情况下它们将服务的一小部分服务器的负载，并通过服务器之间的探测来监控网络状况。当服务器或网络组件被识别为进入可能的过载时，系统激活复制机制来复制热门内容。然后，客户端可以从充当副本的服务器站点检索内容，从而减轻原始过载资源的负载。在P2P方法中，客户端在其浏览器中安装一个插件，该插件与安装在其他客户端浏览器上的类似插件以及分布式目录服务进行通信。客户端缓存其最近下载的内容，并通过插件将缓存的对象通知目录服务。然后，目录服务可以识别最流行的内容以及缓存的位置，并将这些可供下载的替代位置通知其他客户端。通过使用客户端机器来存储和递送热点内容，并且如果对给定内容的请求可以被重定向到客户端机器，则可以消除服务器处的热点。通过这种方式，热点响应变得自组织和自我调节。在我们发展这项救援服务时，有几个问题需要解决。首先，我们将使用实验和分析收集的数据(包括通过公司合作伙伴关系获得的数据集)来开发网络热点活动的因果模型。接下来，我们将分析评估缓存、重定向和迁移等技术对网络热点的服务器和网络负载的影响。最后，我们将实现和评估原型系统，以验证其有效性，无论是在试验床内的模拟热点活动上，还是在可能的情况下，通过与内容提供商的协议在实际热点上验证其有效性。HRS的开发将为哥伦比亚大学的多名学生提供研究机会，它的部署将改善学术机构和非营利性公司提供的内容迄今未由商业性第三方内容交付公司托管的对象的网络性能。

Although Internet traffic is routinely quite heavy, there has usually been more than enough storage,processing and bandwidth capacity to provide acceptable performance. However, it is well known that, moreand more frequently, demands for network resources are mushrooming locally into hotspots or data storms,and in such cases the affected web sites and subnetworks founder almost completely, creating revenuelosses and client dissatisfaction on a large scale. We propose a novel collaborative technology to alleviatethe effects of these hotspots, a technology that we will apply by designing and prototyping a Hotspot RescueService (HRS). This work will build on the past research of the PI's in networking, operating systems, anddistributed caching. A key premise on which the technology is founded lies in the observation that existing Internet band-widthresources are sufficient to deal effectively with hotspots. In other words, rescues of heavily-overloadedsites can be assembled from underutilized resources lying elsewhere. It follows that there is no inherent needfor resources held in reserve uniquely for this purpose, i.e., there is no need for over-provisioned, under-utilized resources such as distributed caches to protect against hotspots. We propose instead a paradigm shift in which efficient mechanisms that we provide will enable communities of participating sites to share their resources to suppress hotspots. The service in action will be transparent, in part self-regulating and will take the form of automated traffic redirection to sites with available bandwidth. The proposed Columbia HRS will be proactive as well as reactive. We will amass hotspot data thatwill be modeled and analyzed with the aim of designing hotspot daemons or plugins, software devices formonitoring traffic behavior and signalling incipient hotspots via hotspot watches and advisories, along withrelevant statistics. We will implement two complementary approaches to the technology, a server-basedapproach and a client-based, peer-to-peer (P2P) approach. In the server-based approach, servers monitor their own loads, the loads of a small set of servers thatthey would service in the event that the other server overloads, and, via probes between servers, networkconditions. When a server or network component is identified as going into possible overload, the systemactivates a replication mechanism to duplicate the hot content. Clients can then retrieve the content fromthe server sites acting as replicas, alleviating the load on the original overloaded resource. In the P2P approach, clients install a plugin into their browser that communicates with similar pluginsinstalled on other clients' browsers, as well as with a distributed directory service. Clients cache their recent downloads, and, through the plugin, inform the directory service of the objects that are cached. The directory service can then identify the most popular content, as well as cached locations and notify additional clients of these alternative locations for download. By using client machines to store and deliver the hot content and if requests for the given content can be redirected to the client machines, the hotspot at the server can be eliminated. In this way, hotspot response becomes self-organizing and self-regulatory. There are several issues that need to be addressed as we develop this rescue service. First, we will useexperimentation and analysis of collected data (including data sets obtained via a company partnerships)to develop models of causes and effects of network hotspot activity. Next, we will analytically evaluatethe effect on server and network load that techniques such as caching, redirecting, and migrating haveupon hotspots within the network. Last, we will implement and evaluate prototype systems to validatetheir effectiveness, either upon simulated hotspot activity within a testbed, or if possible, on actual hotspotsthrough agreements with content providers. Development of the HRS will provide research opportunities to multiple students at Columbia, andits deployment will improve web performance of objects served from academic institutions and non-profitcompanies whose content is to date not hosted by commerical third-party content delivery companies.