Intelligent Internet-Scale Multimedia Storage and Delivery

点击翻译按钮获取中文摘要

Video services, such as Youtube and Netflix, are enormously popular on the Internet nowadays. With the advent of datacenters, cloud computing and increased Internet bandwidth, video delivery is experiencing a major architectural change. As a typical Internet-scale video delivery system, Netflix hosts its content in Amazon Simple Storage Service (S3), which organizes a storage system spanning geographically distributed datacenters. The videos are then delivered to end users through multiple Content Delivery Networks (CDNs) as edge servers. Our grand goal in this proposal is to build intelligence into future video storage and delivery systems at the Internet scale by incorporating data-intensive decision making in computer networking. The proposed system will utilize various types of data collected from the network, including system traces, logs, user reports and even social information, for workload prediction and performance inference. The processed data will then be used to assist content placement, resource management, data transfers and load routing, which can be modelled as data-intensive optimization problems, at multiple levels in the video distribution hierarchy. The key challenges to be addressed include how to learn useful information from data, how to approach various combinatorial optimization problems required by the application, and especially, how to efficiently solve such data-intensive optimizations on-the-fly to satisfy application requirements. In order to scale to very large systems and a large amount of input data, we will particularly investigate novel distributed and parallel algorithms and computation paradigms, which either lead to decentralized decisions in the network or efficient parallel computation through Hadoop and cloud computing services. The proposed system will eventually integrate large-scale computer networking and traffic engineering with data-intensive computing and decision making at the application level. Our study will be based on a large amount of data collected from various video delivery and CDN companies. We aim to test our approaches in real network experiments involving PlanetLab nodes, Amazon EC2 datacenters and server clusters, depending on resource availability.