Intelligent Internet-Scale Multimedia Storage and Delivery

视频服务，如Youtube和Netflix，现在在互联网上非常受欢迎。随着数据中心、云计算和互联网带宽的增加，视频传输正在经历重大的架构变化。Netflix作为一个典型的互联网规模的视频传输系统，将其内容托管在Amazon Simple Storage Service （S3）中，该服务组织了一个跨地理分布数据中心的存储系统。然后，视频通过多个内容交付网络（cdn）作为边缘服务器交付给最终用户。我们在这个提案中的宏伟目标是通过在计算机网络中整合数据密集型决策，将智能构建到未来的互联网规模的视频存储和传输系统中。该系统将利用从网络收集的各种类型的数据，包括系统跟踪、日志、用户报告甚至社会信息，用于工作负载预测和性能推断。处理后的数据将用于帮助内容放置、资源管理、数据传输和负载路由，这些可以建模为数据密集型优化问题，在视频分发层次结构的多个级别上。需要解决的关键挑战包括如何从数据中学习有用的信息，如何处理应用程序所需的各种组合优化问题，特别是如何有效地解决动态数据密集型优化以满足应用程序需求。为了扩展到非常大的系统和大量的输入数据，我们将特别研究新的分布式和并行算法和计算范式，它们要么导致网络中的分散决策，要么通过Hadoop和云计算服务实现高效的并行计算。该系统最终将把大规模计算机网络和交通工程与应用层面的数据密集型计算和决策结合起来。我们的研究将基于从各种视频交付和CDN公司收集的大量数据。我们的目标是根据资源可用性，在涉及PlanetLab节点、Amazon EC2数据中心和服务器集群的真实网络实验中测试我们的方法。

Video services, such as Youtube and Netflix, are enormously popular on the Internet nowadays. With the advent of datacenters, cloud computing and increased Internet bandwidth, video delivery is experiencing a major architectural change. As a typical Internet-scale video delivery system, Netflix hosts its content in Amazon Simple Storage Service (S3), which organizes a storage system spanning geographically distributed datacenters. The videos are then delivered to end users through multiple Content Delivery Networks (CDNs) as edge servers. Our grand goal in this proposal is to build intelligence into future video storage and delivery systems at the Internet scale by incorporating data-intensive decision making in computer networking. The proposed system will utilize various types of data collected from the network, including system traces, logs, user reports and even social information, for workload prediction and performance inference. The processed data will then be used to assist content placement, resource management, data transfers and load routing, which can be modelled as data-intensive optimization problems, at multiple levels in the video distribution hierarchy. The key challenges to be addressed include how to learn useful information from data, how to approach various combinatorial optimization problems required by the application, and especially, how to efficiently solve such data-intensive optimizations on-the-fly to satisfy application requirements. In order to scale to very large systems and a large amount of input data, we will particularly investigate novel distributed and parallel algorithms and computation paradigms, which either lead to decentralized decisions in the network or efficient parallel computation through Hadoop and cloud computing services. The proposed system will eventually integrate large-scale computer networking and traffic engineering with data-intensive computing and decision making at the application level. Our study will be based on a large amount of data collected from various video delivery and CDN companies. We aim to test our approaches in real network experiments involving PlanetLab nodes, Amazon EC2 datacenters and server clusters, depending on resource availability.