CSR: Small: Software Infrastructure for Online Analytics

复杂的分析框架--从大型的、往往是动态的数据源持续提取信息的分布式系统--处于一大类日益重要的应用程序的核心，从信息检索和搜索到大规模科学数据分析。这些框架中的许多框架的一个共同特征是它们的离线性质--底层模型仅以离线方式定期更新，通常通过昂贵的计算(例如，每周到每月更新它们的推荐系统模型)。相比之下，许多新兴应用程序(如情绪分析和个性化)需要在线分析，通常依赖昂贵的计算方法。这些应用程序一直在运行(即，它们不在规定的时间启动和停止)，它们被要求响应实时查询，即使答案是近似的，它们对当前可用的数据进行操作，而不是收集所有可能的数据，并且它们必须适应可用的计算资源。一个专门针对这类重要的在线学习应用程序的连贯的软件系统将大大提高最先进的水平。该项目旨在开发一个软件基础设施，包括应用程序编程接口(API)、运行时系统以及针对可伸缩在线分析的特定领域优化。这项工作对从商业数据分析到科学数据分析等领域的应用具有广泛而深刻的影响。在商业领域，这样的系统将用于分析来自社交网络、经济交易和其他动态信息源的实时数据馈送。在科学领域，这类系统可用于分析来自天体物理观测、高通量仪器和密集感知环境的数据。可扩展分析是当前大规模分布式系统面临的最重要挑战之一。为了应对这些挑战，该项目有以下具体目标：(I)开发适当的领域特定抽象，以及用于规范分析数据流的API；(Ii)支持面向可扩展分析应用的动态更新和用户交互；(Iii)开发用于调度、资源管理、性能和容错的运行时系统基础设施；(Iv)开发重要分析操作的在线版本的内核库；以及(V)通过示范应用进行验证。从理论和系统建设的角度来看，这些目标中的每一个都代表着重大的智力挑战。

Complex analytics frameworks -- distributed systems that continuously extract information from large, often dynamic data sources -- sit at the core of a large class of increasingly important applications, ranging from information retrieval and search, to large-scale scientific data analyses. A common feature of many of these frameworks is their offline nature -- the underlying models are only periodically updated, typically through expensive computations, in an offline manner (e.g., update their recommender system models weekly to monthly). In contrast, many emerging applications (e.g., sentiment analysis and personalization) require online analytics, often relying on expensive computational methods. These applications are constantly running (i.e., they do not start and stop at prescribed times), they are required to respond to real-time queries even if the answer is approximate, they operate on currently available data as opposed to collecting all possible data, and they must adapt to available computational resources. A coherent software system targeted specifically to this important class of online learning applications would significantly enhance the state-of-the-art. This project aims to develop a software infrastructure comprising application programming interfaces (APIs), runtime systems, and domain-specific optimizations for scalable online analytics. This work has broad and deep impact on applications in domains ranging from commercial to scientific data analyses. In commercial domains, such systems would be used to analyze real time data feeds from social networks, economic transactions, and other dynamic information sources. In scientific domains, such systems could be used to analyze data from astrophysical observations, high-throughput instrumentation, and densely sensed environments.Scalable analytics are among the most important current challenges facing large-scale distributed systems. To address these challenges, the project has the following specific aims: (i) development of suitable domain-specific abstractions, along with an API for specification of analytics dataflows; (ii) support for dynamic updates and user-interaction geared towards scalable analytics applications; (iii) development of a runtime system infrastructure for scheduling, resource management, performance, and fault tolerance; (iv) development of a kernel library of online versions of important analytics operations; and (v) validation through exemplar applications. Each of these goals represent significant intellectual challenges from theoretical and systems-building viewpoints.