SHF: Small: Languages and Abstraction for Dynamic Big Data

使用计算机对大型数据集进行分析，又称大数据分析，正在成为科学发现、技术、医疗保健和商业等许多领域的重要工具。这类应用程序中使用的数据集通常是动态的：随着新数据的出现，它们会随时间变化。这种动态更改通常很小，需要类似的小但可能很重要的更新，因为新信息在检测模式或异常时可能至关重要。例如，互联网或社交网络会随着新网页的出现、新链接的添加或现有链接的删除而动态变化。由于这种动态变化，两个先前断开的网站集群可以通过添加单个链接连接起来，例如，表明重要的新闻条目或安全漏洞。不幸的是，在许多现有的大数据系统中，吸收新信息需要对整个数据集进行一次或多次遍历。这种对动态数据的批处理会导致更新缓慢，以及对硬件和能源等资源的利用效率低下，因为（不必要地）执行了许多不受更改影响的子计算。该项目旨在为编程语言和软件系统奠定基础，以支持在现实世界中开发此类应用程序。这项工作有可能改变程序员在动态变化的大数据集上表达计算的方式，通过响应性和高效率的计算，使从大动态数据集中获得新的信息和知识成为可能，并改变我们教授动态数据集计算的设计、分析和实现的方式。该项目还包括开发关于并行性的本科讲座。该项目旨在使用户能够隐式地表达大型数据集中的动态，而无需关心数据更改时结果将如何准确更新，例如，哪些数据依赖于哪些其他数据，哪些数据可能需要更新，哪些依赖关系需要重建。从隐式动态程序开始，软件系统自动有效地构造计算结果的记录，并随着数据集的变化对其进行更新。为了实现这个目标，项目开发了抽象、编程语言、编译器和运行时系统。具体地说，我们期待三组贡献：用于编写在动态变化的大数据集上运行的程序的新颖，强大的抽象和成本模型，以编译器和运行时系统的形式支持编程语言，用于在实际硬件上实现这些抽象，以及有效的算法和实现，用于评估建议和未来的工作。

The analysis of large datasets using computers, a.k.a., big data analytics, is emerging as an important tool in many fields, such as science and discovery, technology, health care, and commerce. The data sets used in such applications are usually dynamic: they change over time as new data becomes available. Such dynamic changes are often small, requiring similarly small but potentially important updates, because new information can be crucial in detecting a pattern or an anomaly. For example, the Internet or a social network changes dynamically as new web pages become available, new links are added, or existing links are removed. As a result of such dynamic changes, two clusters of previously disconnected web sites can become connected by the addition of a single link, indicating for example, an important news item or a security breach. Unfortunately, in many existing big-data systems, absorbing new information involves making one or more passes over the entire dataset. Such batch processing of dynamic data results in slow updates, as well as inefficiencies in the utilization of resources such as hardware and energy, by (unnecessarily) performing many subcomputations that are unaffected by changes. This project aims to lay the groundwork for the programming languages and software systems that can support the development of such applications in the real world. The work has the potential to transform the way the programmers express computations on dynamically changing big data sets, make it possible to derive new information and knowledge from big dynamic data sets by computing with them responsively and efficiently, and transform the way that we teach the design, analysis, and implementations of computations operating for dynamic data sets. The project also includes the development of undergraduate lectures on parallelism.The project aims to enable the user to express the dynamism in large data sets implicitly, without concerning themselves with how exactly the results will be updated when the data changes, e.g., which data depends on which other data, which data may need to be updated, which dependencies need to be reconstructed. Starting with an implicitly dynamic program, a software system automatically and efficiently constructs a record of the computed results and updates it as the dataset changes. To achieve this goal, the project develops abstractions, programming languages, compilers, and run-time systems. Concretely, we expect three sets of contributions: novel, powerful abstractions and cost models for writing programs that operate on dynamically changing large datasets, programming language support in the form of compilers and run-time systems for realizing such abstractions on practical hardware, and efficient algorithms and implementations, to be used to evaluate the proposed and future work.