SHF: Small: Languages and Abstraction for Dynamic Big Data

使用计算机分析大型数据集，又名，大数据分析正在成为许多领域的重要工具，例如科学和发现、技术、医疗保健和商业。这些应用程序中使用的数据集通常是动态的：随着新数据的出现，它们会随着时间的推移而变化。 这种动态变化通常很小，需要同样小但可能重要的更新，因为新信息在检测模式或异常时可能至关重要。 例如，互联网或社交网络随着新网页变得可用、新链接被添加或现有链接被移除而动态地改变。 作为这种动态变化的结果，先前断开的网站的两个集群可以通过添加单个链接而连接起来，例如，指示重要的新闻项目或安全漏洞。 不幸的是，在许多现有的大数据系统中，吸收新信息涉及对整个数据集进行一次或多次传递。 动态数据的这种批处理导致更新缓慢，以及通过（不必要地）执行不受变化影响的许多子计算而导致资源（诸如硬件和能量）的利用效率低下。 该项目旨在为能够支持在真实的世界中开发此类应用程序的编程语言和软件系统奠定基础。这项工作有可能改变程序员在动态变化的大数据集上表达计算的方式，通过响应和有效地计算大动态数据集来获取新的信息和知识，并改变我们教授设计，分析和实现动态数据集计算的方式。 该项目还包括开发关于并行性的本科生讲座。该项目旨在使用户能够隐式地表达大型数据集的动态性，而不关心当数据发生变化时结果将如何准确更新，例如，哪些数据取决于哪些其他数据、哪些数据可能需要更新、哪些依赖关系需要重构。 从隐式动态程序开始，软件系统自动有效地构建计算结果的记录，并随着数据集的变化而更新。 为了实现这一目标，该项目开发了抽象、编程语言、编译器和运行时系统。具体而言，我们期望三组的贡献：新颖的，功能强大的抽象和成本模型，用于编写程序，操作动态变化的大型数据集，编程语言支持的形式，编译器和运行时系统，实现这种抽象的实际硬件，高效的算法和实现，被用来评估建议和未来的工作。

The analysis of large datasets using computers, a.k.a., big data analytics, is emerging as an important tool in many fields, such as science and discovery, technology, health care, and commerce. The data sets used in such applications are usually dynamic: they change over time as new data becomes available. Such dynamic changes are often small, requiring similarly small but potentially important updates, because new information can be crucial in detecting a pattern or an anomaly. For example, the Internet or a social network changes dynamically as new web pages become available, new links are added, or existing links are removed. As a result of such dynamic changes, two clusters of previously disconnected web sites can become connected by the addition of a single link, indicating for example, an important news item or a security breach. Unfortunately, in many existing big-data systems, absorbing new information involves making one or more passes over the entire dataset. Such batch processing of dynamic data results in slow updates, as well as inefficiencies in the utilization of resources such as hardware and energy, by (unnecessarily) performing many subcomputations that are unaffected by changes. This project aims to lay the groundwork for the programming languages and software systems that can support the development of such applications in the real world. The work has the potential to transform the way the programmers express computations on dynamically changing big data sets, make it possible to derive new information and knowledge from big dynamic data sets by computing with them responsively and efficiently, and transform the way that we teach the design, analysis, and implementations of computations operating for dynamic data sets. The project also includes the development of undergraduate lectures on parallelism.The project aims to enable the user to express the dynamism in large data sets implicitly, without concerning themselves with how exactly the results will be updated when the data changes, e.g., which data depends on which other data, which data may need to be updated, which dependencies need to be reconstructed. Starting with an implicitly dynamic program, a software system automatically and efficiently constructs a record of the computed results and updates it as the dataset changes. To achieve this goal, the project develops abstractions, programming languages, compilers, and run-time systems. Concretely, we expect three sets of contributions: novel, powerful abstractions and cost models for writing programs that operate on dynamically changing large datasets, programming language support in the form of compilers and run-time systems for realizing such abstractions on practical hardware, and efficient algorithms and implementations, to be used to evaluate the proposed and future work.