Noria: dynamic, partially-stateful data-flow for high-performance web applications
Noria: dynamic, partially-stateful data-flow for high-performance web applications
复制标题
DOI：
--
复制
发表时间：
2018-10
期刊：
影响因子：
--
通讯作者：
Jon Gjengset;Malte Schwarzkopf;J. Behrens;Lara Timbó Araújo;Martin Ek;E. Kohler;M. Kaashoek;R. Morris
中科院分区：
文献类型：
--
作者：
Jon Gjengset;Malte Schwarzkopf;J. Behrens;Lara Timbó Araújo;Martin Ek;E. Kohler;M. Kaashoek;R. Morris
我们介绍了部分状态数据流，一个新的流数据流模型，支持驱逐和重建的数据流状态的需求。通过避免状态爆炸并支持对数据流图的实时更改，该模型使数据流能够构建长寿命、低延迟的应用程序，例如Web应用程序。我们的实现Noria简化了大量读取的Web应用程序的后端基础设施，同时提高了它们的性能。Noria应用程序提供了一个关系模式和一组参数化查询，Noria将其编译成一个数据流程序，该程序预先计算读取结果并增量地应用写入。Noria可以轻松编写高性能应用程序，无需手动性能调优或维护复杂的缓存层。部分状态性帮助Noria限制其在内存中的状态，而不需要先前的数据流系统对窗口状态的限制，并帮助Noria在联机时使其数据流适应模式和查询更改。与以前的数据流系统不同，Noria还在相关查询之间共享状态和计算，消除了重复工作。在一个真实的Web应用程序的查询中，我们的原型可以扩展到比手动优化的MySQL基线高5倍的负载。Noria还优于典型的MySQL/memcached堆栈和商业数据库的物化视图。它可扩展到多个服务器上每秒数千万次读取和数百万次写入，性能优于最先进的流数据流系统。
We introduce partially-stateful data-flow, a new streaming data-flow model that supports eviction and reconstruction of data-flow state on demand. By avoiding state explosion and supporting live changes to the data-flow graph, this model makes data-flow viable for building long-lived, low-latency applications, such as web applications. Our implementation, Noria, simplifies the backend infrastructure for read-heavy web applications while improving their performance. A Noria application supplies a relational schema and a set of parameterized queries, which Noria compiles into a data-flow program that pre-computes results for reads and incrementally applies writes. Noria makes it easy to write high-performance applications without manual performance tuning or complex-to-maintain caching layers. Partial statefulness helps Noria limit its in-memory state without prior data-flow systems' restriction to windowed state, and helps Noria adapt its data-flow to schema and query changes while on-line. Unlike prior data-flow systems, Noria also shares state and computation across related queries, eliminating duplicate work. On a real web application's queries, our prototype scales to 5× higher load than a hand-optimized MySQL baseline. Noria also outperforms a typical MySQL/memcached stack and the materialized views of a commercial database. It scales to tens of millions of reads and millions of writes per second over multiple servers, outperforming a state-of-the-art streaming data-flow system.