SHF:Medium:A Common Stream Processing Platform

越来越多的娱乐、商业和政府应用程序需要处理连续的数据流，并处理不断增长的数据量。例如，在北美的高峰期，Netflix视频流占互联网流量的近30%。高频交易系统已经执行了美国所有交易的50%以上。纽约市正在联网3000个摄像机，以及数量不详的车牌阅读器和辐射探测器，以防止恐怖袭击。但用现有的软件工具构建和运行这样的流媒体应用程序是困难的；它们只支持一些应用程序或不符合性能要求。这项研究正在开发可重用的软件构件，使其能够更容易地构建和运行高性能的流媒体应用程序。因此，它直接满足了我们社会的关键数据处理需求。在更多的技术上，这项研究正在探索如何通过解耦编写应用程序的流语言、最大化性能和可扩展性的流优化以及管理应用程序执行的低级别方面的流运行时来组件化单片流系统。(1)研究人员正在开发一种用于流处理的中间语言，作为将语言、优化和运行时组件相互组合的粘合剂。(2)研究人员正在探索如何重构语言前端和优化，然后构建可重用的前端和优化组件库。(3)研究人员正在设计一个通用的运行时API，然后开发多处理器和集群的实现，从而实现流媒体系统的无缝扩展。如果成功，这三项研究将产生一个流处理平台，它简化了最先进的流应用程序的开发，同时也促进了流处理领域的未来创新。

More and more applications in entertainment, commerce, and government need to process continuous streams of data, and handle ever increasing data volumes at that. For instance, Netflix video streams account for almost 30% of internet traffic during peak hours in North America. High-frequency trading systems already execute over 50% of all trades in the United Sates. And New York City is networking 3,000 video cameras and an unspecified number of license plate readers and radiological detectors to prevent terrorist attacks. But building and running such streaming applications with existing software tools is hard; they only support some applications or do not meet performance requirements. This research is developing reusable software artifacts that make it possible to more easily build and run high-performance streaming applications in general. Thereby, it is directly addressing critical data processing needs of our society.More technically, this research is exploring how to componentize monolithic streaming systems by decoupling streaming languages, in which applications are written; streaming optimizations, which maximize performance and scalability; and streaming runtimes, which manage the low-level aspects of application execution. (1) The investigator is developing an intermediate language for stream processing that serves as the glue for composing language, optimization, and runtime components with each other. (2) The investigator is exploring how to refactor language front-ends and optimizations and then building a library of reusable front-end and optimization components. (3) The investigator is designing a common runtime API and then developing implementations for both multi-processors and clusters, thereby enabling the seamless scaling of streaming systems. If successful, the three research thrusts result in a stream processing platform that simplifies the development of state-of-the-art streaming applications while also facilitating future innovation in stream processing.