SHF:Medium:A Common Stream Processing Platform

SHF:Medium:通用流处理平台

• 批准号: 1162444
• 负责人: Robert Grimm
• 金额: $ 85万
• 依托单位: New York University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2012
• 资助国家: 美国
• 起止时间: 2012-06-01 至 2015-03-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1162444&HistoricalAwards=false
• 关键词: SHF; Medium; Common; Stream; Processing

More and more applications in entertainment, commerce, and government need to process continuous streams of data, and handle ever increasing data volumes at that. For instance, Netflix video streams account for almost 30% of internet traffic during peak hours in North America. High-frequency trading systems already execute over 50% of all trades in the United Sates. And New York City is networking 3,000 video cameras and an unspecified number of license plate readers and radiological detectors to prevent terrorist attacks. But building and running such streaming applications with existing software tools is hard; they only support some applications or do not meet performance requirements. This research is developing reusable software artifacts that make it possible to more easily build and run high-performance streaming applications in general. Thereby, it is directly addressing critical data processing needs of our society.More technically, this research is exploring how to componentize monolithic streaming systems by decoupling streaming languages, in which applications are written; streaming optimizations, which maximize performance and scalability; and streaming runtimes, which manage the low-level aspects of application execution. (1) The investigator is developing an intermediate language for stream processing that serves as the glue for composing language, optimization, and runtime components with each other. (2) The investigator is exploring how to refactor language front-ends and optimizations and then building a library of reusable front-end and optimization components. (3) The investigator is designing a common runtime API and then developing implementations for both multi-processors and clusters, thereby enabling the seamless scaling of streaming systems. If successful, the three research thrusts result in a stream processing platform that simplifies the development of state-of-the-art streaming applications while also facilitating future innovation in stream processing.

越来越多的娱乐、商业和政府应用需要处理连续的数据流，并处理不断增加的数据量。 例如，Netflix视频流在北美高峰时段占互联网流量的近30%。高频交易系统已经执行了美国50%以上的交易。 纽约市正在将3,000台摄像机和数量不详的车牌识别器和放射性探测器联网，以防止恐怖袭击。 但是，使用现有的软件工具构建和运行这样的流应用程序是很困难的;它们只支持某些应用程序，或者不满足性能要求。 这项研究正在开发可重用的软件工件，使其能够更容易地构建和运行高性能的流媒体应用程序。 因此，它直接解决了我们社会的关键数据处理需求。更技术上，这项研究正在探索如何通过解耦流语言，其中应用程序编写组件化单片流系统;流优化，最大限度地提高性能和可扩展性;和流运行时，管理应用程序执行的低级别方面。 (1)研究人员正在开发一种用于流处理的中间语言，作为相互组合语言、优化和运行时组件的粘合剂。 (2)研究人员正在探索如何重构语言前端和优化，然后构建一个可重用的前端和优化组件库。 (3)研究人员正在设计一个通用的运行时API，然后为多处理器和集群开发实现，从而实现流系统的无缝扩展。 如果成功的话，这三个研究方向将产生一个流处理平台，简化最先进的流应用程序的开发，同时促进流处理的未来创新。