The Zwicky Transient Facility Alert Distribution System
The Zwicky Transient Facility Alert Distribution System
复制标题
DOI：
10.1088/1538-3873/aae904
复制
发表时间：
2018-11
影响因子：
3.5
通讯作者：
M. Patterson;E. Bellm;B. Rusholme;F. Masci;M. Jurić;K. Krughoff;V. Golkhou;M. Graham;S. Kulkarni;Zwicky Transient;Facility Collaboration
中科院分区：
文献类型：
--
作者：
M. Patterson;E. Bellm;B. Rusholme;F. Masci;M. Jurić;K. Krughoff;V. Golkhou;M. Graham;S. Kulkarni;Zwicky Transient;Facility Collaboration
兹威基瞬态设施 (ZTF) 勘测为其广域勘测中发现的光学瞬变、变量和移动物体生成实时警报。我们描述了 ZTF 警报流分发和处理（过滤）系统。该系统使用了业界现有的开源技术：实时流媒体平台Kafka和二进制序列化格式Avro。该系统中使用的技术为 ZTF 用例提供了许多优势，包括（1）内置复制、可扩展性和分发机制的流回绕； (2) 具有严格执行模式和动态类型的结构化消息，用于快速解析； (3) 基于 Python 的流处理接口，类似于熟悉且用户友好的插件过滤系统的批处理，所有这些都在一个模块化的、主要是容器化的系统中。生产部署已成功支持每晚传输多达 120 万个警报或大约 70 GB 的数据，每个警报在警报候选生成后约 10 秒内可供消费者使用。据观察，数据传输率约为每分钟 80,000 个警报。在本文中，我们讨论了这个警报分发和处理系统、框架技术选择的设计动机、生产性能，以及该系统如何普遍适用于其他警报流用例，包括即将推出的大型综合巡天望远镜。
The Zwicky Transient Facility (ZTF) survey generates real-time alerts for optical transients, variables, and moving objects discovered in its wide-field survey. We describe the ZTF alert stream distribution and processing (filtering) system. The system uses existing open-source technologies developed in industry: Kafka, a real-time streaming platform, and Avro, a binary serialization format. The technologies used in this system provide a number of advantages for the ZTF use case, including (1) built-in replication, scalability, and stream rewind for the distribution mechanism; (2) structured messages with strictly enforced schemas and dynamic typing for fast parsing; and (3) a Python-based stream processing interface that is similar to batch for a familiar and user-friendly plug-in filter system, all in a modular, primarily containerized system. The production deployment has successfully supported streaming up to 1.2 million alerts or roughly 70 GB of data per night, with each alert available to a consumer within about 10 s of alert candidate production. Data transfer rates of about 80,000 alerts/minute have been observed. In this paper, we discuss this alert distribution and processing system, the design motivations for the technology choices for the framework, performance in production, and how this system may be generally suitable for other alert stream use cases, including the upcoming Large Synoptic Survey Telescope.