III: Small: Automated Event Classification and Decision Making in Massive Data Streams

随着所有科学（以及现代社会、经济、商业、安全等所有其他领域）中数据量和复杂性的指数级增长，对强大的新工具和方法的需求日益增长，这些工具和方法可以帮助我们从这些海量数据集和数据流中提取知识和理解。新获得的知识经常被用来指导我们的行动，在科学中，这通常意味着随着研究周期的继续，后续的研究和测量。随着数据速率和数据量的增加，有必要将人类从循环中剔除，并开发用于时间关键知识提取的自动化方法，以及对数据处理管道发现的异常或有趣事件的优化响应。这个建议是开发一个系统，它将成为新一代科学实验和方法的一个例子，涉及实时挖掘海量数据流，以及动态的后续策略。该系统将在新兴的时域天文学领域的真实科学情况下进行开发和验证。新一代的天气巡天反复覆盖天空，探测变化或瞬变现象，涵盖广泛的天体物理学，从太阳系和恒星演化，到宇宙学和极端相对论性物体；从太阳系外行星到伽马射线爆发，再到作为暗能量探测器的超新星。当我们探索可观测的参数空间时，就有可能发现新的物体和现象。该系统将使令人兴奋的新天体物理学成为可能，并促进发现。关键是对瞬态事件及其后续观察进行完全自动化的分类和优先级排序。这给应用计算机科学带来了一些有趣的挑战，特别是在机器学习领域，包括只有稀疏、不完整和异构数据可用的自动分类，并且必须在此过程中包含上下文信息和领域专业知识。这一过程必须是动态的，在获得新数据时纳入其中，并相应地修订分类。然后，在有限的资产和资源下，系统将自动生成对最有趣事件的最佳后续行动的决策。该项目将帮助整个天文学界在大型天气巡天时代开发新的科学策略和程序，促进数据共享和重用，并刺激虚拟天文台能力的进一步发展。在这里获得的方法和经验将在公开文献中描述，以便它们可以在天文学之外找到更广泛的用途，无论类似的时间紧迫情况发生在哪里，从而在应用计算机科学和其他领域之间培养建设性的新协同作用。申请者将在科学计算方法和计算思维方面培养本科生、研究生和博士后，并开发有效的EPO材料，涉及新科学和计算。当我们从海量数据流中挖掘信息时，特别是当所研究的现象是短暂的，并且/或者需要快速的后续反应时，在数据丰富的时代，知识提取所带来的挑战变得更加尖锐。潜在的有趣现象和事件必须实时识别、分类和确定优先级，通常使用新的测量和现有的档案数据和模型的组合。然后，必须做出最佳决策，即在任何特定情况下，提供必要新信息的最佳后续行动是什么；如果后续资产稀缺或成本高昂，这一点至关重要。如果时间尺度很短，而数据速率很大，则意味着应该将人类排除在循环之外，并且分类、优先级排序和后续决策过程必须完全自动化。机器学习（ML）和机器智能工具变得必不可少。本提案旨在开发一种新颖的、基于ml的系统，用于瞬态事件的实时分类和优先排序，使用新兴的时域天文学和天气巡天领域作为科学试验平台。这里的分类问题与通常的情况不同：数据是稀疏的和/或不完整的，异构的，并且随着新测量的出现而不断发展；决策过程要考虑到分类过程的不确定性和可用资产；等等。当天空巡天探测到短暂的宇宙事件时，科学的回报来自于它们的直接跟踪。能够对有趣的事件进行分类和优先排序是至关重要的，特别是当我们从目前的Terascale数据流和每晚数十个候选事件转移到未来的Petascale数据体系时，具有数百万个候选事件，只有少数可以跟踪。考虑到数据不完整和稀疏性的问题，提议者将探索使用贝叶斯技术，该技术可以使用当前最好的可用数据，在一组专家开发和基于ml的先验上操作。一些方法上的挑战包括上下文信息和人类专业知识的结合，独立分类器输出的最佳组合，以及本项目开发的新方法。所有的算法开发都将牢记鲁棒性和可扩展性，并在真实的科学用例中进行测试。

As the exponential growth of data volumes and complexity continues in all sciences (and indeed all other fields of the modern society, economy, commerce, security, etc.), there is a growing need for powerful new tools and methodologies which can help us extract knowledge and understanding from these massive data sets and data streams. The newly gained knowledge is often used to guide our actions, and in science that typically means follow-up studies and measurements, as the research cycle continues. As the data rates and volume increase, it becomes necessary to take humans out of the loop, and develop automated methods for time-critical knowledge extraction and optimized response to anomalous or interesting events found by the data processing pipelines. This proposal is to develop a system that will be an example of a new generation of scientific experiments and methods that involve real-time mining of massive data streams, and dynamical follow-up strategies. The system would be developed and validated in the context of real scientific situations from the emerging field of time-domain astronomy. A new generation of synoptic sky surveys covers the sky repeatedly, detecting variable or transient phenomena, over a broad range of astrophysics, from the Solar system and stellar evolution, to cosmology and extreme relativistic objects; from extrasolar planets to gamma-ray bursts and supernovae as probes of the dark energy. As we explore the observable parameter space, there is a real possibility of discovery of new types of objects and phenomena. The system will enable exciting new astrophysics, and facilitate discovery. The key to this is a fully automated classification and prioritization of the transient events, and their follow-up observations. This poses some interesting challenges for applied computer science, especially in the area of Machine Learning, including an automated classification where only a sparse, incomplete, and heterogeneous data are available, and contextual information and domain expertise must be folded in the process. The process must be dynamic, incorporating new data as they become available, and revising the classifications accordingly. The system would then generate automatically decisions for an optimal follow-up of the most interesting events, given the available limited assets and resources. This project will aid the entire astronomical community in developing new scientific strategies and procedures in the era of large synoptic sky surveys, facilitate data sharing and re-use, and stimulate further development of Virtual Observatory capabilities. The methods and experiences gained here will be described in the open literature, so that they may find a broader use outside astronomy, wherever similar time-critical situations occur, thus fostering constructive new synergies between applied computer science and other domains. The proposers will train undergraduate and graduate students and postdocs, in the methods of scientific computing and computational thinking, and develop effective EPO materials, touching on both the new science and computation.The challenges posed by the knowledge extraction in the era of data abundance become even sharper in the time-critical situations where we mine the information from massive data streams, especially when the phenomena under study are short-lived, and/or a rapid follow-up reaction is needed. Potentially interesting phenomena and events must be identified, classified, and prioritized in real time, typically using some combination of the new measurements, and existing archival data and models. Then an optimal decision has to be made as to what is the best follow-up that will provide the essential new information in any given individual case; this can be critical if the follow-up assets are scarce or costly. If the time scales are short, and data rates large, the implication is that humans should be taken out of the loop, and that the classification, prioritization, and follow-up decision process must be fully automated. Machine learning (ML) and machine intelligence tools become a necessity. This proposal is to develop a novel, ML-based system for a real-time classification and prioritization of transient events, using the newly emerging field of time-domain astronomy and synoptic sky surveys as a scientific testbed. The classification problem here is different from the usual situations: the data are sparse and/or incomplete, heterogeneous, and evolving as the new measurements come in; the decision process has to take into account the uncertainties of the classification process, and the available assets; and so on. While the sky surveys detect transient cosmic events, the scientific returns come from their directed follow-up. It is essential to be able to classify and prioritize interesting events, especially as we move from the present Terascale data streams and tens of candidate events per night, to the future Petascale data regime, with literally millions of candidates, only a handful of which can be followed. Given the problem of data incompleteness and sparsity, the proposers will explore the use of Bayesian techniques that can operate on a set of expert-developed and ML-based priors, using the currently best available data. Some of the methodological challenges include incorporation of the contextual information and human expertise and optimal combination of separate classifier outputs, as well as new methods developed in this project. All of the algorithmic developments will be done keeping the robustness and scalability in mind, and tested on real scientific use cases.