Elements: An ML Ecosystem of Filament Detection: Classification, Localization, and Segmentation

自从物体检测算法超越人类以来，十年已经过去了。在此期间，计算机视觉领域取得了前所未有的成就，让许多人相信目标检测是一个解决了的问题。然而，对于显微镜、望远镜、航空、卫星和医学图像等科学图像，通用的目标检测算法远远不是完美的。精确的目标分割和基于纹理特征的物理属性识别仍然是许多跨学科研究领域的突出挑战。太空天气就是这样一个领域。极端空间天气事件与极端陆地事件类似，可能会对人类产生严重的经济和附带影响。对太阳灯丝的持续和自动监测在实现可靠的空间天气预报/预报系统方面发挥着不可或缺的作用，从而为社会的许多基础设施方面，如电网和全球定位系统，提供急需的技术准备。我们的机器学习生态系统为专家带来了对细丝动态行为的自动、准确和可靠的分析。这个生态系统的主要贡献是两个注释细丝的数据产品，以及四个执行这些细丝的注释(定位、识别和分割)的软件产品。这种模块化的生态系统在未来可以很容易地扩展，超过奖励的有效期，因为社区预计将实施更快、更高效的模块，并取代现有的模块。在这个项目的整个开发过程中，我们咨询了国家太阳天文台(NSO)的仪器/数据专家，以便正确利用我们从全球振荡网络集团的六个地面天文台整合的观测图像和元数据，这些图像和元数据一起提供了对太阳的全盘和连续(24/7)覆盖。该项目的主要重点是定位和分割特定的太阳事件，称为暗条，并识别其磁场手性。也就是说，本项目中研究的新概念，如检测算法、增强引擎和对对象的粒度敏感的分割损失函数，仍然与感兴趣的事件/对象的类型无关。此外，发布的注释细丝的数据集可以作为计算机视觉社区的试验床，用于测试旨在高精度分割对象的算法。我们的机器学习生态系统由两个数据产品和四个软件产品组成。最大的手动注解细丝数据集合和不断增长的自动注解细丝集合是两个主要的数据产品。主要的软件产品是(1)为用户提供几乎无限的半真实细丝实例的增强引擎，(2)用于细丝定位、分割和分类的深度神经网络算法，(3)指导分割任务的高精度分割损失函数(对观察到的细丝的粒度敏感)，以及(4)可部署的检测模块，其实时执行定位、分割和分类任务。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Since object-detection algorithms outperformed humans, one decade has passed. During this period, the unprecedented achievements of the Computer Vision domain made many believe that object detection is a solved problem. However, when it comes to scientific imagery such as microscopic, telescopic, aerial, satellite, and medical images, the general-purpose object-detection algorithms are far from perfect. A pixel-precise segmentation of objects and identification of their physical properties based on their texture features are still outstanding challenges in many interdisciplinary areas of research. Space Weather is one such area. Extreme space-weather events, similar to extreme terrestrial events, can have drastic economic and collateral impacts on mankind. Continuous and automatic monitoring of solar filaments plays an integral role in achieving reliable space-weather forecast/prediction systems, which consequently results in the technical preparedness much needed in many infrastructural aspects of the society, such as the power grid and the GPS systems. Our Machine Learning Ecosystem brings automatic, accurate, and reliable analyses of filaments’ dynamic behavior to the experts’ fingertips. The main contributions of this ecosystem are two data products of annotated filaments, and four software products which carry out the annotation (localization, identification, and segmentation) of these filaments. This modular ecosystem can be easily expanded in the future, beyond the lifetime of the award, as faster and more efficient modules are expected to be implemented by the community and replace the existing ones. Throughout the development of this project, we consult with the instrument/data experts from the National Solar Observatory (NSO) for proper utilization of the observation images and metadata we integrate from the six ground-based observatories of the Global Oscillation Network Group, that together provide a full-disk and continuous (24/7) coverage of the Sun.The primary focus of this project is on the localization and segmentation of a specific solar event, called a filament, and the identification of its magnetic field chirality. That said, the novel concepts investigated in this project, such as the detection algorithm, the augmentation engine, and the segmentation loss function which is sensitive to granularities of objects, remain agnostic to the type of the event/object of interest. Moreover, the released datasets of annotated filaments can serve the Computer Vision community as a testbed for algorithms that aim at high-precision segmentation of objects. Our Machine Learning Ecosystem consists of two data products and four software products. The largest collection of manually annotated filaments data, and a continuously-growing collection of automatically annotated filaments are the two main data products. The main software products are (1) an augmentation engine that provides users with practically unlimited semi-real filament instances, (2) a deep neural network algorithm for localization, segmentation, and classification of filaments, (3) a high-precision segmentation loss function (sensitive to granularities of the observed filaments) that guides the segmentation task, and (4) a deployable detection module which carries out the localization, segmentation, and classification tasks in real time.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.