CDS&E: A Deep Learning Framework for Evaluation of Electron Microscopy Images of Chemically-Complex Metallic Materials

这个项目是由材料研究部的凝聚态物质与材料理论和金属与金属纳米结构计划共同资助的。非技术概述电子显微镜可以拍摄纳米尺寸的金属颗粒的美丽和信息，但图像有时很难解释。化学复杂的金属合金纳米粒子(CCA-NPs)在催化、能量转换和存储、生物/等离子体成像等领域具有广泛的应用价值。这项研究项目开发了一种由实验测量支持的多学科建模方法，以跟上原子分辨微观测量日益广泛的应用。本项目的主要目标是利用一种基于机器学习的新的建模框架，从不同成分和尺寸的CCA-NPs的实验高分辨率电子显微镜图像中提取有关原子柱高度和化学元素的信息。尽管目前的工作主要是由纳米颗粒推动的，但该框架是通用的，很容易扩展到其他适合扫描电子显微镜的纳米科学研究，如催化、结晶学和相演化。该团队将在他们的本科生和研究生课程中介绍一些与本项目相关的主题，强调熟悉当前的研究问题，以及对不同计算方法以及开放源代码和商业软件的直接经验。调查人员及其小组成员通过伊利诺伊大学芝加哥分校(UIC)开放日和UIC青年方案，积极参加针对当地高中妇女和代表性不足群体成员的外联活动。带解说的计算机模拟结果将用于课堂教学，还将通过微博和社交网络服务向公众和科学界提供。技术摘要机器和深度学习算法的最新发展，加上原位电子显微镜技术的不断进步，为有效分析各种材料铺平了道路。本研究使用建立在完全卷积神经网络上的深度学习模型来解析原子分辨透射(TEM)和扫描(SEM)电子显微镜图像中所表示的CCA-NPs的元素分布。所提出的神经网络的目的是通过语义分割来学习显微图像的像素强度与CCA-NP结构的原子柱中不同组成元素的原子数之间的非线性相关性。尽管在确定结构和元素分布方面很有必要，但目前的实验工作依赖于试错分析，而且往往涉及许多假设。这是由于透射电子显微镜图像的非线性和复杂性，阻碍了对原子柱高度和元素分布的直接估计。因此，本项目的主要目标是从实验高分辨率电子显微镜(HRTEM)和扫描电子显微镜(STEM)图像中提供不同尺寸和组成的CCA-NPs，包括高熵合金(HEAs)的此类信息。提出了一个基于机器学习(ML)、进化方法(EA)和密度泛函理论(DFT)计算的集成的多学科建模框架，并由HRTEM和STEM测量支持。一个支持目标是提供一系列可获得可靠HRTEM图像的实验条件。该项目采用多学科建模方法对通过实验获得的CCA-NPs的HRTEM/STEM图像的分析和解释提供了范式转变，通过(I)促进用于评估实验获得的图像的深度学习技术的最新技术，(Ii)使用Wulff结构为CCA-NPs生成物理上有意义的和可靠的训练数据，(Iii)使用进化方法评估HEAs中的元素主题，以及(Iv)更深刻地了解显微镜参数(例如，剂量、焦点扩散、散焦等)的影响。关于神经网络预测的质量。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This project is co-funded by the Condensed-Matter-and-Materials-Theory and Metals-and-Metallic-Nanostructures programs in the Division of Materials Research.Nontechnical summaryElectron microscopes take beautiful and informative pictures of metal particles of nanometer size, but the images can sometimes be difficult to interpret. The chemically-complex metallic-alloy nanoparticles (CCA-NPs) that motivate this work are of great interest in a wide range of applications including catalysis, energy conversion and storage, and bio/plasmonic imaging. This research project develops a multi-disciplinary modeling methodology supported by experimental measurements to keep pace with the growing widespread application of atomically resolved microscopic measurements. The main objective of this project is to utilize a novel modeling framework based on machine learning to extract information about atomic column heights and chemical elements from experimental high-resolution electron microscopy images of CCA-NPs of different compositions and sizes. Although the present work is motivated primarily by nanoparticles, the framework is general and easily extendable to other nanoscience research amenable to scanning transmission electron microscopy, such as catalysis, crystallography, and phase evolution.The team will introduce in their undergraduate and graduate courses a number of topics related to the present project, stressing familiarity with current research problems and direct experience with different computational methods as well as open source and commercial software. The investigators and their group members actively participate in outreach activities for local high-school women and members of underrepresented groups through the University of Illinois Chicago (UIC) Open House and the UIC Youth Program. The computer simulation results with narratives will be used for classroom teaching and will also be made available to the public and scientific community via microblogging and social-network services.Technical summaryThe latest developments in machine and deep-learning algorithms, coupled with continuing progress of in-situ electron-microscopy techniques, have paved the way to an effective analysis of a variety of materials. This research uses a deep-learning model built on a fully convolutional neural network to resolve the elemental distribution of CCA-NPs represented in atomic-resolution transmission (TEM) and scanning (SEM) electron microscopy images. The objective of the proposed neural network is to learn, through semantic segmentation, the non-linear correlations between the pixel intensities of microscopy images and the number of atoms of different constituent elements in the atomic columns of CCA-NP structures. In spite of the critical need in determining structures and elemental distributions, current experimental efforts rely on trial-and-error analysis and often involve many assumptions. This is due to the nonlinearity and complexity of TEM images, preventing the straightforward estimation of atomic column heights and elemental distributions. Thus, the main objective of this project is to provide such information from experimental high-resolution TEM (HRTEM) and scanning TEM (STEM) images of CCA-NPs of different sizes and compositions, including high-entropy alloys (HEAs). An integrated, multi-disciplinary modeling framework based on machine learning (ML), an evolutionary approach (EA), and density-functional-theory (DFT) calculations supported by HRTEM and STEM measurements is proposed. A supporting objective is to provide a range of experimental conditions for which reliable HRTEM images may be acquired. This project provides a paradigm shift in the analysis and interpretation of HRTEM/STEM experimentally acquired images of CCA-NPs employing a multi-disciplinary modeling approach through (i) advancing the current state of the art of deep-learning techniques for evaluation of experimentally obtained images, (ii) generation of physically meaningful and reliable training data for CCA-NPs using the Wulff Construction, (iii) evaluation of elemental motifs in HEAs using an evolutionary approach, and (iv) more profound understanding of the influence of the microscope parameters (e.g., dose, focal spread, defocus, etc.) on the quality of neural-network predictions.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.