Material transformers: deep learning language models for generative materials design
Material transformers: deep learning language models for generative materials design
复制标题
DOI：
10.1088/2632-2153/acadcd
复制
发表时间：
2023-03-01
影响因子：
6.8
通讯作者：
Hu, Jianjun
中科院分区：
文献类型：
--
作者：
Fu, Nihang;Wei, Lai;Hu, Jianjun
在大型未标记语料库上预训练的Transformer语言模型（LM）在自然语言处理、有机分子设计和蛋白质序列生成方面产生了最先进的结果。然而，没有这样的模型已被应用到学习的材料组成的生成设计的组成模式。在这里，我们使用ICSD、OQMD和Materials Projects数据库的扩展公式训练了一系列用于材料设计的七种现代Transformer模型（GPT、GPT-2、GPT-Neo、GPT-J、BLMM、BART和RoBERTA）。六个不同的数据集与/出非电中性或EB样本被用来基准的生成设计性能，并揭示现代Transformer模型的材料成分的生成设计的偏见。我们的实验表明，基于因果LM的材料转换器可以生成化学上有效的材料成分，其中高达97.61%是电荷中性的，91.22%是电负性平衡的，与基线伪随机采样算法相比，其富集度高出六倍以上。我们的LM还展示了高世代的新奇，其在新材料发现方面的潜力通过其回收遗留材料的能力得到了证明。我们还发现，所生成的组合物的属性可以通过使用选定的训练集（如高带隙样本）训练模型来定制。我们的实验还表明，不同的模型都有自己的喜好，在所产生的样本的属性和他们的运行时间复杂度有很大的差异。我们已经应用我们的材料转换器发现了一组新材料，并使用密度泛函理论计算进行了验证。
Pre-trained transformer language models (LMs) on large unlabeled corpus have produced state-of-the-art results in natural language processing, organic molecule design, and protein sequence generation. However, no such models have been applied to learn the composition patterns for the generative design of material compositions. Here we train a series of seven modern transformer models (GPT, GPT-2, GPT-Neo, GPT-J, BLMM, BART, and RoBERTa) for materials design using the expanded formulas of the ICSD, OQMD, and Materials Projects databases. Six different datasets with/out non-charge-neutral or EB samples are used to benchmark the generative design performances and uncover the biases of modern transformer models for the generative design of materials compositions. Our experiments show that the materials transformers based on causal LMs can generate chemically valid material compositions with as high as 97.61% to be charge neutral and 91.22% to be electronegativity balanced, which has more than six times higher enrichment compared to the baseline pseudo-random sampling algorithm. Our LMs also demonstrate high generation novelty and their potential in new materials discovery is proved by their capability to recover the leave-out materials. We also find that the properties of the generated compositions can be tailored by training the models with selected training sets such as high-bandgap samples. Our experiments also show that different models each have their own preference in terms of the properties of the generated samples and their running time complexity varies a lot. We have applied our materials transformers to discover a set of new materials as validated using density functional theory calculations.