SHF: Small: Methods, Workflows, and Data Commons for Reducing Training Costs in Neural Architecture Search on High-Performance Computing Platforms

神经网络是强大的人工智能模型，可以自动捕获科学数据中的嵌入式知识。科学家可以利用这些知识来解决物理学、材料科学、神经科学和医学成像等领域的问题。为特定的科学数据集或特定的问题找到准确的神经网络需要很高的训练成本：它需要在大量高性能计算资源上的数千个神经网络中进行搜索。该项目提供了用于降低神经网络训练成本的方法、工作流程和数据共享。这些方法基于参数化建模，能够在训练过程的早期快速终止搜索，使搜索过程更快，更便宜。工作流将搜索与神经网络对不同数据集和问题的准确性预测解耦。数据共享共享神经网络的全部来源，因此其他科学家可以在自己的研究中部署神经网络。神经网络研究的进展对许多科学应用产生了深远的影响。精确的神经网络可用于从原始显微镜数据中提取结构信息，预测业务流程的性能，分析癌症病理学数据，将蛋白质序列映射到折叠，以及预测土壤湿度或作物产量。研究人员建立更广泛的高性能计算专家社区的努力也对人工智能产品的有效设计和使用产生了深远的影响。研究小组通过辅导Systers（田纳西诺克斯维尔大学电气工程和计算机科学领域的妇女组织）的学生，促进代表性不足的学生，特别是妇女的更多参与。此外，研究人员还为计算机科学系以外的科学领域的研究生和本科生开发了量身定制的课程。该项目解决了减少用于神经网络训练的高性能计算资源的迫切需要，同时确保可解释，可重现和接近最佳的神经网络。为此，研究小组提出了一种灵活的适应度预测方法，该方法使用参数建模来预测神经网络的未来适应度，并允许提前终止训练过程。通过这个项目，研究人员为一组不同的适应度曲线创建了一个有效参数函数的索引，包括建模中的边缘情况（例如，从不学习的神经网络或经历学习延迟的神经网络）。研究人员将神经架构搜索实现从紧密耦合的单一软件工具中嵌入搜索和预测，转变为灵活的模块化工作流程，其中搜索和预测是解耦的。项目工作流程使用户能够降低训练成本，增加神经架构搜索吞吐量，并使健身预测适应不同的健身测量，数据集和问题。研究人员构建了一个可搜索和可重复使用的神经网络数据共享记录路径，通过生成，训练和验证阶段捕获神经网络的生命周期，记录神经网络架构，训练数据集以及每个阶段的损失和准确性值。神经网络数据共享使用户能够在训练过程中研究神经网络性能的演变，并识别神经网络的架构与其在具有特定属性的给定数据集上的性能之间的关系，最终支持在一系列真实世界的科学数据集上有效搜索准确的神经网络。此外，数据共享空间为科学界提供了一种资源，用于研究数据集、网络架构和性能之间的关系。为了评估不同数据集的鲁棒性，该项目考虑了众所周知的基准数据集和真实世界的科学数据集，这些数据集包括蛋白质结构分析中的X射线电子激光束的蛋白质衍射图案，精准农业中无人机的作物侦察图像，林业-该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准。

Neural networks are powerful artificial-intelligence models that capture embedded knowledge in scientific data automatically. Scientists can use the knowledge to solve problems in domains such as physics, materials science, neuroscience, and medical imaging, among others. Finding accurate neural networks for a specific scientific dataset or particular problem comes at a high training cost: it requires searching among thousands of neural networks on a large number of high-performance-computing resources. This project delivers methods, workflows, and a data commons for reducing the training cost of neural networks. The methods are based on parametric modeling and enable rapid search termination early in the training process, making the search process faster and cheaper. The workflows decouple the search from the accuracy prediction of neural networks for different datasets and problems. The data commons shares the full provenance of the neural networks so other scientists can deploy the neural networks in their own research. Advances in neural networks research have a far-reaching impact on many scientific applications. Accurate neural networks can be used to extract structural information from raw microscopy data, predict performance of business processes, analyze cancer pathology data, map protein sequences to folds, and predict soil moisture or crop yield. The researchers’ efforts to build a broader community of high-performance-computing experts also have a far-reaching impact on the efficient design and use of artificial-intelligence products. The team of researchers promotes increased participation of underrepresented students, particularly women, through mentoring of students in Systers (the organization for women in Electrical Engineering and Computer Science at the University of Tennessee Knoxville). Furthermore, the researchers also develop curricula tailored for a diverse population of graduate and undergraduate students across scientific domains beyond the department of computer science.This project addresses the urgent need to reduce the use of high-performance-computing resources for the training of neural networks, while assuring explainable, reproducible and nearly-optimal neural networks. To this end, the team of researchers proposes a flexible fitness-prediction method that uses parametric modeling to predict future fitness of neural networks and allow for early termination of the training process. Through this project, the researchers create an index of effective parametric functions for a diverse suite of fitness curves, including edge cases in the modeling (e.g., neural networks that never learn or neural networks that experience a learning delay). The researchers transform neural-architecture search implementations from tightly-coupled, monolithic software tools embedding both search and prediction into a flexible, modular workflow in which search and prediction are decoupled. Project workflows enable users to reduce training cost, increase neural-architecture search throughput, and adapt fitness predictions to different fitness measurements, datasets, and problems. The researchers build a searchable and reusable neural-network data commons of record trails that capture the neural network’s lifespan through generation, training, and validation stages, recording the neural network architecture, the training dataset, and loss and accuracy values throughout each stage. The neural-network data commons enables users to study the evolution of neural-network performance during training and identify relationships between a neural network’s architecture and its performance on a given dataset with specific properties, ultimately supporting effective searches for accurate neural networks across a spectrum of real-world scientific datasets. Furthermore, the data commons provides the scientific community with a resource to study the relationships between datasets, network architectures, and performance. To assess robustness for different datasets, the project considers both well-known benchmark datasets and real-world scientific datasets of protein diffraction patterns from x–ray electron laser beams in protein structural analysis, crop-scouting images from drones in precision farming, and forestry-scouting drone images for wildfire prevention.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.