Collaborative Research: Efficient Bayesian Global Optimization with Applications to Deep Learning and Computer Experiments

这项研究的主要目标是开发全局优化方法，以显着提高复杂科学问题研究的优化效率。这些研究成果将大大加快涉及人工智能和数值模拟的众多科学学科的发现，如机械工程、能源、自动化交通、航空航天工程、环境科学和材料科学。这项研究纳入了一项教育计划，强调对广泛的学生进行跨学科培训，并增加代表人数不足的群体的参与。私人投资机构将从代表性不足的群体中招募女学生和本科生，并积极让她们参与这项研究。研究成果将在会议上传播。此外，研究成果还将被整合到PIS的课程中，为研究生和本科生提供优化和数据分析培训。本文的研究重点是贝叶斯全局优化，它是指通过随机过程先验发展的主动学习策略，用于优化昂贵的黑盒函数。针对深度学习和计算机实验中的全局优化带来的挑战，提出了两种新的贝叶斯主动学习方法，适用于具有条件依赖输入和非高斯随机输出的问题。第一种方法将解决分类问题中随机输出的最优化所产生的一个重要但尚未解决的问题。其创新之处在于基于广义高斯过程开发的预期改进标准，该标准导致了一个易于处理的目标函数，并具有直观的解释。在连续带臂的条件下，系统的渐近收敛性质将得到严格的发展。第二种方法是基于实际中常见的分支和嵌套结构的新的关联函数。期望得到新相关函数有效的充分条件，并系统地构造一类新的最优初始设计。通过严格的贝叶斯全局优化实现深度学习中的自动调整的创新思想，将为黑盒函数的优化提供新的方法，并启发机器学习、优化和空间统计的新研究思路。除了在机器人学中应用于计算机视觉和最优控制之外，设计、建模和优化策略可以为研究具有昂贵的未知函数的复杂优化问题开辟新的途径，并激发理论和应用研究的活力。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The primary objective of this research is to develop global optimization methods which will dramatically enhance the optimization efficiency in the studies of complex scientific problems. The research findings will significantly accelerate discoveries in numerous scientific disciplines involving artificial intelligence and numerical simulations such as mechanical engineering, energy, automated transportation, aerospace engineering, environmental science, and materials science. Integrated into the research is an education plan that emphasizes interdisciplinary training for a broad body of students and increasing participation from underrepresented groups. The PIs will recruit female students and undergraduate students from underrepresented groups and actively involve them in this research. Research findings will be disseminated at conferences. Furthermore, research findings will also be integrated into PIs’ courses to have optimization and data analysis training for graduate and undergraduate students. This research focuses on Bayesian global optimization which refers to active learning strategies developed by stochastic process priors for the optimization of expensive "black box" functions. Motivated by the challenges emerged from global optimization in deep learning and computer experiments, two innovative Bayesian active learning methods will be developed which are applicable to problems with conditionally dependent inputs and non-Gaussian stochastic outputs. The first method will address an important but unresolved issue arising from the optimization of stochastic outputs in classification problems. The novelty lies in an expected improvement criterion developed based on a generalized Gaussian process which leads to a tractable objective function with an intuitive interpretation. The asymptotic convergence properties will be developed rigorously under the continuum-armed-bandit settings. The second method is based on a new correlation function for a branching and nested structure, which occurs commonly in practice. Sufficient conditions on the validity of the new correlation functions is expected to be derived and a new class of optimal initial designs will be systematically constructed. The innovative idea of automatic-tuning in deep learning by a rigorous Bayesian global optimization will shed light on new methodologies for the optimization of "black box" functions and inspire new research ideas in machine learning, optimization, and spatial statistics. Beyond the applications to computer vision and optimal controls in robotics, the design, modeling, and optimization strategies can open new avenues for studying complex optimization problems with expensive unknown functions and energize both theoretical and applied research.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.