Collaborative Research: Efficient Bayesian Global Optimization with Applications to Deep Learning and Computer Experiments

本研究的主要目标是开发全局优化方法，这将大大提高优化效率在复杂的科学问题的研究。这些研究成果将大大加速涉及人工智能和数值模拟的众多科学学科的发现，如机械工程，能源，自动运输，航空航天工程，环境科学和材料科学。这项研究纳入了一项教育计划，该计划强调对广大学生进行跨学科培训，并增加代表性不足群体的参与。研究所将从代表性不足的群体中招募女学生和本科生，并积极让她们参与这项研究。研究结果将在会议上传播。此外，研究成果也将融入PI的课程，为研究生和本科生提供优化和数据分析培训。 本研究的重点是贝叶斯全局优化，这是指主动学习策略的随机过程先验优化昂贵的“黑箱”功能。受深度学习和计算机实验中全局优化所带来的挑战的启发，将开发两种创新的贝叶斯主动学习方法，适用于条件依赖输入和非高斯随机输出的问题。第一种方法将解决一个重要的，但未解决的问题所产生的随机输出的优化分类问题。新颖之处在于预期的改进标准的基础上开发的广义高斯过程，导致一个易于处理的目标函数与直观的解释。渐近收敛性质将严格发展下的连续武装土匪设置。第二种方法是基于一个新的相关函数的分支和嵌套结构，这在实践中经常发生。新的相关函数的有效性的充分条件预计将被导出，一类新的最优初始设计将被系统地构造。通过严格的贝叶斯全局优化在深度学习中进行自动调整的创新思想将为优化“黑箱”函数提供新的方法，并激发机器学习，优化和空间统计的新研究思路。除了在机器人技术中的计算机视觉和最优控制应用外，设计、建模和优化策略还可以为研究具有昂贵未知功能的复杂优化问题开辟新途径，并为理论和应用研究注入活力。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响评审标准进行评估，被认为值得支持。

The primary objective of this research is to develop global optimization methods which will dramatically enhance the optimization efficiency in the studies of complex scientific problems. The research findings will significantly accelerate discoveries in numerous scientific disciplines involving artificial intelligence and numerical simulations such as mechanical engineering, energy, automated transportation, aerospace engineering, environmental science, and materials science. Integrated into the research is an education plan that emphasizes interdisciplinary training for a broad body of students and increasing participation from underrepresented groups. The PIs will recruit female students and undergraduate students from underrepresented groups and actively involve them in this research. Research findings will be disseminated at conferences. Furthermore, research findings will also be integrated into PIs’ courses to have optimization and data analysis training for graduate and undergraduate students. This research focuses on Bayesian global optimization which refers to active learning strategies developed by stochastic process priors for the optimization of expensive "black box" functions. Motivated by the challenges emerged from global optimization in deep learning and computer experiments, two innovative Bayesian active learning methods will be developed which are applicable to problems with conditionally dependent inputs and non-Gaussian stochastic outputs. The first method will address an important but unresolved issue arising from the optimization of stochastic outputs in classification problems. The novelty lies in an expected improvement criterion developed based on a generalized Gaussian process which leads to a tractable objective function with an intuitive interpretation. The asymptotic convergence properties will be developed rigorously under the continuum-armed-bandit settings. The second method is based on a new correlation function for a branching and nested structure, which occurs commonly in practice. Sufficient conditions on the validity of the new correlation functions is expected to be derived and a new class of optimal initial designs will be systematically constructed. The innovative idea of automatic-tuning in deep learning by a rigorous Bayesian global optimization will shed light on new methodologies for the optimization of "black box" functions and inspire new research ideas in machine learning, optimization, and spatial statistics. Beyond the applications to computer vision and optimal controls in robotics, the design, modeling, and optimization strategies can open new avenues for studying complex optimization problems with expensive unknown functions and energize both theoretical and applied research.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.