This research project falls under the EPSRC Artificial Intelligence Technologies category and specifically focuses on Bayesian Optimization

该研究项目属于EPSRC人工智能技术类别，特别关注贝叶斯优化[1]，这是人工智能的一个子领域。贝叶斯优化用于优化未知的“黑盒”函数，其中输入和输出之间的关系是未知的。例如，这样一个“黑盒”函数可以表示一个电子电路的参数和它的速度[2]之间的联系，或者一个神经网络架构和它在给定任务上的性能[3]之间的联系。这些类型的问题经常出现在计算机科学和工程领域，而贝叶斯优化为解决这些问题提供了一个合适的框架。在贝叶斯优化中，第一步是建立一个输入输出关系的模型，然后用这个模型找到最有希望的输入值来进行下一步的尝试。然而，该模型依赖于超参数[4]，这对其性能有很大影响。通常，这些超参数是从数据中估计出来的。然而，在工程问题中，获得新的样品可能是昂贵的。因此，我们掌握的数据通常是有限的，这使得这些估计可能不可靠。到目前为止，还没有提出一个系统的方法来处理这个问题。认识到文献中存在的差距，我们的项目的目标是开发一种在“黑盒”函数模型中选择超参数的原则方法。鉴于有限数据的约束，我们设想了两种应对这一挑战的潜在策略。这两个研究方向是独立的，我们的项目旨在对这两个方向进行全面的探索。一种方法涉及量化与当前超参数估计相关的不确定性。这需要建立一种理论上合理的方法来估计不确定性，并将这些估计纳入优化过程，使其更加稳健。或者，面对有限的数据，我们可以利用以前解决过的类似任务的信息。然而，在实现这一点之前，我们需要自动识别哪些过去的任务与我们正在处理的任务相似。大多数现有文献假设超参数是给定的或可以很容易地从数据中估计出来。因此，我们的方法的新颖性在于解除这个假设，并提出一个关于如何在更一般的情况下进行贝叶斯优化的基本问题。在整个项目中，我们的目标是开发更强大的算法，能够有效地解决工程问题，并以安全的方式处理超参数中的不确定性。[10] Srinivas, Niranjan，等。“匪徒设置中的高斯过程优化：无悔和实验设计。”第27届国际机器学习国际会议论文集。2010. bbbb10Grosnit， Antoine等。“沸腾：逻辑合成的贝叶斯优化。”2022年欧洲设计、自动化与测试会议与展览（日期）。IEEE 2022。[3]Nguyen， Vu等。“顺序和并行神经结构搜索的最佳传输核。”国际机器学习会议。PMLR 2021。[4]威廉姆斯，克里斯托弗·基，和卡尔·爱德华·拉斯穆森。机器学习的高斯过程。卷。2。3号。剑桥，马萨诸塞州：麻省理工学院出版社，2006。

This research project falls under the EPSRC Artificial Intelligence Technologies category and specifically focuses on Bayesian Optimization [1], a subfield of AI. Bayesian Optimization is employed for optimizing unknown "black-box" functions, where the relationship between input and output is not known. For instance, such a "black-box" function could represent the connection between the parameters of an electronic circuit and its speed [2] or a neural network architecture and its performance on a given task [3]. These types of problems often arise across computer science and engineering, and Bayesian Optimization provides a suitable framework for addressing them.In Bayesian Optimization, the first step involves constructing a model of the input-output relationship, which is later used to find the most promising input value to try next. However, this model relies on hyperparameters [4], which significantly influence its performance. Typically, these hyperparameters are estimated from data. However, in engineering problems, obtaining new samples can be costly. As such the data at our disposal is usually limited, making these estimates potentially unreliable. Up to this point there has not been a systematic method proposed for dealing with this issue.Recognizing this existing gap in the literature, our project is driven by the goal of developing a principled approach for selecting hyperparameters in "black-box" function models. Given the constraint of limited data, we envision two potential strategies for addressing this challenge. These two research directions are independent, and our project is designed to explore both comprehensively.One approach involves quantifying the uncertainty associated with current hyperparameter estimates. This entails establishing a theoretically sound method for estimating uncertainty and incorporating these estimates into the optimization process to make it more robust. Alternatively, in the face of limited data, we could leverage information from similar tasks that we have previously solved. However, before this could be achieved, we need to automatically identify which past tasks bear similarity to the one we are tackling.A majority of existing literature assumes that hyperparameters are given or can be easily estimated from data. Therefore, the novelty of our methodology lies in lifting this assumption and posing a fundamental question about how to conduct Bayesian Optimization in a more general case. Throughout this project, our goal is to develop more robust algorithms capable of efficiently solving engineering problems and handling the uncertainty in hyperparameters in a safe manner. [1] Srinivas, Niranjan, et al. "Gaussian process optimization in the bandit setting: no regret and experimental design." Proceedings of the 27th International Conference on International Conference on Machine Learning. 2010.[2] Grosnit, Antoine, et al. "BOiLS: Bayesian optimisation for logic synthesis." 2022 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 2022.[3] Nguyen, Vu, et al. "Optimal transport kernels for sequential and parallel neural architecture search." International Conference on Machine Learning. PMLR, 2021.[4] Williams, Christopher KI, and Carl Edward Rasmussen. Gaussian processes for machine learning. Vol. 2. No. 3. Cambridge, MA: MIT press, 2006.