Derivative-Free Optimization: Algorithmic Developments, Software Design, Applications, and Machine Learning

我的研究重点是无导数优化(DFO)，这在许多工程应用中是必不可少的。更准确地说，我关注的是黑箱优化，当工程优化问题的目标(S)和约束通过被视为黑箱的计算机代码获得时发生。这样的黑盒可能评估成本很高，可能会受到噪音的污染，有时还无法返回值。在此背景下，我正在考虑网格自适应直接搜索算法(MADS)。这项建议是我之前NSERC发现资助的继续，并描述了一个基于MADS的扩展以及DFO和机器学习(ML)之间的联系的五年研究计划。它分为12个非常适合学生的项目，他们将在优化和ML方面发展高级技能。 该计划将涉及算法设计和分析，以及软件开发和在各个领域的应用。 算法项目将开发新的工具，以增强黑盒优化问题的解决。它们包括基于最大似然法的新的替代技术的引入；识别和衡量最重要变量的敏感度分析；约束噪声问题的稳健优化方法；以及基于对近似帕累托前沿质量的不同度量的新的多目标算法。另一个项目涉及ML来调优MADS参数。将对新算法的特点进行数学分析，以证明其收敛。 我研究的一个主要部分是自由软件的设计和传播。特别是，我自2008年以来开发的最先进的Nomad包可以在LGPL许可证下免费获得，网址是www.grad.ca/novad。Nomad正在不断发展，负责项目的学生将在两名从其他来源资助的研究助理的帮助下，将算法开发整合到一揽子计划中。拟议的研究还将提供新版本的sgtelib代用品库，以及两个应用程序代码，用于DFO社区的基准。 我研究的最后一个部分涉及到实际的工程应用。例如，在过去，我致力于合金设计、飞机设计和能源的优化。在这项提案中，我们将利用材料科学中的应用来测试新的替代品；为嘈杂的机械工程问题开发健壮的解决方案；考虑具有两个或三个目标的电气工程问题；开发包含大多数典型黑盒问题特征的太阳能发电场模拟器。最后，我们将DFO技术应用于深度神经网络超参数的优化。 这项提议将有助于培养9名研究生和本科生，他们将接受具有基础性和应用性的多学科组合。

My research focuses on derivative-free optimization (DFO), which is essential in many engineering applications. More precisely, I am concerned with blackbox optimization, which occurs when the objective(s) and constraints of an engineering optimization problem are obtained by a computer code seen as a blackbox. Such blackboxes may be expensive to evaluate, may be contaminated with noise, and sometimes fail to return a value. In this context, I am considering the mesh adaptive direct search algorithm (MADS). This proposal is the continuation of my previous NSERC Discovery Grant and describes a five-year research program based on extensions of MADS and the links between DFO and machine learning (ML). It is divided into 12 projects that are well-suited for students, who will develop advanced skills in both optimization and ML. The program will address algorithm design and analysis, as well as software development and applications to various fields. The algorithmic projects will develop new tools that will enhance the solution of blackbox optimization problems. They include the introduction of new surrogate techniques based on ML; sensitivity analyses to identify and scale the most important variables; robust optimization methods for constrained noisy problems; and a new multiobjective algorithm based on the use of different measures of the quality of an approximate Pareto front. Another project involves ML to tune the MADS parameters. The new algorithmic features will be mathematically analyzed to prove convergence. A major part of my research is the design and dissemination of free software. In particular, the state-of-the-art NOMAD package that I have developed since 2008 is freely available under the LGPL licence at www.gerad.ca/nomad. NOMAD is in constant evolution, and the algorithmic developments will be integrated into the package by the students in charge of the projects with the help of two research associates funded from other sources. The proposed research will also provide new versions of the sgtelib library of surrogates, and two application codes for benchmarking purposes in the DFO community. The last component of my research concerns real engineering applications. For example, in the past, I worked on the optimization of alloy design, aircraft design, and energy. In this proposal, we will use applications in material science to test new surrogates; develop robust solutions for noisy mechanical engineering problems; consider electrical engineering problems with two or three objectives; develop a solar farm simulator that will include most of the typical blackbox-problem characteristics. Finally, we will apply DFO techniques to the optimization of the hyperparameters of deep neural networks. This proposal contributes to the training of 9 graduate and undergraduate students who will receive a multidisciplinary formation with fundamental and applied aspects.