Diagnostic Tools for Evaluating and Comparing Simulation-Optimization Algorithms
Diagnostic Tools for Evaluating and Comparing Simulation-Optimization Algorithms
复制标题
DOI：
10.1287/ijoc.2022.1261
复制
发表时间：
2023-01
期刊：
影响因子：
--
通讯作者：
David J. Eckman;S. Henderson;S. Shashaani
中科院分区：
文献类型：
--
作者：
David J. Eckman;S. Henderson;S. Shashaani
仿真优化涉及优化一些只能通过随机仿真估计的目标函数。许多重要问题可以在这个框架内得到有益的观察。尽管许多求解器（模拟优化算法的实现）已经存在或正在开发中，但求解器之间的比较并不标准化，并且通常在范围上受到限制。这样的比较有助于推进求解器的开发，澄清求解器的相对性能，并识别无法有效解决的问题的类别，以及许多其他用途。我们开发的性能指标和图，和估计，评估和比较求解器和诊断他们的优势和劣势的仿真优化问题的测试平台。我们解释了在这种情况下，两级模拟的需要，并提供支持收敛理论。我们还描述了如何使用自举来获得估计的误差估计。历史：由模拟区域编辑Bruno Tuffin接受。资金：这项工作得到了国家科学基金会的支持[赠款CMMI-2035086，CMMI-2206972和TRIPODS+X DMS-1839346]。补充材料：支持本研究结果的软件可在论文及其补充信息[https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2022.1261]中获得，或可从IJOC GitHub软件库（https://github.com/INFORMSJoC）[http://dx.doi.org/10.5281/zenodo.7329235]获得。
Simulation optimization involves optimizing some objective function that can only be estimated via stochastic simulation. Many important problems can be profitably viewed within this framework. Whereas many solvers—implementations of simulation-optimization algorithms—exist or are in development, comparisons among solvers are not standardized and are often limited in scope. Such comparisons help advance solver development, clarify the relative performance of solvers, and identify classes of problems that defy efficient solution, among many other uses. We develop performance measures and plots, and estimators thereof, to evaluate and compare solvers and diagnose their strengths and weaknesses on a testbed of simulation-optimization problems. We explain the need for two-level simulation in this context and provide supporting convergence theory. We also describe how to use bootstrapping to obtain error estimates for the estimators. History: Accepted by Bruno Tuffin, area editor for simulation. Funding: This work was supported by the National Science Foundation [Grants CMMI-2035086, CMMI-2206972, and TRIPODS+X DMS-1839346]. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplementary Information [ https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2022.1261 ] or is available from the IJOC GitHub software repository ( https://github.com/INFORMSJoC ) at [ http://dx.doi.org/10.5281/zenodo.7329235 ].