Enhancing Ensemble Diversity in Neural Ensemble Search for Uncertainty Quantification

该项目福尔斯属于EPSRC“人工智能技术”的研究领域，旨在研究如何通过深度神经网络的集成来增强不确定性量化。特别是对于自动驾驶或医疗诊断等安全关键型应用，不确定性量化是相关的，然而，神经网络通常校准不良，并且在其预测中显示出过高或过低的置信度。一个流行的非贝叶斯方法来提高神经网络的校准是深度集成，平均预测的神经网络，已经从不同的随机初始化训练，从而实现竞争力的预测精度和calibration.To进一步提高性能，在文献中提出的方法自动构建不同架构的神经网络集成。他们表明，与以前最先进的深度集成相比，架构变化导致更高的集成多样性，从而导致集成具有更高的不确定性校准和鲁棒性。为了自动选择基础学习器架构，他们建议随机搜索与贪婪选择算法（NES-RS）相结合，这是一种易于并行化的方法，基于正则化进化的进化算法（NES-RE）具有更好的性能。与此同时，Wenzel等人提出了超深度集成，该集成聚合了在多个随机超参数上分层的不同随机初始化的集成。然而，它们保持其基本学习器的架构固定。作者表明，与深度集成相比，这种将多样性引入集成的方法也会提高预测精度和校准。因此，问题出现了，对超参数的更直接搜索是否有助于构建更准确，校准良好和鲁棒的神经网络集成，此外，是否有可能通过集成这两种方法并扩展NES-RE以搜索架构和超参数的最佳组合来实现甚至更好的结果（例如，本项目将通过对NES-RS和NES-RE进行不同的修改来探索这种新的方法，这些修改可以同时搜索具有不同架构和超参数的集成。在不确定度量化和鲁棒性方面对当前最先进的NES-RE进行改进。如果是这种情况，调查这种不确定性量化方法是否有助于自动驾驶的安全性

This project falls within the EPSRC `Artificial intelligence technologies' research area.It is an investigation into how to enhance uncertainty quantification through ensembles of deep neural networks.Especially for safety-critical applications such as autonomous driving or medical diagnosis, uncertainty quantification is relevant, however, neural networks are often badly calibrated and display either overly high or low confidence in their predictions. A popular non-Bayesian approach to improve calibration in neural networks is deep ensembles which average predictions of neural networks that have been trained from different random initialisations thereby achieving competitive predictive accuracy and calibration.To further enhance performance, methods proposed in the literature for automatically constructing ensembles of neural networks with varying architectures. They show that architectural variation leads to higher ensemble diversity resulting in ensembles with higher uncertainty calibration and robustness compared to the previous state-of-the-art deep ensembles.To automatically choose base learner architectures they suggest random search in combination with a greedy selection algorithm (NES-RS), an approach that is easily parallelisable, and a more sophisticated evolutionary algorithm based on regularized evolution (NES-RE), which exhibits better performance. Concurrent to this, Wenzel et al. proposed hyper-deep ensembles which aggregate ensembles over different random initialisations stratified over multiple random hyperparameters. However, they keep the architecture of their base learners fixed. The authors show that this approach for inducing diversity into ensembles also leads to increased predictive accuracy and calibration compared to deep ensembles.Therefore, the question arises whether a more directed search for hyperparameters can benefit the construction of more accurate, well-calibrated, and robust ensembles of neural networks, and, moreover, whether it is possible to achieve even better results by integrating the two approaches and extend NES-RE to search for optimal combinations of architecture and hyperparameters (e.g., dropout rate and different L2-regularizers) in base learners.This project will explore this novel approach by experimenting with different modifications to NES-RS and NES-RE that enable searching for ensembles with both varying architectures and hyperparameters simultaneously.Aims and Objectives1. Improve upon the current state-of-the-art NES-RE in terms of uncertainty quantification and robustness.2. If this is the case, investigate whether this approach to uncertainty quantification can aid safety in autonomous driving