Enhancing Ensemble Diversity in Neural Ensemble Search for Uncertainty Quantification

该项目属于EPSRC人工智能技术研究领域。它是关于如何通过深层神经网络集成来增强不确定性量化的研究。特别是对于安全关键应用，如自动驾驶或医疗诊断，不确定性量化是相关的，然而，神经网络经常被错误地校准，并且对其预测表现出过高或过低的置信度。一种流行的改进神经网络校准的非贝叶斯方法是深度集成，它对从不同随机初始化训练的神经网络的预测进行平均，从而获得具有竞争力的预测精度和校准。为了进一步提高性能，文献中提出了自动构建具有不同结构的神经网络集成的方法。为了自动选择基本学习结构，他们建议将随机搜索与贪婪选择算法(NES-RS)相结合，这是一种易于并行化的方法，以及一种更复杂的基于正则化进化的进化算法(NES-RE)，表现出更好的性能。与此同时，温泽尔等人。提出了超深度集合，它在不同的随机初始化上聚合集合，分层在多个随机超参数上。然而，他们将基础学习者的体系结构保持不变。因此，对超参数进行更有针对性的搜索是否有助于构建更准确、更好校准和更稳健的神经网络集成，此外，是否可能通过集成这两种方法来获得更好的结果并扩展NES-RE以搜索体系结构和超参数的最佳组合(例如，本项目将通过对NES-RS和NES-RE进行不同的修改来探索这一新方法，这些修改使得能够同时搜索具有不同结构和超参数的集成。在不确定性、量化和稳健性方面改进目前最先进的NES-RE。如果是这样的话，调查这种不确定性量化方法是否有助于自动驾驶的安全

This project falls within the EPSRC `Artificial intelligence technologies' research area.It is an investigation into how to enhance uncertainty quantification through ensembles of deep neural networks.Especially for safety-critical applications such as autonomous driving or medical diagnosis, uncertainty quantification is relevant, however, neural networks are often badly calibrated and display either overly high or low confidence in their predictions. A popular non-Bayesian approach to improve calibration in neural networks is deep ensembles which average predictions of neural networks that have been trained from different random initialisations thereby achieving competitive predictive accuracy and calibration.To further enhance performance, methods proposed in the literature for automatically constructing ensembles of neural networks with varying architectures. They show that architectural variation leads to higher ensemble diversity resulting in ensembles with higher uncertainty calibration and robustness compared to the previous state-of-the-art deep ensembles.To automatically choose base learner architectures they suggest random search in combination with a greedy selection algorithm (NES-RS), an approach that is easily parallelisable, and a more sophisticated evolutionary algorithm based on regularized evolution (NES-RE), which exhibits better performance. Concurrent to this, Wenzel et al. proposed hyper-deep ensembles which aggregate ensembles over different random initialisations stratified over multiple random hyperparameters. However, they keep the architecture of their base learners fixed. The authors show that this approach for inducing diversity into ensembles also leads to increased predictive accuracy and calibration compared to deep ensembles.Therefore, the question arises whether a more directed search for hyperparameters can benefit the construction of more accurate, well-calibrated, and robust ensembles of neural networks, and, moreover, whether it is possible to achieve even better results by integrating the two approaches and extend NES-RE to search for optimal combinations of architecture and hyperparameters (e.g., dropout rate and different L2-regularizers) in base learners.This project will explore this novel approach by experimenting with different modifications to NES-RS and NES-RE that enable searching for ensembles with both varying architectures and hyperparameters simultaneously.Aims and Objectives1. Improve upon the current state-of-the-art NES-RE in terms of uncertainty quantification and robustness.2. If this is the case, investigate whether this approach to uncertainty quantification can aid safety in autonomous driving