Robustness of Invertible Neural Networks

点击翻译按钮获取中文摘要

Constructing histograms or probability distributions in low dimensions is very simple, and as humans we are able to visualise them in our heads in one, two or maybe even three dimensions. However, the real world is often not constrained to low dimensions, and the probability of an event occurring can be dependent on a large number of factors, resulting in a high-dimensional probability distribution. However, constructing such a histogram in a standard way is often not possible due to the high dimensionality of the problem. Invertible Neural Networks (INN) and Normalising Flows (NFs) offer a way to evaluate the likelihoods of complex high-dimensional distributions. This is achieved by learning a non-linear bijective mapping between a simple latent parametric distribution (which is simple, and we know) and the true distribution (which we don't know and is hard to model). The likelihood of a data sample is then equal to the likelihood of the mapped data point on the simple latent distribution scaled by a normalising constant. Thus, rather than measuring the likelihood on the true distribution, we instead map it to a latent distribution where we can evaluate it easily and then scale it appropriately. Furthermore, INNs allow us to sample from the latent space and perform the inverse mapping to generate data samples. Whilst the theory underpinning these methods is attractive, several open issues remain regarding their usability and accuracy. Addressing these issues will form the core of my thesis and the majority or my research. The first major problem is calculating the normalising constant needed to evaluate the exact likelihood of the data. Naïve calculation is computationally prohibitively expensive, and as such methods to speed up the computation need to leveraged. Standard methods enforce a variety of constraints on the network, potentially harming the expressivity of the mapping. Addressing this issue is the first major objective for my thesis, significant progress has already been made and a method addressing the above issues has been implemented and is awaiting publication. The second issue is enforcing the invertibility of the mapping (to generate samples), this issue forms part of a much broader picture of inferring in input for a given output. For this application, various optimization methods can be employed to obtain the input, however they often do not work well and sometimes not all. This is currently forming the majority of time and preliminary methods have been tested, but none that work successfully across all possible mappings. Thirdly, I will be constructing a thorough investigation into the stability and robustness of INNs and NFs. Recent work suggests that these methods may not be as accurate as they seem. As such part of my ongoing work is to investigate, where and how these methods fail. Thus, allowing us to potentially compensate for their pitfalls. The above three points are currently forming the majority of my research and project work. They are all clearly linked and well-motivated from the literature. Addressing all of them will provide a significant contribution to the community.