Metric-based mutual information estimators and their application to analysing neural recordings Abstract

自从香农提出信息论以来，信息论在各个领域得到了广泛的应用，并取得了巨大的成功。这些领域从数据压缩到语言学，再到热力学。它允许量化随机变量分布中包含的信息，以及这些分布如何相互关联。这些都是基本的、非常重要的想法——解释了它的使用和流行。信息论中的一个关键度量是互信息。这捕获了学习一个变量给你的关于另一个变量的信息量。这种方法在机器学习等领域已经很流行，并且越来越受欢迎。然而，可靠、快速地计算或估计这一数量是具有挑战性的。在实践中，这通常涉及计算整个分布，然后是对数和其他非线性量，这些量可能难以估计，有偏差且收敛缓慢。在（Houghton, 2015） （Houghton, 2019）中，引入了一种估计互信息的新方法。该近似器依赖于在度量空间中嵌入数据，并利用该空间中的距离度量。它是一个Kozachenko-Leonenko估计器，从数据点之间的关系估计互信息，而不是从点本身。这使得它在高维数据上工作得很好，因为它是在估计器中使用的距离，而不是数据点。实际上，使用距离度量意味着该估计器可以处理比传统方法或原始方法小得多的数据集。我们将探讨这种方法的效果和用途，重点是分析神经记录和机器学习。互信息，或信息增益，是机器学习和人工智能中的一个常见数量，无论是隐式的还是显式的。明确的例子包括分裂数据集以构建决策树，以及测量神经网络中的信息流。由于互信息隐含地捕获了两个随机变量中共享信息的数量，机器学习和推理问题通常可以使用信息论和互信息来构建。信息理论在神经科学领域非常有用。大脑的主要功能之一是处理和存储信息，因此应用信息理论方法显然是合适的。随着数据采集和实验技术的进步，神经科学实验提供了大量的时间高维数据，而且往往是不同类型的数据。信息论的模型独立性质意味着它能够捕捉到非常广泛的现象和相互作用，因为它们不受假设模型的限制。这个估计器的一个组成部分是在尖峰列车记录的空间上有一个“合理的”距离测量，这个问题已经提出了多个测量方法，并将研究用于估计互信息的方法。这就给出了一个适合这种估计器的区域；在这个领域，大型数据集可能成本高昂或不可行。我们的目标是将这种互信息估计器应用于马特·琼斯实验室记录的生物数据。这里的目的是，通过估计器，计算出记录的神经元之间的关系，并推断出网络的一些东西。以前也使用过类似的方法，尽管在更有限的环境中，取得了良好的结果，我们探索这种新方法的效果。总体而言，该项目将在记录数据的神经科学背景下，以及在机器学习和人工神经网络的背景下研究和应用该估计器，并发展互信息估计器理论。这项研究属于EPSRC人工智能技术研究领域。霍顿，C.（2015）。计算高速列车的相互信息和其他有距离但没有坐标的数据。

Since Shannon's introduction of Information Theory, it has been applied widely and has found great success in a wide range of fields. These fields range from data compression through linguistics, to thermodynamics. It allows the quantification of the information contained in distributions of random variables, and how these distributions relate to one another. These are fundamental, widely important ideas - explaining its use and popularity. One key measure in Information Theory is mutual information. This captures the amount of information learning one variable gives you about another. This measure has become popular in areas such as machine learning, and is growing in popularity. However, reliably, quickly calculating or estimating this quantity can prove challenging. In practice, this often involves calculating the whole distribution, followed by logarithmic and other non linear quantities, which can be hard to estimate, biased and slow to converge. In (Houghton, 2015) (Houghton, 2019), a new approach to estimating mutual information was introduced. This approximator relies on embedding data in a metric space, and exploiting the distance measure in that space. It is a Kozachenko-Leonenko estimator, estimating the mutual information from the relationship between data points, rather than from the points themselves. This gives the effect that it works well on high dimensional data, as it is this distance used in the estimator, not the data points. Practically, making use of the distance metric means that this estimator can work with a much smaller data set than traditional or naive methods. The effects and uses of this method will be explored, focussing on analysing neural recordings, and in machine learning. Mutual information, or information gain, is a common quantity in machine learning and AI, either implicitly or explicitly. Explicit examples include splitting data sets to construct decision trees, and for measuring information flow in neural networks. Implicitly, as mutual information captures the amount of shared information in two random variables, machine learning and inference problems can often be framed using Information Theory and mutual information. One of the areas Information Theory has been very useful in is neuroscience. One of the primary functions of the brain is to process and store information, so applying Information Theoretic approaches is an obvious fit. Experiments in neuroscience are, as data acquisition and experimental techniques improve, giving a lot of temporal high dimensional data, and often data of different types. The model independent nature of Information Theory means it is able to capture a very wide range of phenomena and interactions, as they are not limited by assuming a model. An integral part of this estimator is having a 'sensible' distance measure on the space of spike train recordings, a problem for which multiple measures have been proposed and will be investigated for use in estimating mutual information. This gives an area that this kind of estimator is suited to; one where large datasets can be costly or unfeasible. We aim to apply this mutual information estimator to biological data, recorded in Matt Jones' lab. The aim here would be, through the estimator, to work out the relationships between recorded neurons, and infer something of the network. Similar approaches have been used before, though in a more restricted setting, to good results, and we explore the effect of this new approach. Overall, this project will study and apply this estimator, in the context of neuroscience on recorded data, as well as in the context of machine learning and for artificial neural networks - and to develop the theory of mutual information estimators. This research falls within the EPSRC Artificial Intelligence Technologies research area. References Houghton, C. (2015). Calculating mutual information for spike trains and other data with distances but no coordinates.