Resilience, Interpretability, and Scale in Large Complex Systems

在我的整个研究生涯中，我一直对图像处理问题感兴趣，对层次或尺度有明确的概念。在过去的五到十年中，多尺度算法已经转变为现在所知的卷积神经网络(CNN)或深度网络，这是一系列方法，现在几乎主导了数据分析和计算机视觉的方方面面。尽管这些深层网络有着卓越的表现，但在某种程度上，这种不可战胜的表象是相当错位的。特别是，它们的非线性和大小意味着这些网络本质上是无法解释的，无法回答某个100层黑匣子中的1亿个参数是如何得出某个结论的。此外，还有一个令人不安的观察，即网络报告99.9%的准确率，实际上在其他非常基本的问题上是失败的。如果这样的网络要用于医疗保健、自动驾驶汽车或无数其他对生命至关重要的应用，那么一定程度的可解释性和网络弹性是必不可少的，事实上，一些政府正在缓慢地强制要求这样的网络。 以下相互关联的摘要概述了在我的探索基金下提出的探索性研究目标： 1.深度网络的弹性和可解释性： 深度学习的不同寻常的方面--在某些测试集上非常健壮，然后在其他测试集上出现令人费解的灾难性错误--暗示着不寻常的学习模式。由于网络本质上生活在1亿维的空间中，将学习的分类边界可视化是完全不可能的，但对学习过程的一些洞察对于产生更具弹性、学习结果更可预测和一定程度的可解释性网络至关重要。 2.规模在大型网络中的作用： 深度网络和相关策略的问题是计算复杂性高，以及该方法何时起作用或不起作用的可预测性有限。机器学习隐含地引入尺度的方式或时间是模棱两可的，尽管众所周知，学习的过滤器经常显示与尺度相关的模式。这项工作的目标是明确地引入规模依赖，无论是在输入端，从其他网络传输，还是显式地引入网络体系结构。 3.大型复杂系统和非线性网络： 从某种意义上讲，深度网络就是异常庞大的复杂非线性系统。然而，复杂系统研究与深度网络研究的联系相对较少。我想考虑复杂系统中弹性的各个方面是否可能导致对大型网络中弹性的相关问题的洞察，不是算法将是相同的，而是对弹性的理解是否可能转化为。 所有这些主题和技能在广泛的行业中都有需求，为HQP带来了出色的培训和未来的就业机会。

Throughout my research career I have been interested in image processing problems with an explicit notion of hierarchy or scale. In the last five to ten years, multi-scale algorithms have transformed into what are now known as convolutional neural networks (CNNs) or deep networks, a family of approaches which now dominates nearly every aspect of data analysis and computer vision. Although these deep networks have remarkable performance, to some extent this appearance of invincibility is quite misplaced. In particular, their nonlinearity and size means that these networks are essentially uninterpretable, that there is no way to answer how the 100-million parameters in some 100-layer black box reached a certain conclusion. Furthermore, there is the unsettling observation that networks, reporting 99.9% accuracy, in fact fail for other very elementary problems. If such networks are to be used in healthcare or autonomous vehicles or countless other life-critical applications, then a degree of interpretability and network resilience is essential, and indeed is slowly being mandated by some governments. The following inter-related summaries outline the exploratory research objectives proposed under my Discovery Grant: 1. Resilience and Interpretability of Deep Networks: The unusual aspects of deep learning - fantastically robust on certain test sets, and then bafflingly catastrophic errors on others - hint at unusual patterns of learning. Since the network essentially lives in a 100-million dimensional space, visualizing the learned classification boundaries is completely out of the question, yet some insight into the learning process is essential in producing more resilient networks with more predictable learning outcomes and some degree of interpretability. 2. The role of Scale in Large Networks: Deep networks and related strategies suffer from a high computational complexity and from a limited predictability as to when the approach will or will not work. It is ambiguous how or when scale may implicitly be introduced by machine learning, although anecdotally it is known that learned filters frequently show scale-related patterns. The goal of this work is to explicitly introduce scale dependence, whether at the inputs, transferred from other networks, or explicitly into the network architecture. 3. Large Complex Systems and Nonlinear Networks: Deep networks are, in a sense, just unusually large complex nonlinear systems. However complex systems research has had relatively little connection to research into deep networks. I would like to consider whether aspects of resilience in complex systems might lead to insights on related questions of resilience in large networks, not that the algorithm would be the same, but whether understanding of resilience might translate. All of these topics and skills are in demand across a wide range of industries, leading to outstanding training and future employment opportunities for HQP.