Information Theoretic Coding for Deep Neural Networks: Frameworks, Theory, and Algorithms

深度神经网络（DNN）在许多应用中越来越普遍，包括计算机视觉、语音识别和自然语言处理。随着我们的数字社会收集到更多的数据，DNN将继续成为工程领域的一个主要增长领域，并改变我们的生活、工作以及与智能机器交互的方式。然而，在DNN可以广泛部署在我们无处不在的通信网络的许多部分之前，DNN的几个关键挑战必须得到解决。例如，看看用于图像分类的DNN。第一个挑战在于什么类型的原始数据将被输入DNN。实际上，在包括数据采集、数据编码（即，在DNN的编码/压缩、数据传输和数据处理/利用中，馈送到每个DNN中的原始数据不是“原始的”;相反，它通常以有损方式被编码/压缩。有损编码如何影响DNN？根据传统观点，为人类感知而设计的现有有损编解码器通常会降低DNN的分类精度。然而，在我们最近的一项工作中，我们通过实验证明了相反的情况--如果可以智能地选择将原始数据的哪个压缩版本输入DNN，DNN的分类精度实际上可以显著提高，同时大大减少传输和存储的比特数。当然，问题是如何为DNN智能地编码原始数据。 第二个挑战是DNN对对抗性示例的脆弱性，恶意修改的输入具有不可感知的扰动，导致DNN产生不正确的输出。对抗性示例的存在和容易构建给DNN带来了重大的安全风险，特别是在安全关键型应用中。第三个挑战在于DNN中的模型参数数量庞大，可能高达数十亿。正是大量的模型参数使得DNN既计算密集型又存储密集型，阻碍了DNN在资源有限的设备中的广泛部署。这也使得在分布式学习中传输和更新模型参数变得困难和昂贵（就带宽而言）。 基于我们早期的成功，在本研究计划中，我们将系统地研究上述挑战，通过将信息理论思想（如软决策量化）引入DNN领域，为DNN提出新的编码框架，发展各自的理论，并为DNN和人类设计新的有效算法，保护DNN免受对抗性攻击，或者联合压缩和训练DNN模型。 我们的研究成果将显著推动信息理论、图像编码、深度学习和计算机视觉领域的发展，并对加拿大及其他地区的相关行业产生重大影响。

Deep neural networks (DNNs) are increasingly becoming ubiquitous in many applications including computer vision, speech recognition, and natural language processing. With more data collected in our digital society, DNNs will continue to be a major area of growth in engineering, and change how we live, work, and interact with each other and intelligent machines. Before DNNs can be widely deployed in many parts of our ubiquitous communications networks, however, several key challenges of DNNs have to be addressed. For example, take a look at DNNs for image classification. The first challenge lies in what type of raw data will be fed into DNNs. Indeed, in the context of ubiquitous communications networks which include the whole pipeline of data acquisition, data encoding (i.e., compression), data transmission, and data processing/utilization, the raw data fed into each DNN is not "raw"; instead, it is generally encoded/compressed in a lossy manner. How does lossy coding impact a DNN? According to the conventional wisdom, existing lossy codecs designed for human perception generally degrade the classification accuracy of the DNN. In one of our recent works, however, we showed experimentally the opposite---if one can choose intelligently which compressed version of the raw data is fed into the DNN, the classification accuracy of the DNN can actually be improved significantly while reducing dramatically the number of bits for transmission and storage. The question is, of course, how to encode raw data intelligently for DNNs. The second challenge is the vulnerability of DNNs to adversarial examples, maliciously modified inputs with imperceptible perturbation that lead DNNs to produce incorrect outputs. The existence and easy construction of adversarial examples pose significant security risks to DNNs, especially in safety critical applications. The third challenge lies in the huge number of model parameters in DNNs, which can be as high as a few billions. It is the huge number of model parameters that makes DNNs both computationally intensive and memory intensive, hindering the wide deployment of DNNs in resource limited devices. It also makes it difficult and costly (in terms of bandwidth) to transmit and update model parameters in distributed learning. Based on our early success, in this research program, we will investigate the challenges mentioned above systematically by introducing information theoretic ideas such as soft decision quantization into the domain of DNN, proposing new coding frameworks for DNNs, developing their respective theories, and designing new effective algorithms for new forms of compression for both DNNs and human, protecting DNNs against adversarial attacks, or jointly compressing and training DNN models. Our research results will significantly advance the fields of information theory, image coding, deep learning, and computer vision, and have great impacts on the related industries in Canada and beyond.