Information Theoretic Coding for Deep Neural Networks: Frameworks, Theory, and Algorithms

点击翻译按钮获取中文摘要

Deep neural networks (DNNs) are increasingly becoming ubiquitous in many applications including computer vision, speech recognition, and natural language processing. With more data collected in our digital society, DNNs will continue to be a major area of growth in engineering, and change how we live, work, and interact with each other and intelligent machines. Before DNNs can be widely deployed in many parts of our ubiquitous communications networks, however, several key challenges of DNNs have to be addressed. For example, take a look at DNNs for image classification. The first challenge lies in what type of raw data will be fed into DNNs. Indeed, in the context of ubiquitous communications networks which include the whole pipeline of data acquisition, data encoding (i.e., compression), data transmission, and data processing/utilization, the raw data fed into each DNN is not "raw"; instead, it is generally encoded/compressed in a lossy manner. How does lossy coding impact a DNN? According to the conventional wisdom, existing lossy codecs designed for human perception generally degrade the classification accuracy of the DNN. In one of our recent works, however, we showed experimentally the opposite---if one can choose intelligently which compressed version of the raw data is fed into the DNN, the classification accuracy of the DNN can actually be improved significantly while reducing dramatically the number of bits for transmission and storage. The question is, of course, how to encode raw data intelligently for DNNs. The second challenge is the vulnerability of DNNs to adversarial examples, maliciously modified inputs with imperceptible perturbation that lead DNNs to produce incorrect outputs. The existence and easy construction of adversarial examples pose significant security risks to DNNs, especially in safety critical applications. The third challenge lies in the huge number of model parameters in DNNs, which can be as high as a few billions. It is the huge number of model parameters that makes DNNs both computationally intensive and memory intensive, hindering the wide deployment of DNNs in resource limited devices. It also makes it difficult and costly (in terms of bandwidth) to transmit and update model parameters in distributed learning. Based on our early success, in this research program, we will investigate the challenges mentioned above systematically by introducing information theoretic ideas such as soft decision quantization into the domain of DNN, proposing new coding frameworks for DNNs, developing their respective theories, and designing new effective algorithms for new forms of compression for both DNNs and human, protecting DNNs against adversarial attacks, or jointly compressing and training DNN models. Our research results will significantly advance the fields of information theory, image coding, deep learning, and computer vision, and have great impacts on the related industries in Canada and beyond.