Streaming Convolutional Neural Networks for End-to-End Learning With Multi-Megapixel Images
Streaming Convolutional Neural Networks for End-to-End Learning With Multi-Megapixel Images
复制标题
DOI：
10.1109/tpami.2020.3019563
复制
发表时间：
2022-03-01
影响因子：
23.6
通讯作者：
Litjens, Geert
中科院分区：
文献类型：
--
作者：
Pinckaers, Hans;van Ginneken, Bram;Litjens, Geert
由于当前硬件的内存限制，大多数卷积神经网络（CNN）都是在亚百万像素图像上训练的。例如，计算机视觉中最流行的数据集包含的图像尺寸远远小于1百万像素（ImageNet为0.09MP， CIFAR-10为0.001MP）。在某些领域，如医学成像，需要数百万像素的图像来准确识别疾病的存在。我们提出了一种新的方法来直接训练卷积神经网络使用任何输入图像大小端到端。该方法利用了现代卷积神经网络中大多数操作的局部性，通过在图像的较小块上执行向前和向后传递。在这项工作中，我们使用高达6600万像素（8192x8192）的图像展示了概念验证，每张图像节省了大约50GB的内存。使用两个公共挑战数据集，我们证明cnn可以学习从这些大图像中提取相关信息，并从提高分辨率中受益。我们将检测乳腺癌转移的受体工作特征曲线下面积从0.580 （4MP）提高到0.706 (66MP) （CAMELYON17）。我们还获得了在TUPAC16数据集上接近最先进性能的Spearman相关度量，从0.485 （1MP）到0.570 （16MP）。重现实验子集的代码可在https://github.com/DIAGNijmegen/StreamingCNN上获得。
Due to memory constraints on current hardware, most convolution neural networks (CNN) are trained on sub-megapixel images. For example, most popular datasets in computer vision contain images much less than a megapixel in size (0.09MP for ImageNet and 0.001MP for CIFAR-10). In some domains such as medical imaging, multi-megapixel images are needed to identify the presence of disease accurately. We propose a novel method to directly train convolutional neural networks using any input image size end-to-end. This method exploits the locality of most operations in modern convolutional neural networks by performing the forward and backward pass on smaller tiles of the image. In this work, we show a proof of concept using images of up to 66-megapixels (8192x8192), saving approximately 50GB of memory per image. Using two public challenge datasets, we demonstrate that CNNs can learn to extract relevant information from these large images and benefit from increasing resolution. We improved the area under the receiver-operating characteristic curve from 0.580 (4MP) to 0.706 (66MP) for metastasis detection in breast cancer (CAMELYON17). We also obtained a Spearman correlation metric approaching state-of-the-art performance on the TUPAC16 dataset, from 0.485 (1MP) to 0.570 (16MP). Code to reproduce a subset of the experiments is available at https://github.com/DIAGNijmegen/StreamingCNN.