Deep learning with limited data

深度神经网络是一种基于类神经元模型的机器学习技术。这里的“深度”指的是具有能够表示复杂功能的许多层的网络。这种模型已经被证明是非常有价值的，因为它们可以从示例中学习分层数据表示法，并预测以前未见过的数据。这些技术使计算机视觉和模式识别领域最近取得的许多进展成为可能，而深度神经网络的进步则是由于大数据集(Big Data)的可用性、算法的一些改进以及使用图形处理单元(GPU)等专用处理器提高计算能力的可用性。谷歌、微软、亚马逊和Facebook等公司，以及许多规模较小的工程公司和市场分析公司都在拥抱这项新技术。 然而，许多应用程序并不拥有大量可用来训练复杂模型的示例。在这里提出的研究计划中，我试图开发适用于数据有限的应用的方法，这包括各种情况，如少量测量、有偏的样本或来自外围数据的样本。数据有限的应用程序很常见。例如，在海洋科学界，与浩瀚的海洋相比，测量范围相当小是很常见的。我的实验室已经开始研究人工数据如何增强预训练，并帮助缩小搜索空间。例如，存在各种海洋动力学的物理模型，例如与大气的二氧化碳交换。虽然这些模型通常被认为不足以在所需的规模上进行预测，但这些模拟可能会为深度网络的预训练提供数据。 另一个有望将训练深度网络的技术提升到一个全新水平的领域是领域专家对学习过程的指导，也称为迁移学习。因此，领域专家包括在互补或类似领域接受过训练的深度神经网络，甚至包括人类专家。虽然迁移学习已经被考虑了一段时间，但利用深度网络学习适当的学习交流渠道为这一领域的许多进展开辟了令人振奋的新可能性。这种专家培训将提供类似于人类学习的网络培训。这样的技术可以进一步大幅减少对大型训练集的需求。 该研究计划的长期目标是开发具体的方法和工具来评估机器学习问题的复杂性和数据需求，并提供一个全面的工具箱，将预学习和专家迁移学习策略应用于数据有限的问题领域。

Deep neural networks is a machine learning technique based on a model of neuron-like elements. The "deep" refers here to networks with many layers that are capable of representing complex functions. Such models have been proven to be extremely valuable as they can learn hierarchical data representations from examples and predict previously unseen data. Such techniques enabled much of the recent progress in computer vision and pattern recognition, and the advancements of deep neural networks was made possible by the availability of large data sets (Big Data), some refinement of algorithms, and the availability of increased computational power with specialized processors such as graphical processing units (GPUs). Companies like Google, Microsoft, Amazon and Facebook, and also numerous smaller engineering firms and market analysis firms are embracing this new technology. However, many applications do not have the luxury of having a large set of examples available to train complex models. In the research program proposed here I am trying to develop methods for applications with limited data, which includes a variety of situations such as small number of measurements, biased examples, or examples from peripheral data. Applications with limited data are common. For example, in the ocean science community it is common to have a fairly small coverage of measurements compared to the vastness of the oceans. My lab has started to investigate how artificial data can augment pre-training and facilitate the narrowing of the search space. For example, there exist a variety of physical models of ocean dynamics such as the exchange of CO2 with the atmosphere. While these models are generally considered to be insufficient for predictions on the required scale, it is possible that these simulations can provide data for the pre-training of deep networks. Another area that promises a whole new level of techniques to training deep networks is the guidance of the learning process by domain experts, also called transfer learning. Domain experts include hereby either deep neural networks that have been trained on complementary or similar domains, or even human experts. While transfer learning has been considered for some time, the use of deep networks to learn appropriate communication channels for learning opened exciting new possibilities for much progress in this area. Such expert training would provide network training that parallels human learning. Such techniques could further drastically reduce the need of large training sets. The long-term goal of this research program is to develop specific methods and tools to both evaluate a machine-learning problem in terms of its complexity and data need, and to provide a comprehensive toolbox to apply pre-learning and expert transfer learning strategies to problem domains that suffer from limited data.