Integration of Heterogeneous Data in Artificial Neural Networks for Image Classification

我们的研究计划的目标是开发机器学习的新方法，这是人工智能的一种，用不同类型的数据组合来执行决策，我们称之为多模式数据。更具体地说，我们将重点放在大型和复杂的图像上，并将开发方法来整合其他类型的信息，以便能够将图像分类或分类为信息性结果。例如，一个人的医学扫描，如磁共振成像(MRI)，可以结合他或她的病史来预测结果(例如，是否有新的症状)。机器学习对这类任务非常有帮助，因为计算机可以浏览非常庞大的信息数据库，比人类更快地识别模式，找到更微妙的联系，让用户更专注于定义和使用结果。然而，机器学习中的一个悬而未决的问题是如何将彼此差异很大的多种类型的数据(即所谓的异构型多模式数据)组合在一起，以便做出最准确的预测。回答这个问题很重要，因为对于给定的预测任务，我们并不总是提前知道哪种数据类型的组合最有用。近年来，一种称为深度学习的机器学习方法被证明在大型图像数据库中提取模式是非常有效的，我们已经开发了一些新的深度学习方法来从大脑磁共振成像中学习模式，这些图像是大型复杂图像。我们研究的下一步是研究如何将自动提取的图像模式与其他类型的数据结合起来，以便计算机能够做出最准确的预测。我们的数据将由大量神经疾病患者的磁共振成像和他们的非成像数据组成，如生物测量、人口统计信息和临床病史。预测结果将是某种形式的重要临床类别，如在一段时间内恶化、稳定或改善。我们将在机器学习文献中最近提出的一些想法的基础上进行构建，但我们预计将不得不做许多新的工作，因为这些想法是新的，而且大多数现有的工作都应用于更小的图像和数据类型之间的关系更清晰的领域。在接下来的几年里，我们预计多模式数据将继续快速扩散，融合了良好集成策略的机器学习方法将开始显示出它们的优势。我们相信，该研究计划产生的方法和知识将对数据科学的这一基本需求产生积极影响。**

The goal of our research program is to develop new methods in machine learning, which is a type of artificial intelligence, to perform decision-making with combinations of different types of data, which we call multi-modal data. More specifically, we focus on large and complex images and will develop methods to integrate other types of information to enable categorization or classification of the images into informative outcomes. For example, a person's medical scans, such as magnetic resonance images (MRIs), and his or her medical history can be used in combination to predict an outcome (e.g., new symptoms or not). Machine learning is very helpful for such tasks, because computers can look through very large databases of information to identify patterns much faster and find more subtle links than humans can, allowing the user to focus more on defining and using the outcomes. However, one of the open questions in machine learning is how to combine multiple types of data that are very different from each other (so called heterogeneous multi-modal data) so that we can make the most accurate predictions. Answering this question is important because we do not always know ahead of time which combination of data types is the most useful for a given prediction task. In recent years, an approach of machine learning called deep learning has been shown to be very effective in extracting patterns in large image databases, and we have developed a number of new deep learning methods for learning patterns from brain MRIs, which are large complex images. The next step in our research is to investigate how to combine automatically extracted image patterns with other types of data, so that the computer can make the most accurate predictions. Our data will consist of large sets of MRIs of persons with neurological disorders and their non-imaging data such as biological measurements, demographic information, and clinical histories. The prediction outcomes will be some form of important clinical category, such as worsening, stable, or improving over a period of time. We will build on some recently proposed ideas in the machine learning literature, but we expect that much new work will have to be done, because of the newness of the ideas, and because most of the existing work is applied to much smaller images and in domains where relationships between data types are much clearer. Over the next several years, we expect a continued rapid proliferation of multi-modal data, and the machine learning methods that incorporate good integration strategies will begin to show their advantage. We are confident that the methods and knowledge produced by this research program will positively impact this fundamental need in data science.**