Integration of Heterogeneous Data in Artificial Neural Networks for Image Classification

我们研究项目的目标是开发机器学习的新方法，这是一种人工智能，用不同类型的数据组合来执行决策，我们称之为多模态数据。更具体地说，我们专注于大型和复杂的图像，并将开发方法来整合其他类型的信息，以使图像分类或分类成为信息丰富的结果。例如，一个人的医学扫描（诸如磁共振图像（MRI））和他或她的病史可以组合使用以预测结果（例如，新的症状或没有）。机器学习对这类任务非常有帮助，因为计算机可以查看非常大的信息数据库，以比人类更快的速度识别模式，并找到更微妙的联系，使用户能够更多地关注定义和使用结果。然而，机器学习中的一个悬而未决的问题是如何组合多种类型的数据，这些数据彼此非常不同（所谓的异构多模态数据），以便我们可以做出最准确的预测。提出这个问题很重要，因为我们并不总是提前知道哪种数据类型的组合对给定的预测任务最有用。近年来，一种名为深度学习的机器学习方法已被证明在提取大型图像数据库中的模式方面非常有效，我们已经开发了许多新的深度学习方法，用于从大脑MRI（大型复杂图像）中学习模式。我们研究的下一步是研究如何将联合收割机自动提取的图像模式与其他类型的数据相结合，以便计算机能够做出最准确的预测。我们的数据将包括大量神经系统疾病患者的MRI及其非成像数据，如生物测量，人口统计学信息和临床病史。预测结果将是某种形式的重要临床类别，例如在一段时间内恶化、稳定或改善。我们将以机器学习文献中最近提出的一些想法为基础，但我们预计还需要做很多新的工作，因为这些想法是新的，而且大多数现有的工作都应用于更小的图像和数据类型之间的关系更加清晰的领域。在接下来的几年里，我们预计多模态数据将继续快速增长，而包含良好集成策略的机器学习方法将开始显示出其优势。我们相信，这项研究计划产生的方法和知识将对数据科学的这一基本需求产生积极影响。

The goal of our research program is to develop new methods in machine learning, which is a type of artificial intelligence, to perform decision-making with combinations of different types of data, which we call multi-modal data. More specifically, we focus on large and complex images and will develop methods to integrate other types of information to enable categorization or classification of the images into informative outcomes. For example, a person's medical scans, such as magnetic resonance images (MRIs), and his or her medical history can be used in combination to predict an outcome (e.g., new symptoms or not). Machine learning is very helpful for such tasks, because computers can look through very large databases of information to identify patterns much faster and find more subtle links than humans can, allowing the user to focus more on defining and using the outcomes. However, one of the open questions in machine learning is how to combine multiple types of data that are very different from each other (so called heterogeneous multi-modal data) so that we can make the most accurate predictions. Answering this question is important because we do not always know ahead of time which combination of data types is the most useful for a given prediction task. In recent years, an approach of machine learning called deep learning has been shown to be very effective in extracting patterns in large image databases, and we have developed a number of new deep learning methods for learning patterns from brain MRIs, which are large complex images. The next step in our research is to investigate how to combine automatically extracted image patterns with other types of data, so that the computer can make the most accurate predictions. Our data will consist of large sets of MRIs of persons with neurological disorders and their non-imaging data such as biological measurements, demographic information, and clinical histories. The prediction outcomes will be some form of important clinical category, such as worsening, stable, or improving over a period of time. We will build on some recently proposed ideas in the machine learning literature, but we expect that much new work will have to be done, because of the newness of the ideas, and because most of the existing work is applied to much smaller images and in domains where relationships between data types are much clearer. Over the next several years, we expect a continued rapid proliferation of multi-modal data, and the machine learning methods that incorporate good integration strategies will begin to show their advantage. We are confident that the methods and knowledge produced by this research program will positively impact this fundamental need in data science.**