Integration of Heterogeneous Data in Artificial Neural Networks for Image Classification

我们研究项目的目标是开发机器学习的新方法，这是一种人工智能，通过不同类型数据的组合来执行决策，我们称之为多模态数据。更具体地说，我们专注于大型和复杂的图像，并将开发方法来整合其他类型的信息，以使图像的分类或分类成为信息结果。例如，一个人的医学扫描，如磁共振图像（mri）和他或她的病史可以结合使用来预测结果（例如，是否有新症状）。机器学习对于这类任务非常有帮助，因为计算机可以通过非常大的信息数据库来识别模式，并找到比人类更快的微妙联系，从而使用户能够更多地专注于定义和使用结果。然而，机器学习中一个悬而未决的问题是如何结合彼此非常不同的多种类型的数据（所谓的异构多模态数据），以便我们可以做出最准确的预测。回答这个问题很重要，因为我们并不总是提前知道哪种数据类型的组合对给定的预测任务最有用。近年来，一种称为深度学习的机器学习方法已被证明在从大型图像数据库中提取模式方面非常有效，我们已经开发了许多新的深度学习方法，用于从大脑核磁共振成像中学习模式，这是大型复杂图像。我们研究的下一步是研究如何将自动提取的图像模式与其他类型的数据结合起来，使计算机能够做出最准确的预测。我们的数据将包括大量神经系统疾病患者的核磁共振成像和他们的非成像数据，如生物测量、人口统计信息和临床病史。预测结果将是某种形式的重要临床类别，如在一段时间内恶化、稳定或改善。我们将以机器学习文献中最近提出的一些想法为基础，但我们预计将有很多新的工作要做，因为这些想法是新颖的，而且因为大多数现有的工作都应用于更小的图像和数据类型之间关系更清晰的领域。在接下来的几年里，我们预计多模态数据将持续快速增长，结合良好集成策略的机器学习方法将开始显示其优势。我们相信，该研究项目产生的方法和知识将对数据科学的这一基本需求产生积极影响

The goal of our research program is to develop new methods in machine learning, which is a type of artificial intelligence, to perform decision-making with combinations of different types of data, which we call multi-modal data. More specifically, we focus on large and complex images and will develop methods to integrate other types of information to enable categorization or classification of the images into informative outcomes. For example, a person's medical scans, such as magnetic resonance images (MRIs), and his or her medical history can be used in combination to predict an outcome (e.g., new symptoms or not). Machine learning is very helpful for such tasks, because computers can look through very large databases of information to identify patterns much faster and find more subtle links than humans can, allowing the user to focus more on defining and using the outcomes. However, one of the open questions in machine learning is how to combine multiple types of data that are very different from each other (so called heterogeneous multi-modal data) so that we can make the most accurate predictions. Answering this question is important because we do not always know ahead of time which combination of data types is the most useful for a given prediction task. In recent years, an approach of machine learning called deep learning has been shown to be very effective in extracting patterns in large image databases, and we have developed a number of new deep learning methods for learning patterns from brain MRIs, which are large complex images. The next step in our research is to investigate how to combine automatically extracted image patterns with other types of data, so that the computer can make the most accurate predictions. Our data will consist of large sets of MRIs of persons with neurological disorders and their non-imaging data such as biological measurements, demographic information, and clinical histories. The prediction outcomes will be some form of important clinical category, such as worsening, stable, or improving over a period of time. We will build on some recently proposed ideas in the machine learning literature, but we expect that much new work will have to be done, because of the newness of the ideas, and because most of the existing work is applied to much smaller images and in domains where relationships between data types are much clearer. Over the next several years, we expect a continued rapid proliferation of multi-modal data, and the machine learning methods that incorporate good integration strategies will begin to show their advantage. We are confident that the methods and knowledge produced by this research program will positively impact this fundamental need in data science.**