From trivial representations to learning concepts in AI by exploiting unique data

一场基于人工智能的革命及其社会经济效益的前景是诱人的。我们希望生活在一个人工智能以高性能和最低风险有效学习的世界里。这样的世界是非常令人兴奋的。我们倾向于认为，人工智能从数据中学习更高级别的概念，但事实并非如此。特别是在图像等数据中，即使提供了数百万个示例，人工智能也会从数据中提取相当琐碎的(低级别)概念。我们经常听说，提供更多具有高度多样性的数据应该有助于提高人工智能可以提取的信息。不过，这种数据积累确实会对隐私和成本产生影响。事实上，还需要对数据进行预处理和清理(即删除不需要的信息)，这也带来了相当大的成本。然而，更关键的是，在几个关键应用程序(例如医疗保健)中，某些事件(例如疾病)可能是罕见的或真正独特的。收集越来越多的数据不会改变这种稀有数据的相对频率。目前的人工智能似乎不能有效利用数据：它不能很好地利用存在于唯一和稀有数据中的信息金矿。这个项目旨在回答一个关键的研究问题：**为什么人工智能与概念斗争，唯一数据的作用是什么？**我们怀疑人工智能与概念斗争的原因有几个：a)我们用来从数据中提取信息的机制(称为表征学习)依赖于非常简单的假设，这些假设不能反映真实数据在世界上的存在方式。例如，我们知道数据是有相关性的，而我们现在做了一个简化的假设，即根本没有相关性。我们建议在我们想要提取的概念中引入更强的因果关系假设。这反过来应该有助于我们提取更好的信息。B)为了学习任何模型，我们确实必须使用优化过程来找到模型的参数。我们在这些过程中发现了一个弱点：独特和稀有的数据没有得到如此多的关注，或者即使它们确实得到了一些关注，也是偶然发生的。这导致了在信息提取方面的相当大的不一致。此外，有时会提取错误的信息，要么是因为我们找到了次优的表示形式，要么是因为我们抓住了一些从消毒过程中逃脱的数据--因为不能总是保证这样完美的过程。我们想了解为什么会存在这种不一致性，并建议设计方法，以确保当我们训练模型时，即使从稀有数据中也能一致地提取信息。B和A之间有着紧密的联系。如果没有新的方法来更好地优化学习函数，我们就不能从稀有数据中可靠地提取表示，因此我们不能强加我们所需的因果关系。关于这项工作还有一个额外的元素，它有助于回答问题的第二部分。稀有而独特的数据可能实际上揭示了独特的因果关系。这是一个非常诱人的前景，我们提出的工作旨在调查。我们提出的工作有相当大和广泛的回报。我们在这里为人工智能奠定了基础，因为它是数据高效的，不应该要求盲目积累数据，而这会给普通公众带来所有隐私方面的担忧。因为它学习高层次的概念，所以它将更熟练地授权决策工具，以支持决策是如何达成的。由于我们在提取这些概念时引入了强大的因果先验，因此我们降低了学习琐碎数据关联的风险。总体而言，人工智能研究社区的一个主要目标是创造出一种人工智能，它可以概括出训练时间内可用的新的看不见的数据。我们希望我们的人工智能将使我们更接近这一目标，从而进一步为人工智能在现实世界中更广泛的部署铺平道路。

The prospect of an AI-based revolution and its socio-economic benefits is tantalising. We want to live in a world where AI learns effectively with high performance and minimal risks. Such a world is extremely exciting. We tend to believe that AI learns higher level concepts from data, but this is not what happens. Particularly in data such as images, AI extracts rather trivial (low-level) notions from the data even when provided with millions of examples. We often hear that providing more data with high diversity should help improve the information that AI can extract. This data amassing does have though privacy and cost implications. Indeed, considerable cost comes also by the need to pre-process and to sanitise data (i.e. remove unwanted information). More critically, though, in several key applications (e.g. healthcare) some events (e.g. disease) can be rare or truly unique. Collecting more and more data will not change the relative frequency of such rare data. It appears that current AI is not data efficient: it poorly leverages the goldmine of information present in unique and rare data.This project aims to answer a key research question: **Why does AI struggle with concepts, and what is the role of unique data? **We suspect there are several reasons why AI struggles with concepts: A) The mechanisms we use to extract information from data (known as representation learning) rely on very simple assumptions that do not reflect how real data exist in the world. For example, we know that data have correlations, and we now make simplified assumptions of no correlation at all. We propose to introduce stronger assumptions of causal relationships in the concepts we want to extract. This should in turn help us extract better information. B) To learn any model, we do have to use optimisation processes to find the parameters of the model. We find a weakness in these processes: data that are unique and rare do not get so much attention, or if they do get some, it happens by chance. This leads to considerable inconsistency in the extraction of information. In addition, sometimes wrong information is extracted, either because we found suboptimal representations or because we latched on some data that escaped from the sanitisation process -since no such perfect process can always be guaranteed. We want to understand why such inconsistency exists and propose to devise methods that can ensure that when we train models, we can consistently extract information even from rare data.There is a tight connection between B and A. Without new methods that better optimise learning functions we cannot extract representations reliably from rare data, and hence we cannot impose the causal relationships we need. There is an additional element about this work that helps answer the second part of the question. Rare and unique data may actually reveal unique causal relationships. This is a very tantalising prospect that the work we propose aims to investigate. There are considerable and broad rewards of the work we propose. We put herein the underpinnings for an AI that, because it is data efficient, should not require blind amassing of data with all the privacy fears this engenders for the general public. Because it learns high-lever concepts it will be more adept to empower decision tools that can support how decisions have been reached. And because we introduce strong causal priors in extracting these concepts, we reduce the risk of learning trivial data associations. Overall, a major goal of the AI research community is to create AI that can generalise to new unseen data beyond what was available during training time. We hope that our AI will bring us closer to this goal, thus further paving the way to broader deployment of AI to the real world.