Towards Open-world Semi-supervised learning

大量经过充分注释的数据是当前深度学习模型成功的主要组成部分之一。然而，在许多应用场景中，这种拥有大量人工提供的注释集合的假设是不现实的，因为收集这些注释的成本可能很高，并且需要专业知识。此外，在现实世界中，新的概念和类别可能会随着时间的推移而出现。因此，为每个可能的概念收集人工注释通常是不可行的。这个项目将集中在如何建立自主代理的问题上，这些代理可以自动对训练时没有提供人工监督的新类别进行推理。目前的半监督学习方法是有限的，因为它们要求所有类别至少有一个示例具有人工注释。因此，在这个开放世界的制度下，可能很难直接采用以前的半监督学习方法。最近出现了一个新的领域，称为新颖类别发现(NCD)，它与该项目密切相关，该领域的重点是如何利用现有已标记数据集中的知识在大型未标记数据集中发现新类别。然而，NCD方法假设未标注集合中的所有类别都是新的，并且只关注这些新类别的性能，在现实世界中，能够识别之前看到的类别和发现新的类别是同等重要的。在这项工作中，我们的目标是开发开放世界半监督学习的算法和解决方案。开发的算法不仅能够识别使用标记和未标记数据的先前看到的类别，而且还能够从未标记数据中发现新的类别。在成功开发这些新方法后，我们的目标也是设计可解释的方法，这些方法可以传达模型认为一组例子是新颖的原因，以及新发现的新颖类别与以前看到的类别有何不同。这些提出的可解释方法将为大量未标记的图像集合提供新的见解，并将推动当前的深度学习方法进入更现实的开放世界环境。在这个项目中，我们将专注于以下目标：(1)分类器学习：学习使用已标记图像和未标记图像来对所见类别和新类别进行分类。(2)类别数目估计：类别数目估计算法应该能够高效地运行，并且与分类器学习过程同时执行。(3)解释发现类别：可解释的模型输出，以便用户知道为什么某些图像被预测以形成新的类别。为了设计这种新的开放世界的半监督学习环境，我们旨在从经典的无监督聚类方法和前沿的深度学习方法中获得启发：(1)层次聚类，这些方法的优点是，通过使用神经网络学习相似性度量函数，自动合并数据点以自适应地形成类别层次结构。我们可以利用它来执行层次聚类，并能够同时学习分类器和估计新类别的数量。(2)利用最先进的视觉语言模型，我们还旨在设计方法来设计方法，自动生成文本解释，以解释模型认为一组例子是新的，以及新类别与人类标记的数据集中所看到的类别有何不同。我们将这些方法应用于现实世界的任务，以更好地理解存在的实际挑战，这将为开发更健壮的模型提供信息。

Large amounts of fully annotated data is one of the major components responsible for the success of current deep-learning models. However, in many application scenarios, this assumption of having an extensive collection of human provided annotations is not realistic, as gathering those annotations can be costly and require expert knowledge. Additionally in real-world settings, new concepts and categories may emerge over time. Thus it is often not feasible to gather human annotations for every possible concept. This project will focus on the question of how to build autonomous agents that can automatically reason about novel categories for which no human supervision is provided at training time.Current semi-supervised learning methods are limited in that they require all categories to have human annotations for at least one example. As a result it can be hard to directly adopt previous semi-supervised learning methods in this open-world regime. Recently a new area called Novel Category Discovery (NCD) has emerged, which is closely related to this project, which focuses on the problem of how to discover novel categories within a large unlabelled dataset by leveraging knowledge from an existing labelled dataset. However, NCD methods assume that all the categories in the unlabelled set are novel, and only focus on the performance of those novel categories.In a real-world scenario, it is of equal importance to be able to recognise both previously seen categories as well as discover novel ones. In this work, we aim to develop algorithms and solutions for open-world semi-supervised learning. The developed algorithms will not only be able to recognise previously seen categories using labelled and unlabelled data, but also be able to discover novel categories from unlabelled data. Upon successful development of these new methods, we also aim to design interpretable methods that can convey 'why' the model thinks a set of examples are novel and `how' a newly discovered novel category differs from previously seen ones. These proposed interpretable methods will give new insight into large unlabeled image collections and will advance current deep-learning approaches into a more realistic open-world setting. In this project, we will focus on the following goals: (1) Classifier learning: Learning to classify both seen and novel categories using labelled and unlabelled images.(2) Category number estimation: The category number estimation algorithm should be able to run efficiently and be performed simultaneously with the classifier learning process.(3) Interpreting discovered categories: Interpretable model outputs so that the user knows why some images are predicted to form a novel category.To design novel methods for this new open-world semi-supervised learning setting, we aim to draw inspiration from classic unsupervised clustering methods and cutting-edge deep learning methods:(1) Hierarchical clustering, these methods have the advantage of automatically merging data points to adaptively form a hierarchy of categories, by learning a similarity measuring function using neural networks. We can leverage it to perform hierarchical clustering and to be able to learn the classifier and estimate the number of novel categories at the same time.(2) Leveraging state-of-the-art visual-language models, we also aim to use recent advancements in visual-language learning to design methods for automatically generating a textual explanation for interpreting 'why' the model thinks a set of examples are novel, and 'how' the novel categories differ from the 'seen' categories from the human labelled dataset.We will apply these methods to real-world tasks to better understand the practical challenges present which will inform the development of more robust models.