Towards Open-world Semi-supervised learning

点击翻译按钮获取中文摘要

Large amounts of fully annotated data is one of the major components responsible for the success of current deep-learning models. However, in many application scenarios, this assumption of having an extensive collection of human provided annotations is not realistic, as gathering those annotations can be costly and require expert knowledge. Additionally in real-world settings, new concepts and categories may emerge over time. Thus it is often not feasible to gather human annotations for every possible concept. This project will focus on the question of how to build autonomous agents that can automatically reason about novel categories for which no human supervision is provided at training time.Current semi-supervised learning methods are limited in that they require all categories to have human annotations for at least one example. As a result it can be hard to directly adopt previous semi-supervised learning methods in this open-world regime. Recently a new area called Novel Category Discovery (NCD) has emerged, which is closely related to this project, which focuses on the problem of how to discover novel categories within a large unlabelled dataset by leveraging knowledge from an existing labelled dataset. However, NCD methods assume that all the categories in the unlabelled set are novel, and only focus on the performance of those novel categories.In a real-world scenario, it is of equal importance to be able to recognise both previously seen categories as well as discover novel ones. In this work, we aim to develop algorithms and solutions for open-world semi-supervised learning. The developed algorithms will not only be able to recognise previously seen categories using labelled and unlabelled data, but also be able to discover novel categories from unlabelled data. Upon successful development of these new methods, we also aim to design interpretable methods that can convey 'why' the model thinks a set of examples are novel and `how' a newly discovered novel category differs from previously seen ones. These proposed interpretable methods will give new insight into large unlabeled image collections and will advance current deep-learning approaches into a more realistic open-world setting. In this project, we will focus on the following goals: (1) Classifier learning: Learning to classify both seen and novel categories using labelled and unlabelled images.(2) Category number estimation: The category number estimation algorithm should be able to run efficiently and be performed simultaneously with the classifier learning process.(3) Interpreting discovered categories: Interpretable model outputs so that the user knows why some images are predicted to form a novel category.To design novel methods for this new open-world semi-supervised learning setting, we aim to draw inspiration from classic unsupervised clustering methods and cutting-edge deep learning methods:(1) Hierarchical clustering, these methods have the advantage of automatically merging data points to adaptively form a hierarchy of categories, by learning a similarity measuring function using neural networks. We can leverage it to perform hierarchical clustering and to be able to learn the classifier and estimate the number of novel categories at the same time.(2) Leveraging state-of-the-art visual-language models, we also aim to use recent advancements in visual-language learning to design methods for automatically generating a textual explanation for interpreting 'why' the model thinks a set of examples are novel, and 'how' the novel categories differ from the 'seen' categories from the human labelled dataset.We will apply these methods to real-world tasks to better understand the practical challenges present which will inform the development of more robust models.