Uncertainty Quantification for Text Classification
Uncertainty Quantification for Text Classification
复制标题
DOI：
10.1145/3539618.3594243
复制
发表时间：
2023-07
期刊：
影响因子：
--
通讯作者：
Dell Zhang;Murat Sensoy;M. Makrehchi;Bilyana Taneva-Popova;Lin Gui;Yulan He
中科院分区：
文献类型：
--
作者：
Dell Zhang;Murat Sensoy;M. Makrehchi;Bilyana Taneva-Popova;Lin Gui;Yulan He
这个全天教程介绍了实用不确定性量化的现代技术，特别是在多类和多标签文本分类的背景下。首先，我们解释估计任意不确定性和认知不确定性对于文本分类模型的有用性。然后，我们描述了几种最先进的不确定性量化方法，并分析了它们对大文本数据的可扩展性：GBDT中的虚拟集成、贝叶斯深度学习（包括深度集成、蒙特卡洛辍学、反向传播贝叶斯及其泛化认知神经网络）、证据深度学习（包括先验网络和后验网络）以及距离感知（包括谱归一化神经高斯过程和深度确定性网络）不确定性）。接下来，我们讨论预训练语言模型不确定性量化的最新进展（包括要求语言模型表达其不确定性、解释基于大规模语言模型的文本分类器的不确定性、文本生成中的不确定性估计、语言模型的校准以及上下文学习的校准）。之后，我们讨论不确定性量化在文本分类中的典型应用场景（包括域内校准、跨域鲁棒性和新类检测）。最后，我们列出了用于评估文本分类中不确定性量化有效性的流行性能指标。为与会者提供了实用的动手示例/练习，让他们在一些现实世界的文本分类数据集（例如 CLINC150）上尝试不同的不确定性量化方法。
This full-day tutorial introduces modern techniques for practical uncertainty quantification specifically in the context of multi-class and multi-label text classification. First, we explain the usefulness of estimating aleatoric uncertainty and epistemic uncertainty for text classification models. Then, we describe several state-of-the-art approaches to uncertainty quantification and analyze their scalability to big text data: Virtual Ensemble in GBDT, Bayesian Deep Learning (including Deep Ensemble, Monte-Carlo Dropout, Bayes by Backprop, and their generalization Epistemic Neural Networks), Evidential Deep Learning (including Prior Networks and Posterior Networks), as well as Distance Awareness (including Spectral-normalized Neural Gaussian Process and Deep Deterministic Uncertainty). Next, we talk about the latest advances in uncertainty quantification for pre-trained language models (including asking language models to express their uncertainty, interpreting uncertainties of text classifiers built on large-scale language models, uncertainty estimation in text generation, calibration of language models, and calibration for in-context learning). After that, we discuss typical application scenarios of uncertainty quantification in text classification (including in-domain calibration, cross-domain robustness, and novel class detection). Finally, we list popular performance metrics for the evaluation of uncertainty quantification effectiveness in text classification. Practical hands-on examples/exercises are provided to the attendees for them to experiment with different uncertainty quantification methods on a few real-world text classification datasets such as CLINC150.