Conformal prediction for text infilling and part-of-speech prediction
Conformal prediction for text infilling and part-of-speech prediction
复制标题
DOI：
10.51387/22-nejsds8
复制
发表时间：
2021-11
期刊：
影响因子：
--
通讯作者：
N. Dey;Jing Ding;Jack G. Ferrell;Carolina Kapper;Maxwell Lovig;Emiliano Planchon;Jonathan P. Williams
中科院分区：
文献类型：
--
作者：
N. Dey;Jing Ding;Jack G. Ferrell;Carolina Kapper;Maxwell Lovig;Emiliano Planchon;Jonathan P. Williams
现代机器学习算法能够给出极为精确的点预测；然而，其统计可靠性仍存疑问。与传统机器学习方法不同，保形预测算法会返回与给定显著性水平相对应的置信集（即集合值预测）。此外，这些置信集是有效的，因为它们保证了对一类错误概率的有限样本控制，使从业者能够选择可接受的错误率。在我们的论文中，针对自然语言数据的文本填充和词性（POS）预测任务，我们提出了归纳保形预测（ICP）算法。我们基于BERT（来自Transformer的双向编码器表征）和BiLSTM（双向长短期记忆）模型，构建了用于词性标注的新型ICP增强算法。对于文本填充，我们设计了一种新的ICP增强BERT算法。我们使用包含57000多个句子的布朗语料库，在模拟中分析了这些算法的性能。我们的结果表明，ICP算法能够生成有效的集合值预测，其规模足够小，可应用于实际场景。我们还给出了一个真实数据示例，说明我们提出的集合值预测如何改进机器生成的音频转录。
Modern machine learning algorithms are capable of providing remarkably accurate point-predictions; however, questions remain about their statistical reliability. Unlike conventional machine learning methods, conformal prediction algorithms return confidence sets (i.e., set-valued predictions) that correspond to a given significance level. Moreover, these confidence sets are valid in the sense that they guarantee finite sample control over type 1 error probabilities, allowing the practitioner to choose an acceptable error rate. In our paper, we propose inductive conformal prediction (ICP) algorithms for the tasks of text infilling and part-of-speech (POS) prediction for natural language data. We construct new ICP-enhanced algorithms for POS tagging based on BERT (bidirectional encoder representations from transformers) and BiLSTM (bidirectional long short-term memory) models. For text infilling, we design a new ICP-enhanced BERT algorithm. We analyze the performance of the algorithms in simulations using the Brown Corpus, which contains over 57,000 sentences. Our results demonstrate that the ICP algorithms are able to produce valid set-valued predictions that are small enough to be applicable in real-world applications. We also provide a real data example for how our proposed set-valued predictions can improve machine generated audio transcriptions.