Feature-Frequency–Adaptive On-line Training for Fast and Accurate Natural Language Processing
Feature-Frequency–Adaptive On-line Training for Fast and Accurate Natural Language Processing
复制标题
DOI：
10.1162/coli_a_00193
复制
发表时间：
2014-09
影响因子：
9.3
通讯作者：
Xu Sun;Wenjie Li;Houfeng Wang;Q. Lu
中科院分区：
文献类型：
--
作者：
Xu Sun;Wenjie Li;Houfeng Wang;Q. Lu
训练速度和准确性是大规模自然语言处理系统的两个主要关注点。通常，我们需要在速度和准确性之间进行权衡。通过牺牲准确性来提高训练速度或通过牺牲速度来提高准确性都是微不足道的。尽管如此，同时提高训练速度和准确性并非易事，这是这项工作的目标。为了实现这一目标，我们提出了一种新的训练方法，即特征频率自适应在线训练，用于快速准确地训练自然语言处理系统。它基于这样的核心思想：频率越高的特征应该具有衰减得越快的学习率。理论分析表明，该方法收敛性好，收敛速度快。实验是基于众所周知的基准任务进行的，包括命名实体识别、分词、短语分块和情感分析。这些任务包括三个结构化分类任务和一个非结构化分类任务，分别具有二进制特征和实值特征。实验结果表明，所提出的方法比现有方法更快，同时更准确，在具有不同特征的任务上取得了最先进的分数。
Training speed and accuracy are two major concerns of large-scale natural language processing systems. Typically, we need to make a tradeoff between speed and accuracy. It is trivial to improve the training speed via sacrificing accuracy or to improve the accuracy via sacrificing speed. Nevertheless, it is nontrivial to improve the training speed and the accuracy at the same time, which is the target of this work. To reach this target, we present a new training method, feature-frequency–adaptive on-line training, for fast and accurate training of natural language processing systems. It is based on the core idea that higher frequency features should have a learning rate that decays faster. Theoretical analysis shows that the proposed method is convergent with a fast convergence rate. Experiments are conducted based on well-known benchmark tasks, including named entity recognition, word segmentation, phrase chunking, and sentiment analysis. These tasks consist of three structured classification tasks and one non-structured classification task, with binary features and real-valued features, respectively. Experimental results demonstrate that the proposed method is faster and at the same time more accurate than existing methods, achieving state-of-the-art scores on the tasks with different characteristics.