Bi-LSTM Model to Increase Accuracy in Text Classification: Combining Word2vec CNN and Attention Mechanism
Bi-LSTM Model to Increase Accuracy in Text Classification: Combining Word2vec CNN and Attention Mechanism
复制标题
DOI：
10.3390/app10175841
复制
发表时间：
2020-09-01
影响因子：
2.7
通讯作者：
Kim, Jong Wook
中科院分区：
文献类型：
--
作者：
Jang, Beakcheol;Kim, Myeonghwi;Kim, Jong Wook
需要从大数据中提取有意义的信息，将其分类为不同的类别，并预测最终用户的行为或情绪。大量数据来自各种来源，如社交媒体和网站。文本分类是自然语言处理领域的一个代表性研究课题，它将非结构化文本数据分类为有意义的类别。用于句子分类的长短期记忆（LSTM）模型和卷积神经网络产生了准确的结果，最近已用于各种自然语言处理（NLP）任务。卷积神经网络（CNN）模型使用卷积层和最大池化或最大超时池化层来提取更高级别的特征，而LSTM模型可以捕获单词序列之间的长期依赖关系，因此更好地用于文本分类。然而，即使使用利用这两种深度学习模型的混合方法，要记住的分类特征数量仍然很大，因此阻碍了训练过程。在这项研究中，我们提出了一个基于注意力的Bi-LSTM+CNN混合模型，它利用了LSTM和CNN的优势，并增加了一个额外的注意力机制。我们使用互联网电影数据库（IMDB）电影评论数据训练模型来评估所提出的模型的性能，测试结果表明，所提出的混合注意力Bi-LSTM+CNN模型产生更准确的分类结果，以及更高的召回率和F1分数，比单独的多层感知器（MLP），CNN或LSTM模型以及混合模型。
There is a need to extract meaningful information from big data, classify it into different categories, and predict end-user behavior or emotions. Large amounts of data are generated from various sources such as social media and websites. Text classification is a representative research topic in the field of natural-language processing that categorizes unstructured text data into meaningful categorical classes. The long short-term memory (LSTM) model and the convolutional neural network for sentence classification produce accurate results and have been recently used in various natural-language processing (NLP) tasks. Convolutional neural network (CNN) models use convolutional layers and maximum pooling or max-overtime pooling layers to extract higher-level features, while LSTM models can capture long-term dependencies between word sequences hence are better used for text classification. However, even with the hybrid approach that leverages the powers of these two deep-learning models, the number of features to remember for classification remains huge, hence hindering the training process. In this study, we propose an attention-based Bi-LSTM+CNN hybrid model that capitalize on the advantages of LSTM and CNN with an additional attention mechanism. We trained the model using the Internet Movie Database (IMDB) movie review data to evaluate the performance of the proposed model, and the test results showed that the proposed hybrid attention Bi-LSTM+CNN model produces more accurate classification results, as well as higher recall and F1 scores, than individual multi-layer perceptron (MLP), CNN or LSTM models as well as the hybrid models.