Unsupervised Neural Text Generation by Stochastic Searching

点击翻译按钮获取中文摘要

Natural language generation (NLG) is an important field of artificial intelligence. NLG aims to synthesize natural language text (e.g., sentences) in a variety of tasks, including text summarization, paraphrase generation, and dialogue systems. State-of-the-art NLG systems are based on deep neural networks, which typically synthesize a sentence by predicting one word at a time in an autoregressive fashion. Such approaches significantly outperform traditional rule/template-based NLG in terms of expressiveness and naturalness. However, existing neural NLG has two major drawbacks: 1) Neural networks are usually data-hungry. For example, millions of pairs of parallel sentences are required to train a neural translation system. 2) These methods suffer from the "error accumulation" problem, i.e., the quality of text could drop drastically as the generation proceeds, due to the autoregressive nature (e.g., left-to-right generation). The long-term goal of this proposed research is to investigate unsupervised approaches to text generation. In our previous studies, we tackled this problem by sampling from the probabilistic continuous latent space of a variational autoencoder. More recently, we proposed a novel Metropolis-Hastings (MH) sampler that directly samples a sentence from the discrete word space. In this way, text generation would be more data-efficient, and could be easily adapted to various real-world applications. Based on our previous work, this proposed research program would systematically explore unsupervised text generation by stochastic search, with the following short-term goals: 1) Development of stochastic searching algorithms. Despite our MH sampler, I plan to explore stochastic search algorithms, such as simulated annealing and genetic algorithms, because most NLG tasks are better formulated as a discrete optimization problem than sampling. Here, we would also design searching operations (e.g., word/phrase editing) suitable for text generation. They perform edits in a distributed way over the entire sentence, so our approach does not suffer from the "error accumulation" problem. 2) Applications of unsupervised text generation. Our searching framework provides a flexible way of text generation, because we can easily manipulate the searching objective function and also because such approach does not require parallel data for training. I plan to address a few important generation tasks in NLP, including text summarization, sentence simplification, and style-transfer text generation. 3) Combining searching and learning for text generation. I would like to integrate our search algorithms into a learnable model. On the one hand, a parametric learning model could not only smooth the manually defined searching objective, but also improve inference efficiency for sentence generation. On the other hand, the search procedure could also help train a learning machine, especially in the reinforcement learning setting.