NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristics
NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristics
复制标题
DOI：
10.18653/v1/2022.naacl-main.57
复制
发表时间：
2021-12
期刊：
影响因子：
--
通讯作者：
Ximing Lu;S. Welleck;Peter West;Liwei Jiang;Jungo Kasai;Daniel Khashabi;Ronan Le Bras;Lianhui Qin;Youngjae Yu;Rowan Zellers;Noah A. Smith;Yejin Choi
中科院分区：
文献类型：
--
作者：
Ximing Lu;S. Welleck;Peter West;Liwei Jiang;Jungo Kasai;Daniel Khashabi;Ronan Le Bras;Lianhui Qin;Youngjae Yu;Rowan Zellers;Noah A. Smith;Yejin Choi
神经文本生成的主要范例是从自回归语言模型进行从左到右的解码。然而，复杂词汇约束下的受限或可控生成需要有远见，提前规划可行的未来路径。从 A^* 搜索算法中汲取灵感，我们提出了 NeuroLogic A*esque，一种结合了未来成本启发式估计的解码算法。我们开发了对大规模语言模型有效的前瞻启发法，使我们的方法成为波束搜索和 top-k 采样等常见技术的直接替代品。为了实现约束生成，我们以 NeuroLogic 解码为基础（Lu et al., 2021），将其在合并逻辑约束方面的灵活性与对未来约束满足的 A*esque 估计相结合。我们的方法在五代任务上优于竞争基准，并在表到文本生成、受限机器翻译和关键字约束生成方面实现了新的最先进的性能。对于需要复杂约束满足或在少样本或零样本设置中的任务，这些改进尤其显着。 NeuroLogic A*esque 展示了解码在改进和实现大规模语言模型新功能方面的力量。
The dominant paradigm for neural text generation is left-to-right decoding from autoregressive language models. Constrained or controllable generation under complex lexical constraints, however, requires foresight to plan ahead feasible future paths. Drawing inspiration from the A^* search algorithm, we propose NeuroLogic A*esque, a decoding algorithm that incorporates heuristic estimates of future cost. We develop lookahead heuristics that are efficient for large-scale language models, making our method a drop-in replacement for common techniques such as beam search and top-k sampling. To enable constrained generation, we build on NeuroLogic decoding (Lu et al., 2021), combining its flexibility in incorporating logical constraints with A*esque estimates of future constraint satisfaction. Our approach outperforms competitive baselines on five generation tasks, and achieves new state-of-the-art performance on table-to-text generation, constrained machine translation, and keyword-constrained generation. The improvements are particularly notable on tasks that require complex constraint satisfaction or in few-shot or zero-shot settings. NeuroLogic A*esque illustrates the power of decoding for improving and enabling new capabilities of large-scale language models.