Robust Performance Evaluation of POMDP-Based Dialogue Systems
Robust Performance Evaluation of POMDP-Based Dialogue Systems
复制标题
DOI：
10.1109/tasl.2010.2076394
复制
发表时间：
2011-05
期刊：
影响因子：
--
通讯作者：
Dongho Kim;J. H. Kim;Kee-Eung Kim
中科院分区：
文献类型：
--
作者：
Dongho Kim;J. H. Kim;Kee-Eung Kim
部分可观察马尔可夫决策过程（pomdp）由于其在不可靠自动语音识别下自然建模对话策略选择问题的能力，在口语对话系统的研究中引起了极大的兴趣。然而，POMDP方法本质上是基于模型的，因此，从POMDP计算的对话策略仍然受制于模型的正确性。在本文中，我们将之前的一些MDP用户模型扩展到pomdp，并评估了用户模型对pomdp计算的对话策略的影响。我们的实验表明，从pomdp计算的策略比从mdp计算的策略表现得更好，并且在不同的用户模型上测试时，从较差的用户模型计算的策略严重失败。本文进一步研究了对话策略的评价方法，提出了一种基于偏方差分析的可靠评价对话性能的方法。
Partially observable Markov decision processes (POMDPs) have received significant interest in research on spoken dialogue systems, due to among many benefits its ability to naturally model the dialogue strategy selection problem under unreliable automated speech recognition. However, the POMDP approaches are essentially model-based, and as a result, the dialogue strategy computed from POMDP is still subject to the correctness of the model. In this paper, we extend some of the previous MDP user models to POMDPs, and evaluate the effects of user models on the dialogue strategy computed from POMDPs. We experimentally show that the strategies computed from POMDPs perform better than those from MDPs, and the strategies computed from poor user models fail severely when tested on different user models. This paper further investigates the evaluation methods for dialogue strategies, and proposes a method based on the bias-variance analysis for reliably estimating the dialogue performance.