A study on multi-modal man-machine interface through spontaneous speech

我们开发了一个由4部分组成的多模态对话系统：语音识别器和触摸屏输入、图形用户界面、自然语言解释器和响应生成器。我们的语音识别器将声学过程与语言过程直接结合在一起，而不需要短语或词格。此外，识别器处理基于未知字处理技术的插话和重启。上下文无关语法是为了能够接受省略后位和倒装的句子，以识别自发语音。虽然我们的自发语音识别器输出了一些由于误认（代换错误）、缺词汇（不认识的单词）和缺语法（不合法的话语）而导致的错误，但语言口译员可以理解错误/非法话语的含义。触摸屏输入用于指定地图在显示器上的位置或从由语音合成器响应的一组项目组成的菜单中选择所需的项目。我们使用显示输出（地图和菜单）和语音合成来响应。用户可以同时使用定位/选择输入和语音输入。在人机交流中，用户想知道他或机器的情况，他从对话中得到了什么信息，或者机器如何解释/理解他的话语，以及语音识别的结果。因此，我们的系统显示对话的历史。这个功能有助于消除用户的不安。实验结果表明，我们的解释机制适合于理解自发语音的识别结果。我们发现带有自发语音和触摸屏的多模态界面是用户友好的。

We developed a malti-modal dialogue system that is composed of 4 parts : input by speech recognizer and touch screen, graphical user interface, natural language interpreter, and response generater.Our speech recognizer intergrates the acoustic process with linguistic process directly without the phrase or word lattice. Furthermore, the recognizer processes interjections and restarts based on an unknown word processing technique.A context free grammar is made to be able to accept sentences with omitted post-positions and inversion of word in order to recognize spontaneous speech.Although our spontaneous speech recognizer outputs some errors caused by misrecognition (substitution errors), out of vacabulary (unknown words) and out of grammar (illegal utterances), the language interpreter can understand the meaning of errorful/illegal utterances.The input by touch screen is used to designate the location of map on the display or to select the desired item form the menu which consists of the set of items responded by a speech synthesizer. We use boht the display output (map and menu) and speech systhesis for the response. User can use the postioning / selecting input and speech input at the same time. On man-machine communication, user wants to know his or machine situation what information he gets from the dialogue or how machine interprets / understands his utterances, as well as the speech recognition result. Therefore our system displays the history of dialogue. This function helps to eliminate the user uneasiness. Experimental evalution showed that our interpretation mechanism was suitable for understanding the recognition result of spontaneous speech. And we found that the multi-modal interface with spontaneous speech and touch screen was user-friendly.