A study on multi-modal man-machine interface through spontaneous speech

点击翻译按钮获取中文摘要

We developed a malti-modal dialogue system that is composed of 4 parts : input by speech recognizer and touch screen, graphical user interface, natural language interpreter, and response generater.Our speech recognizer intergrates the acoustic process with linguistic process directly without the phrase or word lattice. Furthermore, the recognizer processes interjections and restarts based on an unknown word processing technique.A context free grammar is made to be able to accept sentences with omitted post-positions and inversion of word in order to recognize spontaneous speech.Although our spontaneous speech recognizer outputs some errors caused by misrecognition (substitution errors), out of vacabulary (unknown words) and out of grammar (illegal utterances), the language interpreter can understand the meaning of errorful/illegal utterances.The input by touch screen is used to designate the location of map on the display or to select the desired item form the menu which consists of the set of items responded by a speech synthesizer. We use boht the display output (map and menu) and speech systhesis for the response. User can use the postioning / selecting input and speech input at the same time. On man-machine communication, user wants to know his or machine situation what information he gets from the dialogue or how machine interprets / understands his utterances, as well as the speech recognition result. Therefore our system displays the history of dialogue. This function helps to eliminate the user uneasiness. Experimental evalution showed that our interpretation mechanism was suitable for understanding the recognition result of spontaneous speech. And we found that the multi-modal interface with spontaneous speech and touch screen was user-friendly.