SPEECH RECOGNITION WITH SYNCHRONOUS INPUT OF HAND-WRITTEN GESTURES FOR MOBILE DEVICES

移动设备最近在日常生活中经常使用。人们强烈要求具有高精度的用户友好界面。为此，我们提出了一个使用语音和手写手势同时输入的界面。该接口对环境噪声的抵抗能力比纯语音接口强，输入速度也比纯手写手势接口快。我们的目标应用程序是通过输入句子来制作电子邮件。第一年，我们提出了一个界面，在这个界面中，一个句子通过语音输入，而句子中每个短语头部的“平假名”字符通过手写手势输入。我们实现了一种手写手势的识别算法，设计了一种识别两种模式同时输入的方法。利用独立记录的语音数据和手写手势数据进行仿真实验，验证了该方法的有效性。第二年，我们构建了两种模式输入的记录系统，记录了10个主题的530个句子。为了对两种模式进行积分，我们采用了两步过程，即利用第一步语音识别生成的词图进行第二步两种模式的积分。该方法比仅进行语音识别的方法识别准确率提高了2.6个点。在今后的工作中，需要开发一种优化两种模式权重的方法。我们将开发一个实时工作的演示系统，并在噪声环境下对其进行评估。

Mobile devices have recently been often used in daily life. User-friendly interface with high accuracy has been strongly demanded. For this purpose, we propose an interface using simultaneous inputs of speech and hand-written gestures. This interface is more robust against environmental noise than speech-only interface, and its input speed is faster than the interface with only hand-written gestures. Our target application is e-mail making with the input of sentences.First year, we proposed an interface in which a sentence is input by speech while the "hiragana" character at the head of each phrase in the sentence is input by hand-written gestures. We implemented a recognition algorithm for hand-written gestures, designed a method for recognizing the simultaneous inputs of the two modes. The proposed method was evaluated by simulation experiments using speech data and hand-written gesture data, which are recorded independently, and was proved to be effective.Second year, we constructed a recording system for the input of the two modes, and recorded 530 sentences from ten subjects. For integrating the two modes, we employed a two-pass process in which a word graph generated by speech recognition in the first pass is utilized for the integration process of the two modes in the second pass. The proposed method improved the recognition accuracy by 2.6 point over the method only with speech recognition.For future work, a method for optimizing the weights among the two modes should be developed. We are going to develop a demonstration system which works in real time and evaluate it in noisy environment.