Research on an effective method of video communication of sign language in consideration of cognitive characteristics

本研究旨在利用ICT为听障人士提供沟通支持。提供的结果是以下项目。(a)基于N400的语言认知、语义加工过程分析；通过事件相关电位测量来了解日语手语理解机制。结果表明，由于N400在手语和汉字的语义一致性句中被观察到，并且在汉字刺激中潜力较大，并且通过呈现方式显示了加工过程的差异。(b)讨论了手语数字编码电影的质量评价技术。这一结果保证了意见测验可以代替可理解性测验。进一步的研究表明，高帧率的电影比高帧率的电影更容易辨认。另一个结果表明，标识图片的屏幕尺寸对可理解性影响较小。(c)我们关注的是手语运动图像的低比特率数字编码。根据手语的特点，提出了一种新的适用于手语的编码方法。结果表明，与现有的编码方法相比，本文提出的方法更有可能在低比特率下对图像数据进行编码。并且，在相同的参数条件下，与现有的方法相比，我们的方法获得了更清晰的图像。(d)我们还研究了精神工作量的问题，即手语使用者在互动交流时所承受的精神压力。(e)在研究过程中，我们通过视频图像采集语料库，通过三维运动采集电子数据，对手语进行语言分析。电子数据在网络上显示为BoT3D引擎制作的手语动画。我们的研究结果已经出现在期刊上，如IEICE的交易之一，和人类界面学会之一。我们觉得有必要开展这项研究。在此研究成果的基础上，我们通过对手语会话的分析，继续从认知科学的角度进行分析工作。少

This research has aimed at the communication support using ICT for the hearing impaired people. Provided result is the following items.(a) We are concerned with language cognition, semantic processing process analysis based on N400 ; event related potential measurement was examined for the understanding the Japanese sign language comprehension mechanisms. As a result, because it is understood that N400 is observed in semantic congruity sentence by sign language and the character, and the potential is large in the character stimulation, and was shown the difference of the processing process by the presentation modality.(b) We discussed evaluation techniques for the quality of digitally encoded motion picture of sign language. This result guarantees that opinion test can be substituted for intelligibility test. Further investigation shows that motion pictures with high frame rate are more legible than those with high precision. Another result shows that screen size of sign picture doesn' … More t affect comprehensibility.(c) We are concerned with low-bit rate digital encoding of moving picture of sign language. A new encoding method suitable for sign language is presented based on characteristics of sign language. As a result, an encoding image data using our proposed method is more possible at a low bit rate in comparison with an existing method. Moreover, our method in the same parameter condition has been confirmed that it was obtained a clearer image in comparison with an existing method.(d) We have also worked on a problem of mental workload, which is mental stress that sign language users undergo during interactive communication. We have found that NASA-TLX is effective enough for measuring the degree of mental workload(e) In the research process, we collected a corpus by video image and electronic data by three-dimensional movement for linguistic analysis of sign language. The electronic data showed it on Web as sign language animation by the BoT3D engine.Our findings have been appeared in journals such as the one of the transactions on IEICE, and the one of Human Interface Society.We feel necessity to develop this research. On the basis of result of this research, we continue analysis work from the side of cognitive science through conversation analysis of sign language. Less