Research on an effective method of video communication of sign language in consideration of cognitive characteristics

本研究旨在探讨利用资讯科技为听障人士提供沟通支援。研究结果如下：(1)对基于N400的语言认知、语义加工过程进行了分析，考察了事件相关电位测量对日语手语理解机制的影响。因此，由于我们了解到N400在语义一致的句子中通过手语和文字观察到，并且在文字刺激中潜力很大，并通过呈现方式表现出加工过程的差异。(2)讨论了手语数字编码运动图像质量的评价技术。这一结果保证了意见测试可以取代可理解性测试。进一步的研究表明，高帧速率的运动图像比高精度的运动图像更清晰。另一个结果显示，标牌图片的屏幕大小不是…(C)我们关注的是手语运动图像的低比特率数字编码。根据手语的特点，提出了一种新的适合手语的编码方法。因此，与现有方法相比，使用我们提出的方法在低比特率下对图像数据进行编码的可能性更大。此外，我们的方法在相同的参数条件下，与现有的方法相比，得到了更清晰的图像。(D)我们还研究了手语使用者在交互交流过程中所经历的精神负荷问题。我们发现NASA-TLX对测量精神负荷的程度(E)是有效的。在研究过程中，我们通过视频图像和三维运动的电子数据收集了一个语料库，用于手语的语言学分析。这些电子数据由BoT3D引擎在Web上以手语动画的形式显示出来。我们的研究成果已经发表在《IEICE会刊》、《人机界面学会会刊》等期刊上，我们觉得有必要开展这项研究。在本研究结果的基础上，我们继续从认知科学的角度对手语进行会话分析。较少

This research has aimed at the communication support using ICT for the hearing impaired people. Provided result is the following items.(a) We are concerned with language cognition, semantic processing process analysis based on N400 ; event related potential measurement was examined for the understanding the Japanese sign language comprehension mechanisms. As a result, because it is understood that N400 is observed in semantic congruity sentence by sign language and the character, and the potential is large in the character stimulation, and was shown the difference of the processing process by the presentation modality.(b) We discussed evaluation techniques for the quality of digitally encoded motion picture of sign language. This result guarantees that opinion test can be substituted for intelligibility test. Further investigation shows that motion pictures with high frame rate are more legible than those with high precision. Another result shows that screen size of sign picture doesn' … More t affect comprehensibility.(c) We are concerned with low-bit rate digital encoding of moving picture of sign language. A new encoding method suitable for sign language is presented based on characteristics of sign language. As a result, an encoding image data using our proposed method is more possible at a low bit rate in comparison with an existing method. Moreover, our method in the same parameter condition has been confirmed that it was obtained a clearer image in comparison with an existing method.(d) We have also worked on a problem of mental workload, which is mental stress that sign language users undergo during interactive communication. We have found that NASA-TLX is effective enough for measuring the degree of mental workload(e) In the research process, we collected a corpus by video image and electronic data by three-dimensional movement for linguistic analysis of sign language. The electronic data showed it on Web as sign language animation by the BoT3D engine.Our findings have been appeared in journals such as the one of the transactions on IEICE, and the one of Human Interface Society.We feel necessity to develop this research. On the basis of result of this research, we continue analysis work from the side of cognitive science through conversation analysis of sign language. Less