Study on Natural Language Description of Human Behavior from Image Sequences and Its Application to Monitoring Systems

在这个项目中，我们提出了一个系统，它产生的自然语言描述人类行为的图像序列和语音。一般来说，视频监控系统广泛应用于医疗设施的便携式监控电视系统、动物园的珍稀动物监控以及安全系统的状态监控。这些系统在传输监控对象的场景或状态时，需要在有限的通信容量下减少通信量，本文提出的系统依次估计出现在监控图像序列上的人的三维位置和方位。然后，将估计位置的轨迹或系列划分为具体的运动段。对于每个片段，提取出现在监控图像上的人的动作的特征，并将其与最合适的动词相关联，在某些情况下，借助于语音识别来进行。此外，一些动词序列被组合成高级动词。最后，系统使用提取的动词和基于火焰结构语法的相应框架结构表达式来组成表达人的动作的句子。

In this projeact we propose a system, which generates natural language descriptions of human behavior from image sequences and speech. In general, video-monitoring systems are widely used for pelient monitoring television system in medical facilities, monitoring rare animals in a zoo and condition monitoring for security systems. It is necessary for these systems to reduce traffic under limited communication capacity when transmit the scene of monitoring objects or the condition.The proposed system sequentially estimates the 3-D position and orientation of a man appeared on a monitoring image sequence. Then, the trace or series of estimated positions is divided into concrete motion segments. For each segment, features of an action of the man appeared on the monitoring images is extracted and associated with the most suitable verb, in some case carried with the aid of speech recognition. In addition, some sequences of verbs are combined into high-level verbs. Finally the system composes sentences expressing the movements of the man using the extracted verb and the corresponding frame structure expression based on the flame structure grammar.