Real-time Gesture Animation Generation from Speech for Virtual Human Interaction
Real-time Gesture Animation Generation from Speech for Virtual Human Interaction
复制标题
从语音生成实时手势动画以实现虚拟人机交互
DOI：
10.1145/3411763.3451554
复制
发表时间：
2021
期刊：
影响因子：
--
通讯作者：
Krzysztof Pietroszek
中科院分区：
文献类型：
--
作者：
M. Rebol;Christian Gütl;Krzysztof Pietroszek
我们提出了一种直接从语音合成手势的实时系统。我们的数据驱动方法基于生成对抗神经网络来建模语音手势关系。我们利用在线提供的大量演讲者视频数据来训练我们的 3D 手势模型。我们的模型通过获取两秒长度的连续音频输入块来生成特定于说话者的手势。我们在虚拟化身上制作预测手势的动画。我们在音频输入和手势动画之间实现了低于三秒的延迟。
We propose a real-time system for synthesizing gestures directly from speech. Our data-driven approach is based on Generative Adversarial Neural Networks to model the speech-gesture relationship. We utilize the large amount of speaker video data available online to train our 3D gesture model. Our model generates speaker-specific gestures by taking consecutive audio input chunks of two seconds in length. We animate the predicted gestures on a virtual avatar. We achieve a delay below three seconds between the time of audio input and gesture animation.