Sporthesia: Augmenting Sports Videos Using Natural Language
Sporthesia: Augmenting Sports Videos Using Natural Language
复制标题
运动觉：使用自然语言增强运动视频
DOI：
10.1109/tvcg.2022.3209497
复制
发表时间：
2022-09
影响因子：
5.2
通讯作者：
Zhutian Chen;Qisen Yang;Xiao Xie;Johanna Beyer;Haijun Xia;Yingnian Wu;H. Pfister
中科院分区：
文献类型：
--
作者：
Zhutian Chen;Qisen Yang;Xiao Xie;Johanna Beyer;Haijun Xia;Yingnian Wu;H. Pfister
增强型体育视频结合了联合收割机可视化和视频效果，在实际场景中呈现数据，可以轻松地传达见解，因此越来越受到世界各地体育爱好者的欢迎。然而，创建增强的体育视频仍然是一项具有挑战性的任务，需要大量的时间和视频编辑技能。另一方面，体育见解通常使用自然语言进行交流，例如在评论，口头陈述和文章中，但通常缺乏视觉线索。因此，这项工作旨在通过使分析师能够使用以自然语言表达的见解直接创建嵌入视频中的可视化来促进增强体育视频的创建。为了实现这一目标，我们提出了一个三步方法- 1）检测文本中的可视化实体，2）将这些实体映射到可视化中，3）安排这些可视化与视频一起播放-并分析了155个体育视频剪辑和伴随的评论来完成这些步骤。根据我们的分析，我们设计并实现了Sporthesia，这是一个概念验证系统，它将基于球拍的体育视频和文本评论作为输入和输出增强视频。我们在两个示例场景中展示了运动感觉的适用性，即，使用文本来创作增强的体育视频，以及基于听觉评论来增强历史体育视频。一项技术评估表明，运动感觉在检测文本中的可视化实体方面具有很高的准确性（F1得分为0.9）。八位体育分析师的专家评估表明，我们的语言驱动的创作方法具有很高的实用性、有效性和满意度，并为未来的改进和机会提供了见解。
Augmented sports videos, which combine visualizations and video effects to present data in actual scenes, can communicate insights engagingly and thus have been increasingly popular for sports enthusiasts around the world. Yet, creating augmented sports videos remains a challenging task, requiring considerable time and video editing skills. On the other hand, sports insights are often communicated using natural language, such as in commentaries, oral presentations, and articles, but usually lack visual cues. Thus, this work aims to facilitate the creation of augmented sports videos by enabling analysts to directly create visualizations embedded in videos using insights expressed in natural language. To achieve this goal, we propose a three-step approach – 1) detecting visualizable entities in the text, 2) mapping these entities into visualizations, and 3) scheduling these visualizations to play with the video – and analyzed 155 sports video clips and the accompanying commentaries for accomplishing these steps. Informed by our analysis, we have designed and implemented Sporthesia, a proof-of-concept system that takes racket-based sports videos and textual commentaries as the input and outputs augmented videos. We demonstrate Sporthesia's applicability in two exemplar scenarios, i.e., authoring augmented sports videos using text and augmenting historical sports videos based on auditory comments. A technical evaluation shows that Sporthesia achieves high accuracy (F1-score of 0.9) in detecting visualizable entities in the text. An expert evaluation with eight sports analysts suggests high utility, effectiveness, and satisfaction with our language-driven authoring method and provides insights for future improvement and opportunities.