Development of a software system for automatic scene and person indexing in scientific video archives

根据民法，德国广播档案馆(DRA)是一个非营利性基金会，在美因河畔法兰克福和波茨坦-巴布尔斯堡设有办事处。法兰克福档案馆的收藏重点是当代历史的录音和自录音开始以来的音乐，以及历史记录媒体。1994年，DRA扩大到包括前德意志民主共和国(德意志民主共和国)的广播和电视广播档案，最初在柏林的一个地点，今天在波茨坦-巴布尔斯堡。在以前的联合工作中，选定的德意志民主共和国特别电视广播被数字化，并使用基于内容的图像和视频分析的创新方法，使其可以进行搜索。材料包括大约3000小时的视频片段，包括新闻广播《Aktuelle Kamera》、杂志广播和220小时的东德电视电影传统。通过使用和开发基于内容的视频分析的自动化方法，科学家获得了新的可能性来执行他们对所需场景、摄像机镜头和人物或类似图像的搜索。场景分类(视觉概念检测)取得了非常好的效果，其可持续性有待于进一步开发，以供未来使用。在拟议的项目中，将开发一个可供档案馆工作人员使用的软件系统，使DRA和其他档案馆能够轻松地将自动视频分析方法整合到基于内容的图像搜索中。在该软件系统中，将采用深度学习方法，从而可以在提高人和视觉概念检测的同时，将其扩展到电视广播中其他研究密集型部分。具体而言，该项目的目标如下：1.开发一个可持续的软件系统，方便档案馆工作人员对两个词库(概念和人物)的扩充；2.将该软件系统纳入数据档案局的数字化工作流程，使自动视频分析方法适用于档案馆的全部电视广播；3.通过采用深度学习方法提高对概念和人物的识别率；4.将视觉概念词库再扩充约100个概念；5.将人称词典扩大到约100个人；6.通过用户反馈和相似性搜索，提高概念和人物的识别率，7.开发适当的可视化以实现有效的搜索。这样，科学家不仅可以在预定义的概念和人的基础上进行搜索查询，而且还可以轻松地为自己的研究任务扩展视觉概念和人的广泛词典。开发的软件工具将作为开放源码软件提供给其他科学机构。

The German Broadcasting Archive (DRA) is a non-profit foundation under civil law, with offices in Frankfurt am Main and Potsdam-Babelsberg. The collection priorities of the archive at the Frankfurt location are audio recordings of contemporary history and music since the beginning of recording and historical recording media. In 1994, the DRA was extended to include the radio and television broadcasting archives of the former German Democratic Republic (GDR) initially at a location in Berlin, today in Potsdam-Babelsberg. In joint previous work, selected special GDR television broadcasts were digitized, and using innovative methods of content-based image and video analysis, have been made searchable. The material consists of approximately 3,000 hours of video footage, including the newscasts "Aktuelle Kamera", magazine broadcasts and 220 hours of the East German television film tradition. Through the use and development of automated methods for content-based video analysis, scientists have obtained new possibilities to carry out their searches for desired scenes, camera shots and persons, or for similar images. The sustainability of the achieved very good results of scene classification (detection of visual concepts) is intended to be further developed for future use. In the proposed project, a software system usable by archive staff will be developed to enable the DRA and other archives to easily integrate automatic video analysis methods for content-based image search. In this software system, deep learning methods will be employed, thereby making it possible at the same time to improve person and visual concept detection and expand them to other research-intensive parts of the television broadcasts. In particular, the project has the following objectives: 1. development of a sustainable software system for user-friendly expansion of two lexicons (concepts and persons) by archive staff, 2. integration of the software system in the digitalization workflows of the DRA to make automatic video analysis methods applicable to the total stock of television broadcasts in the archive, 3. improvement of the detection rates for concepts and persons by applying deep learning methods, 4. expansion of the visual concept lexicon by about 100 further concepts, 5. expansion of the person lexicon to about 100 persons of the GDR history, 6. improvement of the detection rates for concepts and persons through user feedback and similarity search, 7. development of appropriate visualizations for effective search. In this way, is it not only possible for scientists to carry out search queries on the basis of pre-defined concepts and persons, but they can also easily expand the extensive lexicons for visual concepts and persons for their own research tasks. The developed software tools will be made available to other scientific institutions as open source software.