Development of a software system for automatic scene and person indexing in scientific video archives

德国广播档案馆（DRA）是一个根据民法成立的非营利基金会，在美因河畔法兰克福和波茨坦-巴别尔斯堡设有办事处。法兰克福档案馆的收藏重点是自录音和历史记录媒体开始以来的当代历史和音乐录音。1994年，档案馆扩大到包括前德意志民主共和国（民主共和国）的无线电和电视广播档案，最初在柏林，今天在波茨坦-巴别尔斯堡。在之前的联合工作中，选定的东德特别电视广播被数字化，并使用基于内容的图像和视频分析的创新方法，使其可搜索。这些材料包括大约3 000小时的录像片段，包括新闻广播“Aktuelle camera”、杂志广播和220小时的东德电视电影传统。通过使用和开发基于内容的视频分析的自动化方法，科学家们获得了新的可能性，可以搜索所需的场景、镜头和人物，或类似的图像。在场景分类（视觉概念检测）方面取得的非常好的结果的可持续性有待进一步开发，以供将来使用。在建议的计划中，政府会开发一套可供档案人员使用的软件系统，使资料馆和其他档案能够轻松整合自动视频分析方法，以进行基于内容的图像搜索。在该软件系统中，将采用深度学习方法，从而可以同时改进人和视觉概念检测，并将其扩展到电视广播的其他研究密集型部分。具体而言，该项目有以下目标：1。开发一个可持续的软件系统，方便档案工作人员对两个词汇（概念和人物）进行用户友好的扩展；2 .将软件系统集成到DRA的数字化工作流程中，使自动视频分析方法适用于档案中电视广播的总库存；3 .应用深度学习方法提高概念和人物的检出率；4 .视觉概念词典增加约100个概念；5 .将人物词典扩充到约100人的德意志民主共和国历史；7.通过用户反馈和相似度搜索提高概念和人物的检出率；为有效的搜索开发适当的可视化。这样，科学家不仅可以在预先定义的概念和人物的基础上进行搜索查询，而且可以很容易地为自己的研究任务扩展广泛的视觉概念和人物词汇。开发的软件工具将作为开源软件提供给其他科研机构。

The German Broadcasting Archive (DRA) is a non-profit foundation under civil law, with offices in Frankfurt am Main and Potsdam-Babelsberg. The collection priorities of the archive at the Frankfurt location are audio recordings of contemporary history and music since the beginning of recording and historical recording media. In 1994, the DRA was extended to include the radio and television broadcasting archives of the former German Democratic Republic (GDR) initially at a location in Berlin, today in Potsdam-Babelsberg. In joint previous work, selected special GDR television broadcasts were digitized, and using innovative methods of content-based image and video analysis, have been made searchable. The material consists of approximately 3,000 hours of video footage, including the newscasts "Aktuelle Kamera", magazine broadcasts and 220 hours of the East German television film tradition. Through the use and development of automated methods for content-based video analysis, scientists have obtained new possibilities to carry out their searches for desired scenes, camera shots and persons, or for similar images. The sustainability of the achieved very good results of scene classification (detection of visual concepts) is intended to be further developed for future use. In the proposed project, a software system usable by archive staff will be developed to enable the DRA and other archives to easily integrate automatic video analysis methods for content-based image search. In this software system, deep learning methods will be employed, thereby making it possible at the same time to improve person and visual concept detection and expand them to other research-intensive parts of the television broadcasts. In particular, the project has the following objectives: 1. development of a sustainable software system for user-friendly expansion of two lexicons (concepts and persons) by archive staff, 2. integration of the software system in the digitalization workflows of the DRA to make automatic video analysis methods applicable to the total stock of television broadcasts in the archive, 3. improvement of the detection rates for concepts and persons by applying deep learning methods, 4. expansion of the visual concept lexicon by about 100 further concepts, 5. expansion of the person lexicon to about 100 persons of the GDR history, 6. improvement of the detection rates for concepts and persons through user feedback and similarity search, 7. development of appropriate visualizations for effective search. In this way, is it not only possible for scientists to carry out search queries on the basis of pre-defined concepts and persons, but they can also easily expand the extensive lexicons for visual concepts and persons for their own research tasks. The developed software tools will be made available to other scientific institutions as open source software.