SBIR Phase II: Automatically Generating Domain Specific Structured Ontologies for Video

这个小企业创新研究（SBIR）第二阶段项目的更广泛的影响/商业潜力将是通过大量不同类型的视频内容实现更具体和更细粒度的搜索，解锁世界视频档案和直播视频内容中的信息。当员工在一个更大的视频资产中寻找一个小的、特定的视频片段来满足他们的业务需求时，公司会损失无数的时间和生产力，这花费了机构宝贵的时间和金钱。通过增强视频的可发现性，许多行业，如教育、媒体、在线游戏和其他行业将受益于发布效率的提高，视频搜索结果将更好地反映用户的意图。该项目的好处不仅限于企业。整个社会可以从以前无法获得的档案中的视频更大的可访问性中受益，这可以用来创建一个更好地了解世界事件的社会，并随着曾经无法被公众发现的世界视频档案的开放而发展教育和技能。这个小企业创新研究（SBIR）第二阶段项目开发了一个视频理解框架和知识图谱，以更好地实现企业视频档案和直播视频的视频搜索和发现。利用内容创建者和数据维护者提供的多模态数据和特定领域的结构化本体，该项目建议开发四种新的机器学习和计算机视觉技术。首先，建立基于网络监督内容的视频检索系统，在不提供带注释的训练数据的情况下构建分类器，有效解决视频分类数据冷启动问题。其次，该项目将建立一个多模态本体映射系统，以实现对我们当前技术外部和新颖的语义概念的映射。此外，为了提供暂时理解和形式化对象关系的能力，将开发用于视频的Allen区间代数模块；将其与知识关系学习相结合，以学习构成其知识图的对象之间的时间关系。最后，为了在开发的知识图中限定人/对象之间的动作关系，将开发一种整体视频建模方法，并将其应用于动作/事件识别和其他任务，如字幕或标题，扩展关键识别能力以完成知识图。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The broader impact/commercial potential of this Small Business Innovation Research (SBIR) Phase II project will be to enable more specific and granular search through a large array of different types of video content, unlocking the information within the world's video archives and live stream video content. Countless hours and productivity in corporations are lost when employees search for a small, specific video clip within a larger video asset to meet their business needs, costing institutions valuable time and money. Through the enhanced discoverability within videos that the results of this project will provide, many industries such as education, media, online gaming, and others will benefit from enhanced efficiencies in publishing and video search results better reflecting user intent. The benefits of this project are not confined to enterprises. Society at large can benefit from a greater accessibility of video within archives that were previously unavailable, which can be used to create a society better-informed on world events and develop education and skills as the world's video archives that were once undiscoverable by the general public are made accessible. This Small Business Innovation Research (SBIR) Phase II project develops a video understanding framework and knowledge graph to better enable video search and discovery for enterprise video archives and live stream video. Leveraging multimodal data and domain specific structured ontologies provided by content creators and data maintainers, this project proposes to develop four new machine learning and computer vision technologies. Firstly, a webly supervised content-based retrieval system for video will be created in order to build classifiers without being provided annotated training data, effectively solving the video classification data cold start problem. Second, the project will build a multimodal ontology mapping system to enable mapping semantic concepts external and novel to our current technologies. Furthermore, to provide the ability to understand and formalize object relationships temporally, an Allen interval algebra module for video will be developed; combining it with knowledge relational learning in order to learn temporal relationships between objects that comprise its knowledge graph. Finally, to qualify the action relationships between people/objects in the developed knowledge graph, a holistic video modeling approach will be developed and applied to action/event recognition and other tasks, such as captioning or titling, expanding a key recognition capability to complete the knowledge graph.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.