EAGER: Social Networks Based Concept Learning in Images

在概念层面上对视觉数据的简单、快速和直观搜索的需求是普遍存在的。该项目将探索一种新的基于网络分析的方法来搜索视觉数据，最终导致视觉搜索引擎，使搜索图像像今天通过关键字搜索一样容易。要做到这一点，需要对视觉概念进行机器学习的新方法。该项目提出了一个形式化的框架，将社会网络和语义概念学习的思想统一起来，从而可以高置信度地学习多个语义概念。具体地说，该方法利用概念之间的层次共现相关性作为线索来帮助检测单个视觉概念。健壮性的来源是学习共现模式，类似于社交网络中的社区结构，并随着时间的推移进行改进。概念共现检测的成功将通过自动标注简化个人图像数据的管理。通过语义组织的个人内容，可以学习用户的偏好以提供他/她在线消费的各种内容的个性化。这将对从信息技术到物理、生命和社会科学、情报组织到新闻局的各种应用产生广泛影响。由于人类不必筛选大量数据，对数字数据的取证分析将大大加快。将这些技术扩展到视频、音乐和多模式数据也将为内容消费提供类似的便利。研究团队提供了一个将教育和劳动力发展与研究相结合的环境，并招募和保留了不同的学生群体。与研究活动相辅相成的是教育和公众宣传方面的新举措。该项目开发了一种变革性的方法，系统地探索语义视觉概念的获得和提炼。首先，发现概念的层次化共现模式作为共现网络中的底层社区结构。同现模式在更高的语义水平上扮演着类似于底层场景概念的角色。其次，提出了一种选择视觉上一致的语义概念的方法。由于概念的视觉复杂性不同，通过定量测量概念内的视觉可变性和与其他概念的视觉距离来研究每个概念的视觉语义相关性，从而可以更可靠地建模和更容易地检测它们。第三，该项目引入了一种新颖的图像内容描述符--概念签名，它既能记录语义概念，又能记录从低层特征推断出的相应置信值。最后，该项目提出了通过开发开源技术和软件工具来处理和评估大型数据库性能的可伸缩性技术。这些成果将通过项目网站(http://vislab.ucr.edu/RESEARCH/VisualSemanticConcepts/VSC.php)，通过定期发布软件工具和在IEEE/ACM主要会议上提供教程/讲习班的方式广泛传播。

The need for easy, quick, and intuitive search of visual data at the conceptual level is universal. This project will explore a novel network analysis-based approach to searching visual data, ultimately leading to visual search engines that would make searching images as easy as searching by keywords is today. Accomplishing this requires new approaches to machine learning of visual concepts. This project proposes a formal framework that unifies the ideas from social networks and semantic concept learning so multiple semantic concepts can be learned with high confidence. Specifically, the approach utilizes hierarchical co-occurrence correlation among concepts as cues to help the detection of individual visual concepts. The sources for robustness are the learning of co-occurrence patterns, similar to community structures in a social network, and their refinement over time. The success of concept co-occurrence detection will simplify management of personal image data with automatic tagging. With semantically organized personal content, the preferences of a user can be learned to provide personalization of various contents that he/she consumes online. This will have a broad impact for diverse applications ranging from information technology to physical, life and social sciences to intelligence organizations to news bureaus. Forensics analysis of digital data would be greatly speeded up as humans do not have to sift through large amount of data. Extension of the techniques to video, music, and multi-modal data would also provide similar ease for content consumption. The research team provides an environment integrating education and workforce development with research, and with recruiting and retaining a diverse group of students. Complementing the research activities will be new initiatives in education and public outreach. The project develops a transformative approach to explore the acquisition and refinement of semantic visual concepts systematically. First, it discovers the hierarchical co-occurrence patterns of concepts as underlying community structures in the co-occurrence network. The co-occurrence patterns play roles similar to underlying scene concepts at a higher level of semantics. Second, it proposes an approach for selecting visually-consistent-semantic concepts. Since concepts vary in their visual complexity, visual-semantic relatedness of each concept is investigated by quantitatively measuring the within-concept visual variability and the visual distances to the other concepts such that they can be modeled more reliably and detected more easily. Third, the project introduces a novel image content descriptor called concept signature that can record both the semantic concept and the corresponding confidence value inferred from low-level features. Finally, the project proposes techniques for scalability to handle and evaluate the performance on large databases by developing open source techniques and software tools. The results will be broadly disseminated through the project website (http://vislab.ucr.edu/RESEARCH/VisualSemanticConcepts/VSC.php), via regular releases of software tools and offering tutorials/workshops at major IEEE/ACM conferences.