CHS:Small:Utilizing synergy between human and computer information processing for complex visual information organization and use

CHS：小号：利用人类和计算机之间的协同信息处理复杂的视觉信息组织和使用近年来带来了重要的进展，在使用计算方法自动提取图像的含义（又名，图像语义）。但是，当这些图像与医学等专业领域相关时，理解它们就更具挑战性，因为它依赖于人类的专业知识。该项目汇集了人类和计算机的能力，通过（1）编码人类图像检查和分析行为，代表领域的专业知识，（2）算法融合人类的专业知识与图像数据来发现图像语义。这些成果将有助于为医学、科学和安全情报等领域的复杂图像提供真正有意义的解释。这个跨学科的项目将为本科生和研究生提供广泛的研究机会，并扩大参与计算。团队会利用学院？的成功计划，妇女在计算和博士课程，有一个良好的记录，在招聘学生从代表性不足和文化多元化的群体。该研究将提供新的计算模型，以捕获与执行图像理解任务相关的人类语言和视觉的复杂和独特特征，以及将人类知识数据与图像特征融合的创新概率框架。可解释的知识模式将被提取，以告知人类专业知识的高级抽象，并建立跨模态的关系。分层概率框架将促进多模态知识数据与图像内容的系统融合。通过融合来自多个互补模态的数据，该框架对人类知识数据中的稀疏性、噪声和模糊性是鲁棒的，同时当一个或多个数据模态变得不可用时是灵活的。通过非参数建模，该框架可以解释人类专业知识所产生的新语义，因此密切代表了人类图像理解中基于知识的处理。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

CHS: Small: Utilizing synergy between human and computer information processing for complex visual information organization and useRecent years have brought important advances in the use of computational approaches to automatically extract the meaning of an image (aka, image semantics). But, understanding these images when they relate to specialized areas such as medicine is significantly more challenging because it depends on human expertise. This project brings together human and computer capabilities to discover image semantics by (1) encoding human image inspection and analysis behaviors that represent domain expertise, and (2) algorithmically fusing human expertise with image data. The outcomes will help to provide truly meaningful interpretations of complex images in areas such as medicine, science, and security intelligence. This interdisciplinary project will provide extensive research opportunities for undergraduate and graduate students and for broadening participation in computing. The team will leverage the college?s successful program for Women in Computing and PhD Program that has a strong track record in recruiting students from underrepresented and culturally-diverse groups. The research will contribute novel computational models to capture the complex and unique features of human language and vision related to performing image understanding tasks, and an innovative probabilistic framework to fuse human knowledge data with image features. Interpretable knowledge patterns will be extracted to inform high-level abstractions of human expertise and establish cross-modality relationships. The hierarchical probabilistic framework will promote a systematic fusion of multimodal knowledge data with image content. By fusing data from multiple, complimentary modalities, the framework is robust to sparseness, noise, and ambiguity in human knowledge data while being flexible when one or more data modalities become unavailable. Through nonparametric modeling, the framework can account for the novel semantics resulting from human expertise, hence closely represent the knowledge-based processing in human image understanding.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.