CHS:Small:Utilizing synergy between human and computer information processing for complex visual information organization and use

CHS：小：利用人与计算机信息处理之间的协同作用来处理复杂的视觉信息组织和用户近年来在使用计算方法自动提取图像含义（即图像语义）方面取得了重要进展。但是，当这些图像与医学等专业领域相关时，理解它们就更具挑战性，因为这取决于人类的专业知识。该项目将人类和计算机的能力结合起来，通过(1)编码代表领域专业知识的人类图像检查和分析行为，以及(2)算法将人类专业知识与图像数据融合，来发现图像语义。这些成果将有助于为医学、科学和安全情报等领域的复杂图像提供真正有意义的解释。这个跨学科的项目将为本科生和研究生提供广泛的研究机会，并扩大他们对计算机的参与。球队会利用大学吗？该校的计算机女性项目和博士项目取得了成功，在从代表性不足和文化多样化的群体中招收学生方面有着良好的记录。该研究将提供新的计算模型来捕捉与执行图像理解任务相关的人类语言和视觉的复杂和独特特征，并提供创新的概率框架来融合人类知识数据与图像特征。可解释的知识模式将被提取出来，为人类专业知识的高级抽象提供信息，并建立跨模态关系。分层概率框架将促进多模态知识数据与图像内容的系统融合。通过融合来自多个互补模式的数据，该框架对人类知识数据中的稀疏性、噪声和模糊性具有鲁棒性，同时在一个或多个数据模式不可用时保持灵活性。通过非参数建模，该框架可以解释由人类专业知识产生的新语义，从而紧密地代表了人类图像理解中的基于知识的处理。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

CHS: Small: Utilizing synergy between human and computer information processing for complex visual information organization and useRecent years have brought important advances in the use of computational approaches to automatically extract the meaning of an image (aka, image semantics). But, understanding these images when they relate to specialized areas such as medicine is significantly more challenging because it depends on human expertise. This project brings together human and computer capabilities to discover image semantics by (1) encoding human image inspection and analysis behaviors that represent domain expertise, and (2) algorithmically fusing human expertise with image data. The outcomes will help to provide truly meaningful interpretations of complex images in areas such as medicine, science, and security intelligence. This interdisciplinary project will provide extensive research opportunities for undergraduate and graduate students and for broadening participation in computing. The team will leverage the college?s successful program for Women in Computing and PhD Program that has a strong track record in recruiting students from underrepresented and culturally-diverse groups. The research will contribute novel computational models to capture the complex and unique features of human language and vision related to performing image understanding tasks, and an innovative probabilistic framework to fuse human knowledge data with image features. Interpretable knowledge patterns will be extracted to inform high-level abstractions of human expertise and establish cross-modality relationships. The hierarchical probabilistic framework will promote a systematic fusion of multimodal knowledge data with image content. By fusing data from multiple, complimentary modalities, the framework is robust to sparseness, noise, and ambiguity in human knowledge data while being flexible when one or more data modalities become unavailable. Through nonparametric modeling, the framework can account for the novel semantics resulting from human expertise, hence closely represent the knowledge-based processing in human image understanding.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.