CDI-Type II: Collaborative Research: Joint Image-Text Parsing and Reasoning for Analyzing Social and Political News Events

总结：快速变化的多模式通信技术，从国际卫星电视的全球覆盖，互联网新闻渠道的激增，到YouTube，正在改变新闻行业。平行，？公民新闻在智能手机、社交网络和博客的推动下，互联网正在成为一个由中介事件驱动的巨大信息生态系统？选举、社会运动、自然灾害、疾病流行？丰富的异构数据：文本、图像和视频。与此同时，用户和研究人员的工具和方法没有跟上步伐：系统地访问和研究大量新出现的新闻数据仍然是令人望而却步的劳动密集型。利用加州大学洛杉矶分校正在进行的85，000小时新闻视频的数字收集，包括81亿帧图像和5.3亿字的隐藏字幕，研究团队正在开发一种新的计算范式，用于分析社会和政治新闻事件的大规模数据集：（i）研究联合图像-文本分析法，按主题和事件对新闻进行分类，（2）通过图像-文本联合挖掘来研究说服意图，并对言语和视觉说服技术进行建模;（iii）发现多个介导事件相互作用的时空模式，并分析议程设置模式;（iv）开发交互式多视角新闻界面vrNewsScape，用于可视化和与我们的计算和统计结果交互。智力价值：这个跨学科项目对三个学科做出了创新性贡献。转变社会科学研究。该项目开发了一种数据驱动的范式，用于改变社会科学中的传播研究。通过对大量视觉数据集进行定量研究，研究团队识别并表征了目前研究人员无法获得的大规模新闻调解和说服模式，因为人工分析的成本过高。研究团队超越了传统的对象检测，分割和识别，研究图像中的框架和说服技术，这是计算机视觉中一个未触及的话题。该团队研究对象和场景类别在其社会背景下的语义关联和意义。此外，该团队正在研究图像解析，以填补语义空白？这是图像检索中的一个长期存在的技术障碍，它将从解析树中生成叙事文本描述，以便它们可以与输入文本和隐藏字幕融合，用于主题挖掘。该研究超越了传统的主题挖掘从文本进行综合的文本图像挖掘，偏见检测，并在介导的新闻事件的时空演变模式发现。研究发现并总结争议，挖掘用户生成的内容，分析交际意图和说服效果。更广泛的影响：vrNewsScape正在向研究人员和研究生公开提供。因为新闻媒体报道多个不同专家领域的事件？包括国会和总统政治、国际关系、战争和公众起义、自然灾害和人道主义援助任务、疾病流行和卫生倡议、犯罪活动和法庭案件、名人和文化活动？正在开发的分析工具不仅限于社会、政治和计算机科学的特定研究领域，而且首次允许对理解今天所需的大量数据集进行系统和定量的检查。的中介社会。在教育方面，该项目扩展了加州大学洛杉矶分校？的数字公民学习计划（dcl.sscnet.ucla.edu），一个涉及大学和高中学生的新闻分析计划，从而为全国范围内潜在的大量传播学学生提供教育利益（2004年，433，000名大学生就读于传播与新闻专业，209，000名就读于政治学专业[153]），让他们接触到新一代处理多模态数据的高级工具，并激励他们追求符合NSF的计算思维？的目标。

Summary: Rapidly changing technologies of multi-modal communication, from the global reach of international satellite TV, the proliferation of Internet news outlets, to YouTube, are transforming the news industry. In parallel, ?citizen journalism? is on the rise, enabled by smart phones, social networks, and blogs. The Internet is becoming a vast information ecosystem driven by mediated events ? elections, social movements, natural disasters, disease epidemics ? with rich heterogeneous data: text, image, and video. Meanwhile, the tools and methodologies for users and researchers are not keeping pace: it remains prohibitively labor-intensive to systematically access and study the vast amount of emerging news data. Leveraging UCLA's ongoing digital collection of 85,000 hours of news videos, including 8.1 billion image frames and 530 million words of closed captioning, the research team is developing a new computational paradigm for analyzing massive datasets of social and political news events: (i) Studying joint image-text parsing to categorize news by topics and events, and analyzing selection and presentation biases across networks and media spheres in a statistical and quantitative manner never before possible; (ii) Studying by joint image-text mining to reason the persuasion intents, and modeling the techniques of verbal and visual persuasions; (iii) Discovering spatio-temporal patterns in the interactions of multiple mediated events, and analyzing agenda setting patterns; and (iv) Developing an interactive multi-perspective news interface, vrNewsScape, for visualizing and interacting with our computational and statistical results. Intellectual merit: This interdisciplinary project makes innovative contributions to three disciplines. Transforming social science research. The project develops a data-driven paradigm for transforming communication research in the social sciences. By enabling quantitative studies of massive visual datasets, the research team identifies and characterizes large-scale patterns of news mediation and persuasion currently inaccessible to researchers, due to the prohibitive cost of manual analysis. The research team goes beyond traditional object detection, segmentation, and recognition by studying framing and persuasion techniques in images, an untouched topic in computer vision. The team studies semantic associations and meanings for object and scene categories in their social context. Also, the team is studying image parsing to fill the semantic gap ? a long standing technical barrier in image retrieval, and will generate narrative text descriptions from the parse trees so that they can be fused with the input text and closed captioning for topic mining. The research goes beyond conventional topic mining from text to perform integrative text-image mining, bias detection, and pattern discovery in the spatio-temporal evolution of mediated news events. The research detects and summarizes controversy and mine user-generated content for analyzing communicative intent and persuasive effects. Broader impacts: vrNewsScape is being made publicly available to researchers and graduate students. Because the news media report on events in multiple different expert domains ? including congressional and presidential politics, international relations, war and public uprisings, natural disasters and humanitarian aid missions, disease epidemics and health initiatives, criminal activity and court cases, celebrities and cultural events ? the analytical tools in development are not limited to a particular research domain in social, political and computer sciences, but permit for the first time a systematic and quantitative examination of the massive datasets required to understand today?s mediated society. In education, the project extends UCLA?s Digital Civic Learning initiative (dcl.sscnet.ucla.edu), a program involving college and high-school students in the analysis of news, thus delivering education benefits to potentially a huge number of students nationwide in Communication Studies (in 2004, 433,000 college students were enrolled in Communication and Journalism and 209,000 in Political Science[153]), exposing them to a new generation of high-level tools for handling multimodal data and inspiring them to pursue computational thinking, in line with the NSF?s objectives.