CDI-Type II: Collaborative Research: Joint Image-Text Parsing and Reasoning for Analyzing Social and Political News Events

摘要：快速变化的多模式传播技术，从国际卫星电视的全球覆盖，互联网新闻媒体的激增，到YouTube，正在改变新闻行业。同时，公民新闻？在智能手机、社交网络和博客的推动下，这一数字正在上升。互联网正在成为一个由中介事件驱动的巨大信息生态系统？选举、社会运动、自然灾害、疾病流行？具有丰富的异质数据：文本、图像和视频。与此同时，用户和研究人员的工具和方法没有跟上步伐：系统地访问和研究海量新兴新闻数据仍然是令人望而却步的劳动密集型工作。利用加州大学洛杉矶分校正在进行的8.5万小时新闻视频的数字收集，包括81亿帧图像和5.3亿字的隐藏字幕，研究团队正在开发一种新的计算范式，用于分析社会和政治新闻事件的海量数据集：(I)研究联合图文分析以按主题和事件对新闻进行分类，并以前所未有的统计和定量方式分析跨网络和媒体领域的选择和呈现偏差；(Ii)通过联合图文挖掘来推理说服意图，并对言语和视觉说服的技术进行建模；(Iii)在多个中介事件的互动中发现时空模式，并分析议程设定模式；及(Iv)开发一个互动的多角度新闻界面vrNewsScape，以可视化我们的计算和统计结果并与之互动。智力优势：这个跨学科项目对三个学科做出了创新性的贡献。转变社会科学研究。该项目开发了一种数据驱动的范式，用于改变社会科学中的传播研究。通过对大量可视数据集进行量化研究，研究小组确定并描述了目前研究人员无法获得的大规模新闻调解和说服模式，这是因为人工分析的成本过高。研究团队通过研究图像中的框架和说服技术，超越了传统的对象检测、分割和识别，这是计算机视觉中一个未触及的主题。该团队研究了物体和场景类别在其社会背景下的语义联系和意义。此外，该团队正在研究图像解析以填补语义空白？这是图像检索中的一个长期存在的技术障碍，它将从分析树生成叙述性文本描述，以便它们可以与输入文本和隐藏字幕融合，用于主题挖掘。该研究超越了传统的文本主题挖掘，在媒介新闻事件的时空演化中进行综合的文本-图像挖掘、偏见检测和模式发现。该研究检测和总结争议，并挖掘用户生成的内容，以分析交际意图和说服效果。更广泛的影响：vrNewsScape正在向研究人员和研究生公开提供。因为新闻媒体报道了多个不同专家领域的事件？包括国会和总统政治、国际关系、战争和公共起义、自然灾害和人道主义援助任务、疾病流行和卫生倡议、犯罪活动和法庭案件、名人和文化活动？开发中的分析工具并不局限于社会科学、政治学和计算机科学中的特定研究领域，而是首次允许对理解当今社会所需的海量数据集进行系统和定量的检查？S调解的社会。在教育方面，该项目扩展了加州大学洛杉矶分校S数字公民学习倡议(dcl.sscnet.ucla.edu)，这是一个让大学生和高中生参与新闻分析的项目，从而为全国范围内潜在的大量传播学学生带来教育好处(2004年，433,000名大学生注册学习传播与新闻，209,000名注册学习政治学[153])，使他们接触到处理多模式数据的新一代高级工具，并激励他们追求计算思维，与国家科学基金会？S的目标保持一致。

Summary: Rapidly changing technologies of multi-modal communication, from the global reach of international satellite TV, the proliferation of Internet news outlets, to YouTube, are transforming the news industry. In parallel, ?citizen journalism? is on the rise, enabled by smart phones, social networks, and blogs. The Internet is becoming a vast information ecosystem driven by mediated events ? elections, social movements, natural disasters, disease epidemics ? with rich heterogeneous data: text, image, and video. Meanwhile, the tools and methodologies for users and researchers are not keeping pace: it remains prohibitively labor-intensive to systematically access and study the vast amount of emerging news data. Leveraging UCLA's ongoing digital collection of 85,000 hours of news videos, including 8.1 billion image frames and 530 million words of closed captioning, the research team is developing a new computational paradigm for analyzing massive datasets of social and political news events: (i) Studying joint image-text parsing to categorize news by topics and events, and analyzing selection and presentation biases across networks and media spheres in a statistical and quantitative manner never before possible; (ii) Studying by joint image-text mining to reason the persuasion intents, and modeling the techniques of verbal and visual persuasions; (iii) Discovering spatio-temporal patterns in the interactions of multiple mediated events, and analyzing agenda setting patterns; and (iv) Developing an interactive multi-perspective news interface, vrNewsScape, for visualizing and interacting with our computational and statistical results. Intellectual merit: This interdisciplinary project makes innovative contributions to three disciplines. Transforming social science research. The project develops a data-driven paradigm for transforming communication research in the social sciences. By enabling quantitative studies of massive visual datasets, the research team identifies and characterizes large-scale patterns of news mediation and persuasion currently inaccessible to researchers, due to the prohibitive cost of manual analysis. The research team goes beyond traditional object detection, segmentation, and recognition by studying framing and persuasion techniques in images, an untouched topic in computer vision. The team studies semantic associations and meanings for object and scene categories in their social context. Also, the team is studying image parsing to fill the semantic gap ? a long standing technical barrier in image retrieval, and will generate narrative text descriptions from the parse trees so that they can be fused with the input text and closed captioning for topic mining. The research goes beyond conventional topic mining from text to perform integrative text-image mining, bias detection, and pattern discovery in the spatio-temporal evolution of mediated news events. The research detects and summarizes controversy and mine user-generated content for analyzing communicative intent and persuasive effects. Broader impacts: vrNewsScape is being made publicly available to researchers and graduate students. Because the news media report on events in multiple different expert domains ? including congressional and presidential politics, international relations, war and public uprisings, natural disasters and humanitarian aid missions, disease epidemics and health initiatives, criminal activity and court cases, celebrities and cultural events ? the analytical tools in development are not limited to a particular research domain in social, political and computer sciences, but permit for the first time a systematic and quantitative examination of the massive datasets required to understand today?s mediated society. In education, the project extends UCLA?s Digital Civic Learning initiative (dcl.sscnet.ucla.edu), a program involving college and high-school students in the analysis of news, thus delivering education benefits to potentially a huge number of students nationwide in Communication Studies (in 2004, 433,000 college students were enrolled in Communication and Journalism and 209,000 in Political Science[153]), exposing them to a new generation of high-level tools for handling multimodal data and inspiring them to pursue computational thinking, in line with the NSF?s objectives.