CDI-Type II: Collaborative Research: Joint Image-Text Parsing and Reasoning for Analyzing Social and Political News Events

CDI-Type II：协作研究：用于分析社会和政治新闻事件的联合图文解析和推理

• 批准号: 1028381
• 负责人: Song-Chun Zhu
• 金额: $ 130万
• 依托单位: University of California-Los Angeles
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2010
• 资助国家: 美国
• 起止时间: 2010-10-01 至 2016-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1028381&HistoricalAwards=false
• 关键词: CDI; Type; II; Collaborative; Research

Summary: Rapidly changing technologies of multi-modal communication, from the global reach of international satellite TV, the proliferation of Internet news outlets, to YouTube, are transforming the news industry. In parallel, ?citizen journalism? is on the rise, enabled by smart phones, social networks, and blogs. The Internet is becoming a vast information ecosystem driven by mediated events ? elections, social movements, natural disasters, disease epidemics ? with rich heterogeneous data: text, image, and video. Meanwhile, the tools and methodologies for users and researchers are not keeping pace: it remains prohibitively labor-intensive to systematically access and study the vast amount of emerging news data.Leveraging UCLA's ongoing digital collection of 85,000 hours of news videos, including 8.1 billion image frames and 530 million words of closed captioning, the research team is developing a new computational paradigm for analyzing massive datasets of social and political news events: (i) Studying joint image-text parsing to categorize news by topics and events, and analyzing selection and presentation biases across networks and media spheres in a statistical and quantitative manner never before possible; (ii) Studying by joint image-text mining to reason the persuasion intents, and modeling the techniques of verbal and visual persuasions; (iii) Discovering spatio-temporal patterns in the interactions of multiple mediated events, and analyzing agenda setting patterns; and (iv) Developing an interactive multi-perspective news interface, vrNewsScape, for visualizing and interacting with our computational and statistical results.Intellectual merit: This interdisciplinary project makes innovative contributions to three disciplines. Transforming social science research. The project develops a data-driven paradigm for transforming communication research in the social sciences. By enabling quantitative studies of massive visual datasets, the research team identifies and characterizes large-scale patterns of news mediation and persuasion currently inaccessible to researchers, due to the prohibitive cost of manual analysis. The research team goes beyond traditional object detection, segmentation, and recognition by studying framing and persuasion techniques in images, an untouched topic in computer vision. The team studies semantic associations and meanings for object and scene categories in their social context. Also, the team is studying image parsing to fill the semantic gap ? a long standing technical barrier in image retrieval, and will generate narrative text descriptions from the parse trees so that they can be fused with the input text and closed captioning for topic mining.The research goes beyond conventional topic mining from text to perform integrative text-image mining, bias detection, and pattern discovery in the spatio-temporal evolution of mediated news events. The research detects and summarizes controversy and mine user-generated content for analyzing communicative intent and persuasive effects.Broader impacts: vrNewsScape is being made publicly available to researchers and graduate students. Because the news media report on events in multiple different expert domains ? including congressional and presidential politics, international relations, war and public uprisings, natural disasters and humanitarian aid missions, disease epidemics and health initiatives, criminal activity and court cases, celebrities and cultural events ? the analytical tools in development are not limited to a particular research domain in social, political and computer sciences, but permit for the first time a systematic and quantitative examination of the massive datasets required to understand today?s mediated society.In education, the project extends UCLA?s Digital Civic Learning initiative (dcl.sscnet.ucla.edu), a program involving college and high-school students in the analysis of news, thus delivering education benefits to potentially a huge number of students nationwide in Communication Studies (in 2004, 433,000 college students were enrolled in Communication and Journalism and 209,000 in Political Science[153]), exposing them to a new generation of high-level tools for handling multimodal data and inspiring them to pursue computational thinking, in line with the NSF?s objectives.

摘要：快速变化的多模式通信技术，从国际卫星电视的全球覆盖，互联网新闻媒体的激增，到YouTube，正在改变新闻行业。同时，？公民新闻?在智能手机、社交网络和博客的推动下，互联网正在崛起。互联网正在成为一个由中介事件驱动的庞大信息生态系统。选举、社会运动、自然灾害、流行病？具有丰富的异构数据：文本、图像和视频。与此同时，用户和研究人员的工具和方法没有跟上步伐：系统地访问和研究大量新出现的新闻数据仍然需要大量的劳动。利用加州大学洛杉矶分校正在进行的85,000小时新闻视频的数字收集，包括81亿帧图像和5.3亿字的封闭字幕，研究团队正在开发一种新的计算范式，用于分析社会和政治新闻事件的大量数据集：(i)研究图像-文本联合分析，根据主题和事件对新闻进行分类，并以前所未有的统计和定量方式分析网络和媒体领域的选择和呈现偏差；（ii）通过图像-文本联合挖掘来研究说服意图，并对口头说服和视觉说服技术进行建模；（三）发现多重中介事件相互作用的时空格局，分析议程设置模式；（iv）开发交互式多视角新闻界面vrNewsScape，以便将我们的计算和统计结果可视化并与之交互。知识价值：本项目为三个学科做出了创新性贡献。转变社会科学研究。该项目开发了一种数据驱动的范式，用于转变社会科学中的传播研究。通过对大量视觉数据集进行定量研究，研究小组确定并描述了目前由于人工分析成本过高，研究人员无法获得的新闻调解和说服的大规模模式。研究小组超越了传统的目标检测、分割和识别，研究了图像中的框架和说服技术，这是计算机视觉中一个未触及的话题。该团队研究了社会背景下物体和场景类别的语义关联和含义。此外，该团队正在研究图像解析以填补语义空白。这是图像检索中长期存在的技术障碍，并且将从解析树中生成叙述性文本描述，以便它们可以与输入文本和主题挖掘的封闭字幕融合。该研究超越了传统的文本主题挖掘，在媒介新闻事件的时空演变中进行综合文本-图像挖掘、偏见检测和模式发现。研究发现并总结争议，挖掘用户生成的内容来分析交际意图和说服效果。更广泛的影响：vrNewsScape正在向研究人员和研究生公开提供。因为新闻媒体报道的事件涉及多个不同的专家领域？包括国会和总统政治、国际关系、战争和公众起义、自然灾害和人道主义援助任务、疾病流行和卫生倡议、犯罪活动和法庭案件、名人和文化活动？正在开发的分析工具并不局限于社会、政治和计算机科学的特定研究领域，而是首次允许对理解当今世界所需的大量数据集进行系统和定量的检查。S中介社会。在教育方面，该项目扩展了加州大学洛杉矶分校？数字公民学习计划（dcl.sscnet.ucla.edu），一个让大学和高中学生参与新闻分析的项目，从而为全国范围内潜在的大量传播学学生提供教育效益（2004年，43.3万名大学生注册了传播学和新闻学，20.9万名学生注册了政治学[153]）；让他们接触到处理多模态数据的新一代高级工具，并激励他们追求计算思维，这符合国家科学基金会的要求？年代的目标。