Exploiting Communicative Signals to Summarize Information Graphics

为了应对信息超载，近年来人们在文本文档的摘要、分类和检索方面投入了大量的精力。然而，很少有人注意到出现在文本文档中的图形，尽管它们是一种重要的资源，并且经常传达任何随附文本所没有的信息。在这个项目中，PI将开发一种方法，用于自动总结信息（非图形）图形，如条形图和线形图，这些图形传达实体的属性和实体之间的关系。尽管一些信息图形仅用于显示数据，但大多数出现在报纸、杂志和正式报告中的信息图形旨在传达信息。此消息捕获图形的总体内容，并可作为图形的摘要，以便存储、索引和检索。该项目的基本假设是，信息图形包含可用于识别图形所传达的信息的交流信号。这些信号包括不同感知任务所需的相对努力（因为最简单的任务是观众自然会执行的任务，这将对图形传达的信息贡献最大），设计选择，如图形类型，着色，注释，饼图中的楔形爆炸等（所有这些都可以用来吸引人们对图形中某些实体和关系的注意），以及图形标题的元素（尽管由于其通常格式不正确，标题的效用有限）。由交流信号提供的关于图形信息的证据将被输入贝叶斯网络，该网络将试图推断图形的信息。作为研究的一部分，为了计算贝叶斯网络所需的概率表，将收集大量的信息图形语料库并对其识别的信息进行注释。将进行评估实验，以评估系统识别信息图形信息的能力以及信息作为图形摘要的有效性。除了简单的条形图和折线图外，该项目还将研究复杂图形的总结，如分组图和由几个相互关联的图形组成的复合图。虽然该项目只是研究信息图形的摘要，但是PI希望这里开发的一般方法和方法也将为其他类型的图形的摘要方法提供见解。更广泛的影响：该项目将创造能够访问数字图书馆中的信息图形的技术，从而使个人能够更好地利用以图形形式提供的丰富信息。此外，这项研究的成果将包括供HCI社区其他人使用的资源（例如，收集和注释的信息图形语料库将提供给研究社区）。这项研究也将有助于培养未来的科学家，通过提供交叉学科的论文主题，重叠计算机科学和认知科学。

In an effort to cope with information overload, much effort has been devoted in recent years to summarization, categorization, and retrieval of text documents. Yet relatively little attention has been paid to the graphics that appear in text documents, although they are an important resource and often convey information that is absent from any accompanying text. In this project the PI will develop a methodology for automatically summarizing informational (non-pictorial) graphics, such as bar charts and line graphs, that convey attributes of entities and relationships among the entities. Although some informational graphics are only intended to display data, the majority of them that appear in newspapers, magazines, and formal reports are intended to convey a message. This message captures the graphic's overall content and can serve as a summary of the graphic for storage, indexing, and retrieval. The underlying hypothesis of the project is that information graphics contain communicative signals that can be utilized to identify the message that the graphic conveys. These signals include the relative effort required for different perceptual tasks (since the easiest tasks are the ones a viewer will naturally perform and that will contribute the most to the message conveyed by the graphic), design choices such as graphic type, coloring, annotations, an exploded wedge in a pie chart, etc. (all of which may be used to draw attention to certain entities and relations in the graphic), and elements of the graphic's caption (although captions are of limited utility due to their generally ill-formed nature). The evidence about the graphic's message provided by the communicative signals will be entered into a Bayesian network that will attempt to deduce the graphic's message. As part of the research, a large corpus of information graphics will be collected and annotated with their identified messages, in order to compute the probability tables required by the Bayesian network. Evaluation experiments will be conducted to assess the system's ability to identify the message of an information graphic and the effectiveness of the message as a summary of the graphic. In addition to simple bar charts and line graphs, the project will investigate the summarization of complex graphics such as grouped charts and composite graphs consisting of several interrelated graphics. Although the project is only investigating the summarization of informational graphics, the PI expects the general approach and methodology developed here will provide insights into summarization methods for other kinds of graphics as well.Broader Impacts: The project will create technology that enables access to informational graphics in a digital library and thereby empowers individuals to better utilize the wealth of information available in graphical form. In addition, the outcomes of this research will include resources for use by others in the HCI community (e.g., the corpus of collected and annotated informational graphics will be made available to the research community). The research will also contribute to the training of future scientists, by providing interdisciplinary thesis topics that overlap computer science and cognitive science.