Exploiting Communicative Signals to Summarize Information Graphics

为了应对信息过载，近年来在文本文档的摘要、分类和检索方面投入了大量的努力。然而，出现在文本文档中的图形却相对较少受到关注，尽管它们是一种重要的资源，并且通常传达的信息在任何附带的文本中都不存在。在这个项目中，PI将开发一种方法，用于自动汇总传达实体属性和实体之间关系的条形图和折线图等信息(非图示)图形。虽然有些信息性图形仅用于显示数据，但出现在报纸、杂志和正式报告中的大多数信息图形都旨在传达信息。此消息捕获图形的整体内容，并可用作图形的摘要以供存储、索引和检索。该项目的基本假设是，信息图形包含可用于识别图形所传达的信息的交流信号。这些信号包括不同感知任务所需的相对努力(因为最简单的任务是观看者自然执行的任务，这将对图形所传达的信息做出最大贡献)、设计选择，例如图形类型、颜色、注释、饼图中的分解楔形等(所有这些都可用于将注意力吸引到图形中的某些实体和关系)，以及图形标题的元素(尽管由于其通常格式不良的性质，标题的用处有限)。由通信信号提供的关于图形消息的证据将被输入到试图推断图形消息的贝叶斯网络中。作为研究的一部分，将收集大量信息图形语料库，并用它们识别的消息进行注释，以便计算贝叶斯网络所需的概率表。将进行评价实验，以评估该系统识别信息图形信息的能力，以及该信息作为图形摘要的有效性。除了简单的条形图和折线图外，该项目还将研究复杂图形的汇总，如分组图表和由几个相互关联的图形组成的复合图形。虽然该项目只是调查信息图形的摘要，但PI希望这里开发的一般方法和方法也将为其他类型的图形的摘要方法提供见解。广泛的影响：该项目将创建能够访问数字图书馆中的信息图形的技术，从而使个人能够更好地利用图形形式的丰富信息。此外，这项研究的结果将包括供人机界面社区其他人使用的资源(例如，将向研究社区提供收集和注释的信息图形语料库)。这项研究还将通过提供重叠计算机科学和认知科学的跨学科论文主题，为未来科学家的培训做出贡献。

In an effort to cope with information overload, much effort has been devoted in recent years to summarization, categorization, and retrieval of text documents. Yet relatively little attention has been paid to the graphics that appear in text documents, although they are an important resource and often convey information that is absent from any accompanying text. In this project the PI will develop a methodology for automatically summarizing informational (non-pictorial) graphics, such as bar charts and line graphs, that convey attributes of entities and relationships among the entities. Although some informational graphics are only intended to display data, the majority of them that appear in newspapers, magazines, and formal reports are intended to convey a message. This message captures the graphic's overall content and can serve as a summary of the graphic for storage, indexing, and retrieval. The underlying hypothesis of the project is that information graphics contain communicative signals that can be utilized to identify the message that the graphic conveys. These signals include the relative effort required for different perceptual tasks (since the easiest tasks are the ones a viewer will naturally perform and that will contribute the most to the message conveyed by the graphic), design choices such as graphic type, coloring, annotations, an exploded wedge in a pie chart, etc. (all of which may be used to draw attention to certain entities and relations in the graphic), and elements of the graphic's caption (although captions are of limited utility due to their generally ill-formed nature). The evidence about the graphic's message provided by the communicative signals will be entered into a Bayesian network that will attempt to deduce the graphic's message. As part of the research, a large corpus of information graphics will be collected and annotated with their identified messages, in order to compute the probability tables required by the Bayesian network. Evaluation experiments will be conducted to assess the system's ability to identify the message of an information graphic and the effectiveness of the message as a summary of the graphic. In addition to simple bar charts and line graphs, the project will investigate the summarization of complex graphics such as grouped charts and composite graphs consisting of several interrelated graphics. Although the project is only investigating the summarization of informational graphics, the PI expects the general approach and methodology developed here will provide insights into summarization methods for other kinds of graphics as well.Broader Impacts: The project will create technology that enables access to informational graphics in a digital library and thereby empowers individuals to better utilize the wealth of information available in graphical form. In addition, the outcomes of this research will include resources for use by others in the HCI community (e.g., the corpus of collected and annotated informational graphics will be made available to the research community). The research will also contribute to the training of future scientists, by providing interdisciplinary thesis topics that overlap computer science and cognitive science.