RI: Text-to-Picture Synthesis

标题：文本到图片合成个人资料：朱晓金(曾傑瑞)和查尔斯·戴尔研究所：威斯康星大学麦迪逊分校摘要人工智能的一个挑战是通过多种形式实现人与计算机之间的自然交互。通常需要在不同的模式之间转换信息。一个例子是使用语音合成和语音识别在文本和语音之间进行转换。然而，在其他模式之间进行这样的转换是罕见的。特别是，从一般文本到图形表征的转换研究相对较少。这个项目将开发通用的文本到图片合成算法，从自然语言句子自动生成图片，以便图片传达文本的主要含义。与以前的系统不同，算法需要手工制作场景的叙事描述，算法将生成代表重要对象、空间关系和一般文本动作的静态或动画图片。关键组件包括从文本中提取重要信息、为每条信息生成相应的图像、将图像组合成连贯的图像以及评估。该方法使用统计机器学习，并借鉴了自动机器翻译、文本摘要、文本到语音合成、计算机视觉和图形学的思想。这项研究将产生计算方法和工作系统。文本到图片的合成可能会产生一些重要的广泛影响。首先，它有可能提高一系列群体的识字水平，包括在学习阅读方面需要额外支持的儿童，以及正在学习第二语言的成年人。其次，它可以作为阅读障碍和脑损伤等残疾人的辅助交流工具，也可以作为一种通用语言，当许多说不同语言的人同时需要交流时。第三，它可以作为快速浏览长文本文档的摘要工具。这项研究将促进计算机科学和其他学科的研究人员之间的合作，包括心理学和教育学。该项目的成果将通过技术出版物、公共网页和软件、研讨会和讲座以及课堂教育进行传播。网址：http://www.cs.wisc.edu/~jerryzhu/ttp/

Title: Text-to-Picture SynthesisPIs: Xiaojin (Jerry) Zhu and Charles DyerInstitution: University of Wisconsin-MadisonAbstractOne challenge in artificial intelligence is to enable natural interactions between people and computers via multiple modalities. It is often desirable to convert information between modalities. One example is the conversion between text and speech using speech synthesis and speech recognition. However, such conversion is rare between other modalities. In particular, relatively little research has considered the transformation from general text to pictorial representations. This project will develop general-purpose Text-to- Picture synthesis algorithms that automatically generate pictures from natural language sentences so that the picture conveys the main meaning of the text. Unlike prior systems that require hand-crafted narrative descriptions of a scene, algorithms will generate static or animated pictures that represent important objects, spatial relations, and actions for general text. Key components include extracting important information from text, generating corresponding images for each piece of information, composing the images into a coherent picture, and evaluation. The proposed approach uses statistical machine learning and draws ideas from automatic machine translation, text summarization, text-to-speech synthesis, computer vision, and graphics. This research will produce computational methods as well as working systems.Text-to-picture synthesis is likely to have a number of important broad impacts. First, it has the potential for improving literacy across a range of groups including children who need additional support in learning to read, and adults who are learning a second language. Second, it may be used as an assistive communication tool for people with disabilities such as dyslexia and brain damage, and as a universal language when communication is needed simultaneously to many people who speak different languages. Third, it can be a summarization tool for rapidly browsing long text documents. This research will foster collaboration between researchers in computer science and other disciplines, including psychology and education. Results of the project will be disseminated through technical publications, public web pages and software, seminars and talks, and classroom education.URL: http://www.cs.wisc.edu/~jerryzhu/ttp/