RI: Text-to-Picture Synthesis

职务名称：文本到图片合成PI：Xiaojin（Jerry）Zhu和Charles Dyer研究所：威斯康星大学麦迪逊分校摘要人工智能的一个挑战是通过多种方式实现人与计算机之间的自然交互。 通常希望在模态之间转换信息。 一个例子是使用语音合成和语音识别在文本和语音之间的转换。 然而，这种转换是罕见的其他模式之间。 特别是，相对较少的研究已经考虑到从一般文本到图形表示的转换。 该项目将开发通用的文本到图片合成算法，该算法可以从自然语言句子中自动生成图片，以便图片传达文本的主要含义。 与需要手工制作的场景叙事描述的现有系统不同，算法将生成静态或动画图片，这些图片代表重要的对象，空间关系和一般文本的动作。 关键组件包括从文本中提取重要信息，为每条信息生成相应的图像，将图像合成为连贯的图片，以及评估。 所提出的方法使用统计机器学习，并从自动机器翻译，文本摘要，文本到语音合成，计算机视觉和图形中汲取灵感。 这项研究将产生计算方法以及工作系统。文本到图片合成很可能有一些重要的广泛影响。 首先，它有可能提高一系列群体的识字率，包括需要额外支持学习阅读的儿童和正在学习第二语言的成年人。 第二，它可以作为一种辅助沟通工具，为残疾人，如阅读障碍和脑损伤，并作为一种通用语言时，需要同时沟通的许多人谁讲不同的语言。 第三，它可以是一个快速浏览长文本文档的摘要工具。 这项研究将促进计算机科学和其他学科（包括心理学和教育学）研究人员之间的合作。 该项目的成果将通过技术出版物、公共网页和软件、研讨会和讲座以及课堂教育传播。

Title: Text-to-Picture SynthesisPIs: Xiaojin (Jerry) Zhu and Charles DyerInstitution: University of Wisconsin-MadisonAbstractOne challenge in artificial intelligence is to enable natural interactions between people and computers via multiple modalities. It is often desirable to convert information between modalities. One example is the conversion between text and speech using speech synthesis and speech recognition. However, such conversion is rare between other modalities. In particular, relatively little research has considered the transformation from general text to pictorial representations. This project will develop general-purpose Text-to- Picture synthesis algorithms that automatically generate pictures from natural language sentences so that the picture conveys the main meaning of the text. Unlike prior systems that require hand-crafted narrative descriptions of a scene, algorithms will generate static or animated pictures that represent important objects, spatial relations, and actions for general text. Key components include extracting important information from text, generating corresponding images for each piece of information, composing the images into a coherent picture, and evaluation. The proposed approach uses statistical machine learning and draws ideas from automatic machine translation, text summarization, text-to-speech synthesis, computer vision, and graphics. This research will produce computational methods as well as working systems.Text-to-picture synthesis is likely to have a number of important broad impacts. First, it has the potential for improving literacy across a range of groups including children who need additional support in learning to read, and adults who are learning a second language. Second, it may be used as an assistive communication tool for people with disabilities such as dyslexia and brain damage, and as a universal language when communication is needed simultaneously to many people who speak different languages. Third, it can be a summarization tool for rapidly browsing long text documents. This research will foster collaboration between researchers in computer science and other disciplines, including psychology and education. Results of the project will be disseminated through technical publications, public web pages and software, seminars and talks, and classroom education.URL: http://www.cs.wisc.edu/~jerryzhu/ttp/