RI: Medium: Collaborative Research: From Text to Pictures

该奖项是根据2009年美国复苏和再投资法案（公法111-5）资助的。研究人员正在开发新的理论模型和技术，以自动将描述性文本转换为代表文本的3D场景。年代的意思。他们通过基于场景的词汇知识资源（SBLR）来实现这一点，这是一个他们从现有资源（PropBank, WordNet, FrameNet）和维基百科和其他未注释文本的自动挖掘中创建的资源。除了谓词-参数结构和语义角色之外，SBLR还包括必要的角色、典型的角色填充、上下文元素和活动姿势，这些元素可以对输入句子进行深度分析，并从3D对象库中组装适当的元素，以描绘句子所隐含的更完整的场景。例如：？特里吃早餐了吗？没有告诉我们在哪里（厨房、餐厅、餐厅）或他吃了什么（麦片、甜甜圈、米饭、梅博什和纳豆）。这些元素必须由关于典型角色填充符的知识提供，这些角色填充符适用于输入中指定的信息。请注意，SBLR有一个因文化背景而异的组成部分。文本生成的3D场景将对人机交互产生深远的、范式转换的影响，使图形设计不熟练的人能够直接用自然语言表达意图和约束——绕过标准的低级直接操作技术。这项研究将为更大的人群和更广泛的应用程序打开3D场景创建的世界。这项研究将特别针对那些需要提高沟通能力的中学生，包括那些母语不是英语或有学习困难的学生：在纽约的一个课外项目中进行的实地研究将测试使用该系统是否能提高读写能力。正如与K-12教师的互动所表明的那样，这项技术也有可能让更多样化的人群在早期就对计算机科学感兴趣。

This award is funded under the American Recovery and Reinvestment Act of 2009 (Public Law 111-5).The researchers are developing new theoretical models and technology to automatically convert descriptive text into 3D scenes representing the text?s meaning. They do this via the Scenario-Based Lexical Knowledge Resource (SBLR), a resource they are creating from existing sources (PropBank, WordNet, FrameNet) and from automated mining of Wikipedia and other un-annotated text. In addition to predicate-argument structure and semantic roles, the SBLR includes necessary roles, typical role fillers, contextual elements, and activity poses which enables analysis of input sentences at a deep level and assembly of appropriate elements from libraries of 3D objects to depict the fuller scene implied by a sentence. For example, ?Terry ate breakfast? does not tell us where (kitchen, dining room, restaurant) or what he ate (cereal, doughnut, or rice, umeboshi, and natto). These elements must be supplied from knowledge about typical role fillers appropriate for the information that is specified in the input. Note that the SBLR has a component that varies by cultural context.Textually-generated 3D scenes will have a profound, paradigm-shifting effect in human computer interaction, giving people unskilled in graphical design the ability to directly express intentions and constraints in natural language -- bypassing standard low-level direct-manipulation techniques. This research will open up the world of 3D scene creation to a much larger group of people and a much wider set of applications. In particular, the research will target middle-school age students who need to improve their communicative skills, including those whose first language is not English or who have learning difficulties: a field study in a New York after-school program will test whether use of the system can improve literacy skills. The technology also has the potential for interesting a more diverse population in computer science at an early age, as interactions with K-12 teachers have indicated.