RI: Medium: Collaborative Research: From Text to Pictures

该奖项是根据2009年美国复苏和再投资法案（公法111-5）资助的。研究人员正在开发新的理论模型和技术，以自动将描述性文本转换为代表文本的3D场景？的意义。 他们通过基于场景的词汇知识资源（SBLR）来实现这一点，SBLR是他们从现有资源（PropBank，WordNet，FrameNet）以及维基百科和其他未注释文本的自动挖掘中创建的资源。 除了谓词论元结构和语义角色之外，SBLR还包括必要的角色、典型的角色填充物、上下文元素和活动姿势，这使得能够在深层次上分析输入句子，并从3D对象库中组装适当的元素来描绘句子所暗示的更完整的场景。 例如，？泰瑞吃早餐了吗没有告诉我们在哪里（厨房，餐厅，餐厅）或他吃了什么（麦片，甜甜圈，或米饭，梅饼，纳豆）。这些元素必须由关于适合于输入中指定的信息的典型角色填充符的知识提供。 请注意，SBLR的一个组成部分会因文化背景而异。文本生成的3D场景将在人机交互中产生深刻的、范式转换的效果，使不熟悉图形设计的人能够直接用自然语言表达意图和约束--绕过标准的低级直接操作技术。这项研究将为更多的人和更广泛的应用程序打开3D场景创建的世界。特别是，这项研究将针对需要提高交际技能的中学生，包括那些第一语言不是英语或有学习困难的学生：纽约课后项目的实地研究将测试使用该系统是否可以提高识字技能。这项技术也有可能在早期吸引更多样化的计算机科学人群，正如与K-12教师的互动所表明的那样。

This award is funded under the American Recovery and Reinvestment Act of 2009 (Public Law 111-5).The researchers are developing new theoretical models and technology to automatically convert descriptive text into 3D scenes representing the text?s meaning. They do this via the Scenario-Based Lexical Knowledge Resource (SBLR), a resource they are creating from existing sources (PropBank, WordNet, FrameNet) and from automated mining of Wikipedia and other un-annotated text. In addition to predicate-argument structure and semantic roles, the SBLR includes necessary roles, typical role fillers, contextual elements, and activity poses which enables analysis of input sentences at a deep level and assembly of appropriate elements from libraries of 3D objects to depict the fuller scene implied by a sentence. For example, ?Terry ate breakfast? does not tell us where (kitchen, dining room, restaurant) or what he ate (cereal, doughnut, or rice, umeboshi, and natto). These elements must be supplied from knowledge about typical role fillers appropriate for the information that is specified in the input. Note that the SBLR has a component that varies by cultural context.Textually-generated 3D scenes will have a profound, paradigm-shifting effect in human computer interaction, giving people unskilled in graphical design the ability to directly express intentions and constraints in natural language -- bypassing standard low-level direct-manipulation techniques. This research will open up the world of 3D scene creation to a much larger group of people and a much wider set of applications. In particular, the research will target middle-school age students who need to improve their communicative skills, including those whose first language is not English or who have learning difficulties: a field study in a New York after-school program will test whether use of the system can improve literacy skills. The technology also has the potential for interesting a more diverse population in computer science at an early age, as interactions with K-12 teachers have indicated.