RI: Small: A Data-Driven Framework to Sketch-to-Text Generation

该项目旨在解决目前自然语言生成系统的局限性，寻求新的数据驱动的方法来模拟文本构成的上下文和创造性方面。通过将大量在线文本作为修辞模式和语言创造力的非结构化数据库，该项目开发了一个统计生成引擎，该引擎能够以比以前可能的语言创造力和复杂性更高的水平撰写文本。该项目将草图到文本的生成作为概念框架，将图像字幕和产品说明的自动合成作为应用场景进行研究。该项目还探索了人机协作写作的新可能性，开发了一个基于搜索的交互式编辑器，帮助学生作家从大量其他人的作品中学习。该项目的技术成果有可能在两个方面造福我们的社会：首先，通过推进各种日常照片的自动图像字幕，它可以为视障人士平等地访问网络做出贡献。其次，通过在大量在线写作语料库上启用互动搜索渠道，它可以为培养学生的写作技能创造新的教育体验。该项目有助于支持国际学生联合会在吸引和教育代表不足群体的学生方面所作的持续努力。这项拟议的研究是基于这样一个前提，即如果使用得当，大规模在线写作可以成为草图到文本生成的促成因素。该项目由三项基础研究活动组成。首先，该项目提出了一种新的概念形式主义，以组合框架和元素来组织修辞模式作为构建块，并开发了无监督算法来从大规模特定领域语料库中提取修辞模式。其次，该项目开发了区分字面语言和比喻语言的统计方法，具体目标是控制生成的语言的字面程度和创造力。最后，针对约束优化问题，设计了可扩展的、健壮的组合推理算法。技术贡献包括将与研究界共享的几个独特资源，包括一个新的大规模图像-字幕对语料库，以及学习的构图框架和元素数据库。

The project aims to address the limitations of the current natural language generation (NLG) systems by seeking new data-driven approaches to modeling the contextual and creative dimensions of text composition. By taking a large collection of online text as an unstructured database of rhetorical patterns and linguistic creativity, the project develops a statistical generation engine that is capable of composing text with a new level of linguistic creativity and sophistication than what has been previously possible. Formulating sketch-to-text generation as a conceptual framework, the project investigates automatic composition of image captions and product descriptions as application scenarios. The project also explores new possibilities of human-computer collaborative writing, by developing an interactive search-based editor that will assist student writers to learn from a large collection of other people's writings. The technical outcome of the project has the potential to benefit our society in two ways: first, by advancing automatic image captioning for a wide variety of everyday photographs, it can contribute toward equal web access for visually impaired. Second, by enabling interactive search channels over a large-corpus of online writings, it can create new education experiences for training students' writing skills. The project is instrumental for supporting the PI's ongoing efforts in attracting and educating students from underrepresented groups. The proposed research is based on the premise that large-scale online writings, if used correctly, can be an enabling factor for sketch-to-text generation. The project consists of three fundamental research activities. First, the project proposes composition frames and elements as a new conceptual formalism to organize rhetorical patterns as building blocks, and develops unsupervised algorithms to extract them from a large-scale domain-specific corpus. Second, the project develops statistical approaches to differentiate literal language from figurative, with the specific goal of controlling the degree of literalness and creativity in the generated language. Finally, the proposed work designs scalable and robust inference algorithms for composition formulated as constrained optimization. Technical contributions include several unique resources to be shared with the research community, including a new large-scale corpus of image-caption pairs, and the database of learned composition frames and elements.