RI: Small: A Data-Driven Framework to Sketch-to-Text Generation

该项目旨在解决当前自然语言生成（NLG）系统的局限性，寻求新的数据驱动方法来对文本组成的上下文和创造性维度进行建模。通过将大量在线文本作为修辞模式和语言创造力的非结构化数据库，该项目开发了一个统计生成引擎，该引擎能够以比以前更高的语言创造力和复杂性来组合文本。该项目将素描到文本的生成作为一个概念框架，研究图像标题和产品描述的自动合成作为应用场景。该项目还探索了人机协作写作的新可能性，通过开发一个交互式的基于搜索的编辑器，帮助学生作家从大量其他人的作品中学习。该项目的技术成果有可能在两个方面使我们的社会受益：首先，通过推进各种日常照片的自动图像字幕，它可以为视障人士平等地访问网络做出贡献。其次，通过在大量在线文章的语料库上启用交互式搜索渠道，它可以为培养学生的写作技能创造新的教育体验。该项目有助于支持PI在吸引和教育来自代表性不足群体的学生方面的持续努力。提出的研究是基于一个前提，即大规模的在线写作，如果使用得当，可以成为素描到文本生成的一个有利因素。该项目包括三项基础研究活动。首先，该项目提出了组合框架和元素作为一种新的概念形式主义来组织修辞模式作为构建块，并开发了无监督算法来从大规模的特定领域语料库中提取它们。其次，该项目开发了统计方法来区分字面语言和比喻语言，其具体目标是控制生成语言的字面性和创造性程度。最后，本文设计了可扩展且鲁棒的组合推理算法，并将其表述为约束优化。技术贡献包括与研究界共享的一些独特资源，包括一个新的大规模图像标题对语料库，以及学习合成框架和元素的数据库。

The project aims to address the limitations of the current natural language generation (NLG) systems by seeking new data-driven approaches to modeling the contextual and creative dimensions of text composition. By taking a large collection of online text as an unstructured database of rhetorical patterns and linguistic creativity, the project develops a statistical generation engine that is capable of composing text with a new level of linguistic creativity and sophistication than what has been previously possible. Formulating sketch-to-text generation as a conceptual framework, the project investigates automatic composition of image captions and product descriptions as application scenarios. The project also explores new possibilities of human-computer collaborative writing, by developing an interactive search-based editor that will assist student writers to learn from a large collection of other people's writings. The technical outcome of the project has the potential to benefit our society in two ways: first, by advancing automatic image captioning for a wide variety of everyday photographs, it can contribute toward equal web access for visually impaired. Second, by enabling interactive search channels over a large-corpus of online writings, it can create new education experiences for training students' writing skills. The project is instrumental for supporting the PI's ongoing efforts in attracting and educating students from underrepresented groups. The proposed research is based on the premise that large-scale online writings, if used correctly, can be an enabling factor for sketch-to-text generation. The project consists of three fundamental research activities. First, the project proposes composition frames and elements as a new conceptual formalism to organize rhetorical patterns as building blocks, and develops unsupervised algorithms to extract them from a large-scale domain-specific corpus. Second, the project develops statistical approaches to differentiate literal language from figurative, with the specific goal of controlling the degree of literalness and creativity in the generated language. Finally, the proposed work designs scalable and robust inference algorithms for composition formulated as constrained optimization. Technical contributions include several unique resources to be shared with the research community, including a new large-scale corpus of image-caption pairs, and the database of learned composition frames and elements.