CRII: CHS: Learning Procedural Modeling Programs for Computer Graphics from Examples

程序建模用于以编程方式为指令、模拟、动画、视觉效果、建筑、图形设计和其他应用程序生成可视内容。一个有效的程序模型可以产生各种详细的、视觉上有趣的甚至令人愉快的令人惊讶的结果。不幸的是，这样的模型很难创作，既需要视觉创造力，也需要编程专业知识。如果能够从实例中推断出程序模型，就可以使更多的人有能力创建和使用程序模型。当前的项目将解决计算机图形学中这一长期悬而未决的问题，方法是在PI先前工作的基础上开发一个研究计划，通过将概率程序与神经网络相结合来研究从示例中学习过程模型的新方法；程序具有足够的表达能力来表示各种可视内容，而神经网络则提供从数据中灵活的学习。项目成果将通过允许用户使用示例而不是通过编写代码来创建过程模型，从而帮助过程建模的民主化，以便更广泛的创意专业人员和爱好者可以参与其中。所有产生的代码和数据都将作为开源发布，以允许其他研究人员和开发人员应用和扩展新技术。由于图形内容通常是分层的，(概率)语法通常用于对其进行程序化建模。然而，这样的内容也具有连续属性的特征：颜色、仿射变换等。虽然可以对语法进行扩展以支持其中一些属性，但没有通用的方法来从示例中学习此类模型。现有的方法要么忽略连续属性，要么专门用于一种类型的内容(例如，建筑立面)。这项研究提出了一种新的通用方法，即基于实例的过程模型学习，该方法生成具有连续属性的离散层次结构。关键的见解是将程序模型表示为概率程序，其控制流和数据流可以由神经网络管理。像语法一样，这样的程序可以自然地表示(可能是递归的)层次结构。程序的神经网络逻辑可以表示复杂的函数，这些函数生成连续的属性，如变换。该模型可以用基于随机梯度的方法有效地学习，并且具有从小样本到大数据集的扩展潜力。最初的重点将是学习3D场景图形的程序模型，3D场景图形是由部件层次结构组成的3D对象。然后，研究将扩展到从示例的大数据集中学习程序模型，将技术应用到3D场景图以外的领域，并利用非结构化输入(如图像)作为示例。项目成果将包括从实例中学习程序模型的新数学框架，有效解决学习问题的算法，以及对学习模型生成的内容质量的评估。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Procedural modeling is used to programmatically generate visual content for instruction, simulation, animation, visual effects, architecture, graphic design, and other applications. An effective procedural model can produce a variety of detailed, visually interesting, and even pleasantly surprising results. Unfortunately, such models are difficult to author, requiring both visual creativity and programming expertise. More people could be empowered to create and use procedural models were it possible to deduce them from examples. The current project will tackle this long-standing open problem in computer graphics by building on the PI's prior work to develop a research program investigating new approaches to learning procedural models from examples by combining probabilistic programs with neural nets; programs are expressive enough to represent a variety of visual content, while neural networks provide flexible learning from data. Project outcomes will help democratize procedural modeling by allowing users to create procedural models with examples rather than by writing code, so that a wider demographic of creative professionals and enthusiasts can participate. All code and data produced will be released as open-source, to allow other researchers and developers to apply and extend the new techniques.Because graphical content is often hierarchical, (probabilistic) grammars are typically used to procedurally model it. However, such content is also characterized by continuous attributes: colors, affine transformations, and so on. While grammars can be extended to support some of these attributes, there are no general-purpose methods for learning such models from examples. Existing approaches either ignore continuous attributes or are specialized to one type of content (e.g., building facades). This research presents a new general-purpose approach for example-based learning of procedural models which generate discrete hierarchical structures with continuous attributes. The key insight is representing a procedural model as a probabilistic program whose control flow and data flow can be governed by neural networks. Like a grammar, such a program can naturally represent (possibly recursive) hierarchical structure. The neural network logic of the program can represent complex functions which generate continuous attributes such as transformations. The model is efficiently learnable with stochastic-gradient-based methods and has the potential to scale from small numbers of examples to large datasets. The initial focus will be on learning procedural models of 3D scene graphs, which are 3D objects composed of a hierarchy of parts. The research will then expand into learning procedural models from large datasets of examples, applying the techniques to domains beyond 3D scene graphs, and leveraging unstructured inputs such as images as examples. Project outcomes will include new mathematical frameworks for learning procedural models from examples, algorithms for efficiently solving the learning problem, and evaluations of the quality of content generated by learned models.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.