CRII: CHS: Learning Procedural Modeling Programs for Computer Graphics from Examples

过程建模用于以编程方式生成用于教学、模拟、动画、视觉效果、建筑、图形设计和其他应用的视觉内容。 一个有效的过程模型可以产生各种详细的，视觉上有趣的，甚至令人惊喜的结果。 不幸的是，这样的模型很难创作，需要视觉创造力和编程专业知识。 如果能够从例子中推导出过程模型，那么更多的人可以被授权创建和使用过程模型。 目前的项目将通过建立在PI之前的工作来解决计算机图形学中这个长期存在的开放性问题，以开发一个研究计划，通过将概率程序与神经网络相结合来研究从示例中学习过程模型的新方法;程序具有足够的表达能力来表示各种视觉内容，而神经网络提供灵活的数据学习。 项目成果将通过允许用户使用示例而不是编写代码来创建过程模型，从而有助于过程建模的民主化，以便更广泛的创意专业人士和爱好者可以参与其中。 所有生成的代码和数据都将作为开源发布，以允许其他研究人员和开发人员应用和扩展新技术。（概率）语法通常用于对其进行过程建模。然而，此类内容也具有连续属性的特征：颜色、仿射变换等。虽然语法可以扩展以支持其中的一些属性，但没有通用的方法可以从示例中学习这些模型。 现有的方法或者忽略连续属性或者专用于一种类型的内容（例如，建筑立面）。 本研究提出了一种新的通用方法，用于基于实例的过程模型学习，该过程模型生成具有连续属性的离散层次结构。 关键的见解是将过程模型表示为概率程序，其控制流和数据流可以由神经网络管理。 像语法一样，这样的程序可以自然地表示（可能是递归的）层次结构。 程序的神经网络逻辑可以表示生成连续属性（如转换）的复杂函数。 该模型可以通过基于随机梯度的方法有效地学习，并且有可能从少量的示例扩展到大型数据集。 最初的重点将是学习3D场景图的程序模型，这些场景图是由零件层次结构组成的3D对象。 然后，研究将扩展到从大型示例数据集学习过程模型，将这些技术应用于3D场景图之外的领域，并利用图像等非结构化输入作为示例。 项目成果将包括从示例中学习程序模型的新数学框架、有效解决学习问题的算法以及对学习模型生成的内容质量的评估。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估来支持。

Procedural modeling is used to programmatically generate visual content for instruction, simulation, animation, visual effects, architecture, graphic design, and other applications. An effective procedural model can produce a variety of detailed, visually interesting, and even pleasantly surprising results. Unfortunately, such models are difficult to author, requiring both visual creativity and programming expertise. More people could be empowered to create and use procedural models were it possible to deduce them from examples. The current project will tackle this long-standing open problem in computer graphics by building on the PI's prior work to develop a research program investigating new approaches to learning procedural models from examples by combining probabilistic programs with neural nets; programs are expressive enough to represent a variety of visual content, while neural networks provide flexible learning from data. Project outcomes will help democratize procedural modeling by allowing users to create procedural models with examples rather than by writing code, so that a wider demographic of creative professionals and enthusiasts can participate. All code and data produced will be released as open-source, to allow other researchers and developers to apply and extend the new techniques.Because graphical content is often hierarchical, (probabilistic) grammars are typically used to procedurally model it. However, such content is also characterized by continuous attributes: colors, affine transformations, and so on. While grammars can be extended to support some of these attributes, there are no general-purpose methods for learning such models from examples. Existing approaches either ignore continuous attributes or are specialized to one type of content (e.g., building facades). This research presents a new general-purpose approach for example-based learning of procedural models which generate discrete hierarchical structures with continuous attributes. The key insight is representing a procedural model as a probabilistic program whose control flow and data flow can be governed by neural networks. Like a grammar, such a program can naturally represent (possibly recursive) hierarchical structure. The neural network logic of the program can represent complex functions which generate continuous attributes such as transformations. The model is efficiently learnable with stochastic-gradient-based methods and has the potential to scale from small numbers of examples to large datasets. The initial focus will be on learning procedural models of 3D scene graphs, which are 3D objects composed of a hierarchy of parts. The research will then expand into learning procedural models from large datasets of examples, applying the techniques to domains beyond 3D scene graphs, and leveraging unstructured inputs such as images as examples. Project outcomes will include new mathematical frameworks for learning procedural models from examples, algorithms for efficiently solving the learning problem, and evaluations of the quality of content generated by learned models.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.