CHS: Small: Appearance Modeling by Synthesis

外观建模旨在创建材料的数字表示，范围从艺术家绘制的想象材料到物理材料的复制品。到目前为止，创建逼真的数字材质最成功的方法是数据驱动，从物理材质样本中捕获的数据用于重建数字表示。大多数以前的工作都假设测量完全限制了重建过程。这个项目将探索一种新的范式，将外观建模视为一个受限的合成过程，而不是重建/内插过程。这种范式的改变需要重新思考基本的外观建模假设。首先，受约束的合成生成可能解决方案的分布，而不是经典外观建模方法中的单个最可能解决方案。这为用户提供了参与外观建模过程的机会(例如，根据主观或艺术标准选择“最佳”解决方案)。此外，约束合成不是用额外的外观测量来构建解决方案，而是减少每次额外测量的解决方案分布；也就是说，它促进了外观建模的减法方法。外观建模的约束合成方法还提供了一种优雅且可伸缩的解决方案，用于从不充分或不完整的数据重现材质的外观。这项研究建立在计算机视觉和机器学习方法的基础上，并有可能推动这两个领域的最新进展。更广泛地说，所开发的重建和合成方法将适用于对高维数据进行建模且难以获得样本的领域。拟议研究活动的结果将被纳入新的和现有的课程、研究生和本科水平的招聘活动以及向少数群体推广STEM的外联活动。该项目将通过探索生成性对抗网络(GAN)的三个研究推进受限合成作为外观建模的新范式：(1)无约束材料生成器，(2)约束材料生成器，以及(3)赋予用户创作新材料生成器的能力。在这些研究的每一次推进中，提供从高维目标分布到一个或多个较低维源分布的映射的(可能是非线性的)投影的概念发挥了核心作用。将鉴别器与每个投影相关联产生了新颖的GaN结构，其将训练数据、条件和目标分布的空间去耦合。基于投影和由此产生的分布空间分离的概念，这三项研究的每一项都致力于为外观建模中的不同子领域做出贡献：(1)合成空间变化材料的新实例，(2)根据可变数量的观测或不完整的测量重建材料，以及(3)通过根据用户偏好限制一般材料生成器的输出来设计新的材料生成器。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Appearance modeling aims to create digital representations of materials, ranging from imaginary materials drawn by artists to reproductions of physical materials. To date, the most successful methods for creating realistic digital materials have been data-driven, where data captured from a physical material sample is used to reconstruct a digital representation. Most prior work assumes that the measurements fully constrain the reconstruction process. This project will explore a new paradigm that views appearance modeling as a constrained synthesis process instead of a reconstruction / interpolation process. This change in paradigm requires a rethinking of fundamental appearance modeling assumptions. First, constrained synthesis produces a distribution of possible solutions rather than a single most likely solution as in classic appearance modeling methods. This creates opportunities for users to participate in the appearance modeling process (for example, by selecting the "best"' solution according to subjective or artistic criteria). Furthermore, instead of building up a solution with additional appearance measurements, constrained synthesis reduces the solution distribution with each additional measurement; that is to say, it promotes a subtractive approach to appearance modeling. A constrained synthesis approach to appearance modeling also offers an elegant and scalable solution to reproducing a material's appearance from insufficient or incomplete data. This research builds on methods from computer vision and machine learning, and has the potential to advance the state-of-the-art in both fields. More broadly, the reconstruction and synthesis methods developed will be applicable to fields that model high dimensional data and for which it is difficult to obtain samples. The results from the proposed research activities will be incorporated in new and existing courses, recruitment activities at the graduate and undergraduate level, and outreach activities promoting STEM to minorities.This project will advance constrained synthesis as a new paradigm for appearance modeling through three research thrusts that explore generative adversarial networks (GANs) for: (1) unconstrained material generators, (2) constrained material generators, and (3) empowering users to author new material generators. In each of these research thrusts, the concept of a (potentially non-linear) projection that provides a mapping from the high dimensional target distribution to one or more lower-dimensional source distributions plays a central role. Associating a discriminator with each projection yields novel GAN architectures that decouple the spaces of the training data, conditions, and the target distribution. Building on the concept of projections and the resulting decoupling of distribution spaces, each of the three research thrusts endeavors to contribute to a different subfield in appearance modeling: (1) synthesizing novel instances of spatially varying materials, (2) reconstructing a material from a variable number of observations or from incomplete measurements, and (3) designing novel material generators by restricting the output of a general material generator based on user preferences.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.