RI: Small: Integrating physics, data, and art-based insights for controllable generative models

生成性模型是指一大类机器学习技术，可以从文本提示、草图或其他用户提供的图像等输入生成用户指定的媒体-包括图像、视频、3D环境和文本。新的产生式模型正在迅速开发，并被视为在许多不同的应用中越来越重要，例如在聊天机器人和自动化方面。目前的生成性模型的特点是在网络规模的数据上训练的非常大的模型，但仔细检查发现，在至关重要的背景下是不可靠的。这个项目的重点是视觉媒体的生成模型，其中当前的生成模型将通过利用关于视觉特征如何通过物理和统计定律来描述的先验知识来改进。将利用的知识来源包括基于物理的知识、来自传统内容创作技术的见解，以及使用新的几何方法建模潜在空间的进展。预期的好处包括更健壮的模型、更小规模的模型以及更具解释性和模块化的模型。本研究系统地考察了产生式-对抗性网络的基础知识。第一个任务考虑输入概率分布的作用，从输入概率分布中提取样本，将其推广到非参数分布，以减少样本混合下的分布失配。第二项任务涉及细节分层方面的体系结构新颖性，其中合成被分解为一系列更简单的体系结构。第三个任务集中于开发简化参数鉴别器模型，使用正交型约束作为照明、纹理和变形等物理变量的代理。第四个任务集中于开发形状感知体系结构，使用可学习的多项式基函数来更直接地表示形状。这些方法的应用包括扩充训练集以在诸如制造和健康等难以收集大训练集的环境中创建可信的机器学习模型。课程创新包括为非STEM学生创造使用这些方法的机会，在名为面向媒体艺术的机器学习的课程中。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Generative models refer to a large class of machine learning techniques that can generate user-specified media – including images, video, 3D environments, and text – from inputs such as text prompts, sketches, or other user provide images. New generative models are rapidly being developed and are seen as increasingly important in many different applications such as in chatbots and automation. Current generative models are characterized by extremely large models trained on web-scale data, but on closer inspection are found to be unreliable in critically important contexts. This project focuses on generative models for visual media, where current generative models will be advanced by leveraging prior knowledge about how visual features can be described by physical and statistical laws. The sources of knowledge that will be leveraged include physics-based knowledge, insights from traditional content creation techniques, and advances in modeling latent-spaces using novel geometric methods. The anticipated benefits include more robust models, smaller scale models, and more interpretable and modular models. This research systematically investigating the basics of generative-adversarial networks. The first task considers the role of the input probability distribution from which samples are drawn, generalizing to non-parametric distributions tuned to reduce distribution mismatch under sample mixing. The second task involves architectural novelty in terms of detail layering, where synthesis is broken into a series of simpler architectures. The third task focuses on developing reduced parameter discriminator models, using orthogonality-type constraints as a proxy for physical variables like lighting, texture, and deformation. The fourth task focuses on developing shape-aware architectures, using learnable polynomial basis functions to represent shape more directly. Applications for these methods include augmenting training-sets to create trustworthy machine learning models in contexts such as manufacturing and health, where it is difficult to gather large training sets. Curricular innovations include creating access to these approaches for non-STEM students, in a class titled Machine Learning for Media Arts.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.