RI: Small: Integrating physics, data, and art-based insights for controllable generative models

生成模型是指一大类机器学习技术，可以从文本提示、草图或其他用户提供的图像等输入生成用户指定的媒体，包括图像、视频、3D环境和文本。新的生成模型正在迅速开发，并且在聊天机器人和自动化等许多不同的应用中越来越重要。 目前的生成模型的特点是在网络规模的数据上训练的非常大的模型，但仔细检查后发现，在非常重要的背景下是不可靠的。该项目的重点是视觉媒体的生成模型，其中当前的生成模型将通过利用有关如何通过物理和统计定律描述视觉特征的先验知识来改进。将被利用的知识来源包括基于物理的知识，传统内容创建技术的见解，以及使用新的几何方法建模潜在空间的进展。预期的好处包括更强大的模型，更小规模的模型，以及更可解释和模块化的模型。本研究系统地研究了生成对抗网络的基础知识。第一个任务考虑输入概率分布的作用，从该输入概率分布中提取样本，推广到非参数分布，以减少样本混合下的分布失配。第二个任务涉及到细节分层方面的建筑新奇，其中合成被分解为一系列更简单的架构。第三个任务的重点是开发减少参数的三维模型，使用正交型约束作为物理变量，如照明，纹理和变形的代理。第四个任务侧重于开发形状感知架构，使用可学习的多项式基函数更直接地表示形状。这些方法的应用包括增强训练集，以在制造和健康等难以收集大型训练集的环境中创建值得信赖的机器学习模型。课程创新包括为非STEM学生创建访问这些方法，在一个名为“媒体艺术机器学习”的课程中。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Generative models refer to a large class of machine learning techniques that can generate user-specified media – including images, video, 3D environments, and text – from inputs such as text prompts, sketches, or other user provide images. New generative models are rapidly being developed and are seen as increasingly important in many different applications such as in chatbots and automation. Current generative models are characterized by extremely large models trained on web-scale data, but on closer inspection are found to be unreliable in critically important contexts. This project focuses on generative models for visual media, where current generative models will be advanced by leveraging prior knowledge about how visual features can be described by physical and statistical laws. The sources of knowledge that will be leveraged include physics-based knowledge, insights from traditional content creation techniques, and advances in modeling latent-spaces using novel geometric methods. The anticipated benefits include more robust models, smaller scale models, and more interpretable and modular models. This research systematically investigating the basics of generative-adversarial networks. The first task considers the role of the input probability distribution from which samples are drawn, generalizing to non-parametric distributions tuned to reduce distribution mismatch under sample mixing. The second task involves architectural novelty in terms of detail layering, where synthesis is broken into a series of simpler architectures. The third task focuses on developing reduced parameter discriminator models, using orthogonality-type constraints as a proxy for physical variables like lighting, texture, and deformation. The fourth task focuses on developing shape-aware architectures, using learnable polynomial basis functions to represent shape more directly. Applications for these methods include augmenting training-sets to create trustworthy machine learning models in contexts such as manufacturing and health, where it is difficult to gather large training sets. Curricular innovations include creating access to these approaches for non-STEM students, in a class titled Machine Learning for Media Arts.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.