RI: Small: Integrating physics, data, and art-based insights for controllable generative models

生成模型指的是一大类机器学习技术，它可以从文本提示、草图或其他用户提供的图像等输入中生成用户指定的媒体——包括图像、视频、3D环境和文本。新的生成模型正在迅速发展，并且在许多不同的应用中被视为越来越重要，例如在聊天机器人和自动化中。目前的生成模型的特点是在网络规模的数据上训练的非常大的模型，但仔细检查发现在至关重要的背景下是不可靠的。该项目侧重于视觉媒体的生成模型，目前的生成模型将通过利用关于如何用物理和统计定律描述视觉特征的先验知识来推进。将被利用的知识来源包括基于物理的知识，来自传统内容创建技术的见解，以及使用新颖几何方法对潜在空间建模的进展。预期的好处包括更健壮的模型，更小的规模模型，以及更多的可解释性和模块化模型。本研究系统地探讨了生成对抗网络的基础知识。第一个任务考虑从样本抽取的输入概率分布的作用，推广到非参数分布，以减少样本混合下的分布不匹配。第二个任务涉及细节分层方面的体系结构新颖性，其中综合被分解为一系列更简单的体系结构。第三个任务侧重于开发减少参数鉴别器模型，使用正交性类型约束作为物理变量（如照明、纹理和变形）的代理。第四个任务侧重于开发形状感知架构，使用可学习的多项式基函数更直接地表示形状。这些方法的应用包括增强训练集，以在制造业和健康等难以收集大型训练集的环境中创建可信的机器学习模型。课程创新包括在名为“媒体艺术机器学习”的课程中为非stem学生提供这些方法。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Generative models refer to a large class of machine learning techniques that can generate user-specified media – including images, video, 3D environments, and text – from inputs such as text prompts, sketches, or other user provide images. New generative models are rapidly being developed and are seen as increasingly important in many different applications such as in chatbots and automation. Current generative models are characterized by extremely large models trained on web-scale data, but on closer inspection are found to be unreliable in critically important contexts. This project focuses on generative models for visual media, where current generative models will be advanced by leveraging prior knowledge about how visual features can be described by physical and statistical laws. The sources of knowledge that will be leveraged include physics-based knowledge, insights from traditional content creation techniques, and advances in modeling latent-spaces using novel geometric methods. The anticipated benefits include more robust models, smaller scale models, and more interpretable and modular models. This research systematically investigating the basics of generative-adversarial networks. The first task considers the role of the input probability distribution from which samples are drawn, generalizing to non-parametric distributions tuned to reduce distribution mismatch under sample mixing. The second task involves architectural novelty in terms of detail layering, where synthesis is broken into a series of simpler architectures. The third task focuses on developing reduced parameter discriminator models, using orthogonality-type constraints as a proxy for physical variables like lighting, texture, and deformation. The fourth task focuses on developing shape-aware architectures, using learnable polynomial basis functions to represent shape more directly. Applications for these methods include augmenting training-sets to create trustworthy machine learning models in contexts such as manufacturing and health, where it is difficult to gather large training sets. Curricular innovations include creating access to these approaches for non-STEM students, in a class titled Machine Learning for Media Arts.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.