Content-based Search for Deep Generative Models
Content-based Search for Deep Generative Models
复制标题
DOI：
10.1145/3610548.3618189
复制
发表时间：
2022-10
期刊：
影响因子：
--
通讯作者：
Daohan Lu;Sheng-Yu Wang;Nupur Kumari;Rohan Agarwal;David Bau;Jun-Yan Zhu
中科院分区：
文献类型：
--
作者：
Daohan Lu;Sheng-Yu Wang;Nupur Kumari;Rohan Agarwal;David Bau;Jun-Yan Zhu
定制和预训练的生成模型的不断增长使得用户无法完全了解现有的每个模型。为了满足这一需求，我们引入了基于内容的模型搜索的任务：给定一个查询和大量的生成模型，找到最匹配查询的模型。由于每个生成模型都会产生一个图像分布，因此我们将搜索任务制定为一个优化问题，以选择生成类似内容的概率最高的模型作为查询。我们引入一个公式来近似这种概率，给定来自不同模态的查询，例如，图像、草图和文本。此外，我们提出了一个对比学习框架模型检索，学习适应各种查询模态的功能。我们证明了我们的方法优于几个基准生成模型动物园，一个新的基准，我们创建的模型检索任务。
The growing proliferation of customized and pretrained generative models has made it infeasible for a user to be fully cognizant of every model in existence. To address this need, we introduce the task of content-based model search: given a query and a large set of generative models, finding the models that best match the query. As each generative model produces a distribution of images, we formulate the search task as an optimization problem to select the model with the highest probability of generating similar content as the query. We introduce a formulation to approximate this probability given the query from different modalities, e.g., image, sketch, and text. Furthermore, we propose a contrastive learning framework for model retrieval, which learns to adapt features for various query modalities. We demonstrate that our method outperforms several baselines on Generative Model Zoo, a new benchmark we create for the model retrieval task.