PT-MMD: A Novel Statistical Framework for the Evaluation of Generative Systems
PT-MMD: A Novel Statistical Framework for the Evaluation of Generative Systems
复制标题
DOI：
10.1109/ieeeconf44664.2019.9048757
复制
发表时间：
2019-10
期刊：
影响因子：
--
通讯作者：
Alexander Potapov;Ian Colbert;K. Kreutz-Delgado;A. Cloninger;Srinjoy Das
中科院分区：
文献类型：
--
作者：
Alexander Potapov;Ian Colbert;K. Kreutz-Delgado;A. Cloninger;Srinjoy Das
现在，基于随机采样的生成神经网络（例如受限的玻尔兹曼机器和生成对抗网络）现在用于诸如剥离，图像遮挡去除，模式完成和运动合成之类的应用。在涉及使用这些模型执行此类推理任务的方案中，确定允许在预先指定的实施约束下进行模型选择和/或维护必要生成性能的指标至关重要。在本文中，我们提出了一个新的指标，用于评估基于最大平均差异（MMD）和基于置换（基于PT）的重新采样的p值的生成模型性能，我们将其称为PT-MMD 。我们证明了该指标在两种情况下的有效性：（1）选择位宽度和激活功能复杂性，以实现受限制的玻尔兹曼机器的最小功率； （2）通过两种类型的生成对抗网络（PGAN和WGAN）生成的图像进行定量比较，以促进模型选择，以最大程度地提高生成的图像的保真度。对于这些应用，我们的结果使用欧几里得和基于HAAR的内核进行了PT-MMD两个样本假设检验。这证明了距离函数在将生成的图像与其相应的地面真理对应物进行比较中的关键作用与人类用户所感知的。
Stochastic-sampling-based Generative Neural Networks, such as Restricted Boltzmann Machines and Generative Adversarial Networks, are now used for applications such as denoising, image occlusion removal, pattern completion, and motion synthesis. In scenarios which involve performing such inference tasks with these models, it is critical to determine metrics that allow for model selection and/or maintenance of requisite generative performance under pre-specified implementation constraints. In this paper, we propose a new metric for evaluating generative model performance based on p-values derived from the combined use of Maximum Mean Discrepancy (MMD) and permutation-based (PT-based) resampling, which we refer to as PT-MMD. We demonstrate the effectiveness of this metric for two cases: (1) Selection of bitwidth and activation function complexity to achieve minimum power-at-performance for Restricted Boltzmann Machines; (2) Quantitative comparison of images generated by two types of Generative Adversarial Networks (PGAN and WGAN) to facilitate model selection in order to maximize the fidelity of generated images. For these applications, our results are shown using Euclidean and Haar-based kernels for the PT-MMD two sample hypothesis test. This demonstrates the critical role of distance functions in comparing generated images against their corresponding ground truth counterparts as what would be perceived by human users.