On Characterizing the Trade-off in Invariant Representation Learning
On Characterizing the Trade-off in Invariant Representation Learning
复制标题
DOI：
--
复制
发表时间：
2021-09
期刊：
影响因子：
--
通讯作者：
Bashir Sadeghi;Sepehr Dehdashtian;Vishnu Naresh Boddeti
中科院分区：
文献类型：
--
作者：
Bashir Sadeghi;Sepehr Dehdashtian;Vishnu Naresh Boddeti
表示学习的许多应用，如隐私保护、算法公平性和领域自适应，都需要对丢弃的语义信息进行显式控制。该目标被表述为满足两个目标：最大化预测目标属性的效用，同时与已知的语义属性保持不变(独立)。不变表示学习(IRepL)问题的解决方案导致了效用和不变性在竞争时的权衡。虽然现有的工作研究局限于这种权衡，但有两个问题仍然悬而未决：1)效用和不变性之间的确切权衡是什么？以及2)哪些编码者(将数据映射到表示)实现了权衡，我们如何从训练数据中进行估计？本文讨论了再生核Hilbert空间中IREPL的这些问题，在高维数据的低维投影的分布近似为正态分布的假设下，给出了再生核Hilbert空间中编码器基本优化问题的全局最优解的闭式解。这产生了近乎最佳的折衷、对应的最佳表示维度和对应的编码器(S)的闭合公式。我们还对典型问题的权衡进行了数值量化，并将它们与基线IRepL算法所获得的结果进行了比较。
Many applications of representation learning, such as privacy preservation, algorithmic fairness, and domain adaptation, desire explicit control over semantic information being discarded. This goal is formulated as satisfying two objectives: maximizing utility for predicting a target attribute while simultaneously being invariant (independent) to a known semantic attribute. Solutions to invariant representation learning (IRepL) problems lead to a trade-off between utility and invariance when they are competing. While existing works study bounds on this trade-off, two questions remain outstanding: 1) What is the exact trade-off between utility and invariance? and 2) What are the encoders (mapping the data to a representation) that achieve the trade-off, and how can we estimate it from training data? This paper addresses these questions for IRepLs in reproducing kernel Hilbert spaces (RKHS)s. Under the assumption that the distribution of a low-dimensional projection of high-dimensional data is approximately normal, we derive a closed-form solution for the global optima of the underlying optimization problem for encoders in RKHSs. This yields closed formulae for a near-optimal trade-off, corresponding optimal representation dimensionality, and the corresponding encoder(s). We also numerically quantify the trade-off on representative problems and compare them to those achieved by baseline IRepL algorithms.