Minimax Learning for Distributed Inference
Minimax Learning for Distributed Inference
复制标题
DOI：
10.1109/tit.2020.3029182
复制
发表时间：
2020-10
影响因子：
2.5
通讯作者：
Cheuk Ting Li;Xiugang Wu;Ayfer Özgür;A. El Gamal
中科院分区：
文献类型：
--
作者：
Cheuk Ting Li;Xiugang Wu;Ayfer Özgür;A. El Gamal
监督学习的经典问题是使用一组标记的训练样本从测量变量X$中推断出目标变量Y$的准确估计。由于数据和决策的日益分布式，本文考虑了这一经典问题的一种变体，其中推理分布在两个节点之间，例如，移动设备和云，它们之间的通信具有速率约束。移动设备观察到$X$并将$X$的描述$M$发送到云，云计算出$Y$的估计值$\hat {Y}$。我们采用最近的极大极小学习方法来研究这个推理问题，并表明它对应于一个单次极大极小噪声有损源编码问题。然后，我们建立了风险率拉格朗日代价的信息理论边界，从而得到了设计近似最优描述-估计对的一般方法。证明我们的结果的一个关键因素是先前用于建立几个一次性源编码定理的强函数表示引理的改进版本。我们的研究结果表明，对于速率约束推理，朴素估计压缩方案通常不是最优的。当$(X，Y)$的分布已知并且误差由对数损失测量时，我们的风险率拉格朗日成本的界限为信息瓶颈提供了一种新的一次性操作解释。我们还演示了一种方法来约束用我们的方法得到的描述-估计对的超额风险。
The classical problem of supervised learning is to infer an accurate estimate of a target variable $Y$ from a measured variable $X$ using a set of labeled training samples. Motivated by the increasingly distributed nature of data and decision making, this paper considers a variation of this classical problem in which the inference is distributed between two nodes, e.g., a mobile device and a cloud, with a rate constraint on the communication between them. The mobile device observes $X$ and sends a description $M$ of $X$ to the cloud, which computes an estimate $\hat {Y}$ of $Y$ . We follow the recent minimax learning approach to study this inference problem and show that it corresponds to a one-shot minimax noisy lossy source coding problem. We then establish information theoretic bounds on the risk-rate Lagrangian cost, leading to a general method for designing a near-optimal descriptor-estimator pair. A key ingredient in the proof of our result is a refined version of the strong functional representation lemma previously used to establish several one-shot source coding theorems. Our results show that a naive estimate-compress scheme for rate-constrained inference is not optimal in general. When the distribution of $(X,Y)$ is known and the error is measured by the logarithmic loss, our bounds on the risk-rate Lagrangian cost provide a new one-shot operational interpretation of the information bottleneck. We also demonstrate a way to bound the excess risk of the descriptor-estimator pair obtained by our method.