Provable Pathways: Learning Multiple Tasks over Multiple Paths
Provable Pathways: Learning Multiple Tasks over Multiple Paths
复制标题
DOI：
10.48550/arxiv.2303.04338
复制
发表时间：
2023-03
期刊：
影响因子：
--
通讯作者：
Yingcong Li;Samet Oymak
中科院分区：
文献类型：
--
作者：
Yingcong Li;Samet Oymak
构建跨大量任务的有用表示是样本高效智能系统的关键要求。多任务学习(MTL)的传统思想是在任务之间建立一个共享的表征，然后通过调整最后一层来适应新的任务。使用共享的一刀切表示法的一个理想改进是构造特定于任务的表示法。为此，最近的PathNet/MUNet体系结构将单个任务表示为更大的超网中的路径。由路径诱导的子网络可以看作是超网计算图中模块组成的特定于任务的表示。这项工作从统计学习的角度探讨了路径建议：我们首先为学习多个路径上的多个任务的经验风险最小化问题(多路径MTL)开发了新的泛化界限。同时，我们正式确定了当适应新的下游任务时所产生的多路径表示的好处。我们的界限是用高斯复杂性表示的，为线性表示类带来了有形的保证，并为多路径表示的质量和好处提供了新的见解。当计算图是一棵树时，多路径MTL对任务进行层次聚类，并建立特定于聚类的表示。我们对分层MTL进行了进一步的讨论和实验，并严格地确定了多路径MTL明显优于传统的浅超网MTL方法的条件。
Constructing useful representations across a large number of tasks is a key requirement for sample-efficient intelligent systems. A traditional idea in multitask learning (MTL) is building a shared representation across tasks which can then be adapted to new tasks by tuning last layers. A desirable refinement of using a shared one-fits-all representation is to construct task-specific representations. To this end, recent PathNet/muNet architectures represent individual tasks as pathways within a larger supernet. The subnetworks induced by pathways can be viewed as task-specific representations that are composition of modules within supernet's computation graph. This work explores the pathways proposal from the lens of statistical learning: We first develop novel generalization bounds for empirical risk minimization problems learning multiple tasks over multiple paths (Multipath MTL). In conjunction, we formalize the benefits of resulting multipath representation when adapting to new downstream tasks. Our bounds are expressed in terms of Gaussian complexity, lead to tangible guarantees for the class of linear representations, and provide novel insights into the quality and benefits of a multipath representation. When computation graph is a tree, Multipath MTL hierarchically clusters the tasks and builds cluster-specific representations. We provide further discussion and experiments for hierarchical MTL and rigorously identify the conditions under which Multipath MTL is provably superior to traditional MTL approaches with shallow supernets.