Embeddings as epistemic states: Limitations on the use of pooling operators for accumulating knowledge
Embeddings as epistemic states: Limitations on the use of pooling operators for accumulating knowledge
复制标题
作为认知状态的嵌入：使用池算子来积累知识的限制
DOI：
10.1016/j.ijar.2023.108981
复制
发表时间：
2023
影响因子：
3.9
通讯作者：
Schockaert S
中科院分区：
文献类型：
--
作者：
Schockaert S
各种神经网络架构都依赖于池化算子来聚合来自不同来源的信息。在这种情况下，通常隐含地假设向量编码认知状态，即向量捕获已经获得的关于某些感兴趣的属性的证据，并且汇集这些向量产生结合这些证据的向量。我们研究了一些标准池化算子在什么条件下与这个思想兼容，我们称之为认知池化原则。虽然我们发现所有考虑的池化算子都可以满足认知池化原则，但这仅在嵌入足够高维时成立，并且对于大多数池化算子来说，当嵌入满足特定约束（例如具有非负坐标）时成立。我们进一步表明，这些约束对如何在实践中使用嵌入具有重要意义。特别是，我们发现当满足认知池原则时，在大多数情况下，不可能使用线性评分函数来验证命题公式的满足性，只有两种例外：(i)具有上界嵌入的最大池化和（ii）具有非负嵌入的Hadamard池化。这一发现有助于澄清为什么图神经网络有时在推理任务中表现不佳。最后，我们还研究了将认知池化原理扩展到加权认知状态，这在非单调推理中是很重要的，在非单调推理中，最大池化是最合适的算子。
Various neural network architectures rely on pooling operators to aggregate information coming from different sources. It is often implicitly assumed in such contexts that vectors encode epistemic states, i.e. that vectors capture the evidence that has been obtained about some properties of interest, and that pooling these vectors yields a vector that combines this evidence. We study, for a number of standard pooling operators, under what conditions they are compatible with this idea, which we call the epistemic pooling principle. While we find that all the considered pooling operators can satisfy the epistemic pooling principle, this only holds when embeddings are sufficiently high-dimensional and, for most pooling operators, when the embeddings satisfy particular constraints (e.g. having non-negative coordinates). We furthermore show that these constraints have important implications on how the embeddings can be used in practice. In particular, we find that when the epistemic pooling principle is satisfied, in most cases it is impossible to verify the satisfaction of propositional formulas using linear scoring functions, with two exceptions: (i) max-pooling with embeddings that are upper-bounded and (ii) Hadamard pooling with non-negative embeddings. This finding helps to clarify, among others, why Graph Neural Networks sometimes under-perform in reasoning tasks. Finally, we also study an extension of the epistemic pooling principle to weighted epistemic states, which are important in the context of non-monotonic reasoning, where max-pooling emerges as the most suitable operator.