Visualizing the Obvious: A Concreteness-based Ensemble Model for Noun Property Prediction
Visualizing the Obvious: A Concreteness-based Ensemble Model for Noun Property Prediction
复制标题
DOI：
10.48550/arxiv.2210.12905
复制
发表时间：
2022-10
期刊：
影响因子：
--
通讯作者：
Yue Yang;Artemis Panagopoulou;Marianna Apidianaki;Mark Yatskar;Chris Callison-Burch
中科院分区：
文献类型：
--
作者：
Yue Yang;Artemis Panagopoulou;Marianna Apidianaki;Mark Yatskar;Chris Callison-Burch
神经语言模型编码有关实体及其关系的丰富知识，可以使用探测从其表示中提取这些知识。然而，与其他类型的知识相比，名词的共同属性（例如红草莓、小蚂蚁）的提取更具挑战性，因为它们很少在文本中明确说明。我们假设这主要是对于通信参与者来说显而易见的感知属性的情况。我们建议从图像中提取这些属性并将其用于集成模型中，以补充从语言模型中提取的信息。我们认为感知属性比抽象属性更具体（例如，有趣、完美）。我们建议使用形容词的具体性得分作为校准每个来源（文本与图像）的贡献的杠杆。我们在排名任务中评估我们的集成模型，其中名词的实际属性需要比其他不相关的属性排名更高。我们的结果表明，与强大的基于文本的语言模型相比，所提出的文本和图像的组合极大地改善了名词属性预测。
Neural language models encode rich knowledge about entities and their relationships which can be extracted from their representations using probing. Common properties of nouns (e.g., red strawberries, small ant) are, however, more challenging to extract compared to other types of knowledge because they are rarely explicitly stated in texts. We hypothesize this to mainly be the case for perceptual properties which are obvious to the participants in the communication. We propose to extract these properties from images and use them in an ensemble model, in order to complement the information that is extracted from language models. We consider perceptual properties to be more concrete than abstract properties (e.g., interesting, flawless). We propose to use the adjectives' concreteness score as a lever to calibrate the contribution of each source (text vs. images). We evaluate our ensemble model in a ranking task where the actual properties of a noun need to be ranked higher than other non-relevant properties. Our results show that the proposed combination of text and images greatly improves noun property prediction compared to powerful text-based language models.