POQue: Asking Participant-specific Outcome Questions for a Deeper Understanding of Complex Events
POQue: Asking Participant-specific Outcome Questions for a Deeper Understanding of Complex Events
复制标题
DOI：
10.48550/arxiv.2212.02629
复制
发表时间：
2022-12
期刊：
影响因子：
--
通讯作者：
Sai Vallurupalli;Sayontan Ghosh;K. Erk;Niranjan Balasubramanian;Francis Ferraro
中科院分区：
文献类型：
--
作者：
Sai Vallurupalli;Sayontan Ghosh;K. Erk;Niranjan Balasubramanian;Francis Ferraro
关于结果的知识对于复杂事件的理解至关重要，但很难获得。我们表明，通过预先识别复杂事件中的参与者，人群工作者能够（1）推断构成该情况的突出事件的集体影响，（2）注释参与者在导致该情况时的意志参与，以及（3）将该情况的结果置于参与者的状态变化中。通过创建一个多步骤的接口和一个仔细的质量控制策略，我们收集了一个高质量的注释数据集的8 K短新闻专线叙述和ROC故事与高注释者之间的协议（0.74- 0.96加权Fleiss Kappa）。我们的数据集POQUe（参与者结果问题）使我们能够探索和开发解决语义理解多个方面的模型。在实验中，我们表明，目前的语言模型落后于人类的表现在微妙的方式，通过我们的任务制定目标的抽象和具体的理解一个复杂的事件，其结果，以及参与者的影响力的事件高潮。
Knowledge about outcomes is critical for complex event understanding but is hard to acquire.We show that by pre-identifying a participant in a complex event, crowdworkers are ableto (1) infer the collective impact of salient events that make up the situation, (2) annotate the volitional engagement of participants in causing the situation, and (3) ground theoutcome of the situation in state changes of the participants. By creating a multi-step interface and a careful quality control strategy, we collect a high quality annotated dataset of8K short newswire narratives and ROCStories with high inter-annotator agreement (0.74-0.96weighted Fleiss Kappa). Our dataset, POQUe (Participant Outcome Questions), enables theexploration and development of models that address multiple aspects of semantic understanding. Experimentally, we show that current language models lag behind human performance in subtle ways through our task formulations that target abstract and specific comprehension of a complex event, its outcome, and a participant’s influence over the event culmination.