Current browse context:
cs.CV
Change to browse by:
References & Citations
Computer Science > Computer Vision and Pattern Recognition
Title: Visual Goal-Step Inference using wikiHow
(Submitted on 12 Apr 2021 (v1), last revised 10 Sep 2021 (this version, v2))
Abstract: Understanding what sequence of steps are needed to complete a goal can help artificial intelligence systems reason about human activities. Past work in NLP has examined the task of goal-step inference for text. We introduce the visual analogue. We propose the Visual Goal-Step Inference (VGSI) task, where a model is given a textual goal and must choose which of four images represents a plausible step towards that goal. With a new dataset harvested from wikiHow consisting of 772,277 images representing human actions, we show that our task is challenging for state-of-the-art multimodal models. Moreover, the multimodal representation learned from our data can be effectively transferred to other datasets like HowTo100m, increasing the VGSI accuracy by 15 - 20%. Our task will facilitate multimodal reasoning about procedural events.
Submission history
From: Yue Yang [view email][v1] Mon, 12 Apr 2021 22:20:09 GMT (24128kb,D)
[v2] Fri, 10 Sep 2021 03:10:13 GMT (16528kb,D)
Link back to: arXiv, form interface, contact.