We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: What do Models Learn From Training on More Than Text? Measuring Visual Commonsense Knowledge

Abstract: There are limitations in learning language from text alone. Therefore, recent focus has been on developing multimodal models. However, few benchmarks exist that can measure what language models learn about language from multimodal training. We hypothesize that training on a visual modality should improve on the visual commonsense knowledge in language models. Therefore, we introduce two evaluation tasks for measuring visual commonsense knowledge in language models and use them to evaluate different multimodal models and unimodal baselines. Primarily, we find that the visual commonsense knowledge is not significantly different between the multimodal models and unimodal baseline models trained on visual text data.
Comments: Accepted to the ACL Student Research Workshop 2022
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2205.07065 [cs.CL]
  (or arXiv:2205.07065v1 [cs.CL] for this version)

Submission history

From: Lovisa Hagström [view email]
[v1] Sat, 14 May 2022 13:37:50 GMT (6581kb,D)

Link back to: arXiv, form interface, contact.