We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:

References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computer Vision and Pattern Recognition

Title: Automatic Modeling of Social Concepts Evoked by Art Images as Multimodal Frames

Abstract: Social concepts referring to non-physical objects--such as revolution, violence, or friendship--are powerful tools to describe, index, and query the content of visual data, including ever-growing collections of art images from the Cultural Heritage (CH) field. While much progress has been made towards complete image understanding in computer vision, automatic detection of social concepts evoked by images is still a challenge. This is partly due to the well-known semantic gap problem, worsened for social concepts given their lack of unique physical features, and reliance on more unspecific features than concrete concepts. In this paper, we propose the translation of recent cognitive theories about social concept representation into a software approach to represent them as multimodal frames, by integrating multisensory data. Our method focuses on the extraction, analysis, and integration of multimodal features from visual art material tagged with the concepts of interest. We define a conceptual model and present a novel ontology for formally representing social concepts as multimodal frames. Taking the Tate Gallery's collection as an empirical basis, we experiment our method on a corpus of art images to provide a proof of concept of its potential. We discuss further directions of research, and provide all software, data sources, and results.
Comments: First International Workshop on Multisensory Data and Knowledge at the 3rd Conference on Language, Data and Knowledge (2021)
Subjects: Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL); Digital Libraries (cs.DL); Social and Information Networks (cs.SI)
Cite as: arXiv:2110.07420 [cs.CV]
  (or arXiv:2110.07420v1 [cs.CV] for this version)

Submission history

From: Delfina Sol Martinez Pandiani [view email]
[v1] Thu, 14 Oct 2021 14:50:22 GMT (5027kb,D)

Link back to: arXiv, form interface, contact.