We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

eess.IV

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Electrical Engineering and Systems Science > Image and Video Processing

Title: Vision-Language Pretraining Enables Radiographs and Reports to be Learned without Curation

Abstract: Recent advances in vision-language pre-training have demonstrated astounding performances in diverse vision-language tasks, shedding a light on the long-standing problems of a comprehensive understanding of both visual and textual concepts in artificial intelligence research. However, there have been limited successes in the application of vision-language pre-training in the medical domain, as the current vision-language models and learning strategies for photographic images and captions are not optimal to process the medical data that are usually insufficient in the amount and the diversity. To address this, here we present medical X-VL, a novel model tailored for efficient vision-language pre-training that exploits cross attention in the radiological images and reports' common feature space in a symmetric manner. We experimentally demonstrate that the pre-trained medical X-VL model outperforms the current state-of-the-art models in various vision-language tasks in medical domains. We also demonstrate novel clinical usages in the diagnosis of newly emerging diseases and human error detection, which suggests the potential of the model for widespread applicability in different medical applications.
Subjects: Image and Video Processing (eess.IV); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
Cite as: arXiv:2208.05140 [eess.IV]
  (or arXiv:2208.05140v2 [eess.IV] for this version)

Submission history

From: Jong Chul Ye [view email]
[v1] Wed, 10 Aug 2022 04:35:58 GMT (12638kb,D)
[v2] Fri, 2 Sep 2022 01:03:58 GMT (23725kb,D)

Link back to: arXiv, form interface, contact.