We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

eess.IV

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Electrical Engineering and Systems Science > Image and Video Processing

Title: Generalized Radiograph Representation Learning via Cross-supervision between Images and Free-text Radiology Reports

Abstract: Pre-training lays the foundation for recent successes in radiograph analysis supported by deep learning. It learns transferable image representations by conducting large-scale fully-supervised or self-supervised learning on a source domain. However, supervised pre-training requires a complex and labor intensive two-stage human-assisted annotation process while self-supervised learning cannot compete with the supervised paradigm. To tackle these issues, we propose a cross-supervised methodology named REviewing FreE-text Reports for Supervision (REFERS), which acquires free supervision signals from original radiology reports accompanying the radiographs. The proposed approach employs a vision transformer and is designed to learn joint representations from multiple views within every patient study. REFERS outperforms its transfer learning and self-supervised learning counterparts on 4 well-known X-ray datasets under extremely limited supervision. Moreover, REFERS even surpasses methods based on a source domain of radiographs with human-assisted structured labels. Thus REFERS has the potential to replace canonical pre-training methodologies.
Comments: Accepted by Nature Machine Intelligence. The official version is at this https URL Codes are available at this https URL
Subjects: Image and Video Processing (eess.IV); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
DOI: 10.1038/s42256-021-00425-9
Cite as: arXiv:2111.03452 [eess.IV]
  (or arXiv:2111.03452v2 [eess.IV] for this version)

Submission history

From: Hong-Yu Zhou [view email]
[v1] Thu, 4 Nov 2021 14:28:22 GMT (8517kb,D)
[v2] Thu, 27 Jan 2022 03:41:54 GMT (14748kb,D)

Link back to: arXiv, form interface, contact.