References & Citations
Computer Science > Computer Vision and Pattern Recognition
Title: Lesion Guided Explainable Few Weak-shot Medical Report Generation
(Submitted on 16 Nov 2022 (v1), last revised 17 Nov 2022 (this version, v2))
Abstract: Medical images are widely used in clinical practice for diagnosis. Automatically generating interpretable medical reports can reduce radiologists' burden and facilitate timely care. However, most existing approaches to automatic report generation require sufficient labeled data for training. In addition, the learned model can only generate reports for the training classes, lacking the ability to adapt to previously unseen novel diseases. To this end, we propose a lesion guided explainable few weak-shot medical report generation framework that learns correlation between seen and novel classes through visual and semantic feature alignment, aiming to generate medical reports for diseases not observed in training. It integrates a lesion-centric feature extractor and a Transformer-based report generation module. Concretely, the lesion-centric feature extractor detects the abnormal regions and learns correlations between seen and novel classes with multi-view (visual and lexical) embeddings. Then, features of the detected regions and corresponding embeddings are concatenated as multi-view input to the report generation module for explainable report generation, including text descriptions and corresponding abnormal regions detected in the images. We conduct experiments on FFA-IR, a dataset providing explainable annotations, showing that our framework outperforms others on report generation for novel diseases.
Submission history
From: Jinghan Sun [view email][v1] Wed, 16 Nov 2022 07:47:29 GMT (451kb,D)
[v2] Thu, 17 Nov 2022 06:48:04 GMT (449kb,D)
Link back to: arXiv, form interface, contact.