A novel and compact Efficient Multi-Granularity Knowledge Transfer (\textbf{EMGKT}) method to address the above issues and develops a mixture of disease diagnosis expert classifiers to enhance knowledge extraction.
Abstract
Radiology report generation can automatically generate clinical descriptions from X-ray images, thereby significantly improving the efficiency of radiologists. This task is challenging because it requires medical knowledge to accurately identify diseases and describe them in a professional manner. However, existing methods often overlook the importance of enhancing medical knowledge in describing pivotal areas, a capability that requires models to effectively extract and aggregate knowledge at multiple levels of granularity. Accordingly, we herein propose a novel and compact Efficient Multi-Granularity Knowledge Transfer (\textbf{EMGKT}) method to address the above issues. First, we encode global knowledge embeddings using a medical vision-language model, which provides contextual medical knowledge. Moreover, we devise a novel Fine-Grained Knowledge Distillation (FGKD) training task which efficiently extract fine-grained knowledge. Specifically, the FGKD training task contains teacher embeddings and student embeddings. Teacher embeddings are encoded using extra priors; while student embeddings are learned from the teacher embeddings through knowledge distillation. During inference, the student embeddings are used to enhance fine-grained knowledge while the teacher embeddings are discarded, resulting in negligible computational costs and no need for extra priors. Finally, we further develop a mixture of disease diagnosis expert classifiers to enhance knowledge extraction. The classifiers are initialized using disease embeddings and are modeled as different experts to address various granularity features. Notably, \textbf{EMGKT} can be efficiently applied to most existing methods. Extensive experiments are conducted on two widely-used public datasets and various baselines, which demonstrates the effectiveness and transferability of \textbf{EMGKT}.
Computed tomography (CT) provides diverse patient information and various clinical tasks, for which numerous deep learning models have been investigated. Recent years have seen the development of vision-language foundation models that can comprehensively extract clinical features, driven by representation learning that...
RG-ICL is introduced, a training-free inference framework that retrieves query-aligned demonstrations using frozen encoders, without task-specific parameter updates, and indicates that for medical image interpretation, curating which reference cases an MLLM sees is a practical alternative to retraining it.
Min-Da Zhao, Fang-Yu Hu, Yan Luo et al.· arXiv.org· 0 citations
A Concept Clause Decomposition method is designed to extract semantically complete descriptions of pathological findings or radiology manifestations from medical reports as medical concept clauses, which are then utilized within a multi-granularity cross-modal alignment framework to enhance medical concept perception a...
Xiang-Min Kong, Xi-Bin Jia, Da-Wei Yang et al.· IEEE journal of biomedical a...· 0 citations
Chest X-ray report generation systems are valuable for assisting disease diagnosis and improving healthcare efficiency. However, existing methods still face two key challenges. First, multiple diseases often co-occur, leading to a combinatorial explosion of label combinations and sparse supervision for learning a gener...
Hong-Ze Zhu, Hong Liu, Ya-Wen Huang et al.· IEEE Transactions on Medical...· 0 citations
Radiology report generation (RRG) aims to convert medical images into diagnostic texts to assist in clinical decision-making and alleviate the workload of physicians. Although existing methods have made extensive progress in cross-modal interaction and the incorporation of external priors, the distribution shift of und...
Er-Jian Zhang, Jia-Yuan Ma, Lie-Jun Wang et al.· 0 citations
Automated radiology report generation requires models to do more than produce fluent clinical narratives: they must explicitly state supported diagnoses while avoiding unsupported ones. However, existing medical vision-language models (MedVLMs) are typically trained with token-level maximum likelihood objectives, which...
Ignacy Kolton, Wen-Xuan Li, Zong-Wei Zhou et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.