UMed-LVLM is a medical large vision-language model built to unveil and localize abnormalities in medical images, a capability existing Med-LVLMs handle poorly. Its novel contribution is a data-plus-training recipe: a Medical Abnormalities Unveiling (MAU) dataset created by prompting GPT-4V to produce diagnoses conditioned on identified abnormal regions, and a two-stage procedure combining Abnormal-Aware Instruction Tuning with Abnormal-Aware Rewarding, whose Relevance, Abnormal-Localization, and Vision-Relevance rewards push the model to ground its answers in the actual pathological regions. The result improves abnormality identification by 58% over baseline and, notably, shows that sharpening abnormality detection also raises general medical-image understanding and cross-dataset generalization.
This paper improves Medical Large Vision-Language Models (Med-LVLMs) at localizing and interpreting abnormalities in medical images, a known weakness of existing models. The authors build UMed-LVLM v…