A medical vision foundation model that yields clinically meaningful and spatially consistent patch-level representations, providing a stronger visual foundation for downstream medical LVLMs.
Fine-grained visual representations are essential for medical image analysis, particularly when diagnostically relevant evidence is subtle and spatially localized. Modern transformer-based medical vision encoders must therefore learn patch-level representations that are both clinically meaningful and spatially consistent. Without these properties, large vision-language models (LVLMs) operate on an ambiguous visual foundation, limiting their ability to generate clinically reliable and spatially grounded responses.
However, existing training strategies for medical vision encoders rarely achieve both objectives. Image-text alignment provides clinically meaningful supervision primarily at the image level, leaving the spatial localization of diagnostic evidence weakly constrained. In contrast, self-supervised learning promotes spatial consistency but lacks the semantic supervision needed to distinguish visually similar yet clinically distinct regions.
To address this gap, we present LoFi, a medical vision foundation model built on location-aware fine-grained representation learning. LoFi trains a vision encoder with a lightweight large language model under grounding and grounded captioning objectives. Because these objectives require predicting location from clinical text and vice versa, spatial consistency emerges without any explicit patch-level regularization. To enable training at scale, we construct MedG, a large-scale medical grounding dataset of 4.48M image-text-box triplets curated from 84 datasets spanning 7 modalities. Across phrase grounding, visual question answering, and region-based organ classification under perturbations, LoFi consistently outperforms general-purpose and medical vision foundation models as well as state-of-the-art LVLMs.
Reliable medical LVLMs require a vision encoder that yields fine-grained representation: patch-level features that are both clinically meaningful and spatially consistent. Current medical vision foundation models fall into three categories based on their training signal — instruction-tuned, image-text-aligned, and self-supervised — but none achieves both of these properties (Fig. 1).
PCA maps and PaCMAP projections confirm this: the instruction-tuned Lingshu and the image-text-aligned MedSigLIP are spatially inconsistent, whereas the self-supervised RadioDINO is spatially consistent but cannot separate organs of similar intensity, such as the liver and heart (Figs. 2 and 3).
We construct MedG, a large-scale medical grounding dataset of 4.48M image-text-box triplets curated from 84 datasets across 7 modalities. We propose LoFi, trained with a grounding loss that predicts bounding boxes from clinical text and a grounded captioning loss that reverses this — letting spatially consistent, clinically meaningful features emerge jointly without any explicit patch-level constraint (Fig. 4).
LoFi outperforms all prior models across every downstream dataset (Fig. 5). On perception-centric datasets, models with strong spatial consistency (e.g., DINO) perform best among prior work; on semantics-centric SLAKE, models with explicit image-text alignment (e.g., MedSigLIP) score highest. LoFi attains both properties, which explains its consistent gains across dataset types.
LoFi's patch-level features are highly consistent within anatomical structures and effectively separate small targets from surrounding structures (Fig. 6).
Full quantitative results and ablations are in the paper.
If you find LoFi useful, please consider citing:
@article{kang2026location,
title = {Location-Aware Fine-Grained Representation Learning for Medical Vision Foundation Models},
author = {Kang, Myeongkyun and Yang, Yanting and Li, Xiaoxiao},
journal = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
year = {2026}
}
This research is supported, in part, by the NSERC Discovery Grant, NSERC Alliance Grant, and a Canada Research Chair Fellowship (Tri-Agency Canada); Canada CIFAR AI Chair Awards; Genome British Columbia, the Terry Fox Research Institute, and Health Research BC; Google Gemini Research Awards; the IITP grant funded by the Korea government (MSIT); and the NRF grant funded by the Korea government (MSIT).