Unsupervised Shortcut Detection and Mitigation in Vision Transformers
An unsupervised pipeline that finds and suppresses shortcut features (spurious cues a model exploits instead of the true signal) in a vision transformer without group labels or retraining. It clusters the transformer's patch activations to isolate a candidate shortcut cluster, extracts prototypical patches, and uses a multimodal LLM to caption them so a human can name the spurious feature; at inference the shortcut is removed by ablating the offending image patches. Demonstrated to lift worst-group and average accuracy on medical-imaging and ImageNet-W benchmarks while running on consumer hardware.
Efficient Unsupervised Shortcut Learning Detection and Mitigation in Transformers Lukas Kuhn∗1,3,4
Introduces an unsupervised pipeline for finding and removing shortcuts, the spurious image features (e.g. a patient's age or a ruler in a skin-lesion photo) that a vision transformer latches onto ins…