SAFER: Layer-Selective Sharpness-Aware Fine-Tuning for Robust Vision Transformers
A defense that makes vision transformers more adversarially robust while curbing the adversarial overfitting caused by their large parameter counts. Instead of fine-tuning the whole network, SAFER identifies the small subset of layers most susceptible to overfitting and applies sharpness-aware minimization (seeking flat, low-sharpness loss regions that generalize better) to only those layers, freezing the rest. This layer-selective sharpness-aware fine-tuning consistently improves both clean and adversarial accuracy over baseline defenses, with typical gains around 5% and up to 20% across multiple ViT architectures and datasets.
SAFER: Sharpness Aware layer-selective Finetuning for Enhanced Robustness in vision transformers
A method for making vision transformers (ViTs) more robust to adversarial attacks while reducing the adversarial overfitting that their large parameter counts induce during adversarial training. Rath…