Deep-learning image classifiers can be fooled by small adversarial perturbations, and such perturbations often transfer to models the attacker never accessed - but transfer is weak across architectures, for example from a CNN surrogate to a Vision Transformer. This paper proposes Spatial Adversarial Alignment (SAA), which fine-tunes the surrogate model to share features with a separate witness model so that attacks target architecture-agnostic common features. SAA combines two alignment terms: spatial-aware alignment minimizes global and local feature divergence between the surrogate and the witness, and adversarial-aware alignment adds a self-adversarial constraint that aligns features on adversarial inputs as well. Adversarial examples crafted on the SAA-aligned surrogate transfer substantially better across diverse ImageNet models, including CNN-to-ViT settings, than examples from standard surrogates. Accepted at NeurIPS 2025.
Deep-learning image classifiers can be fooled by small adversarial perturbations, and such perturbations often transfer to models the attacker never accessed - but transfer is weak across architectur…