Cross-Layer Cache Aggregation for Token Reduction in Vision Transformers
Two plug-and-play modules that let a Vision Transformer keep very few image-patch tokens while preserving accuracy on ultra-fine-grained recognition. A Cross-Layer Aggregation classification head feeds intermediate-layer features directly into the classifier, and a Cross-Layer Cache stores and later retrieves features that token reduction would otherwise discard, recovering fine-grained information lost when the token keep-rate is pushed as low as 10%.
2501.00243
Ultra-fine-grained image recognition (UFGIR) classifies images into sub-categories finer than species -- for example cultivars of a plant -- where inter-class differences are tiny and intra-class var…