Context Length and Attention-Free Operators for Biomedical Image Analysis
A systematic study showing that model performance on biomedical imaging tasks (segmentation, denoising, classification) depends strongly on transformer context length, tuned via Vision Transformer patch size and Swin Transformer window size, and that sub-quadratic attention-free operators such as Hyena and the Mamba state-space model match self-attention accuracy while cutting compute by over 80%, making long-context transformers practical on high-resolution medical images.
2501.00619
An empirical study of how input context length affects deep-learning models for biomedical image analysis and whether attention-free operators can replace self-attention efficiently. Because the self…