Embedding-Dimension Reduction for Memory-Efficient Transformers in Deep Learning
A model-compression strategy for transformer networks that lowers memory footprint and inference latency by halving the embedding dimension and reducing the number of attention heads directly in the architecture, rather than relying only on post-training pruning or quantization. Students learn how embedding size and head count drive a transformer's parameter count and per-layer compute cost, and how trading them off enables deployment on resource-constrained edge devices with modest accuracy loss.
2501.00042
A modified transformer encoder architecture aimed at deployment on resource-constrained edge devices, which shrinks the model by halving the embedding dimension and reducing the number of attention h…