A model-compression approach in which entire transformer encoder layers are removed from a pre-trained BERT model (top, middle, or bottom pruning) and the smaller model is fine-tuned for downstream text classification. Empirically, for low-resource languages, middle-layer pruning of a monolingual BERT retains accuracy competitive with the full model at 50-80% size reduction and outperforms same-size models trained from scratch, making pruning a faster, cheaper route to compact classifiers than training small models from scratch.
An empirical study of layer pruning as a model-compression method for BERT-based text classifiers in the low-resource Marathi language. Whole transformer encoder layers are removed from a pre-trained…