Hyperparameter Trade-offs in Fine-Tuning DistilBERT for Text Classification
How the fine-tuning hyperparameters of a DistilBERT text classifier -- learning rate, batch size, and number of epochs -- jointly and non-linearly determine accuracy, F1-score, and loss. Polynomial regression in absolute and relative-to-baseline frameworks reveals that optimizing one metric typically trades off against others and that the epoch-by-batch-size interaction is decisive for F1, so hyperparameters must be tuned together rather than in isolation.
2501.00241
An empirical study of how three fine-tuning hyperparameters -- learning rate, batch size, and number of epochs -- affect a DistilBERT text classifier (the distilbert-base-uncased-finetuned-sst-2-engl…