The minimum description length principle selects, among classifiers of acceptable performance, the one requiring the fewest bits to describe — placing model complexity and prediction error on a common informational footing so that the total cost of encoding the model plus encoding its mistakes on the training data can be minimized jointly. Since a more complex model needs more bits to specify but fewer bits to record its errors, and a simpler one the reverse, the principle formalizes the complexity–accuracy trade-off; it can be shown to be a proper Bayesian procedure, related to maximum-likelihood and maximum-a-posteriori estimation, and underlies a family of model-selection criteria. The concept sits in machine learning and information theory, and it introduces the broader point that machine learning is an empirical discipline requiring exploratory data analysis and designed experiments rather than purely analytical solution.
The minimum description length principle selects, among classifiers of acceptable performance, the one requiring the fewest bits to describe — placing model complexity and prediction error on a commo…