Subset selection addresses the principal weakness of ordinary least squares: while the least-squares fit is unbiased when the linear model is correct, it can carry high variance, and constraining some coefficients to zero trades a small increase in bias for a substantial reduction in variance by shrinking the number of estimated parameters. The two motivations are prediction accuracy and interpretability — a model expressed in a handful of retained variables is intelligible in a way that one with hundreds of coefficients is not. Best subset selection evaluates every candidate subset exhaustively and is combinatorial in the number of predictors, offering no inclusion property that would let smaller optima be extended into larger ones; forward stepwise selection is its greedy approximation, beginning from the intercept and admitting at each stage the variable that most improves fit, with backward and hybrid add/drop variants — the topic belongs to supervised learning in machine learning, specifically model selection for linear regression.
Subset selection addresses the principal weakness of ordinary least squares: while the least-squares fit is unbiased when the linear model is correct, it can carry high variance, and constraining som…