Evaluation in supervised learning must distinguish the objective function optimised during training (squared error, cross entropy, regularization penalties, impurity criteria such as the Gini index) from the measure used to judge the resulting model, whose proper target is expected performance over the unknown underlying joint data distribution P(X, Y) rather than over the training sample. For classification the principled evaluation measure is the 0-1 loss, i.e. misclassification error, and for regression the squared error; adopting the training objective as the evaluation measure biases comparisons in favour of the method that optimises it. Because the distribution is accessible only through finite samples, evaluation asks two distinct questions — how good a particular fitted parameter set is, and how good the parameter-finding procedure is on unseen data — and answers them by resampling schemes (multiple training sets, cross validation, bootstrap) whose averaging reduces the variance of the error estimate relative to a single train/test split.
Evaluation in supervised learning must distinguish the objective function optimised during training (squared error, cross entropy, regularization penalties, impurity criteria such as the Gini index) …