The classification problem is the supervised learning setting in which the output G is drawn from a finite discrete space 𝒢 of cardinality K while the input lies in R^p, and its loss is specified not by squared error but by a K × K loss matrix L whose (k, l) entry is the cost of predicting class l when the true class is k; the 0-1 loss is the special case with zeros on the diagonal and ones everywhere else. Minimizing the expected prediction error pointwise — conditioning on X and, because the output distribution is discrete, writing the inner expectation as a finite sum of L(k, f̂(x)) weighted by P(k | X = x) — yields the Bayes optimal classifier: predict the class with the highest conditional probability P(G = k | X = x), since under 0-1 loss that choice zeroes out the largest term in the sum. Because the conditional distribution is unknown it must be estimated, and estimating it by class frequencies among the k nearest neighbours and taking the maximum reduces exactly to the majority-vote k-nearest-neighbour classifier, while encoding the classes numerically and fitting a linear regression estimates P(G = 1 | X) directly and thresholds it, showing classification and regression to be instances of one formulation.
The classification problem is the supervised learning setting in which the output G is drawn from a finite discrete space 𝒢 of cardinality K while the input lies in R^p, and its loss is specified not…