Multiple linear regression can be decomposed into a sequence of univariate regressions: when the input columns are mutually orthogonal, each coefficient is simply the univariate regression of the response on that column, and when they are not, successive orthogonalization (a Gram-Schmidt procedure) replaces each predictor by its residual after regressing out all previously considered predictors. This construction gives each coefficient its formal interpretation as the contribution of that variable to the response after adjusting for all other inputs, and it exposes the ill-conditioning that arises when predictors are nearly collinear, since the orthogonal residual becomes small and its reciprocal inflates the estimate. Expressed in matrix form the same procedure yields the QR decomposition of the design matrix into an orthonormal basis and an upper-triangular reconstruction matrix, connecting regression in statistics and machine learning to standard numerical linear algebra; because the derivation uses only inner products, it generalizes to any inner-product space.
Multiple linear regression can be decomposed into a sequence of univariate regressions: when the input columns are mutually orthogonal, each coefficient is simply the univariate regression of the res…