The least squares method provides a mechanism to solve inconsistent linear systems $Ax = b$ by minimizing the Euclidean norm $\|Ax - b\|$ between two vectors, which corresponds to finding the orthogonal projection of $b$ onto the column space of matrix $A$. This concept relies on fundamental properties from linear algebra and functional analysis regarding norms in Hilbert spaces, specifically utilizing definitions involving vector orthogonality and subspace projections. It serves as a critical theoretical bridge within applied mathematics for handling overdetermined systems where exact solutions do not exist, ensuring minimal error estimation under specific convexity conditions.
The core principle is that for any matrix $A$ with linearly independent columns, there exists a unique symmetric orthogonal projection matrix $P = A(A^TA)^{-1}A^T$ mapping vectors onto the column spa…