Refers to a general estimation technique that selects the parameter value to minimize the squared difference between two quantities, such as the observed value of a variable, and the expected value of that observation conditioned on the parameter value. Gaussian linear models are fit by least squares and least squares is the idea underlying the use of mean-squared-error (MSE) as a way of evaluating an estimator.
Overview
Refers to a general estimation technique that selects the parameter value to minimize the squared difference between two quantities, such as the observed value of a variable, and the expected value of that observation conditioned on the parameter value. Gaussian linear models are fit by least squares and least squares is the idea underlying the use of mean-squared-error (MSE) as a way of evaluating an estimator.
Formulation
Given a set of data $(x_1,y_1),...,(x_n,y_n)$ where $x_i \in \mathbb{R}^{p}$ and a vector of coefficients $\beta$, the least squares estimate is the solution to the equation:
$$\widehat{\beta}_{LS} = \underset{\beta} {\text{arg min}} \sum\limits_{i=1}^{n}(y_i - \sum\limits_{j=1}^{p}x_{i,j}\beta_{j})^2 = || {\bf y - X\beta}||^2$$
Using linear algebra, one can find the least squares hyperplane:
$$ {\bf \widehat{\beta} = (X^TX)^{-1}X^{T}y} $$
References
Least squares methods are treated in many introductory statistics resources and textbooks, but there are also advanced resources dedicated only to the subject, for example:
- Data Analysis Using the Method of Least Squares by John Wolberg
- Numerical Methods for Least Squares Problems by Åke Björck (for computational perspectives)
