The residuals of a model are the actual values minus the predicted values. Many statistical models make assumptions about the error, which is estimated by the residuals.
In the context of regression, the $i^{th}$ residual is defined as:
$$\epsilon_i = y_i - \hat{y}_i$$
Where $y_i$ is the actual value of the $i^{th}$ observation and $\hat{y}_i$ is its estimated or fitted value.
The sum of residuals in a least squares regression is 0, i.e. $\sum_i \epsilon_i = 0$. And the goal of least squares is to find the $\beta$ which minimizes the sum of squared residuals (SSR), i.e.
$$\hat{\beta}_{OLS} = \underset{\beta} {\text{argmin}} \sum_i \epsilon_i^2 = \underset{\beta} {\text{argmin}} \sum_i ( y_i - \sum_j x_{ij}\beta_j)^2$$
