Generally speaking, can RMSE be smaller than MAE?
I am calculating RMSE and MAE for my results. In two out of five methods, the RMSE is smaller than MAE. Note that I am using the same data, the same script, and the same code to calculate RMSE and MAE. The only difference is the algorithm to create the models.
I have been over my script several times and everything is ok with the code.
Thank you.
We should normally have $$MAE \leq RMSE$$
This is a consequence of the triangle inequality (in a similar way as 'square of the mean values < mean of the squared values')
$$ \overbrace{\left(\frac{\vert r_1 \vert + \vert r_2\vert + \dots + \vert r_n \vert}{n} \right)^2}^{MAE^2} \leq \overbrace{\frac{\vert r_1 \vert^2 + \vert r_2\vert^2 + \dots + \vert r_n \vert^2}{n}}^{RMSE^2}$$
We can derive this more explicitly as following by expressing the absolute value of the error terms $\vert r_i \vert$ as a sum of two components: the mean of the absolute value of the error terms and the variation relative to that mean, $\vert r_i \vert = \mu + \delta_{i}$.
$$\begin{array}{} \overbrace{(r_1^2 + r_2^2 + \dots + r_n^2)}^{n \cdot RMSE^2} &=& \vert r_1 \vert^2 + \vert r_2\vert^2 + \dots + \vert r_n \vert^2 \\ & = & (\mu+\delta_1)^2 + (\mu+\delta_2)^2 +\dots + (\mu+\delta_n)^2\\ & = & n \mu^2 + 2 \mu (\delta_1 + \delta_2 +\dots + \delta_n) + \delta_1^2 + \delta_2^2 + \dots + \delta_n^2\\ & = & n \mu^2 + \delta_1^2 + \delta_2^2 + \dots + \delta_n^2 \geq \underbrace{n\mu^2}_{n \cdot MAE^2} \end{array}$$
The step where $2 \mu (\delta_1 + \delta_2 +\dots + \delta_n)$ is removed comes from $\overline{\delta_{\vert r_i \vert}}=0$. Note that this must be the case for $\mu$ to be the mean of ${\vert r_i \vert}$, since $\overline{\vert r_i \vert} = \overline{\mu + \delta_i} = \overline{\mu} + \overline{\delta_i} = \mu + \overline{\delta_i}$)
The equality arises when you have all $\delta_i=0$, this is the case when $\vert r_i \vert = \mu$, or when $r_i = \pm \mu$.
Normally you compute the mean operation in RMSE and MAE by division with $n$ and then you get the inequality
$$\frac{(r_1^2 + r_2^2 + \dots + r_n^2)}{n} \geq \left( \frac{\vert r_1\vert + \vert r_2\vert + \dots + \vert r_n\vert }{n} \right)^2$$
But it is possible that you use a division by $n-1$ or $n-p$. And you compare
$$\frac{(r_1^2 + r_2^2 + \dots + r_n^2)}{n-p} \quad\text{versus} \quad \left( \frac{\vert r_1\vert + \vert r_2\vert + \dots + \vert r_n\vert }{n-p} \right)^2$$
In this case it is possible that the right side is larger than the left side. Maybe this is the case for your code where you mention different methods to compute MAE and RMSE.
Example: let $r_1 = r_2 = 1$ and $r_3 = r_4 = -1$, let $n=4$ and $p=1$, then
$$\begin{array}{rcccl} RMSE &=& \sqrt{\frac{1^2+1^2+(-1)^2+(-1)^2}{3}} &=& \sqrt{\frac{4}{3}} &\approx& 1.1547 \\ MAE &=& \frac{\vert 1 \vert+\vert 1 \vert+\vert -1 \vert+\vert -1 \vert}{3} &=& {\frac{4}{3}} &\approx& 1.3333 \end{array}$$
As a complement to Sextus' answer (+1), here is how I convinced myself that RMSE >= MAE:
We know that $$ \mathbf{Var}\left[f(X)\right] = \mathbf{E}\left[f(X)^2 \right] - \mathbf{E}\left[f(X) \right]^2 \geq 0 $$
If we use this fact with $f$ as the absolute value function and the model's residuals (errors) as our $X$, we get
$$ \mathbf{E}\left[\text{abs}(\text{error})^2 \right] \geq \mathbf{E}\left[\text{abs}(\text{error}) \right]^2 $$ $$ \mathbf{E}\left[\text{error}^2 \right] \geq \text{MAE}^2 $$ $$ \text{MSE} \geq \text{MAE}^2 $$ $$ \text{RMSE} \geq \text{MAE} $$
An example where they are equal: we have errors $(-1, -1, 1, 1)$, and our RMSE and MAE are both equal to 1. This should happen whenever $\mathbf{Var}\left[\text{abs}(\text{error})\right] = 0$. Another example would be errors $(-0.5, -0.5, 0.5)$, in which case our RMSE and MAE are both $0.5$.
In plain English, it is not possible that the MAE > Standard Error. The MAE is essentially fairly close to the Median absolute error (the 50th percentile). The Standard Error is around the 68th percentile. Both statements, assume that the errors in your model are not too far off from being normally distributed. If your model is fairly well specified, it is quite likely.
