Loss surface visualisation/intuition

Question

I'm trying to wrap my head around a loss surface in pytorch. This is for work, not a homework assignment.

let's say we have a model

y = model(x)
error = y - y_label

The most simple of loss functions, absolute error

   error.abs().mean().backwards()

Looks like this:

The "industry standard" loss function looks like this

(error * error).mean().backwards() # error.pow(2) also works

In my mind, two things are happening here:

1. The errors are being weighted by the magnitude of the errors
2. The error function is now non linear, and the gradient is dependant on the error squared

So my question is: Can anyone tell me (like I'm a 5 year old), what their intuition is about the difference between mse_loss and the function below is?

(error * error.detach()).mean().backwards()

Answer 1

Using your notation, the error $e$ is the difference between the model's predictions $\hat y$ and the true target $y$. $$ \texttt{error} = e = \hat{y} - y $$

The MSE loss function $$ L_\text{MSE} = (\hat y -y)^2 $$ has gradient $$ \frac{ \partial L_\text{MSE} }{ \partial \hat y} = 2(\hat y - y). $$

The proposed loss function

(error * error.detach()).mean().backwards()

is not MSE because error.detach() removes the reference to this error from the autograd graph. This means that, from the perspective of autograd, error.detach() is a constant.

So from the perspective of autograd, we can write the loss as

$$ L_\text{proposed} = (\hat y - y)e $$

where $e$ is a constant that just happens to have the exact value $\hat y - y$. This has gradient $$ \frac{\partial L_\text{proposed}}{\partial \hat y} = e=\hat{y} - y $$ which is not the same as MSE, so using $L_\text{proposed}$ will not produce the same updates, or yield the same model as MSE.