Constant pull of gravity
There's a "constant frictional force" as well. In fact, the magnet stays stuck due to friction, not directly magnetic force. The magnetic force will exert a normal force $N$ on the fridge (and vice versa by action reaction). The fridge will exert an upwards static frictional force $\leq \mu_sN$, where $\mu_s$ is the coefficient of static friction for the magnet-fridge interface. $\mu_s$ should be large enough so that $\mu_sN\geq mg$, so the frictional force will be sufficient to keep the magnet steady.
Now let's talk about it falling off. There are two ways this can happen: Either $\mu_s$ reduces due to smoothing of the surface, or $N$ reduces due to demagnetization. Demagnetization can slowly happen whenever the magnet undergoes some change, like rusting, being heated, or being jarred. It can also happen if the magnetic field in the area abruptly changes. In this case, the abrupt change is the one that happens in the Earth's magnetic field, but before that happens, the magnet will most probably have rusted or flaked off. I had a normal (iron) magnet which became extremely weak after ten-ish years. So most probably your fridge magnet will fall off before the Earth gets swallowed up. I'd give it a few decades, since it's a neodymium magnet and it's not really exposed to the air.