1

Just curious to hear any thoughts on weighting over prediction in mean absolute error to minimize the penalty since I'm more interested in under prediction, if that makes sense. Basically, I'm forecasting to find optimal staffing needs so I'm not as worried if the model over predicts too much. On the other hand, I do care about under prediction.

I have basically times the absolute error by half (totally arbitrary) to minimize the penalty on over predictions. I'm more seeking to discuss rather than find a solution since I've already implemented something.

theduker
  • 13

2 Answers2

0

The good news: if you use the MAE, chances are good you are already under-predicting. Specifically, the MAE elicits the conditional median, not the mean, and if your series has a skewed conditional distribtion (usually a good bet), the median is already lower than the mean, and your MAE-optimal forecast is likely lower than an unbiased expectation forecast. (Note that this is different from "underforecasting more often than overforecasting".) This may be of interest.

To find an optimal error measure, you would need to specify more precisely what a "good" underforecast is (Kolassa, 2020). If you want the conditional median (which, per above, is probably already lower than the conditional expectation), then stick with the MAE - no need to scale it. If your data are nonzero, you can use the Mean Absolute Percentage Error (MAPE), which is typically optimized by heavily biased forecasts. I personally have never seen a business use that would be best served by a MAPE-optimal forecast. If you want an underforecast of the form "75% of the expectation", then use the (Root) Mean Squared Error to elicit unbiased expectation forecasts, then multiply these by $0.75$.

The most common approach to getting point forecasts that are not measures of tendency per se would be quantile forecasts. You could aim for a 40% quantile forecast, for instance. (Note that if you have weirdly skewed observations, such a 40% quantile forecast can still be higher than an unbiased expectation forecast.) To evaluate such a quantile forecast for a given quantile level, use the pinball or hinge loss, see also Gneiting (2011).

Stephan Kolassa
  • 123,354
0

You should ideally look up the economics of the situation involved.

What's the actual financial cost of being understaffed by one? Is it a schedule slippage of one month, which cases late entry to market, which decreases lifetime profits by $100,000?

What about understaffing by two? Maybe the problem compounds and you get four months delayed, at a cost of $600,000. Do this for a few points and linearly interpolate.

The cost of overstaffing is usually easier to estimate: it's the fully loaded cost of staff times the error.

As you can guess, these error functions can get quite detailed and complicated. They may even teach you a thing or two about the problem you're trying to solve.

As a word of caution: don't make the error function in the model too complicated. Usually, the prediction uncertainty itself is so large that you can tolerate significant imprecision in the error function.

kqr
  • 814