Combining Forecasts: Best Information to Solicit from Forecasters?

Question

Suppose

• Statistician $m=1$ produces a set of $h$-step-ahead point forecasts $\hat{x}_{t+h|t, 1}$ of $x_{t+h}$ where $x_{t+h} \in [0,1]$.
• Also, this point forecast could come with:
  • a predictive density $f_{{x}_{t+h|t,1}}(x|x_t,...,x_1)$
  • a predictive error $\hat{\sigma}_{t+h|t, 1}$
  • a $(1-\alpha)\%$ predictive interval $(c_{t+h|t,1,\alpha,lb},c_{t+h|t,1,\alpha,ub})$
  • or no measure of certainty in forecast at all
• After seeing this first forecast $\hat{x}_{t+h|t, 1}$, there are $M$ statistically unsophisticated forecasters with domain knowledge who could each produce their own point forecasts $\hat{x}_{t+h|t, m}$, $m=2,...,M+1$.
• After seeing all $\hat{x}_{t+h|t, m}$, $m=1,...,M+1$, a combined forecast is constructed $\bar{x}_{t+h|t, m} = \sum_{m=1}^{M+1} w_{t+h|t,m}\hat{x}_{t+h|t, m}$ where $w_{t+h|t,m} \geq 0$ and $\sum_{m=1}^{M+1}w_{t+h|t,m}=1 $

Q: What additional piece of info would you solicit from the forecasters $m=2,...,M+1$ to help you construct the optimal forecast combination weights $w_{t+h|t,m}$? How would you use that info to construct the weights?

Suppose additionally,

• A backtested measure of model forecast accuracy (e.g., MAPE) could exist for $m=1$ but does not exists for $m=2,...,M+1$
• These forecasters are statistically unsophisticated, so asking for their $\hat{\sigma}_{t+h|t, m}$ might not be as meaningful as asking for their predictive interval
• These forecasters don't have a lot of time and, so, are unlikely to produce predictive densities for each of their forecasts
• Forecaster $m$ usually only produces a forecast in periods $t$ where they believe the forecast $m=1$ is flawed. So $\hat{x}_{t+h|t, m}$ won't exist for all $t$

I think the leading candidates for additional info are:

• predictive interval for the $m>1$ forecast (which, theoretically, would be constructed independently from the $\hat{x}_{t+h|t,1}$)

• probability that the $m>1$ forecast is closer to the actual than the $m=1$ forecast (i.e., $\Pr\{|\hat{x}_{t+h|t, m}-x_{t+h}| < |\hat{x}_{t+h|t, 1}-x_{t+h}|\}$ where $m>1$)

• probability that the the the actual is greater than the $m=1$ forecast (i.e., $\Pr\{x_{t+h} > \hat{x}_{t+h|t, 1}\})$

But I'm not really sure how one would use this info to construct optimal weights (even less so with the second and third pieces of info.) The top answer to Assigning Weights to an Averaged Forecast has a lot of great suggestions on how to combine forecasts. But they all assume some track record of error or a full predictive distribution, which doesn't exist here. Averaging forecasts and predictive bounds would be so simple. But it really isn't sufficient here. There needs to be a way to down-weight uncertain predictions and, with time, forecasters who've developed a track-record of inaccuracy.