Predict probability of outcome from continuous variables

Question

We have a model which predicts the start time of an event (lets call it predicted_start). We also have a default start time for an event (lets call it default_start), but it's usually not correct and that's why we made the model to predict a more correct start time.

The model is doing great, but sometimes it's wrong and predicted_start differs greatly from the actual start (lets call it actual_start). Also, sometimes it's right and predicted_start can differ greatly from default_start and still be correct.

It would be nice to know the probability of prediction_start being correct and close to actual_start. It's not a random guess and there has to be a probability distribution somewhere ... right? This validation would also probably be dependant on offset from defualt_start and maybe previous event's offset from defualt_start, not sure, maybe this doesn't need to be that complicated?

Can't really wrap my head around this and would greatly appreciate any pointers.

EDIT: I have considered logistic regression of some sort, but was hoping someone knew a better solution.

Answer 1

What about predicting the "corrections" for the default time? If $y$ is your time to predict, and $\tilde y$ the "default" time you know, so instead of predicting $y$, you would predict $y - \tilde{y}$ (i.e. $\tilde{y}$ would be an offset variable in regression).

If this doesn't work, you can use logistic regression, as you mentioned, or use a single model for this. The model could be something like

$$ y = \pi \hat y + (1-\pi) \tilde y + \varepsilon $$

where $\pi \in [0, 1]$. So $\pi$ would tell you about models' confidence about the prediction $\hat y$. By doing this in single model, it could just learn when to use $\tilde y$, and when not care for those values and use $\hat y$ values. This would simplify the task for predicting $\hat y$ as well.

To predict $\hat y$ and $\pi$ you could modify the model to

$$ (\mu_1, \mu_2) = f(\mathbf{X}) \\ \hat y = \mu_1, \qquad \pi = \sigma(\mu_2) $$

where $f$ is some model, and $\sigma$ is sigmoid function.

The simplest case for $f$ could be linear regression model with $k-1$ features and intercept in $\mathbf{X}$

$$ \overbrace{\boldsymbol{\mu}}^{(n \times 2)} = \overbrace{\mathbf{X}}^{(n \times k)} \overbrace{\boldsymbol {\beta}}^{(k \times 2)} $$

But if you want to consider changes over time, you could use something like

$$ (\mu_1, \mu_2) = \mathsf{LSTM}(\mathbf{X}) $$

where $\mathsf{LSTM}$ is LSTM-based recurrent neural network.

If you cannot incorporate the model that makes the $\hat y$ predictions into single model, as described above, you can build higher-level model that would either choose between predictions $\hat y$ and the default values $\tilde y$, or compute weighed average of them. In the first case, you would use classifier like logistic regression, or random forest, to make the choice. Alternatively, you may build a model that would learn to weight the two outcomes by $\pi$ weights, as described above. Such model would predict the weights, and train it by minimizing loss (e.g. squared) between weighed mean of the predicted and default values, and the true values.