Do repeated measures affect point estimates of predicted probabilities or AUC from logistic regression?

Question

I understand the presence of repeated measures affects standard errors and hence inferences using p-values, confidence intervals, etc.

However, do repeated measures affect the point estimates of predicted probabilities or the summary measure of accuracy using AUC-ROC (c statistic)? That is, do they affect prediction if that is a singular goal of the modeling?

I would think repeated measures could affect the regression coefficients which then will affect the predicted probabilities - but I have not seen extensive discussion of this. Thank you.

Answer 1

Yes, repeated measurements do "affect prediction (even) if that is a singular goal of the modeling".

Repeated measurement matter because they reflect a structure in the underlying data. They effectively impose a clustering/nesting that we need to account for. If we do not account for this clustering we will have a two-fold issue:

1. Our estimates about performance will be optimistically biased. We will unintentionally leak information about the within-cluster structure. Especially if we have longitudinal data this might be outright wrong as our training data might incorporate information for future values of our test data.
2. Our predictions of unseen data will be suboptimal in terms of generalisability. We will use our model in a setting other than the one it has trained upon. Deploying the model will be problematic. New "clusters" will appear but our model was always taught to rely on some "cluster-specific" information (instead of learning how to generalise the known "cluster-specific" patterns to new unseen clusters), and in absence of this information it will not do a good job.

Please note that the fix of this situation, in most cases, is relatively easy. We will treat as a unit of analysis, not an individual measurement, but rather the cluster of repeated measurements. If you have repeated measurements that are longitudinal, there are some more specific algorithms that might want to explore (I give more information about them in the thread: Xgboost and repeated measures) but at first instance, sampling at "cluster"-level should be adequate to get a more realistic AUC-ROC value. Finally, it goes without saying that standard approaches for clustered data like GLMMs should always be considered too.

Answer 2

I would hope that taking repeated measures will result in a reduction of the MSE and since, for example, a formula for the prediction interval for a simple regression model can be expressed as (per a reference):

$ y_k \text{ (per fitted regression model at } x_k) \text{ +/- }\text{ } t_{(\alpha/2,n-2)} \sqrt{MSE(1+ \frac{1}{n}+ \frac{(x_k - x_m)^2 }{\sum(x_i - x_m)^2})}$

So the prediction interval could be, accordingly, reduced as well (as n is increased and the MSE may be lower).

The comment, and a more generalized version of the formula, can at least be extended to ANOVA models which can be presented and solved in a regression model format.

[EDIT] Did find a reference on repeated measures regression model, to quote:

Repeated measures regression exists, but isn't generally a very good model (e.g., because it eats up degrees of freedom estimating slopes for each person).

I would suggest a multilevel model implemented in the linear mixed model commands in SPSS. Another option is generalised estimating equations (also implemented in SPSS). See linked examples.

http://www.ats.ucla.edu/stat/spss/library/gee.htm http://www.ats.ucla.edu/stat/spss/topics/MLM.htm

Per this reference, my opinion now is that the prediction error question can likely be very case-specific (relating to loss in degrees of freedom), but per my prior comments above, the prediction error should be quantifiable. Further, I would not be surprised if there are experimental designs with repeated measures, that are likely well represented by the computed prediction variance. The latter can be verified by attempting to set up a representative experimental design with known parameters and employing Monte Carlo simulation of, say, random deviates from a Normal distribution, and tabulating the distribution of prediction errors, to compare to the theoretical expected.