2

I have a problem of estimating some kind of a test-retest reliability/intraclass correlation of the regression coefficients. I have only seen test-retest reliability apply to outcome variables, simply calculating a ratio of between-variance to the total (outcome variable measured at two time points, for example). We have a problem where the regression coefficients are of interest and we would like to calculate the reliability of the estimated regression coefficients. The experimental setup has two time points - test and re-test, and the model has multiple predictors with a binary dependent variable.

The approach I have seen is estimating the regression coefficients at each time point in the first, then treating the estimated coefficients as outcome and estimating the between- and within variance components in the second stage. I am not quite ok with this approach, since the second stage analysis ignores the uncertainty in the estimated coefficients.

Any thoughts on what the right approach should be?

user151310
  • 91
  • My understanding of reliability is that it is related to the consistency or precision of a measurement that is repeated under the same or similar conditions. As you note, the psychometric literature is focused on outcomes, not coefficients. While I would be interested in any papers that formally develop an approach to the reliability of regression coefficients, I'm not aware of any. This doesn't mean that they don't exist. – user78229 Nov 18 '15 at 15:11
  • 1
    In their absence, why not develop a defensible heuristic of your own? For instance, by leveraging a bootstrapped or random forests-like, "divide and conquer" approach to model building, you can build up a distribution of how the coefficients change as a function of the underlying random data resampling. A coefficient of variation would provide evidence towards a measure of their consistency and variability as a function of those changes. – user78229 Nov 18 '15 at 15:11
  • Thanks for the proposed solution. I agree that parametric bootstrap does sound like a viable option. I looked into the measurement error literature, and essentially the second stage analysis (ICC) would be in the realm of analysis when the outcome variable (regression coefficients) have measurement error. A book "Measurement error: models methods and applications" by Buonaccorsi talks about the application of parametric bootstrap in such cases. The parametric bootstrap can be done sampling from the parameter means and variances obtained from the first stage regressions. – user151310 Nov 19 '15 at 15:45
  • A nice side effect is that I would then get a bootstrap sample of the ICCs, which would allow me to get the confidence intervals as well. – user151310 Nov 19 '15 at 15:45

0 Answers0