Using a regression model to make prediction: When to stop?

Question

I computed a simple linear regression model from my experiment measures in order to make predictions. I have read that you should not calculate predictions for points that depart too far from the available data. However, I could not find any guidance to help me know how far I can extrapolate. For example, if I calculate the reading speed for a disk size of 50GB, I guess the result will be close to the reality. What about a disk size of 100GB, 500GB? How do I know if my predictions are close to the reality?

The details of my experiment are:

I am measuring the reading speed of a software by using different disk size. So far I have measured it with 5GB to 30GB by increasing the disk size of 5GB between experiments (6 measures in total).

My results are linear and the standard errors are small, in my opinion.

Answer 1

The term you're searching for is 'extrapolation'. The problem is that no matter how much data you have, and how many intermediate levels you have between your endpoints on disk size (i.e., between 5 and 30), it is always possible that there is some degree of curvature in the true underlying function, that you simply don't have the power to detect. As a result, when you extrapolate far out from the endpoint, what was a small degree of curvature becomes magnified, in that the true function moves further and further away from your fit line. Another possibility is that the true function really is perfectly straight within the range examined, but that there is perhaps a change-point at some distance from the end point in your study. These sorts of things are impossible to rule out; the question is, how likely are they and how inaccurate would your prediction be if they turn out to be real? I don't know how to provide an analytical answer to those questions. My hunch is that 500 is an awfully long way off when the range under study was [5, 30], but there is no real reason to think my hunches are more worthwhile than yours. Standard formulas for computing prediction intervals will show you an expanding interval as you move away from $\bar{x}$, seeing what that interval looks like might be helpful. Nonetheless, you need to bear in mind that you are making a theoretical assumption that the line really is perfectly straight, and remains such all the way out to the $x$-value you will use for the prediction. The legitimacy of that prediction is contingent on both the data & fit, and that assumption.

Answer 2

Let me add a few points to @gung's excellent answer:

• Depending on your field, there may be relevant norms (as in DIN/EN or ISO). This is probably not an issue with predicting hard disk reading speed, but e.g. in analytical chemistry the rule is no extrapolation. Period. If you want to go as far as 500 GB, then go and do some measurements up to including 500 GB.

• The usual way of setting up a linear model has two important assumptions

  • Obviously, that the function is linear. In practice it is usually not a very good assumption that linearity extends to infinity. E.g. can you expect still to find linearity if you read larger amounts than the hard disk volume?

  • Usually, also homoskedasticity is assumed. This means that the absolute amount of error/noise does not depend on the dependent ($x$) variable, here: the amount of data to be read. I'm not sure about hard disk readings, but I experience (chemistry/chemometrics) usually something between constant absolute and constant relative noise (or more complicated behaviour due to different sources of noise).
    Any deviation from the constant absolute amount of noise regime will mean that the prediction intervals for the extrapolation are grossly wrong -- usually they will be far too narrow.

• Even if these assumptions are met, consider how big the prediction interval actually is for that kind of extrapolation:

  

  (I took some real calibration data of a very nice measurement I had and adapted it to your problem).
  Note that the prediction interval at $x$ = 500 is already twice as large as the total difference in $t$ your calibration data spans! If you don't have such an exceedingly nice linear data set, the prediction interval will just "explode".