1

Today, I came across an interpretation of 95% confidence interval (thereafter "CI"). The text has been written from education science perpective saying that, beside a (1.) random sampling error of randomly selected cases from a given population, CI also accounts for "(2.) different conditions in which the test was administrated and (3.) which tasks were included into a test (4.) etc."

I am bit confused now, since I have never seen that CI could also account for this other sources of uncertanaity (lets say all of them have stochastic character). Is my knowledge about CI not complete or the author just misinterpreted the meaning of CI?

Thank you in advance.

Nothingman
  • 111
  • 1
    Can you please include (as an edit to the Q, not only as a comment) a link or reference to that text? – kjetil b halvorsen Jun 15 '22 at 13:27
  • 3
    A procedure to produce a CI can, on the face of it, correctly account only for the uncertainty represented in the underlying statistical model. Thus, given a suitable model along with a demonstration of its applicability, a CI procedure can account for any or all of (1) through (4). However, to the extent to which any of these potential sources of variation exist, they can cause the data to vary--and most CI procedures will estimate (correctly or not) the magnitude of that variation and incorporate it within their results. – whuber Jun 15 '22 at 13:52
  • 2
    On a slightly cynical note (but one amply grounded in experience), one shouldn't look too closely at any account of statistical concepts or procedures written from an education science perspective. For careful, accurate, and rigorous explanations, look to the statistics literature itself. – whuber Jun 15 '22 at 13:54
  • I am sorry, but I am not able to provide a link for this citation. The text hasn't been writter in english, but in rather "minority" language, therefore I do not think it will be particulary helpful. At least I can provide my translation: "The measurement of students' knowledge ... through tests is always associated with a certain statistical error, which is given by what tasks were included in the test, what conditions prevailed on the day of testing, students participated in testing, etc. To capture the uncertainties associated with testing ... [use ] 95% confidence intervals." – Nothingman Jun 15 '22 at 14:08
  • 1
    Continuing: "If the testing took place on a different day, if other pupils were present, or if a test with a different order of assignments were to be taken, the schools could achieve a slightly different result than they recorded in this particular survey. The range in which their result would most likely be found during repeated testing is expressed by the confidence intervals." (Now the second part is obv. wrong interpretation, but it is for another discussion.) – Nothingman Jun 15 '22 at 14:13
  • 1
    Those are helpful quotations. Provided the dataset has sampled all these different conditions in a way that can be rigorously construed as random, then a CI might reflect all those forms of variation. But, for instance, if testing occurred on just one day, how could a CI possibly reflect variation over time, which a fortiori has not occurred within the dataset? – whuber Jun 15 '22 at 14:29

1 Answers1

1

(1.) random sampling error of randomly selected cases from a given population...

Random sampling error is what most statistical methods are about.

The other sources of error,

...also accounts for "(2.) different conditions in which the test was administrated and (3.) which tasks were included into a test (4.) etc."

, are not part of the random sampling procedure*. These are more like systematic errors. When confidence intervals are computed/estimated then these type of errors are typically not accounted for (e.g an interval based on the t-statistic does not).

* At least they are often are not part of the sampling. You can make it part of the random sampling by performing tests with different settings like changing conditions and tasks. But the text using a phrase like 'beside a (1.) random sampling error...' seems to regard these errors implicitly as different types of error that are not random sampling error. Potentially it could be that they see them as different from 'random error' as they can be seen as 'random effects' and the text might wants to put them aside because of that? There is some duality to random effects and they can be seen as both part of the deterministic part of the model as well as the random part of the model. For these potential subtleties it is better when you provide a link to the original text, even when it is not English.

Sextus Empiricus
  • 77,915