Statistical Analysis Stack Exchange
Statistical Analysis Stack Exchange

Questions tagged [reliability]

A measure is said to have a high reliability if it produces similar results under consistent conditions. DO NOT confuse reliability with validity (see tag wiki). DO NOT use for inter-rater reliability which has its own tag inter-rater

Reliability refers to the overall consistency of a measurement, rather than its accuracy (how well it reflects some real, external quantity - i.e. validity).

Note that the meaning of this term varies slightly from one discipline to another. It may either refer to reliability in engineering, the bias variance trade-off illustrated below when characterizing the properties of an estimator or a statistical predictive model as a whole, or the consistency of a subjective or biological measure, both in time and across raters. In the latter case, related tags are , , , .

Reference: Wikipedia

Reliability vs Validity illustration
(Figure adapted from this image, created by Nevit Dilmen
under the Creative Commons Attribution-Share Alike 3.0 license.)

547 questions
6
votes
2 answers

Comparing test-retest reliabilities

Imagine I have a test for a disease that returns either a positive or a negative result. I administer the test to one group twice in a single session; in another group, I administer it twice with two weeks separating the tests. The hypothesis is…
Matt Parker
  • 6,017
4
votes
2 answers

Standard error of measurement versus minimum detectable change

I am currently calculating reliability estimates for test-retest data. My question is regarding the difference between standard error of measurement (SEM) versus minimum detectable change (MDC) when seeking to determine if there is a 'real'…
Stephen
  • 41
3
votes
0 answers

Can I use a measure when it has low reliability?

I would like to use a 5-item subscale to measure the effects of a behavioural intervention program on the use of behavioural strategies. I obtain significant results when I analyse it in models however the reliability of this measure is very low at…
Melissa Duncombe
  • 79
3
votes
2 answers

How to test reliability of an index when the index is composed of distinct facets?

I am measuring a concept (such as 'friendship') that is multidimensional in nature. To capture the multidimensional nature of the concept, I have measured the concept through several aspects, each of which focus on one part (or dimension) of the…
Adhesh Josh
  • 3,255
2
votes
1 answer

How to calculate inter-rater reliability for just one sample?

I'm trying to compute a reliability rating for potentially malicious emails. Essentially a system where multiple people will view a suspect email, and assign it a "malice score". The idea is for emails where the raters reliably rate it malicious, to…
scuzzy-delta
  • 121
2
votes
0 answers

How can test-retest reliability be high when internal reliability is very low at the second session?

We did a study on the reliability of a task where we measure for how long (milliseconds) people look at certain images. The internal reliability (consistency) at the first session was very high (cronbach's alpha = .90) but at the second session it…
Puzzled researcher
  • 21
2
votes
0 answers

Does measure of reliability apply to regression coefficients?

I have a problem of estimating some kind of a test-retest reliability/intraclass correlation of the regression coefficients. I have only seen test-retest reliability apply to outcome variables, simply calculating a ratio of between-variance to the…
user151310
  • 91
2
votes
0 answers

Split thirds correlation

You hear about split-half correlations often as a measure of reliability. But is there a reason that split-third correlations (or split-fourth, split-fifth, etc.) are never used? Shouldn't they provide a better sense of the internal reliability if…
japem
  • 141
1
vote
0 answers

can the same item be used in multiple constructs?

I am constructing a survey with around 10 multi-item scales. My piloting indicates that one of the most important multi-item scales (COP) has very poor reliability (.49), but there is an item in another scale (REX) which seems to hang together…
fiverliver
  • 11
1
vote
1 answer

Inter-rater reliability

In my research, I use a questionnaire so that students mention some classmates according to a group of characteristics. Each item is a different characteristic, and for each one they must name the classmate they consider to have it. In this case can…
Magalys Reyes
  • 33
1
vote
0 answers

Reliable Change Index Question

I was wondering if someone could help me with a reliable change index question. I am looking to do pre/post analysis at the individual participant level. Is this method only appropriate for measures such as surveys where internal consistency…
user348153
  • 11
1
vote
1 answer

Relationship between SD of MC test scores and test reliability?

The scores on a 80 question multiple-choice test had a mean 70% correct and standard deviation (SD) of 10 (SD of % correct). This SD is 2.5 times higher than the theoretical SD of 4 (based on N, p and q; SD = sqrt of Npq). I take this higher value…
Joel W.
  • 3,306
1
vote
0 answers

Can I use G theory for this case? How? Comparing Waldorf and traditional schools

I have my data in the format of: In my study I examined 20 waldorf student and 30 traditional student using concept map. Concept maps can be scored by different scoring methods. How can I determine which scoring is the best (most reliable) for…
Natália László
  • 11
1
vote
1 answer

Pre-test/Post-test Reliabilty

I calculated Cronbach's α to estimate reliability using both pre-test and post-test scores before and after a training. I fully expected the pre-test reliability estimates to be low, but what could cause the post-test reliability coefficients to be…
user3700285
  • 11
1
vote
0 answers

Reliability of Composite Variable Made of 4 Measures?

I am conducting some analyses with a composite variable constructed by averaging 4 separate measures. I would like to know the reliability of the composite variable. Is Cronbach's alpha appropriate in this case (even though I have entire scale…
Alice
  • 11
1
2 3