1

For reasons of economy, a college has moved from live lectures to videotaped.

They have exam results from the year/years before the change, and the year/years after.

They want to be reassured that the new method delivers exam results THE SAME AS, OR NOT WORSE THAN, the old.

We can assume they have taken proper steps to have all the exams comparable.

How is this to be framed technically, and what tests can be applied?

Community
  • 1
Bob
  • 21
  • 1
    This is called equivalence testing. Look at this topic for more info: http://stats.stackexchange.com/questions/139385/inverted-nonparametric-rank-test – alesc Apr 22 '15 at 07:35
  • Many thanks, alesc, especially for the topic name. I see a difficulty that distributions of exam results don't have exactly-paired values, but perhaps that can be overcome by grouping. Now to study the area. – Bob Apr 24 '15 at 04:52
  • Do the exams have different questions each year, are the questions different or do they partly change? I ask because one thing that could be done is test equating: http://en.wikipedia.org/wiki/Equating that would make the tests have a common scale. However this needs you to have at least some questions to be answered by the same students or at least students with equivalent level of ability/knowledge. If you provide more detailed description I can write more on this method. – Tim Apr 24 '15 at 10:31
  • Hi Tim; thanks for the suggestion - I read the Wikipedia article and I remember studying common item equating. That's what we have here - exam questions are drawn from pools of similar. Each year a fresh batch of students takes an equivalent exam. – Bob Apr 26 '15 at 06:24
  • Hi @Tim; thanks - I read the Wikipedia article and remember studying common item equating. That's what we have here - exam questions drawn from pools of similar. Each year a fresh batch of students takes an equivalent exam. But I have to assume equivalent exams set to equivalent groups of students, ie that standardization has been done already by the subject specialists. The data I have is a sequence of distributions (ie the exam results for each year: how many students scored n%, n=1-100). If test equating can be applied to this, I'd be happy to know. – Bob Apr 26 '15 at 06:35
  • This comment just to notify @alesc – Bob Apr 26 '15 at 06:37
  • Sorry, I am unfamiliar with Equating. But from reading Wikipedia, I think this is not intended for equivalence testing, but rather for some form of normalizing the test results in order to make results from various groups using different exam questions comparable. You can still use this to pre-process your data, but as far as I understood this term, It won't give you your desired answer (i.e. are the results the same or worse than before). – alesc Apr 26 '15 at 08:11
  • @Bob after re-reading your question I start to wonder if equating is applicable to this data. You have different students and different tests, some items re-appear in subsequent tests but you want to compare paper-and-pencil tests to the computerized ones. It is possible that even if you have the same items on paper and computerized tests they behave differently, so you probably should not assume that the items are equivalent without any external justification. – Tim Apr 26 '15 at 09:44
  • alesc's link led me eventually to confusedcoder at non-parametric two-sample equivalence tests with unequal sample sizes, who had found the works by Wellek cited in the answer. – Bob May 03 '15 at 05:02

1 Answers1

1

The appropriate tests are in the book by Stefan Wellek (2010) Testing Statistical Hypotheses of Equivalence and Noninferiority, Second Edition: "6.2 Mann-Whitney test for equivalence" (pp126-135) and "7.5 A non-parametric k-sample test for equivalence" (pp 231-234).

The software package accompanying the book is at https://www.crcpress.com/downloads/K10408/TSHEQ_2ndEd_Programs_etc.zip. The software consists of scripts for the freeware R Statistical Package (equivalent SAS macros are also included). The script for the Mann-Whitney test for equivalence is called mawi. According to Wellek (2010) p128, the input data required for mawi are the significance level, the two sample sizes and the name of a file containing the raw data.

The original published source of the Mann-Whitney test for equivalence may be more convenient: Wellek, S. (1996), A New Approach to Equivalence Assessment in Standard Comparative Bioavailability Trials by Means of the Mann-Whitney Statistic. Biom. J., 38: 695–710. doi: 10.1002/bimj.4710380608 {onlinelibrary.wiley.com/doi/10.1002/bimj.4710380608/abstract}.

This extract from the abstract gives the idea of Wellek's approach:

"Let the two distribution functions be given by F(x) = P[X ≤ x], G(y) = P[Y ≤ y] with (X, Y) denoting an independent pair of real-valued random variables. An intuitively appealing way of putting the notion of equivalence of F and G into nonparametric terms can be based on the distance of the functional P[X > Y] from the value it takes if F and G coincide. This leads to the problem of testing the null hypothesis Ho P[X > Y] ≤ 1/2 - ε1 or P[X > Y] ≥ 1/2 + ε2 versus H1 : 1/2 − ε1 < P[X > Y] < 1/2 + ε2, with sufficiently small ε1, ε2 in (0, 1/2). The testing procedure we derive for (H0, H1), and propose to term the Mann-Whitney test for equivalence, consists of carrying out in terms of the U-statistics estimator of P[X > Y] the uniformly most powerful level a test for an interval hypothesis about the mean of a Gaussian distribution with fixed variance. The test is shown to be asymptotically distribution-free with respect to the significance level. In addition, results of an extensive simulation study are presented which suggest that the new test controls the level even with sample sizes as small as 10. For normally distributed data, the loss in power as against the optimal parametric procedure is found to be almost as small as in comparisons between the Mann-Whitney and the t-statistic in the conventional one or two-sided setting, provided the power of the parametric test does not fall short of 80%."

Clearly there are many other approaches to the problem, but Wellek's solution seems particularly suitable to this case.

Bob
  • 21