2

I have one population of people to whom I am asking one Likert scale question (1 = strongly disagree; 5 = strongly agree).

I want to demonstrate that people mostly agree with the statement being rated.

Specifically, I want to demonstrate that more people answer 'agree' or 'strongly agree' than we would expect if people were choosing answers at random. In other words, there is a significant trend for people to choose 'agree' or 'strongly agree' over the other options.

All the resources I have found talk about comparing two groups. I only have one group.

My first thought was to use a Binomial test, but the possible outcomes are not equally likely. For example, if we consider 'success' to mean 'chose agree or strongly agree,' then there is a 2/5 chance of success and a 3/5 chance of failure... and that's assuming we don't account for central tendency bias. If I understand correctly, this means I can't use a Binomial test in the traditional way... though perhaps someone who understands it better can modify it to account for the unequal probabilities of success and failure.

Can someone please let me know the appropriate test to use?

sgware
  • 25
  • 1
    Pleas bear in mind that regardless of the test, interpreting significance as a strong evidence in this case is problematic in practice: Since the differences in how people would interpret the question are likely a huge source of bias, the test IMHO brings little additional value over just reporting the summary statistics. Also the assumption of uniform null distribution is certainly severely broken in any real dataset and while better nulls could probably be made (e.g. from ordered logistic distribution), the interpretation problem is IMHO insurmountable anyway. Best of luck and be careful! – Martin Modrák May 23 '19 at 14:59

1 Answers1

1

If you have $n = 100$ subjects choosing among the five categories at random, then the number of Successes (Likert 4's or 5's) is $X \sim \mathsf{Binom}(n = 100, p = 2/5 = .4)$

Exact binomial test. You want to test $H_0: p = 0.4$ vs. $H_a: p > 0.4.$ You might use an exact test that rejects $H_0$ in favor of $H_a,$ if $X \ge 49.$

Significance level. That test has significance level 4.23% because $$P(X \ge 49|p=.4) = 1 - P(X \le 48|p=.4) = 0.0423,$$ as computed in R statistical software, where pbinom is a binomial CDF.

1 - pbinom(48, 100, .4)
[1] 0.04230142

Power. The power against the specific alternative that there are 60% Successes, is 99%. (The power against alternative $p = 0.55$ is about 90%.)

$$P(X \ge 49|p=.6) = 1 - P(X \le 48|p=.6) = 0.99.$$

 1 - pbinom(48, 100, .6)
 [1] 0.9899949
 1 - pbinom(48, 100, .55)
 [1] 0.9040484

Graphical summary. The upper panel of the figure below shows the null distribution $\mathsf{Binom}(100,.4);$ the significance level is the sum of the heights to the right of the vertical red line. Similarly the lower panel shows the alternative distribution $\mathsf{Binom}(100, .6),$ illustrating the power against alternative $p = 0.6.$

enter image description here

Notes: (1) Similar tests can be constructed for different numbers $n$ of subjects. (2) For sufficiently large $n$ (say $n\ge 25)$, you could use a normal approximation to obtain good approximations of the relevant probabilities. (3) Many statistical computer programs have procedures for doing binomial tests, sometimes called 'test of a single proportion'. In R, the test is binom.test. Minitab's output for such a test is as follows:

Test and CI for One Proportion 

Test of p = 0.4 vs p > 0.4

                                              Exact
Sample   X    N  Sample p  95% Lower Bound  P-Value
1       51  100  0.510000         0.423411    0.017
BruceET
  • 56,185
  • The Question was how to distinguish between random choices among the five Likert responses one the one hand and a greater tendency to choose 4s and 5s on the other That is what I answered. // We don't know the subject matter or the design of of the questionnaire, so it does not seem fruitful to speculate whether random responses would ever happen.or whether pos responses are likely. @MartinModrák's comment seems to say more about the Q than about the A. Binomial and multinomial tests are widely used to analyze Likert data, Better than assuming Likert to be a numerical scale and using t tests. – BruceET May 23 '19 at 14:26
  • OK, but it is less confusing to discuss a Question with comments below the Q, and to discuss an Answer with comments below the A. – BruceET May 23 '19 at 14:42
  • 1
    You are correct, I was mostly speaking about the Q, deleting my comments from here and putting it at the Q. – Martin Modrák May 23 '19 at 14:59
  • Thank you @BruceET for the very in-depth explanation! My alternative hypothesis is "Based on the responses of 100 subjects, we observed 49 agree and strongly agree answers, which is more than we would expect to see if people were choosing answers to the question at random." If I've understood your answer correctly, the R code for this test would be binom.test(49, 100, p=0.4, alternative="greater") ? Based on a quick test, 49 is the smallest number that gives p < 0.05. – sgware May 23 '19 at 18:13
  • Yes, and a direct computation of the P-value in R is: 1- pbinom(48, 100, .4) which returns 0.04230142. rounded in binom.test output to 0.0423. – BruceET May 24 '19 at 01:51