Can my LIkert data say anything about my intervention's effectiveness?

Question

I have Likert data from a survey that unfortunately I did not get to design, and I would like to see if there are any ways to assess if an intervention is having an effect (or rather, to assess participants' perceptions of whether the intervention is having an effect). I don't have baseline data, only post-intervention; I recognize the many major limitations of the data but I'm hoping something is salvageable!

Our "intervention:" we made medical specialists available via web chat to generalist clinicians that did not previously have access to specialists, so that they could ask for advice regarding complicated patients.

Evaluation data: we have no "before" data, and this was obviously not blinded or randomized. We just have survey data from clinicians who used this resource. Data is currently in Google Docs, hah, but could be exported somewhere else (Stata and Excel ideally). The questions we asked are as follows:

I changed elements of my patient's care based on what I learned from the specialist.

1 = Strongly disagree (my patient's care did not change)

5 = Strongly agree (my patient's care changed)

What I learned from the specialist allowed me to implement care which improved my patient's condition.

1 - Strongly disagree (there was no improvement)

5 - Strongly agree (there was improvement)

My knowledge about a particular medical condition improved as a result of the WhatsApp case discussion.

1 - Strongly disagree

5 - Strongly agree

The predicament: how should we present this data?!? We can calculate very simple summary stats (e.g. the mean), but it would be nice for a medical education paper we'd like to write to say something more statistically meaningful about whether our intervention was better than doing nothing. I thought about just comparing our mean to a null hypothesis of m=3 (or m=1 -- we debated about what the null should really be) but I've read that Likert data doesn't work that way. Here are the two options suggested to me so far:

Two proportion test: "bin" the data (either 1-2-3 vs 4-5, or 1-2 vs 3-4-5) and compare our proportion of "successful" responses to whatever portion we think would have occurred if our intervention had no impact or a negative impact. (If we did this, any recommendations on how to choose the null hypothesis proportion?)

Chi square test: if our null hypothesis is that the responses are random, and our intervention isn't causing any positive effect so people are just selecting their survey answers randomly, then we could use a Chi square test to compare the distribution of our answers to a hypothetical normal distribution centered around 3. I'm intrigued by this idea, but I'm not sure how to actually implement it mathematically.

Do those ideas sound reasonable? Are there other ideas to analyze our Likert data to show that what we're doing is better than nothing... without actually having a control group / counterfactual / baseline data?

In case it wasn't clear, I have decent classroom training in statistics thanks to my medical education but have not worked with them on real projects before. Will happily delve into any resources you all can provide, though! Thank you so much.

Answer 1

I think you want to use the very simple one sample Z-test and its relative the one sample Z-test for proportion. Also, that link shows you how simple it is to do in R.

To be clear, the one sample portion of those names refers to the very fact that you only have one group (ie. no control group).

Suggested Method 1

If you want to consider each question individually, you may consider the one sample proportion test for each question. I think I'm describing what you called a two proportion test.

For a question that has a list of answers ranging from 1 to 5 like

$$[1,3,2,5,4,1,...]$$

If you consider a successful intervention as being a very effective one, you might say that the extreme answers of 4 and 5 are considered a "success" (coded by 1) while 1,2 and 3 are considered a "fail" (coded by 0). Then you would transform the list according to this rule and get $$[0,0,0,1,1,0,...]$$

From here on out it is just a regular Z-test, since the mean of the transformed list equals the proportion of 1's in the transformed list.

A null hypothesis might be that the intervention is successful 30% of the time (equivalently, the proportion of successes is 0.3). You could then calculate a one-tailed p value from the Z-test to indicate whether the mean of your transformed survey answers is significantly greater than 0.3 (or, significantly less).

Suggested Method 2

You may also want to consider the questions together, in which case you could calculate a score for each survey taken and use a one sample Z-test.

So, each question has a list of answers in order of survey (ie. the first item in each list corresponds to the first survey taken),

$$\begin{align}\text{q1 answers} &=[1,4,2,5,3,...] \\ \text{q2 answers} &= [3,2,5,2,1,...] \\ \text{q3 answers} &= [4,5,2,3,1,...] \end{align}$$

Then you score each survey taken by considering its mean across questions $$\text{scores for each survey taken} = [2.66,3.66,3,3.33, 1.66,...]$$

A null hypothesis might be that the mean score is 4, and you test whether the mean of the survey scores is significantly different than 4 with a regular Z-test.

This is basically addressing your concern,

I thought about just comparing our mean to a null hypothesis of m=3 (or m=1 -- we debated about what the null should really be) but I've read that Likert data doesn't work that way.

When doing it like this, I think doing this with your data perfectly reasonable.

When choosing the null, consider the scale and meaning of your questions. Since answering 1 on the survey seems to be the same as answering "no change" (or similar), testing if the average survey answer is significantly greater than m = 1 is like testing if there was any significant change at all. Testing if the average is significantly greater than m = 3 is like testing if the intervention had a significantly extreme effect. The same goes for choosing your "rule" for success when doing a proportion test.

Answer 2

A few comments...

What might be easiest is the chi-square approach. If you do this, however, you won't be able to say in which direction the effect occurs. You will only be able to say there appears to be a difference between intervention and control. You would also have to verify that there are more than 5 people in each cell.

What might be best is to model the outcomes as a function of the intervention groups. Since your data is essentially categorical, an ordinal logistic regression sounds appropriate. Stata has this functionality if I remember correctly.

EDIT: I have misread the question.

A proportions test wouldn't work because you have no null hypothesis. Asking "what should my null be" is really only a question you can answer as the expert in the field, but I'm afraid whatever you'll choose will just be out of convenience rather than being a bonda fide hypothesis.

I can't think of much that can be done aside from perhaps using this as pilot data to do a future experiment. You could reference it and say "from pilot data, we estimate the mean to be...".