2

I'm helping a friend design his research. He'll use a Google Forms questionnaire with several questions (from 0 to 5, Likert scale) to assess anxiety of students' before taking two different tests. The questionnaire has about 10 questions, all underlie anxiety and all go from 0 (strongly disagree) to 5 (strongly agree), and students will take either test A or test B (the tests are the same, but in different languages). The question is to verify if anxiety can be explained by type of test. Hence, I'm thinking about performing an Ordinal regression (using R's clm) to test it in a second moment.

  • The thing is, after we have the data, what would be an appropriate approach to combine all results into a single "anxiety" measure? in order to be able to fit the model afterwards?

  • Hypothetical data:

ID      test   Q1 ... Q10 
PART1    A      1      3
PART2    A      2      4
PART3    B      5      1
PART4    B      4      2

We'd like to model "Anxiety" taking into consideration Q1 to Q10:


clm(ANXIETY ~ test)

Notes:

  • sts will take either test A or test B, so this is not a repeated-measures design
  • I've thought of running a pilot study with a few sts and run a Cronbach's alpha to assess reliability/consistency, but I still don't know what to do afterwards to combine all questions into a single variable.
  • This is more of a theoretical question, but R tips would be much appreciated. Thanks!

I've seen many similar questions here, hopefully this isn't a duplicate, but I still don't know what path would be the most adequate. like this from 2015, this from 2012,this from 2017 and this from 2015

Larissa Cury
  • 741
  • Seems like a fairly straightforward application of structural equation modeling... – BigBendRegion Nov 26 '22 at 16:30
  • One aspect of this is that one can wonder to what extent this should actually be determined by the data, i.e., to what extent the data contain information about how an overall scale should be put together. For example, if two items are highly correlated, this may mean that they're measuring the same thing (in which case they should probably be implicitly downweighted, as PCA will do), but it may also mean that they measure really different aspects of anxiety that happen to be correlated in the data, in which case they should probably not be downweighted. – Christian Hennig Nov 29 '22 at 11:48
  • But why do you need to combine the 10 items into one measure (so early)? If Q1 in A correspinds to Q1 in B (and so further), why not test the effect of AvsB on each item separetely? – ttnphns Dec 04 '22 at 11:30

1 Answers1

2
  • Your question is, basically, all of the field of item response theory. A set of questionnaire items along with the algorithm to combine them is called a "scale". There are many popular scale examples, like the CES-D of depression or the 3MSE of cognition.
  • Before a "scale" is usable for research purposes, people are often interested to know whether the scale is validated. This form of validation is separate from the forms of model validation often discussed by machine learning and data science folks. Unfortunately, the same data shouldn't be used to both validate a scale and produce meaningful summaries - not that you won't find examples of this being done in the literature, but the weight of evidence should be considered "shifted" by a keen reviewer.
  • Whether a set of questionnaire items can develop a powerfully predictive scale, and whether the sum of those ordinal scales is an adequately powerful predictor in its own right are separate questions.
  • Cumulative link models are a nice conceptual starting place because they open the door to consider latent variable modeling - where the nuisance parameters, that is the "Intercepts", allow an unknown but positive score to separately predict the probability of achieving the next response level. That said, it's the completely wrong place to look, and you often find the most problematic aspect of the design are the particular questions themselves and how they should be scored. Consider that surveys sometimes mix questions with "reverse anchoring": i.e. q1: My doctor cares about my feelings (1 worst - 10 best) q2: my doctor is late (1 none of the time - 10 all of the time).
  • The Rasch model (particularly the polytomous Rasch - which is perhaps inappropriately named as they are referring to ordinal outcomes) are a super flexible way of analyzing "Likert" style responses. In practice, I find there's rarely much difference between the results there and, say, a continuous analysis, or dichotomizing a Likert variable using a "top box" response (i.e. 4 or 5 out of a 1-5 scale). I don't believe in spurious dichotomization, but seeing a billion Likert answers, people usually do not give intermediate responses, or, if they do, most analysts agree to treat a "3" as a negative result.
AdamO
  • 62,637