I am in the process of developing a scale. I have already performed EFA on the first dataset I collected, which showed strong support for a 4 factor structure. I am now planning to collect another dataset to do a CFA, but I have some conceptual problems:
This scale supposedly measures construct X, with the 4 factors as subtypes of construct X. If the scale measures construct X, I should be able to sum the scores across these 4 factors to create a total score that measures construct X. However, I have a problem where a score on construct X is not actually interpretable. Some scales, like anxiety scales, are clearly interpretable because a higher total score indicates higher anxiety while a lower score indicates lower anxiety. My construct X is ambiguous, for eg. "Feelings about COVID-19"; a higher score could supposedly indicate greater feelings about COVID-19, which does not mean much.
What would you suggest I do in this situation? I can certainly still test the four factor structure I found in the next dataset using CFA, but is it still advisable to sum up the scores across the factors? Is second order CFA (with correlated first order factors and construct X as the second order factor) the way to go in testing whether a total score across the four factors make sense?
In sum, here are my questions:
- How do I know when it's okay to sum the scores across factors to create a total score on a scale (eg. perhaps by specifying a second order factor in CFA)?
- I'm not even sure the total score summing across factors would make any sense. Should I still bother testing whether a second order factor is at play?
Thanks so much! Really appreciate any help I can get.
