comparing 2 surveys on same question but with 2 different scales

Question

I have results from 2 surveys asking the same question. One allows for a single answer, the other allows for up to 2 answers. Is there any way to compare the 2 surveys?

He are the numbers:

Survey A: "why do you do x?" Answer up to 2 options

option 1a) 78%
option 2a) 43%
option 3a) 21%
option 4a) 17%
option 5a) 13%
option 6a) 8%
option 7a) 7%
option 8a) 7%
option 9a) 2%

Survey B: "why do you do x?" (same question as survey A) Answer exactly one option

option 1b) 60% (is the same answer as 2a)
option 2b) 38% (is the same answer as 1a)
option 3b) 2%

(How) can I compare the options 1a) and 2b)? Is this even possible?

Why do you want to compare them, i.e. what question do you aim to answer? — Scholar, Apr 11 '19 at 13:53
Are you only interested in these 2 questions across the 2 surveys - or are there multiple questions across the 2 surveys that you would like to do this for? — Craig K. Van Pay, Apr 11 '19 at 14:00
Apples contain less vitamin C than oranges. Are you looking for such kind of comparison? — Sextus Empiricus, Apr 11 '19 at 15:35

morphist · Answer 1 · 2019-04-18T08:20:04.403

You cannot and you should not compare them. Although the text of the question and two of the answer options may be identical, there is no way of comparing the choices. The main problem is that the options cannot be seen as independent survey items, because they depend on the presence of the other answer options.

Consider this: We ask people which flavor of ice cream they like best. In Survey A we give them 5 options: Vanilla, Chocolate, Strawberry, Hazelnut, and Stracciatella. In Survey B we, only give them three options: Vanilla, Chocolate, and Blueberry.

If we tried to compare whether people in Survey A or Survey B like Vanilla more, we have no way of knowing, because implicitly we ask which flavor out of the ones presented they like best. People in both surveys might favor Stracciatella, but in survey B, this option does not exist, so people might fall back to the next best thing, which is Vanilla, with the result that people in Survey B seem to like Vanilla more. Even with a "none of the above" or "other" option, the problem will probably remain.

In short: You cannot compare single (or double for that matter) choice survey items with different sets of options, because the options are not independent of each other.

A comment to add to this excellent answer - no, it does not make sense to compare 1a) and 2b) with a statistical test. But you may still be able to form some sensible commentary/suggestions/hypotheses from comparing the two surveys without using a statistical test - e.g. did the second survey have some other options which perhaps added nuance to option 2a, thus making option 2a less popular, whereas without such nuance in the second survey, option 1b was the closest respondents could get to the truth? The next step then would be to design a survey capable of testing your hypotheses... — Izy, Apr 18 '19 at 08:40

Sextus Empiricus · Answer 2 · 2019-04-11T15:57:02.883

The outcome is based on hidden variables.

Say the questionnaire would have asked the more detailed question like

Survey C: 'Why do you do x? answer with your first option and second option'

then you could get a list like

                first      second     sum 1st + 2nd
option 1        60         18         78
option 2        38         5          43
option 3        2          19         21
option 4                   17         17
option 5                   13         13
option 6                   8          8
option 7                   7          7
option 8                   7          7
option 9                   2          2
other                      4          4

The survey A gives you only the third column. The survey B gives you only the first column. The second column you do not know with neither survey A or B.

(note: It is not exactly right to equate the surveys A and B with C, with a different way of asking the question the people may answer slightly different)

The question is what and how you wish to compare. (The point of the comparison is not clear)

case 1 When you perform the two different questionnaires with different populations, conditions or other changes then it makes little sense to compare the first column of the one questionnaire with the third column of the other questionnaire (except that you may expect the third column > first column).
case 2 When you perform the two questionnaires/sampling with the same conditions, like same population and circumstance, and the sample is large enough (sufficient accuracy). Then you could do a comparison like combining the two questionnaires and deduce the second (unasked) column from the two.

comparing 2 surveys on same question but with 2 different scales

2 Answers2