1

I have a stats conundrum

I have an MSc thesis in which two groups were manipulated to either include others in their group, or manipulated to exclude others (c1 inclusive and c2 exclusive).

My dependent variable is how much income tax the participant would be willing to pay for the other group, with the prediction from the literature that when primed inclusively, people would pay a higher amount of tax for them (we pay for our own).

A one-way ANOVA did find a significant difference between the groups. Also, there is a significant difference (in the same direction) between groups in how much they identify with the group they need to pay for (superordinate identity, which is supposed to reflect the manipulation).

At this point I got excited. However, there is no significant correlation between superordinate identity and my DV (how much income tax willing to pay). I’m really not sure what this means? Does this mean my manipulation isn't effective?

I took measures for other things which may be important to the DV. Many of these were strongly significantly correlated with the DV. For example, attitude towards paying taxes (makes sense), attitudes towards the other group (makes sense), how much of a threat the other group is (results didn't make much sense in context). However, there was no significant difference in these measurements between the two groups, showing that the manipulation had no significant impact on these factors.

I attempted a multiple regression to see which predictors best predicted the DV and these other strongly correlated predictors were included in the model, but not my predictor of interest (superordinate identity).

Now I really don’t know what this means.

  • How is there only a significant difference between groups on the DV and the scale which reflects the manipulation (superordinate identity) but no significant differences between groups on the other measurements?
  • Additionally, why is my superordinate identity scale (reflecting the manipulation) finding a significant difference between groups, but is not correlated with the DV or included in a MR model to predict the DV?

I spent a restless night with it going round in my head, over and over, but still cannot understand what has happened. I'm petrified that my whole project might be a waste of time.

Gala
  • 8,501
richard
  • 79
  • I don't understand the multiple regression model, why didn't you include “superordinate identity” in the model? – Gala Jul 08 '13 at 07:06
  • I tried to clarify the title, I hope it's a good description of your question. – Gala Jul 08 '13 at 11:08

1 Answers1

1

Perhaps other will offer a more detailed explanation on what exactly is going on but here are some practical tips/ideas:

  • Finding a difference on the manipulation check and on the variable of interest is already very encouraging. It is certainly good enough for many accomplished researchers and prestigious publications so don't worry about the project being a waste of time because of that.
  • This could be an example of the “ecological fallacy”. When you try to correlate the manipulation check (superordinate identity) and the taxes your participants would be willing to spend, you are looking at individual-level data, which can have another pattern than group-level data.
  • Don't obsess over statistical significance or put too much emphasis on the significance threshold. For example, correlation can be high or low, not only present or absent, and it is meaningful to interpret that kind of differences. A related idea is that “the difference between significant and not significant is not itself significant”. Concretely this means that the p-value of the F-test is something like .03 and the correlation is not significantly different from 0 with p = .08, both results are in fact very similar.
  • As a consequence of the previous points, you should in any case create plots and interpret them (possibly post them here as well if you wish) rather than pour over lists of p-values.
  • The Pearson correlation coefficient reflects linear relationships. It's possible than the relationship between a quantitative measure of the manipulation and the outcome has another shape, explaining the null result, even if the ANOVA suggests the manipulation does produce some sort of effect.
  • The absence of a significant difference should not be interpreted as definitive evidence that there was no effect (as you appear to do in several places). If the sample size is small it often means that you just don't know precisely the sign or the magnitude of the correlation, but not necessarily that it is very close to 0.
Gala
  • 8,501
  • Hello Gaël. Thank you for taking the time to read my post, and thank you even more for taking the additional time to reply. To be more specific about my problem - The social identity approach for supporting others suggests that people will be more willing to support those included in the ingroup. I manipulated a two-group experiment with British participants by either emphasizing European identification, or deemphasizing European identification. It was my prediction that those emphasized with a shared EU identification would be more willing to pay taxes to support a new member of the EU..... – richard Jul 09 '13 at 07:52
  • ….The manipulation created significant differences between the amount the groups would pay in tax, British identification 2.86 vs. European identification 3.18, p = <.05. Also there was a significant difference in a ‘superordinate scale’ (which measured how much the groups identified with the European identification, reflecting the manipulation) British identification 5.3 vs. EU Identification 5.8, p = <.05. However, as mentioned previously, the ‘superordinate scale’ does not statistically correlate with the DV in either group (group 1. r = -.137, p = .417 / group 2. r = .238, p = .144)... – richard Jul 09 '13 at 07:53
  • ....This is why I am so confused because the manipulation affected the DV, and created a significant difference in scores between groups in the ‘superordinate scale’. But can I infer that it is this manipulation which caused the difference in superordinate scale, and therefore a change in the DV? If not then I don’t think I have supported my hypothesis. Thanks again for your help, and sorry for the length of my reply. – richard Jul 09 '13 at 07:54
  • Within- and between-group correlations simply have a different meaning. What this could mean is that your manipulation “moves” the mean of the whole group but in each group those with the highest European identification are not necessarily those who would pay more. You really ought to read more about the ecological fallacy. – Gala Jul 09 '13 at 08:46
  • The most important piece of evidence to support the effectiveness of your manipulation is that you manipulated your variable experimentally. Correlations are sometimes relevant, especially in observational studies, but really the most important thing is that group assignment was random and that the manipulation really did impact the variable you intended to manipulate. – Gala Jul 09 '13 at 08:49
  • One last point: A p-value over .05 (presumably from a test that the correlation is different from 0) is not a proof that the correlation is exactly equal to 0. The numbers you gave suggest a sample size of about 40 participants per group. It's certainly possible that there is some moderate positive correlation in both groups but that you didn't have enough power to detect it. – Gala Jul 09 '13 at 10:08
  • "what this could mean is that your manipulation “moves” the mean of the whole group but in each group those with the highest European identification are not necessarily those who would pay more." Thank you very much for this comment. It really makes sense! For example, it could be that after a certain point in the identification scale, higher identification actually reduces the amount of tax you will pay (just an example)? Or it could be that my manipulation affects tax, and affects how much identify with Europe, but this identification does not impact directly on willingness to pay ... – richard Jul 09 '13 at 10:43
  • ...though this would be a scary thought because it is opposite to my hypothesis that 1) emphasis on EU identity increases identity with this group 2) This increased identity results in increased amount of tax would pay. Really it would be emphasis on EU manipulation increases identity with this group. HOWEVER 2) The manipulation will also increase amount of tax (rather than there being a link between them). – richard Jul 09 '13 at 10:45
  • Or it could be that the manipulation cannot be reliably measured by the superordinate scale (which is also bad news because the manipulation could be doing anything). Thank you for all of your helpful comments so far :) – richard Jul 09 '13 at 11:13
  • One general advice would be to actually look at the data, creating several plots (strip charts for each group, scatter plots within each group, etc.) instead of relying mainly on tests to understand what is going on. – Gala Jul 09 '13 at 11:19
  • Thank you Gaël. I think the main thing I can get from my plots are that although largely the more someone identifies with the EU the more they will pay, there are always some cases of people with high identification and low tax. These cases must reduce the effect. My manipulation had an effect on how much would pay which is always interesting to talk about. I just have to hope it was for the reason I suggested. :) thanks again for you helpful comments – richard Jul 10 '13 at 11:31