0

I ran a between-subjects experiment in which 114 people evaluated 1 innovative idea, and I manipulated the source of the idea (IV1 = source; categorical: internal vs. external) to see if it affected the idea evaluation (DV).

My main hypothesis is individuals with a high level of innovator role identification (IV2 = identity; continuous) will evaluate ideas in the external source condition lower.

I ran an OLS in R and I got the following results:

enter image description here

QUESTION 1: Do I interpret the coefficients correctly:

Coefficient 1: 2.6938 1.4225 1.894 0.06090 .**

The idea evaluation increases by 2.69 points on average when the source switches from internal to external, while the level of role identification equals 0 (OR is held constant??) Is this coefficient meaningful, as the level of role identification in a firm is never 0?

**Coefficient 2: 0.8085 0.2425 3.334 0.00117 ****

The idea evaluation increases by 0.8 points when the source is 0 = internal.

**Coefficient 3: -0.7152 0.3395 -2.106 0.03744 ***

With each one-unit increase in the level of innovator role identification, the idea evaluation decreases by 0.7 points on average when the source switches from internal to external.

QUESTION 2: The confidence bands do not overlap at the higher values of identity. Can I say the difference may be significant?

enter image description here

kjetil b halvorsen
  • 77,844
Daisy
  • 1
  • 1
    Never look at overlap. Compute the confidence band for the difference. A general approach to do that is the R rms package contrast function: https://www.rdocumentation.org/packages/rms/versions/6.8-0/topics/contrast.rms# – Frank Harrell Mar 27 '24 at 11:25
  • Thanks @FrankHarrell - i'll try that one out! Any comments on my first question, with regard to the interpretation? – Daisy Mar 27 '24 at 11:51
  • 1
    Does this answer your question? Why is mean ± 2*SEM (95% confidence interval) overlapping, but the p-value is 0.05? it relates to the idea behind overlapping intervals and how they are not the same as a hypothesis test or p-value. Your case follows the same principle, although it has the additional complication of being the case for a difference for with a specific condition. – Sextus Empiricus Mar 27 '24 at 12:08
  • @Sextus Yes, the same principle applies, but in this context the confidence intervals are dependent. Thus, a more specific analysis is needed and requires the full variance-covariance matrix of the coefficient estimates as well as the Mahalanobis distance matrix of all explanatory variables. – whuber Mar 27 '24 at 14:07

1 Answers1

1

Regarding your first question: Your interpretation is correct. Is the first coefficient "meaningful"? Well you can use it (and the other coefficients) to get the predicted level of overall quality for any combinations of the IVs, so, in that sense, it is meaningful.

If you make a graph of the predicted values vs. the two IVs, you can find where that number sort of "shows up". But if you want to make it more, shall we say, realistic, you can center role identification at 0 before doing the analysis. The the first coefficient will still be for when "role identification = 0" but that will now be the mean value of role identification.

This won't change the meaning of the analysis, but it may make the output easier to interpret. But .... It might make it harder, as well. I think it depends on a) How well known the measure of role identification is, in your audience. If it is very well known, I think not centering has some advantages as the numbers are known. b) How sophisticated your audience is. Can the grasp the notion of mean centering? A lot of people hate math and statistics. Are you talking to some of them? If so, I wouldn't center. And, if they are really math phobic, I might just present graphs and tables and not present a table of coefficients at all.

(If you are writing for an academic journal, you may have to write for a mix of audiences, implying more explanation. But you may also have word limits. No one said academic writing was easy!)

Peter Flom
  • 119,535
  • 36
  • 175
  • 383
  • 1
    Thanks a lot, @Peter Flom! Someone from my audience interpreted the second coefficient as the unique effect of identity on idea evaluation in general, not on the evaluation when the source is 0 (internal). But If treatment contrasts for a categorical variable are present in a model, the estimation of further effects is based on the reference level of the categorical variable, right? – Daisy Mar 28 '24 at 11:02
  • When there is an interaction, the main effects are only for the case wheh the other variable is 0 – Peter Flom Mar 28 '24 at 11:08
  • Exactly, thanks a lot! Any chance you could help me out here as well: https://stats.stackexchange.com/questions/643750/which-strategy-to-use-to-analyze-my-repeated-measures-data-se-clustered-at-eval – Daisy Mar 28 '24 at 11:11