2

I'm looking at a multinomial logistic regression analysis of deer behavioural responses to camera traps. The levels of the response variable are no reaction, reaction and strong reaction. I've selected a number of models based on their AIC values. However, I've found that the p values gain and lose significance within differing models. For instance, with only season as a predictor:

Season               Summer  Winter

Coefficient r         0.8     0.03
Coefficient sr        0.4     0.5
P value     r         0.041   0.925
P value     sr        0.18*   0.024

AIC  1005.023
* P value of interest

With species, camera model, and season as predictors:

Species, Camera model and Season

                    Muntjac  Roe   Sika  Bushnellb   Reconyx  Summer Winter

Coefficients  r       0.94   .43  -0.56    0.76        0.71    1.6    0.5
Coefficients  sr      1.11   .19  -0.19    0.28        0.77    0.95   0.8
P value       r       0.1    0.4   0.4     0.13        0.1     0.008  0.3
P value       sr      0.005  0.6   0.6     0.4         0.01    0.03*  0.02

AIC 1006.618
* P value of interest

Am I right to assume that this may be due to collinearity inflating the significance of this p value, and therefore when analysing the probability of the predictor variable summer influencing the response variable sr, using the p value from the first model? Is it correct to assume that the p value from the 2nd model is due to collinearity and thus not representative of the actual significance of this variable (or level?) summer? Thanks!

Nick Stauner
  • 12,342
  • 5
  • 52
  • 110
user29836
  • 77
  • 4
  • 1
    Thanks for your reply. I'm comparing the model season with the model species, camera and season. I've noticed that when I run models with for instance season included the p values can change. For instance within my first model season, the p value for summer sr is 0.18 and thus insignificant but within the 2nd model, species, camera and season the p value for summer sr is 0.03 and thus significant. I'm wondering if it is correct to assume that the p value from the 2nd model is due to collinearity and thus not representative of the actual significance of this variable (or level?) summer? – user29836 Aug 13 '14 at 20:43
  • You should use an ordinal GLM instead of a multinomial model, because your response variable is clearly ordered, but a multinomial model fails to take advantage of this info (see When do we use multinomial regression and Poisson regression?). Also, I don't understand how you've coded your predictors. Do you have a third season as your reference level, and reference levels for species and camera model? If you're using dummy coding, you should, but I suspect that you only have two levels of season... – Nick Stauner Aug 13 '14 at 20:59
  • Thanks for editing my question, I really appreciate it. My supervisor suggested a multinomial model as I'm comparing 3 response variables (no reaction, reaction and strong reaction)? No reaction is set as a reference level. I have a reference level for all of the predictor variables, so for instance spring for season, fallow deer for species – user29836 Aug 13 '14 at 21:09
  • Those (no reaction, reaction and strong reaction) are not variables; they are three levels of one ordinal variable (call it "strength of reaction" if you will). Your supervisor should have recommended ordinal logistic regression instead. It is not necessary to choose a reference level for the dependent variable in an ordinal GLM, but thanks for clarifying the reference levels of your predictors. I recommend redoing this analysis as an ordinal GLM (and editing your question if you still have any interpretive issues) rather than concerning yourself with interpretation of these results. – Nick Stauner Aug 13 '14 at 21:12
  • Okay cool, I didn't realise that, thanks. – user29836 Aug 13 '14 at 21:17
  • You can expect p values for your predictors to change depending on what other predictors you include in the model. The difference is essentially a matter of what other variables' predictive relationships you control statistically when estimating a given predictor's relationship with the response. – Nick Stauner Aug 13 '14 at 21:20
  • Okay cool, so the difference in p values between the predictor variable summer sr (0.18) in the model season and the predictor variable summer sr (0.03) in the model species, season and camera isn't related to species but to the relationship between the predictor variables in the second model? And therefore is the p value in the first model the one that is actually representative of the significance of the predictor variable summer in relation to the response level sr? Sorry for all the questions, I really appreciate the help! – user29836 Aug 13 '14 at 21:36

2 Answers2

2

Strong multicollinearity tends to inflate standard errors, so no, that's probably not why the summer coefficient is more significant when controlling other variables. Response variance due to differences in species or camera model may be suppressing the relationship between season and responses. Controlling that noise may give you a clearer sense of how season would relate in isolation, but that may not represent the reality of the broader scenario in which species and camera model may vary (and possibly interact) freely.

It's important that you decide which hypothesis you want to test. Do you want to control the effects of species and camera model when describing the predictive relationship of season, or would you rather leave all other variables uncontrolled? Some other variables will always be uncontrolled, so in my view, controlling them is optional, but may be helpful depending on what you're really trying to learn from your analysis.

P.S. Again, ordinal logistic regression would've been a better analytic choice here. If you have the option, I'd redo the analysis and focus on interpreting those results instead.

Community
  • 1
Nick Stauner
  • 12,342
  • 5
  • 52
  • 110
  • I really appreciate the advice but unfortunately I don't have the time (or expertise) to change over :) I want to show the effect of all predictor variables in isolation included in the analysis based on their AIC and p values when evaluated in isolation. I was thinking I would report the p values from the isolated models and how this affects my hypotheses. I would also include those models with multiple predictor variables which have the lowest AIC values overall and discuss these models, the isolated models and an all predictor variable model. I hope that makes sense :) – user29836 Aug 13 '14 at 23:17
1

I seriously doubt that even your best selected model will be able to successfully deal with so many variable and model problems (collinearity, normality assumption, independent,..). If you really need a functional model, not just something that impressive on paper, construct a hypothetical data set with strong, neutral and negative responses in a spreadsheet. Maintain the number of variables and total number of data points. You know the underlying true values and the selected amount of noise. Does your model work at all in recovering the known underlying values? My experience is seldom. I would recommend reduce the dimensionality by using one variable, and weight the one variable models together. Try taking two variables and similarly combining the models. Are the results much better now in getting back to the known underlying?

Trying to understand why the models in higher dimensions work poorly doesn't help one in getting meaningfully accurate results if that is one true intent. Violating underlying model assumptions (seldom discussed or even revealed) and sensitivity to numerical analysis issues (examples, taking the inverse of a matrix with a determinant value that is small) can be very detrimental to the analysis.

AJKOER
  • 2,180