5

I am trying to increase my model accuracy by taking into account interaction effect of relevant variables. I am choosing variables to interact based more on the common sense than on trying every combination. So far most interaction effects with good p value ( p<.0001) and chi-square have resulted in an increase of true positives. However, the last interaction effect, while having good p values and chi square values, resulted in decreased true positives by 10.

  • Should this happen?
  • How do I interpret this?
  • Shouldnt a variable which is significant always increase my true positives?
Jeromy Anglim
  • 44,984
ayush biyani
  • 1,617

2 Answers2

6

No it should not. e.g. for logistic regression, which appears to be the case here, it may be that the greater p-value (I'm assuming you mean the right kind of p-value here, like from a likelihood ratio test) comes from increasing the odds for (correctly predicted) observations that are already relatively extreme (e.g. all observations that had predicted probability 0.8 become 0.9 and all those that had 0.2 become 0.1), while the ones that were only just on the right side of the 50% threshold are now just on the other side. As a result, the extremities are now predicted with more confidence, but there are more misclassifications.

In general, good fit does not guarantee good prediction (or the other way around) - even though that's the way most 'scientific' publications work these days :-(

I would advise you to look into a more evolved technique like LASSO or elastic net for variable selection... It will also easily allow you to optimize for some predictive measure like missclassification. In R, try glmnet.

Nick Sabbe
  • 12,819
  • 2
  • 37
  • 47
  • 1
    You are making a rather large assumption that all the users of the model are in complete agreement about the choice of a cutoff, i.e., that they have identical utility functions. As this would be the exception rather than the rule, a continuous proper scoring rule is usually called for. And keep in mind that lasso, glmnet, etc. optimize a proper scoring rule (as they should), not proportion "correct". – Frank Harrell Jun 05 '11 at 14:05
4

"True positives", like proportion classified correctly, requires arbitrary and information losing categorization of the predicted values. These are improper scoring rules. An improper scoring rule is a criterion that when optimized leads to a bogus model. Also, watch out when using P-values in any way to guide model selection. This will greatly distort the inference from the final model.

Frank Harrell
  • 91,879
  • 6
  • 178
  • 397
  • @frank-- I understand this point partly..would be great if you can elaborate the "improper scoring rule" and "bogus model" thing please. – ayush biyani May 20 '11 at 05:35
  • 1
    I have a simulated dataset example where adding a very important variable (by any other measure) makes the % classified correctly decrease. This is an example of how an improper scoring rule can mislead. – Frank Harrell May 21 '11 at 00:48