1

I'm following this tutorial to fit a logistic regression model on to my data which has a binary response. I've understood the reasoning behind each step, apart from why the author checks the first 5 probabilities within glm.probs and observes they are close to 50% and so sets glm.pred to the following:

glm.pred <- ifelse(glm.probs > 0.5, "Up", "Down")

Lets say you observe the first 5 or even take the mean of probabilities and you obtain a value of 0.25. Does that now mean you set glm.pred to:

glm.pred <- ifelse(glm.probs > 0.25, "Up", "Down")

Also another question I have is regarding the removal of variables. From my understanding, a variable with a p-value > 0.05 means we cannot conclude the effect the variable has on the model. Some say you shouldn't remove the variable itself as you lose important information. So how do we then try to improve the model?

Ali
  • 209
  • 3
    Just as the significance level, the break value used for dichotomisation is not dependent on your data. In fact, one could argue that you shouldn't even do this step. Anyway, this is off-topic here. – Roland Mar 29 '19 at 11:32
  • Would it not produce a higher classification rate and solve the problem of overfitting by testing the model with new data? – Ali Mar 29 '19 at 11:34
  • 2
    What you asking is theory, rather than code. Try stackexchange instead, and search for logistic regression probability cut-off points, there you will find the answers to these questions. – Felipe Alvarenga Mar 29 '19 at 11:39

2 Answers2

1

Logistic regression returns the probability of the outcome conditioned on the covariates (or the features, as it is sometimes said in ML). In this example, the logistic regression returns the probability of "Up" given the covariates.

When the author does glm.pred <- ifelse(glm.probs > 0.5, "Up", "Down"), what they are doing implicitly is saying that if the model returns a probability greater than 50%, then assume that the "Up" event will occur. Thus, your prediction will be "Up". If the probability is less than or equal to 0.5, predict "Down".

Demetri Pananos
  • 36,121
0

One way to determine the threshold is to find out where the sum of sensitivity and specificity is maximal. Here is an example https://stackoverflow.com/q/23240182/11131830

For your second question, a variable with a p-value > 0.05, you can conduct a test to determine whether or not you should remove the variable. One example is to compare the AIC between models with and without the variable. The model with lower AIC means it is a better model with smaller information loss.

Yolanda Pan
  • 18