8

I'm working on a project using logistic regression to predict student retention. The data were collected through three self-report instruments. We are trying to find out which predictors are powerful enough to predict the at-risk students. I came across some articles saying a balanced sample (50% stay, 50% dropouts) is desirable for such study, e.g. Glynn, J.G., Sauer, P.L., & Miller, T.E. (2003). Signaling Student Retention With Prematriculation Data, NASPA Journal, 41 (1), 41-67:

A problem however, is that the distribution of the dependent variable is likely to be highly skewed toward persistence. For example, if 85% of the analysis sample were persistors, a classification model that classified every student as a persistor would have a success rate of 85%, or would classify 85% of student correctly. To resolve this issue, the maintenance of relative balance between the number of dropouts and the number of persistors (about 50% each) in the analysis sample was desirable.

Is this true? Our sample only has about 25%-30% dropout students. Will this affect the results?

kjetil b halvorsen
  • 77,844
MingL
  • 103
  • 1
  • 5
  • Do you have a link/citation to those articles? – Gala Aug 07 '13 at 15:39
  • 3
    You should include more information about your model and what data you have. There is no assumption about the proportion of 0's and 1's in the dependent variable of a standard logistic regression, but other more specialized uses of logistic regression may be different. – caburke Aug 07 '13 at 15:42
  • @Laurans: --Signaling student retention with prematriculation data, Glynn, Sauler, and Miller, 2003. – MingL Aug 07 '13 at 15:57
  • @caburke: Thanks. The article saying "A problem however, is that the distribution of the dependent variable is likely to be highly skewed toward persistence. For example, if 85% of the analysis sample were persistors, a classification model that classified every student as a persistor would have a success rate of 85%, or would classify 85% of student correctly. To resolve this issue, the maintenance of relative balance between the number of dropouts and the number of persistors ( about 50% each)in the analysis sample was desirable. "Does this make sense to you? – MingL Aug 07 '13 at 15:58
  • @MingL A link is one that works. A citation is one that would be acceptable in a professional publication. Your reference is neither.... – Nick Cox Aug 07 '13 at 16:04
  • 1
    @NickCox:http://www.uccs.edu/Documents/retention/2003%20Signaling%20Student%20Retention%20With%20Prematriculation%20Data.pdf – MingL Aug 07 '13 at 16:06
  • 4
    @MingL It is true that a model that constantly predicts a student to be a persistor would classify 85% of the students in the sample correctly, but including just some of the information available to you should result in a better in-sample classification rate. This only suggests that looking at the in-sample classification rate independent of the data obtained is not a good assessment of goodness-of-fit or predictive ability of the model. Using the Deviance Information Criterion or cross validation to evaluate the predictive ability of the model would be much better. – caburke Aug 07 '13 at 16:18
  • For prediction, I find that training on a balanced dataset leads to superior models. – Seanosapien Mar 04 '18 at 16:48

3 Answers3

15

This is not so much a problem of logistic regression per se as it is a problem with classification accuracy as a performance measure. Note that balancing the data set is not necessarily the only valid approach. If one of the classes is actually much more common in the population (and not merely in your sample), a naive model (classifying everything as belonging to the most common category) really is a good guess. If the error costs are not symmetric, balancing the data set might lead you to err in the wrong direction (the more costly one).

The problem also often comes up the other way around: Training/evaluating on some artificially balanced data set before using the resulting model in a strongly unbalanced situation (think detecting fraud or diagnosing a rare disease) where the usefulness of the model is not nearly as high as the raw accuracy would suggest. It all depends on your objectives and your cost structure.

Gala
  • 8,501
  • Thank you for your explanation. It helps a lot! I like what you said it all depends on the objectives. If my objective is to predict at-risk students (how many potential dropouts can we predict based on the model we have?), do you have any suggestions/thoughts about model selection (e.g. how to obtain a parsimonious model? Only based on r square, and strength of coefficients?)Sorry I'm new to logistic regression :). – MingL Aug 07 '13 at 19:14
  • 4
    Using classification accuracy to judge the performance of a model is discouraged in this situation for precisely this reason. A better alternative is the area under the ROC curve. – orizon Aug 08 '13 at 09:04
1

Yes; it will affect the results. Logistic regression fits an MLE by minimizing an objective function which is evaluated at all the data points. If the data is unbalanced then the minimization will be unbalanced too.

While your example is not extreme, you will get different answers if you re-balance.

A good explanation of this and how to address it is in King and Zeng, http://gking.harvard.edu/files/gking/files/0s.pdf.

Nick Cox
  • 56,404
  • 8
  • 127
  • 185
Chris
  • 625
0

Class imbalance can be a real problem. An alternative to down-sampling would be to assign costs to the different classes, which is supported in popular toolkits.

E.g. look for the -j parameter in SvmLight (for support-vector regression), or the -w in LibLinear (for different kinds of linear regression).

Nick Cox
  • 56,404
  • 8
  • 127
  • 185
benroth
  • 305