R - Classification ctree {party} - Testing sample and leaf attribution with unbalanced data

Question

Let's start with data description of the website visits I analyse :

• 6M rows
• Dependant variable quotation is binary and takes values 0 and 1 with 1% of value 1
• The other 3 variables are temperature, humidity and minute of the day

The objective is to identify quotation trend based on the weather to optimize communication campaigns and not to determine if for a given visit there will be a quotation.

To avoid overfitting problems due to the large dataset I decided to cross-validate my tree-models to determine the right one.

My questions :

Due to the low probability of quotation = 1 even the best leaf-node gets a 5% with the training sample. Therefore, if I do a predict() on my Testing sample I get only 0 for all nodes.

1. Is there a way with the party package to attribute the corresponding node to each value of the Testing sample
2. Is that the right method to evaluate my different models since predict() doesn't seem to work for me (0 for all observations)?

I went there but every suggestions are based on predict which is I feel of no help in my case...

Answer 1

Run this example:

library(party)

set.seed(15)

# example data
dt = data.frame(y = c(rbinom(n=2000,size=1,prob=0.1), 
                      rbinom(n=2000,size=1,prob=0.2),
                      rbinom(n=2000,size=1,prob=0.3)),
                group = c(rep("A",3000), rep("B",3000)),
                x = c(sort(rnorm(3000,50,2)), sort(rnorm(3000,70,3), decreasing = T)))

dt$y = as.factor(dt$y)

# separate train and test set (50/50 split here)
rn = sample(1:nrow(dt), 3000)

dt_train = dt[rn,]
dt_test = dt[-rn,]

# build model
model = ctree(y~group+x, data = dt_train)

# visualise model
plot(model, type="simple")

# predict new data
dt_test$predClass = predict(model, newdata=dt_test, type="response")    # obtain the class (0/1)
dt_test$predProb = sapply(predict(model, newdata=dt_test,type="prob"),'[[',2)  # obtain probability of class 1 (second element from the lists)
dt_test$predNode = predict(model, newdata=dt_test, type="node")   # obtain the predicted node (in case you need it)

You will see that ALL predClass values in dt_test are 0. You can use column predProb to create your own classification based on your threshold. For example:

table(dt_test$predClass, dt_test$y)  # everything is classified as 0

#      0    1
# 0 2392  608
# 1    0    0

# pick a threshold of 0.2
dt_test$predClass2 = 0
dt_test$predClass2[dt_test$predProb >= 0.2] = 1


table(dt_test$predClass2, dt_test$y)  # you have some cases classified as 1

#      0    1
# 0 1342  229
# 1 1050  379

If you want to see how good your model is you can use an ROC curve. No need to use thresholds to classify now, as the process will use your predicted probabilities as they are:

library(ROCR)

# plot ROC
roc_pred <- prediction(dt_test$predProb, dt_test$y)
perf <- performance(roc_pred, "tpr", "fpr")
plot(perf, col="red")
abline(0,1,col="grey")

# get area under the curve
performance(roc_pred,"auc")@y.values