Reference: https://www.kaggle.com/c/liberty-mutual-fire-peril/details/evaluation
Submissions are evaluated on the normalized, weighted Gini co-efficient. The weights used in the calculation are represented by Comp_Earned_Count in the dataset.
To calculate the normalized weighted Gini, your predictions are sorted from largest to smallest. This is the only step where the explicit prediction values are used (i.e. only the order of your predictions matters). We then move from largest to smallest, asking "In the leftmost x% of the data, how much of the actual observed, weighted loss (the target multiplied by the provided weight) have you accumulated?" With no model, you expect to accumulate 10% of the loss in 10% of the predictions, so no model (or a "null" model) achieves a straight line. The area between your curve and this straight line is the Gini coefficient.
There is a maximum achievable area for a perfect model. The normalized Gini is obtained by dividing the weighted Gini coefficient of your model by the weighted Gini coefficient of a perfect model.
Submission Format
Your submission should be in a CSV file with exactly 47,896 rows (including the header) and 2 columns:
The first column is the unique row 'id' provided within the dataset
The second column is the predicted claim frequency for the corresponding id. It should be non-negative.
The CSV file would have a format like the following:
id,predictions
1,0.45
2,2.1
3,0.01
4,0.56
etc.
Sample code to generate a CSV of holdout predictions for an intercept-only model is as follows:
# Import data
require(data.table)
train = fread("compDatasetTrain.csv")
test = fread("compDatasetTest.csv")
# Construct intercept-only model
glmMod = glm(Comp_Claim_Count ~ 1, data = train, family = poisson(link = "log"), offset = log(Comp_Earned_Count))
# Order the test dataset by id
test = test[order(test$id),]
# Compute predictions - predicted frequency is equal to the predicted comp claim count divided by the comp earn count
predictions = predict(glmMod, newdata = test,
type = "response")/test$Comp_Earned_Count
# Save predictions
write.csv(data.frame("id" = test$id, "predictions" = predictions), "mySubmission.csv", row.names = F)
The above code will output a file titled "mySubmission.csv" in your working directory. Submit it by going to the "Make a Submission" tab (on the left pane), uploading the "mySubmission.csv" file, and clicking "Submit". If the submission is successful (i.e. in the correct format), it will be scored and you will be re-directed to the Public Leaderboard -- at this point, if the submission is better than your previous submission, the score on the Public Leaderboard will be revised to reflect the better score; if it is not better than the previous submission, your score on the Public Leaderboard will not change.
Let's revise the sample code to generate predictions for a simple model with Accident Year and 2 randomly selected predictors (Driver_Age, and Comp_Deductible):
# Import data
require(data.table)
train = fread("compDatasetTrain.csv")
test = fread("compDatasetTest.csv")
# Impute Driver_Age in train and test with median Driver_Age - of the three selected predictors, it is the only 1 with missing values
medianDriverAge = median(c(train$Driver_Age, test$Driver_Age), na.rm = T)
train[is.na(train$Driver_Age), "Driver_Age"] = medianDriverAge
test[is.na(test$Driver_Age), "Driver_Age"] = medianDriverAge
# Construct model with Accident year and 2 randomly selected predictors
glmMod = glm(Comp_Claim_Count ~ Accident_Year + Driver_Age + Comp_Deductible, data = train, family = poisson(link = "log"), offset = log(Comp_Earned_Count))
# Order the test dataset by id
test = test[order(test$id),]
# Compute predictions - predicted frequency is equal to the predicted comp claim count divided by the comp earn count
predictions = predict(glmMod, newdata = test, type = "response")/test$Comp_Earned_Count
# Save predictions
write.csv(data.frame("id" = test$id, "predictions" = predictions), "mySubmission2.csv", row.names = F)
The above code will output a file titled "mySubmission2.csv" in your working directory. Perform the same process as for "mySubmission.csv" to score this submission.
Note: The above model is a crude model mainly for the purpose of illustration. In reality, you may choose to fit accident year as a categorical variable instead of a continuous one.