We( @yasufuminakama, @currypurin, @hidehisaarai1213 ) would like to thank Booz Allen Hamilton for the very interesting competition and to all the participants for giving us a lot of ideas.
\n\npython\ndef decaying_counter(x, days):\nreturn max(0.xx, 1-days/30) * x\nFeature generation code is made public at https://github.com/koukyo1994/kaggle-dsb2019/blob/master/src/features/past_summary3_decay.py.\n* Features based on public kernels\nAlthough it worked well, it can also be a cause of overfitting since the distribution of count based features differs between train and test. Therefore, I also applied decaying factor to count them or get average of them, which had already been proved to be effective to avoid overfitting in Y.Nakama's experiment.\nDecaying of count features in Y.Nakama's and mine is slightly different, since Y.Nakama applied decaying to every assessments while I applied decaying to every sessions.\nNote that some of those features which appeared to be not useful so much based on LightGBM importance or showed high correlation with other features were deleted from feature generation code.\n* Past Assessment features\n * {mean, var, last} of {success_ratio, n_failure, accuracy_group} of the same assessment type in the past\n * time to get success\n * {mean, var} of interval of actions (event_code 4020, 4025)\n* Past Game features\n * {mean, var, last} of {correct, incorrect} (decayed) count of each Game\n * {mean, var, last} of success ratio of each Game\n * {mean, var} of interval of actions in each Game\n* Past Activity features\nFew handcrafted features of some of the activities.
As we apply decay function to Count features, we could augment data by using different decay functions. The aim is that oblivion curve is different from each people by age or something.
\n\nFeature selection using LightGBM / CatBoost importance were applied before training. About 80 - 90% of the features were deleted at this step and the resulting number of features are around 400. Feature selection was effective especially to NN model (probably because of high dropout rate in NN model) and bumped up the oof score around 0.005 for GBDT model and 0.01 for NN model.
\n\nApplying certain transformation to the output of multiclass classification gives us better result\ncompared to regression. The transformation is as follows.\npython\nprediction @ np.arange(4) # the format of prediction should be (n_samples, 4)\n
My team tried several objectives. Cross entropy and multiclass worked, used it for the final model.\n* Final model used three models\n * Lightgbm: cross entropy\n * Lightgbm: multiclass\n * Catboost: multiclass\n* cross entropy\n * Divide the target by 3 and convert from 0 to 1, then learn with cross entropy (objective: xentropy). In the final model, this model's weight was the largest.\n* multiclass\n * In multiclass, after calculating the probabilities of the target class from 0 to 3, the following calculation is performed to make continuous values.\n * preds @ np.arange(4)
Our NN model is simple 3 layer MLP. The implementation is here (DSBOvR is the model we used).\nWe used training of one-vs-rest fashion, so the output of the model is a (n_batch, 4) shape tensor and each column represents the probability of each class. torch.nn.BCELoss was used for loss function and after getting the output tensor, following transformation is applied to get (pseudo-)regression value.\npython\nvalid_preds = valid_preds / np.repeat(\n valid_preds.sum(axis=1), 4).reshape(-1, 4) # normalization\nvalid_preds = valid_preds @ np.arange(4) / 3 # conversion to get pseudo-regression value\n\nthis pseudo-regression value can be used for threshold optimization. Note that we normalized this value to be in the range of (0.0, 1.0) while training.\nBefore training, feature selection using LightGBM importance (about 80-90% of the features were deleted), preprocessing (fillna, log transformation for those feature which showed high skewness, feature scaling with StandardScaler) was applied. When training, Adam optimizer is used with CosineAnnealing lr scheduler and for each fold we trained the model 100 epochs. At the end of each epoch we calculate QWK using threshold optimization to pseudo-regression value and saved the weights if the best score is achieved. Final oof and prediction to test data was made with the weights which achived the best QWK score at each fold.\nWe've also prepared NN only kernel here.
Ensemble using all oof is not appropriate to maximize truncated CV.\nTherefore, We sampled the training data at the same ratio as when truncate.\nIn particular, sampling weight is 1/(Assessment Count) for each installation_id.\nBlend is performed based on this sampled data. We also tried stacking by Ridge regression, but we don't think there is a big difference from blending.\nThe threshold is also determined so that the truncated cv of this sampled data is maximized.
\n\nBoth public LB score and oof score was not very helpful to judge if a change in our submission is effective or not. Therefore we used truncation to train data to mimic the generation process of test data. This truncation is mostly the same as that shared in common in discussion (select 1 assessment from each installation_id). Since this score is a bit unstable we repeated the sampling & scoring process 1000 times and calulated the mean of the score.
\n\nWe( @yasufuminakama, @currypurin, @hidehisaarai1213 ) would like to thank Booz Allen Hamilton for the very interesting competition and to all the participants for giving us a lot of ideas.
\n\npython\ndef decaying_counter(x, days):\nreturn max(0.xx, 1-days/30) * x\nFeature generation code is made public at https://github.com/koukyo1994/kaggle-dsb2019/blob/master/src/features/past_summary3_decay.py.\n* Features based on public kernels\nAlthough it worked well, it can also be a cause of overfitting since the distribution of count based features differs between train and test. Therefore, I also applied decaying factor to count them or get average of them, which had already been proved to be effective to avoid overfitting in Y.Nakama's experiment.\nDecaying of count features in Y.Nakama's and mine is slightly different, since Y.Nakama applied decaying to every assessments while I applied decaying to every sessions.\nNote that some of those features which appeared to be not useful so much based on LightGBM importance or showed high correlation with other features were deleted from feature generation code.\n* Past Assessment features\n * {mean, var, last} of {success_ratio, n_failure, accuracy_group} of the same assessment type in the past\n * time to get success\n * {mean, var} of interval of actions (event_code 4020, 4025)\n* Past Game features\n * {mean, var, last} of {correct, incorrect} (decayed) count of each Game\n * {mean, var, last} of success ratio of each Game\n * {mean, var} of interval of actions in each Game\n* Past Activity features\nFew handcrafted features of some of the activities.
As we apply decay function to Count features, we could augment data by using different decay functions. The aim is that oblivion curve is different from each people by age or something.
\n\nFeature selection using LightGBM / CatBoost importance were applied before training. About 80 - 90% of the features were deleted at this step and the resulting number of features are around 400. Feature selection was effective especially to NN model (probably because of high dropout rate in NN model) and bumped up the oof score around 0.005 for GBDT model and 0.01 for NN model.
\n\nApplying certain transformation to the output of multiclass classification gives us better result\ncompared to regression. The transformation is as follows.\npython\nprediction @ np.arange(4) # the format of prediction should be (n_samples, 4)\n
My team tried several objectives. Cross entropy and multiclass worked, used it for the final model.\n* Final model used three models\n * Lightgbm: cross entropy\n * Lightgbm: multiclass\n * Catboost: multiclass\n* cross entropy\n * Divide the target by 3 and convert from 0 to 1, then learn with cross entropy (objective: xentropy). In the final model, this model's weight was the largest.\n* multiclass\n * In multiclass, after calculating the probabilities of the target class from 0 to 3, the following calculation is performed to make continuous values.\n * preds @ np.arange(4)
Our NN model is simple 3 layer MLP. The implementation is here (DSBOvR is the model we used).\nWe used training of one-vs-rest fashion, so the output of the model is a (n_batch, 4) shape tensor and each column represents the probability of each class. torch.nn.BCELoss was used for loss function and after getting the output tensor, following transformation is applied to get (pseudo-)regression value.\npython\nvalid_preds = valid_preds / np.repeat(\n valid_preds.sum(axis=1), 4).reshape(-1, 4) # normalization\nvalid_preds = valid_preds @ np.arange(4) / 3 # conversion to get pseudo-regression value\n\nthis pseudo-regression value can be used for threshold optimization. Note that we normalized this value to be in the range of (0.0, 1.0) while training.\nBefore training, feature selection using LightGBM importance (about 80-90% of the features were deleted), preprocessing (fillna, log transformation for those feature which showed high skewness, feature scaling with StandardScaler) was applied. When training, Adam optimizer is used with CosineAnnealing lr scheduler and for each fold we trained the model 100 epochs. At the end of each epoch we calculate QWK using threshold optimization to pseudo-regression value and saved the weights if the best score is achieved. Final oof and prediction to test data was made with the weights which achived the best QWK score at each fold.\nWe've also prepared NN only kernel here.
Ensemble using all oof is not appropriate to maximize truncated CV.\nTherefore, We sampled the training data at the same ratio as when truncate.\nIn particular, sampling weight is 1/(Assessment Count) for each installation_id.\nBlend is performed based on this sampled data. We also tried stacking by Ridge regression, but we don't think there is a big difference from blending.\nThe threshold is also determined so that the truncated cv of this sampled data is maximized.
\n\nBoth public LB score and oof score was not very helpful to judge if a change in our submission is effective or not. Therefore we used truncation to train data to mimic the generation process of test data. This truncation is mostly the same as that shared in common in discussion (select 1 assessment from each installation_id). Since this score is a bit unstable we repeated the sampling & scoring process 1000 times and calulated the mean of the score.
\n\n