![\"results\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F347731%2F71393d4b9490bd66d06d71e03ea90aa1%2Fimage.png?generation=1579958134153508&alt=media\")
Congratulations to all winners!\nHere is brief summary of our solution.
\n\nUsing QWK for tuning models was too difficult, so we decided to evaluate only the RMSE for the model performance.\nGroup 5 folds is used as validation method. We applied truncation to validation set.\n- 1st level\n - lgbm: CV 1.0395 +/- 0.031\n - objective rmse\n - 3 random seed averaging\n - xgb: CV 1.0457 +/- 0.028\n - objective rmse\n - 3 random seed averaging\n - catboost: CV 1.0430 +/- 0.028\n - objective rmse\n - NN: CV 1.0423 +/- 0.029\n - rmse + smooth l1 loss\n - RNN-layer: GRU + Attention\n - sequence of last 6 histories as input\n - Dense-layer\n - 3 random seed blending\n - NN model has the almost same performance as boosting tree models, but has a low correlation.\n- 2nd level: PublicLB 0.538 PrivateLB 0.556\n - ElasticNet: CV 1.0361 +/- 0.028\nIn addition, we cloud not include it though, lgbm with accuracy classification is best model in our experiments.
\n\nThe most time was spent on how to determine the threshold. We prepared some ides and experimented.\nWe did sampling with replacement 10 times from each fold's OOF for each installation_id (we called it as OTV).\nThen, we applied following thresholding methods with 5 folds CV of OOF.\n- 1. Confirm label distribution of validation set to OOF, OTV true label distribution (many kernel did)\n- 2. Apply Optimized Rounder to oof, otv and get thresholds from them, then confirm the validation label distribution to optimized OOF, OTV label distribution.\n- 3. Apply Optimized Rounder to OOF, OTV and get thresholds from them, then apply thresholds to validation set.\nIn our experiments, No. 2 method with otv almost got top score, but sometimes No.3 with OOF did.\nSo we chose two thresholding methods as final submissions.\nExperiment results close to PrivateLB.\nThe results of the above experiment gave almost the same results as PrivateLB.
\n\n
The extension of train set by test set was used only for training each 5 folds, and was not used for determining the thresholds or as validation set.
", "comments": [ { "id": 735071, "author_name": "NorwayPing", "author_url": "", "post_date": "2020-02-02T14:17:15.613000", "content": "Thanks for sharing!
", "votes": 0, "replies": [] } ], "comment_count": 1, "messages": [ { "id": 728914, "postDate": "2020-01-25T13:17:53.343Z", "content": "Congratulations to all winners!\nHere is brief summary of our solution.
\n\nUsing QWK for tuning models was too difficult, so we decided to evaluate only the RMSE for the model performance.\nGroup 5 folds is used as validation method. We applied truncation to validation set.\n- 1st level\n - lgbm: CV 1.0395 +/- 0.031\n - objective rmse\n - 3 random seed averaging\n - xgb: CV 1.0457 +/- 0.028\n - objective rmse\n - 3 random seed averaging\n - catboost: CV 1.0430 +/- 0.028\n - objective rmse\n - NN: CV 1.0423 +/- 0.029\n - rmse + smooth l1 loss\n - RNN-layer: GRU + Attention\n - sequence of last 6 histories as input\n - Dense-layer\n - 3 random seed blending\n - NN model has the almost same performance as boosting tree models, but has a low correlation.\n- 2nd level: PublicLB 0.538 PrivateLB 0.556\n - ElasticNet: CV 1.0361 +/- 0.028\nIn addition, we cloud not include it though, lgbm with accuracy classification is best model in our experiments.
\n\nThe most time was spent on how to determine the threshold. We prepared some ides and experimented.\nWe did sampling with replacement 10 times from each fold's OOF for each installation_id (we called it as OTV).\nThen, we applied following thresholding methods with 5 folds CV of OOF.\n- 1. Confirm label distribution of validation set to OOF, OTV true label distribution (many kernel did)\n- 2. Apply Optimized Rounder to oof, otv and get thresholds from them, then confirm the validation label distribution to optimized OOF, OTV label distribution.\n- 3. Apply Optimized Rounder to OOF, OTV and get thresholds from them, then apply thresholds to validation set.\nIn our experiments, No. 2 method with otv almost got top score, but sometimes No.3 with OOF did.\nSo we chose two thresholding methods as final submissions.\nExperiment results close to PrivateLB.\nThe results of the above experiment gave almost the same results as PrivateLB.
\n\n
The extension of train set by test set was used only for training each 5 folds, and was not used for determining the thresholds or as validation set.
", "rawMarkdown": "Congratulations to all winners!\nHere is brief summary of our solution.\n\n### Feature Engineering\n- worked\n - is 1st assessment or not\n - Normalized counting feature: count for each event codes and ids / game session duration\n - Whether assessment is solved in order of the game design, or not\n - aggregations\n - durations for each type (mean, std, min)\n - the number of records for each type (mean, std, min)\n- not worked\n - last activity, last game statistics\n - corrects, incorrects, misses, rounds, levels..\n - last type, title history sequence\n\n### Models\nUsing QWK for tuning models was too difficult, so we decided to evaluate only the RMSE for the model performance.\nGroup 5 folds is used as validation method. We applied truncation to validation set.\n- 1st level\n - lgbm: CV 1.0395 +/- 0.031\n - objective rmse\n - 3 random seed averaging\n - xgb: CV 1.0457 +/- 0.028\n - objective rmse\n - 3 random seed averaging\n - catboost: CV 1.0430 +/- 0.028\n - objective rmse\n - NN: CV 1.0423 +/- 0.029\n - rmse + smooth l1 loss\n - RNN-layer: GRU + Attention\n - sequence of last 6 histories as input\n - Dense-layer\n - 3 random seed blending\n - NN model has the almost same performance as boosting tree models, but has a low correlation.\n- 2nd level: PublicLB 0.538 PrivateLB 0.556\n - ElasticNet: CV 1.0361 +/- 0.028\nIn addition, we cloud not include it though, lgbm with accuracy classification is best model in our experiments.\n\n### Thresholding\nThe most time was spent on how to determine the threshold. We prepared some ides and experimented.\nWe did sampling with replacement 10 times from each fold's OOF for each installation_id (we called it as OTV).\nThen, we applied following thresholding methods with 5 folds CV of OOF.\n- 1. Confirm label distribution of validation set to OOF, OTV true label distribution (many kernel did)\n- 2. Apply Optimized Rounder to oof, otv and get thresholds from them, then confirm the validation label distribution to optimized OOF, OTV label distribution.\n- 3. Apply Optimized Rounder to OOF, OTV and get thresholds from them, then apply thresholds to validation set.\nIn our experiments, No. 2 method with otv almost got top score, but sometimes No.3 with OOF did.\nSo we chose two thresholding methods as final submissions.\nExperiment results close to PrivateLB.\nThe results of the above experiment gave almost the same results as PrivateLB.\n\n\n\n### train dataset augmentation\nThe extension of train set by test set was used only for training each 5 folds, and was not used for determining the thresholds or as validation set.\n", "votes": 25 }, { "id": 735071, "postDate": "2020-02-02T14:17:15.613Z", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!" } ], "complete": true, "source": "live", "fetched_at": "2026-10-23T23:50:31Z", "competition_id": 16531, "competition_slug": "data-science-bowl-2019", "team_id": 3925649, "solution_url": "/competitions/data-science-bowl-2019/writeups/data-science-bowz-20th-place-solution" }