First of all, I would like to thank Booz Allen Hamilton and Kaggle for hosting such an interesting competition! I moved up from about 800th place to 50th place in 9 days before the end of the competition, and I moved up from about 80th place to 23rd place on the final day of the competition. So this week was very thrilling for me.
\n\nMy solution was very simple. The summary is below.\n1. Feature engineering (almost aggregation features)\n2. Make some diverse models and stacking\n3. Threshold tuning with random search
\n\nI introduce my solution's key points briefly.
\n\nMany kernels used get_data function that deals with the user behaviour data sequentially.But I thought this function made it difficult for me to make and manage features.So I made a new approach for making train dataset. Let me show this. At first, to each game session, I assigned the number of assessment that a user tried until that game session . Below is an example.\n\ntrain_gs_assess_dict = {}\nfor ins_id, user_sample in tqdm(train.groupby('installation_id')):\n assess_count = 0\n for gs, session in user_sample.groupby('game_session', sort=False):\n if session['type'].iloc[0] == 'Assessment':\n assess_count += 1\n train_gs_assess_dict[gs] = assess_count\ntrain['assess_count'] = train['game_session'].map(train_gs_assess_dict)\n\nThen I calculated the aggregation features for the subset of user activities before the assessment. The duration for creating train and test dataset became longer than kernel's. But this made implementation and management of features very easy.
I created the 8 models at first level.\n|model|type|target|eval metrics|corr with accuracy group|kendall's tau|\n|:---:|:---:|:---:|:---:|:---:|:---:|\n|LightGBM|gbdt|accuracy group|rmse|0.621|0.460|\n|LightGBM|goss|accuracy group|rmse|0.568|0.433|\n|LightGBM|dart|accuracy group|rmse|0.619|0.459|\n|LightGBM|gbdt|accuracy|rmse|0.615|0.457|\n|LightGBM|gbdt|accuracy group>2| auc|0.598|0.452|\n|LightGBM|gbdt|accuracy group>1| auc|0.615|0.456|\n|LightGBM|gbdt|accuracy group>0| auc|0.597|0.441|\n|NN|-|accuracy group|rmse|0.600|0.444|
\n\nAnd I used Ridge Regression for stacking.\n|model|type|target|eval metrics|corr coef with accuracy group|kendall's tau|\n|:---:|:---:|:---:|:---:|:---:|:---:|\n|Ridge Regression|-|accuracy group|-|0.628|0.467|
\n\nStrangely, the weight of prediction which had the best correlation coefficient with accuracy group became 0. But this stacking was so effective. It pushed me up near the gold zone.
\n\nA threshold was very important in this competition. At first, I used OptimizedRounder which many kernels used. But I found this function depended on the initial value, and output was likely to fall into a local solution from my experiments. So I used a random search for deciding thresholds. This approach pushed me up about 800th place. And I thought public and private dataset was very similar because my adversarial validation's AUC was around 0.5. So I selected thresholds that maximize mean-QWK for 100 datasets which were truncated randomly from train dataset.
", "comments": [ { "id": 732695, "author_name": "Zichuan Liu", "author_url": "", "post_date": "2020-01-30T05:51:17.533000", "content": "Thank you for sharing. Can I learn the code？
", "votes": 0, "replies": [ { "id": 732973, "author_name": "keeeeei79", "author_url": "", "post_date": "2020-01-30T14:07:16.717000", "content": "OK. I'll open the kernel of my whole code. Please wait for a while.
", "votes": 0, "replies": [] }, { "id": 733346, "author_name": "Zichuan Liu", "author_url": "", "post_date": "2020-01-31T02:15:01.513000", "content": "Thanks!!!
", "votes": 0, "replies": [] } ] }, { "id": 728829, "author_name": "Miyabon", "author_url": "", "post_date": "2020-01-25T10:52:36.097000", "content": "Congrats & Thanks for sharing your solution🎉 😄 👍
", "votes": 0, "replies": [] }, { "id": 728841, "author_name": "", "author_url": "", "post_date": "2020-01-25T11:15:03.513000", "content": "", "votes": 0, "replies": [] }, { "id": 730202, "author_name": "", "author_url": "", "post_date": "2020-01-27T08:03:55.620000", "content": "Congrats !! Thanks for sharing !
", "votes": 3, "replies": [] }, { "id": 733271, "author_name": "corochann", "author_url": "", "post_date": "2020-01-30T23:03:16.080000", "content": "Congrats and thank you for sharing
", "votes": 0, "replies": [] } ], "comment_count": 7, "messages": [ { "id": 728738, "postDate": "2020-01-25T07:04:15.687Z", "content": "First of all, I would like to thank Booz Allen Hamilton and Kaggle for hosting such an interesting competition! I moved up from about 800th place to 50th place in 9 days before the end of the competition, and I moved up from about 80th place to 23rd place on the final day of the competition. So this week was very thrilling for me.
\n\nMy solution was very simple. The summary is below.\n1. Feature engineering (almost aggregation features)\n2. Make some diverse models and stacking\n3. Threshold tuning with random search
\n\nI introduce my solution's key points briefly.
\n\nMany kernels used get_data function that deals with the user behaviour data sequentially.But I thought this function made it difficult for me to make and manage features.So I made a new approach for making train dataset. Let me show this. At first, to each game session, I assigned the number of assessment that a user tried until that game session . Below is an example.\n\ntrain_gs_assess_dict = {}\nfor ins_id, user_sample in tqdm(train.groupby('installation_id')):\n assess_count = 0\n for gs, session in user_sample.groupby('game_session', sort=False):\n if session['type'].iloc[0] == 'Assessment':\n assess_count += 1\n train_gs_assess_dict[gs] = assess_count\ntrain['assess_count'] = train['game_session'].map(train_gs_assess_dict)\n\nThen I calculated the aggregation features for the subset of user activities before the assessment. The duration for creating train and test dataset became longer than kernel's. But this made implementation and management of features very easy.
I created the 8 models at first level.\n|model|type|target|eval metrics|corr with accuracy group|kendall's tau|\n|:---:|:---:|:---:|:---:|:---:|:---:|\n|LightGBM|gbdt|accuracy group|rmse|0.621|0.460|\n|LightGBM|goss|accuracy group|rmse|0.568|0.433|\n|LightGBM|dart|accuracy group|rmse|0.619|0.459|\n|LightGBM|gbdt|accuracy|rmse|0.615|0.457|\n|LightGBM|gbdt|accuracy group>2| auc|0.598|0.452|\n|LightGBM|gbdt|accuracy group>1| auc|0.615|0.456|\n|LightGBM|gbdt|accuracy group>0| auc|0.597|0.441|\n|NN|-|accuracy group|rmse|0.600|0.444|
\n\nAnd I used Ridge Regression for stacking.\n|model|type|target|eval metrics|corr coef with accuracy group|kendall's tau|\n|:---:|:---:|:---:|:---:|:---:|:---:|\n|Ridge Regression|-|accuracy group|-|0.628|0.467|
\n\nStrangely, the weight of prediction which had the best correlation coefficient with accuracy group became 0. But this stacking was so effective. It pushed me up near the gold zone.
\n\nA threshold was very important in this competition. At first, I used OptimizedRounder which many kernels used. But I found this function depended on the initial value, and output was likely to fall into a local solution from my experiments. So I used a random search for deciding thresholds. This approach pushed me up about 800th place. And I thought public and private dataset was very similar because my adversarial validation's AUC was around 0.5. So I selected thresholds that maximize mean-QWK for 100 datasets which were truncated randomly from train dataset.
", "rawMarkdown": "First of all, I would like to thank Booz Allen Hamilton and Kaggle for hosting such an interesting competition! I moved up from about 800th place to 50th place in 9 days before the end of the competition, and I moved up from about 80th place to 23rd place on the final day of the competition. So this week was very thrilling for me.\n\nMy solution was very simple. The summary is below.\n1. Feature engineering (almost aggregation features)\n2. Make some diverse models and stacking\n3. Threshold tuning with random search\n\nI introduce my solution's key points briefly.\n# How to make train and test dataset\nMany kernels used `get_data` function that deals with the user behaviour data sequentially.But I thought this function made it difficult for me to make and manage features.So I made a new approach for making train dataset. Let me show this. At first, to each game session, I assigned the number of assessment that a user tried until that game session . Below is an example.\n```\ntrain_gs_assess_dict = {}\nfor ins_id, user_sample in tqdm(train.groupby('installation_id')):\n assess_count = 0\n for gs, session in user_sample.groupby('game_session', sort=False):\n if session['type'].iloc[0] == 'Assessment':\n assess_count += 1\n train_gs_assess_dict[gs] = assess_count\ntrain['assess_count'] = train['game_session'].map(train_gs_assess_dict)\n```\nThen I calculated the aggregation features for the subset of user activities before the assessment. The duration for creating train and test dataset became longer than kernel's. But this made implementation and management of features very easy.\n\n# Model and stacking\nI created the 8 models at first level.\n|model|type|target|eval metrics|corr with accuracy group|kendall's tau|\n|:---:|:---:|:---:|:---:|:---:|:---:|\n|LightGBM|gbdt|accuracy group|rmse|0.621|0.460|\n|LightGBM|goss|accuracy group|rmse|0.568|0.433|\n|LightGBM|dart|accuracy group|rmse|0.619|0.459|\n|LightGBM|gbdt|accuracy|rmse|0.615|0.457|\n|LightGBM|gbdt|accuracy group>2| auc|0.598|0.452|\n|LightGBM|gbdt|accuracy group>1| auc|0.615|0.456|\n|LightGBM|gbdt|accuracy group>0| auc|0.597|0.441|\n|NN|-|accuracy group|rmse|0.600|0.444|\n\nAnd I used Ridge Regression for stacking.\n|model|type|target|eval metrics|corr coef with accuracy group|kendall's tau|\n|:---:|:---:|:---:|:---:|:---:|:---:|\n|Ridge Regression|-|accuracy group|-|0.628|0.467|\n\nStrangely, the weight of prediction which had the best correlation coefficient with accuracy group became 0. But this stacking was so effective. It pushed me up near the gold zone.\n\n# Threshold tuning\nA threshold was very important in this competition. At first, I used OptimizedRounder which many kernels used. But I found this function depended on the initial value, and output was likely to fall into a local solution from my experiments. So I used a random search for deciding thresholds. This approach pushed me up about 800th place. And I thought public and private dataset was very similar because my adversarial validation's AUC was around 0.5. So I selected thresholds that maximize mean-QWK for 100 datasets which were truncated randomly from train dataset.", "votes": 29 }, { "id": 732695, "postDate": "2020-01-30T05:51:17.533Z", "content": "Thank you for sharing. Can I learn the code？
", "rawMarkdown": "Thank you for sharing. Can I learn the code？", "replies": [ { "id": 732973, "postDate": "2020-01-30T14:07:16.717Z", "content": "OK. I'll open the kernel of my whole code. Please wait for a while.
", "rawMarkdown": "OK. I'll open the kernel of my whole code. Please wait for a while." }, { "id": 733346, "postDate": "2020-01-31T02:15:01.513Z", "content": "Thanks!!!
", "rawMarkdown": "Thanks!!!" } ] }, { "id": 728829, "postDate": "2020-01-25T10:52:36.097Z", "content": "Congrats & Thanks for sharing your solution🎉 😄 👍
", "rawMarkdown": "Congrats & Thanks for sharing your solution🎉 😄 👍 " }, { "id": 728841, "postDate": "2020-01-25T11:15:03.513Z", "rawMarkdown": "", "isDeleted": true }, { "id": 730202, "postDate": "2020-01-27T08:03:55.620Z", "content": "Congrats !! Thanks for sharing !
", "rawMarkdown": "Congrats !! Thanks for sharing !", "votes": 3 }, { "id": 733271, "postDate": "2020-01-30T23:03:16.080Z", "content": "Congrats and thank you for sharing
", "rawMarkdown": "Congrats and thank you for sharing" } ], "complete": true, "source": "live", "fetched_at": "2026-10-23T23:49:56Z", "competition_id": 16531, "competition_slug": "data-science-bowl-2019", "team_id": 3863281, "solution_url": "/competitions/data-science-bowl-2019/writeups/keeeeei79-9th-place-solution" }