First, I would like to thank the host, Kaggle, and everyone for this great competition. I would like to share my deepest gratitude to my teammates @alijs1 @johnpateha @kazanova . It was a really good time and joyful period for me.
\n\nI will briefly brief our solution, and if I miss something, my teammates can add in their perspective.
\n\nBesides some popular features available in public kernels, we have some more custom features, such as ratio of good actions / all actions in Activity sessions, ratio of misclick or misdrag action / count actions (or session duration), count of some specific event codes since the previous Assessment session. The most interesting features, would be stats on the same assessment with regards to each data sample. This class of features helps tree models to converge quicker and reduce the importance of Assessment title.\nAs a separate solution, Marios (@kazanova) crafted his own feature set which took into consideration the train/test mismatch. He can share more about this if needed.
\n\nIt is surprising that we did not realize the possibility to augment train data by a lot of test samples with true labels. Whenever a test installation_id (hereinafter referred to as “id”) has more than 0 prior assessment, we can trim the user history and make extra training samples. This augmentation helped us a lot in the blend.
\n\nBesides using all original data (17690 rows) as the main base model, we noticed that training 5 separate submodels for each type can give a boost if properly blended with the main base model. And since we also have the option of data augmentation, it results in 4 different training strategies in general.
\n\nFind threshold: We use a simple optimizer to find threshold based on pure CV. People care too much about searching for right threshold, but we don’t. Instead we care more about modelling and ensembling, so thresholding would cast less effect.
Train with sample weights\nWe saw a significant LB boost if using appropriate sample weights in training. CV might not be boosted a lot, but LB is. We use the number of prior assessment as a criterion for assigning weight for each sample. The problem here is that we cannot naively use histograms of this criterion between train/test to calculate weights. The issue is that samples from the same id are much similar, so the effect of each individual sample in a single id should drop. For instance, if the ratio of 0-prior-assessment samples in train data is 1/4, and the ratio of 0-prior-assessment samples in test data is 1/2, then we cannot just simply assign weight=2 for all 0-prior-assessment samples in train, but a smaller value. In the end, we did not come up with a theoretically concrete strategy on how to get the weights, but just to roughly estimate it. We chose [1.65 , 1.09, 0.87, 0.77, 0.57, 0.47] as the weight for samples with 0-prior, 1-prior, 2-prior, 3-prior, 4-prior, and more-than-4-prior assessments, respectively.
| Abs(model1_int – model2_int) | model1_int | model2_int | Classifier | Result |\n| --- | --- | --- | --- | --- |\n| 0 | | | | model1_int |\n|1| 0 (or 1)| 1 (or 0)| A >= 0.2|1|\n|1| 0 (or 1)| 1 (or 0)| A < 0.2 |0|\n|1| 1 (or 2) |2 (or 1) |B >= 0.5| 2|\n|1 |1 (or 2) |2 (or 1) |B < 0.5 |1|\n|1| 2 (or 3)| 3 (or 2)| C >= 0.85| 3|\n|1 |2 (or 3)| 3 (or 2)| C < 0.85 |2|\n|> 1 | | | |(model1_int + model2_int) / 2|
\n\nThanks for reading, and hope you like this write-up. Our kernel is posted here. \nhttps://www.kaggle.com/khahuras/bowl-2201-a?scriptVersionId=27403894
", "comments": [ { "id": 726477, "author_name": "narsil (jobs-in-data.com)", "author_url": "", "post_date": "2020-01-23T03:02:47.787000", "content": "Great write-up and many interesting ideas to try in the future. Congrats on keeping your gold. It was not easy this time.
\n\nI must confess: during the competition I was actually thinking of sending you an email to merge our teams. I decided to give a shot for a solo-gold however. Maybe team up next time? :)
", "votes": 2, "replies": [ { "id": 726652, "author_name": "Kha Vo", "author_url": "", "post_date": "2020-01-23T06:33:00.730000", "content": "Congrats on your impressive result @narsil. I am glad for your result. Also happy to hear your intent to merge. Yes next time I look forward to team together with you in a good time ;)
", "votes": 0, "replies": [] } ] }, { "id": 726814, "author_name": "Mohammed Tayor", "author_url": "", "post_date": "2020-01-23T08:53:09.097000", "content": "Congrats and thanks for sharing. \nIMO, the best tip to avoid shake-ups as you said is :
\n\n\n", "votes": 1, "replies": [] }, { "id": 726792, "author_name": "Iurii Uspangaliev", "author_url": "", "post_date": "2020-01-23T08:38:48.090000", "content": "Diversity is important, and is the key factor to avoid shake-up. We tried to bag training a lot, with lots of feature sets and models from each member.
\n
Congratz and thanks for really useful write-up! What was your CV strategy?
", "votes": 1, "replies": [ { "id": 727772, "author_name": "Kha Vo", "author_url": "", "post_date": "2020-01-24T03:22:27.753000", "content": "We did just truncated CV like everybody else (1000 times).
", "votes": 0, "replies": [] } ] }, { "id": 726505, "author_name": "Ailurophile", "author_url": "", "post_date": "2020-01-23T03:20:44.077000", "content": "Congratulations\nGreat Write-Up\nThanks for sharing your Approach & Insights!! @khahuras
", "votes": 1, "replies": [ { "id": 726656, "author_name": "Kha Vo", "author_url": "", "post_date": "2020-01-23T06:35:11.060000", "content": "Thanks Veerala.
", "votes": 0, "replies": [] } ] }, { "id": 726303, "author_name": "Patrick Chan", "author_url": "", "post_date": "2020-01-23T00:55:08.640000", "content": "Congrats... Many overfitting solutions simply do not have the oversight of blending the models correctly. I did a simple RNN and reached the top 8% model. Will be great if you can share more insights on your RNN model and features. Cheers. Patrick
", "votes": 1, "replies": [ { "id": 726654, "author_name": "Kha Vo", "author_url": "", "post_date": "2020-01-23T06:34:38.210000", "content": "Impressive RNN! Thanks Patrick. \nI extracted count of event codes for each session, served as features for 1 time step. A sequence of recent 20 sessions is fed into a simple LSTM, then concat the output to the dense features used for LGB.
", "votes": 1, "replies": [] } ] }, { "id": 728239, "author_name": "sagar pundir", "author_url": "", "post_date": "2020-01-24T14:21:16.920000", "content": "That's the most logical,intuitive and organic approach I have seen by now for this competition. Can you elaborate slightly on 'The most interesting features, would be stats on the same assessment with regards to each data sample'???
", "votes": 0, "replies": [] }, { "id": 732577, "author_name": "corochann", "author_url": "", "post_date": "2020-01-30T00:33:35.250000", "content": "Congrats and thanks for sharing.
", "votes": 0, "replies": [] } ], "comment_count": 11, "messages": [ { "id": 726288, "postDate": "2020-01-23T00:45:35.047Z", "content": "First, I would like to thank the host, Kaggle, and everyone for this great competition. I would like to share my deepest gratitude to my teammates @alijs1 @johnpateha @kazanova . It was a really good time and joyful period for me.
\n\nI will briefly brief our solution, and if I miss something, my teammates can add in their perspective.
\n\nBesides some popular features available in public kernels, we have some more custom features, such as ratio of good actions / all actions in Activity sessions, ratio of misclick or misdrag action / count actions (or session duration), count of some specific event codes since the previous Assessment session. The most interesting features, would be stats on the same assessment with regards to each data sample. This class of features helps tree models to converge quicker and reduce the importance of Assessment title.\nAs a separate solution, Marios (@kazanova) crafted his own feature set which took into consideration the train/test mismatch. He can share more about this if needed.
\n\nIt is surprising that we did not realize the possibility to augment train data by a lot of test samples with true labels. Whenever a test installation_id (hereinafter referred to as “id”) has more than 0 prior assessment, we can trim the user history and make extra training samples. This augmentation helped us a lot in the blend.
\n\nBesides using all original data (17690 rows) as the main base model, we noticed that training 5 separate submodels for each type can give a boost if properly blended with the main base model. And since we also have the option of data augmentation, it results in 4 different training strategies in general.
\n\nFind threshold: We use a simple optimizer to find threshold based on pure CV. People care too much about searching for right threshold, but we don’t. Instead we care more about modelling and ensembling, so thresholding would cast less effect.
Train with sample weights\nWe saw a significant LB boost if using appropriate sample weights in training. CV might not be boosted a lot, but LB is. We use the number of prior assessment as a criterion for assigning weight for each sample. The problem here is that we cannot naively use histograms of this criterion between train/test to calculate weights. The issue is that samples from the same id are much similar, so the effect of each individual sample in a single id should drop. For instance, if the ratio of 0-prior-assessment samples in train data is 1/4, and the ratio of 0-prior-assessment samples in test data is 1/2, then we cannot just simply assign weight=2 for all 0-prior-assessment samples in train, but a smaller value. In the end, we did not come up with a theoretically concrete strategy on how to get the weights, but just to roughly estimate it. We chose [1.65 , 1.09, 0.87, 0.77, 0.57, 0.47] as the weight for samples with 0-prior, 1-prior, 2-prior, 3-prior, 4-prior, and more-than-4-prior assessments, respectively.
| Abs(model1_int – model2_int) | model1_int | model2_int | Classifier | Result |\n| --- | --- | --- | --- | --- |\n| 0 | | | | model1_int |\n|1| 0 (or 1)| 1 (or 0)| A >= 0.2|1|\n|1| 0 (or 1)| 1 (or 0)| A < 0.2 |0|\n|1| 1 (or 2) |2 (or 1) |B >= 0.5| 2|\n|1 |1 (or 2) |2 (or 1) |B < 0.5 |1|\n|1| 2 (or 3)| 3 (or 2)| C >= 0.85| 3|\n|1 |2 (or 3)| 3 (or 2)| C < 0.85 |2|\n|> 1 | | | |(model1_int + model2_int) / 2|
\n\nThanks for reading, and hope you like this write-up. Our kernel is posted here. \nhttps://www.kaggle.com/khahuras/bowl-2201-a?scriptVersionId=27403894
", "rawMarkdown": "First, I would like to thank the host, Kaggle, and everyone for this great competition. I would like to share my deepest gratitude to my teammates @alijs1 @johnpateha @kazanova . It was a really good time and joyful period for me. \n\nI will briefly brief our solution, and if I miss something, my teammates can add in their perspective.\n\n## **FEATURE ENGINEERING**\nBesides some popular features available in public kernels, we have some more custom features, such as ratio of good actions / all actions in Activity sessions, ratio of misclick or misdrag action / count actions (or session duration), count of some specific event codes since the previous Assessment session. The most interesting features, would be stats on the same assessment with regards to each data sample. This class of features helps tree models to converge quicker and reduce the importance of Assessment title.\nAs a separate solution, Marios (@kazanova) crafted his own feature set which took into consideration the train/test mismatch. He can share more about this if needed.\n\n## **DATA AUGMENTATION**\nIt is surprising that we did not realize the possibility to augment train data by a lot of test samples with true labels. Whenever a test installation_id (hereinafter referred to as “id”) has more than 0 prior assessment, we can trim the user history and make extra training samples. This augmentation helped us a lot in the blend.\n\n## **MODELLING**\n+ **Model 1**: Main Model for all assessments\n+ **Model 2**: Five separate models for each assessment, then concat result.\n\nBesides using all original data (17690 rows) as the main base model, we noticed that training 5 separate submodels for each type can give a boost if properly blended with the main base model. And since we also have the option of data augmentation, it results in 4 different training strategies in general. \n\n+ **Find threshold:** We use a simple optimizer to find threshold based on pure CV. People care too much about searching for right threshold, but we don’t. Instead we care more about modelling and ensembling, so thresholding would cast less effect.\n\n+ **Train with sample weights**\nWe saw a significant LB boost if using appropriate sample weights in training. CV might not be boosted a lot, but LB is. We use the number of prior assessment as a criterion for assigning weight for each sample. The problem here is that we cannot naively use histograms of this criterion between train/test to calculate weights. The issue is that samples from the same id are much similar, so the effect of each individual sample in a single id should drop. For instance, if the ratio of 0-prior-assessment samples in train data is 1/4, and the ratio of 0-prior-assessment samples in test data is 1/2, then we cannot just simply assign weight=2 for all 0-prior-assessment samples in train, but a smaller value. In the end, we did not come up with a theoretically concrete strategy on how to get the weights, but just to roughly estimate it. We chose [1.65 , 1.09, 0.87, 0.77, 0.57, 0.47] as the weight for samples with 0-prior, 1-prior, 2-prior, 3-prior, 4-prior, and more-than-4-prior assessments, respectively.\n\n## **ENSEMBLING**\n+ **Blend by Classifier Logic**\nWe found a nice way to combine the main model result with 5-submodel result. We trained 3 simple classifiers with AUC loss: A) classify between class 0/1, B) between class 1/2, and C) between class 2/3. Then we use the following custom logic to combine 2 float predictions of model 1 and model 2 to class label: \n\n| Abs(model1_int – model2_int) | model1_int | model2_int | Classifier | Result |\n| --- | --- | --- | --- | --- |\n| 0 | | |\t| model1_int |\n|1|\t0 (or 1)|\t1 (or 0)|\tA >= 0.2|1|\n|1|\t0 (or 1)|\t1 (or 0)|\tA < 0.2\t|0|\n|1|\t1 (or 2)\t|2 (or 1)\t|B >= 0.5|\t2|\n|1\t|1 (or 2)\t|2 (or 1)\t|B < 0.5\t|1|\n|1|\t2 (or 3)|\t3 (or 2)|\tC >= 0.85|\t3|\n|1\t|2 (or 3)|\t3 (or 2)|\tC < 0.85\t|2|\n|> 1\t|\t|\t|\t|(model1_int + model2_int) / 2|\n\n+ **Stack**\nStacking also worked for us, both in CV and LB. As a result, we chose 1 final submission for the classifier logic, and the other 1 for stacking. For stacking, we tried 2 approaches: 4 stackers average, and extra-tree regressor. The latter performed better in CV and private LB, but we did not choose it and instead chose the blend of classifier logic + stack, which is bad in private LB.\n\n## **WHAT WORKED IN PRIVATE LB BUT NOT PUBLIC LB**\n- Blend by histogram matching (use prediction histogram of the best public LB submission to rectify private test predictions): very bad public LB, but very good private LB. \n- Extra Trees Regressor Stacking.\nWe would have finished \"In The Money\" zone if we chose this submission. However we don't regret.\n\n## **WHAT DID NOT WORK**\n-\tRanking average the predictions.\n-\tPseudo label from unused train ids. Indeed, we observed high CV boost when using pseudo samples from unused train ids, but LB decreased. Instead we doubt we did not do it properly enough due to our code’s complexity.\n\n## **WHAT WE DID NOT FINISH IN TIME**\n-\tWe also developed an RNN model, which has CV 0.53x. This RNN takes as input two kinds of features: 1) the sequence of sessions as sequential data. Each session’s features are just count of different event codes. And 2) dense features which is same as those in LGB modelling. Indeed, this model can contribute in the blending, but we only finished this in the last day, so it was hard to combine into the code. We believe it would boost our score significantly.\n\n## **WHAT DISTILLED IN MY MEMORY**\n-\tMy first time to work with 3 great grandmasters in a big competition. It is my pleasure and great opportunity to learn from all of my teammates. Thanks a lot guys.\n- I, personally feel happy with this result since we are one of the only 3 teams that can keep gold. Disappointment is overwhelmed by joy of lucky.\n-\tWe sometimes felt that we a little bit hate kernel 😊 just because it run for 8 hours then failed at the end due to some minor error. However in the end I think it’s a good way for Kaggle competition: people cannot use black magic too much, and a concrete code base is needed, which makes room for coding skills enhancement for competitors. \n-\tCombining solutions from team members is not a joke, especially if merging is late, like in our case (Evgeny and Marios only joined in the last week). It needs tons of efforts from all members. But in the end, if diversity of solutions is ensured one can expect a huge leap.\n-\tIt is good that no extensive public sharing or any scandal appeared during this competition. \n-\tPublic LB/CV correlation is a mystery, which makes the competition more interesting.\n-\tDiversity is important, and is the key factor to avoid shake-up. We tried to bag training a lot, with lots of feature sets and models from each member. \n- We have no concrete sign (for example, CV and public LB) to select our best private LB, so in general we don't regret the result too much. The gold position is somehow the result of our hard work and general sense of shakeup. So we will enjoy this gold medal a lot!\n\nThanks for reading, and hope you like this write-up. Our kernel is posted here. \nhttps://www.kaggle.com/khahuras/bowl-2201-a?scriptVersionId=27403894\n", "votes": 52 }, { "id": 726477, "postDate": "2020-01-23T03:02:47.787Z", "content": "Great write-up and many interesting ideas to try in the future. Congrats on keeping your gold. It was not easy this time.
\n\nI must confess: during the competition I was actually thinking of sending you an email to merge our teams. I decided to give a shot for a solo-gold however. Maybe team up next time? :)
", "rawMarkdown": "Great write-up and many interesting ideas to try in the future. Congrats on keeping your gold. It was not easy this time. \n\nI must confess: during the competition I was actually thinking of sending you an email to merge our teams. I decided to give a shot for a solo-gold however. Maybe team up next time? :)", "votes": 2, "replies": [ { "id": 726652, "postDate": "2020-01-23T06:33:00.730Z", "content": "Congrats on your impressive result @narsil. I am glad for your result. Also happy to hear your intent to merge. Yes next time I look forward to team together with you in a good time ;)
", "rawMarkdown": "Congrats on your impressive result @narsil. I am glad for your result. Also happy to hear your intent to merge. Yes next time I look forward to team together with you in a good time ;)" } ] }, { "id": 726814, "postDate": "2020-01-23T08:53:09.097Z", "content": "Congrats and thanks for sharing. \nIMO, the best tip to avoid shake-ups as you said is :
\n\n\n", "rawMarkdown": "Congrats and thanks for sharing. \nIMO, the best tip to avoid shake-ups as you said is :\n\n> Diversity is important, and is the key factor to avoid shake-up. We tried to bag training a lot, with lots of feature sets and models from each member.\n", "votes": 1 }, { "id": 726792, "postDate": "2020-01-23T08:38:48.090Z", "content": "Diversity is important, and is the key factor to avoid shake-up. We tried to bag training a lot, with lots of feature sets and models from each member.
\n
Congratz and thanks for really useful write-up! What was your CV strategy?
", "rawMarkdown": "Congratz and thanks for really useful write-up! What was your CV strategy?", "votes": 1, "replies": [ { "id": 727772, "postDate": "2020-01-24T03:22:27.753Z", "content": "We did just truncated CV like everybody else (1000 times).
", "rawMarkdown": "We did just truncated CV like everybody else (1000 times)." } ] }, { "id": 726505, "postDate": "2020-01-23T03:20:44.077Z", "content": "Congratulations\nGreat Write-Up\nThanks for sharing your Approach & Insights!! @khahuras
", "rawMarkdown": "Congratulations\nGreat Write-Up\nThanks for sharing your Approach & Insights!! @khahuras ", "votes": 1, "replies": [ { "id": 726656, "postDate": "2020-01-23T06:35:11.060Z", "content": "Thanks Veerala.
", "rawMarkdown": "Thanks Veerala. " } ] }, { "id": 726303, "postDate": "2020-01-23T00:55:08.640Z", "content": "Congrats... Many overfitting solutions simply do not have the oversight of blending the models correctly. I did a simple RNN and reached the top 8% model. Will be great if you can share more insights on your RNN model and features. Cheers. Patrick
", "rawMarkdown": "Congrats... Many overfitting solutions simply do not have the oversight of blending the models correctly. I did a simple RNN and reached the top 8% model. Will be great if you can share more insights on your RNN model and features. Cheers. Patrick", "votes": 1, "replies": [ { "id": 726654, "postDate": "2020-01-23T06:34:38.210Z", "content": "Impressive RNN! Thanks Patrick. \nI extracted count of event codes for each session, served as features for 1 time step. A sequence of recent 20 sessions is fed into a simple LSTM, then concat the output to the dense features used for LGB.
", "rawMarkdown": "Impressive RNN! Thanks Patrick. \nI extracted count of event codes for each session, served as features for 1 time step. A sequence of recent 20 sessions is fed into a simple LSTM, then concat the output to the dense features used for LGB. ", "votes": 1 } ] }, { "id": 728239, "postDate": "2020-01-24T14:21:16.920Z", "content": "That's the most logical,intuitive and organic approach I have seen by now for this competition. Can you elaborate slightly on 'The most interesting features, would be stats on the same assessment with regards to each data sample'???
", "rawMarkdown": "That's the most logical,intuitive and organic approach I have seen by now for this competition. Can you elaborate slightly on 'The most interesting features, would be stats on the same assessment with regards to each data sample'???" }, { "id": 732577, "postDate": "2020-01-30T00:33:35.250Z", "content": "Congrats and thanks for sharing.
", "rawMarkdown": "Congrats and thanks for sharing." } ], "complete": true, "source": "live", "fetched_at": "2026-10-23T23:48:04Z", "competition_id": 16531, "competition_slug": "data-science-bowl-2019", "team_id": 3856274, "solution_url": "/competitions/data-science-bowl-2019/writeups/alijs-khavo-evgeny-marios-14th-place-solution" }