Thanks to Booz Allen Hamilton, Kaggle and everyone for this wonderful competition. And also thanks to my teammate @oyxuan . Congratulations to all winners!
\n\nOur selected score is based on a single lightgbm (average on multi-seeds 5 fold).
\n\nThe model score : \nprivate qwk 0.568, public qwk 0.563\ncv weighted qwk 0.591, cv weighted rmse 1.009
\n\nIn the early game , we find that the LB score is unstable and has low correlation with the local cv, so we decide to focus on the local cv score only. We have tried several ideas to make the local cv stable. Below are two validation sets we use:
\n\n2.1 GroupK CV : We use the 5 times * 5-fold GroupK by installation_id, each time with random groupk split seed and random column order. However qwk is still not so stable on our local cv, so we mainly concern the weighted rmse when validating our ideas and ignore qwk. For the weighted loss , the weight is the sample prob for each sample (We use full data, for the test part, we calculate the expectation of the sample prob as weight).
\n\n2.2 Nested CV: Usually, the GroupK cv above works well. When we think the GroupK cv's decision has low confidence (eg. inconsistent with our common sense), we will use another nested set for double check: We simulate the train-test split on the local data : random select 1400 users with full history for the nested training and 2200 users with truncated history for the nested testing. We repeat it for 50~100 times and calculate the mean score for validation.
\n\nMost of our time are spending on feature engineer. We generate around 20,000 features these days, and use the null importance method to select the top 500 features.
\n\nLots of stats (mean/sum/last/std/max/slope) from true attempts ratio, correct true ratio, correct feedback ratio etc. Stats based on same assessment or similar game are highest important (Similar game : we map each game to the corresponding similar assessment, since they are similar task)
We extract features from different parts of the child history data : 1) full history part, 2) last 5/12/48 hour, 3) from last assessment to the current assessment. Since here are some shared devices phenomenon, add different part info may help model.
Event interval features (next event timestamps - current event timestamps) : Stats (mean/last) of event interval groupby event_id / event_code. Several event interval features show high importance.
Video skip prop ratio : clip event interval / clip length provided by organizer. (Does the child skip the video? If so, when does he skip?)
Event data feature : Stats(mean/sum/last) of all numerical args in event data X event_id / event code combination. We get the combination and args type from the specs file. \neg. event_code2030_misses_mean.
Congrats! So feature engineering was key after all!
", "votes": 7, "replies": [ { "id": 729999, "author_name": "30CrMnSiA", "author_url": "", "post_date": "2020-01-27T00:38:22.307000", "content": "I think \"focus on the local cv score only\" is the key
", "votes": 7, "replies": [] } ] }, { "id": 727888, "author_name": "Yassine Alouini", "author_url": "", "post_date": "2020-01-24T06:50:01.353000", "content": "Thanks for sharing your solution! By the way, for those that aren't familiar with the null importance method mentioned, here is a kernel: https://www.kaggle.com/ogrellier/feature-selection-with-null-importances \nLet me know if this is the one you had in mind. Cheers!
", "votes": 7, "replies": [ { "id": 727968, "author_name": "Marco Gorelli", "author_url": "", "post_date": "2020-01-24T09:17:15.793000", "content": "Thanks, was wondering what that was
", "votes": 1, "replies": [] }, { "id": 728629, "author_name": "zr", "author_url": "", "post_date": "2020-01-25T01:26:58.313000", "content": "Thanks! and thanks for @ogrellier 's great kernel, we use it in the competition and it works well. I've added the link to the summary.
", "votes": 2, "replies": [] } ] }, { "id": 729073, "author_name": "None", "author_url": "", "post_date": "2020-01-25T17:36:46.717000", "content": "It is very impressive to reach that level of performance without many models or other complicated ideas like we have seen in the other gold solutions. This probably means your feature engineering was absolutely on point. We are trying to produce a stronger submission to learn more about the process. The rough ideas you describe for creating features are something we also came up with but we didn't reach that level. Can you give some more specifics on the features, it would really help us. Thank you for sharing, kaggle is amazing.
", "votes": 3, "replies": [ { "id": 730397, "author_name": "zr", "author_url": "", "post_date": "2020-01-27T12:41:15.843000", "content": "Thanks! Here is a feature list we use in our final model, sorted by null importance score, most of features are mentioned in our summary, you can check it for more details.
", "votes": 4, "replies": [] }, { "id": 730600, "author_name": "None", "author_url": "", "post_date": "2020-01-27T17:12:22.447000", "content": "Great Thanks!
", "votes": 1, "replies": [] } ] }, { "id": 727882, "author_name": "None", "author_url": "", "post_date": "2020-01-24T06:41:46.253000", "content": "Congratulations, very impressive!\nWill you open source your code?
", "votes": 3, "replies": [ { "id": 728637, "author_name": "zr", "author_url": "", "post_date": "2020-01-25T01:58:02.140000", "content": "Thanks. Our code is so messy and we can't to open source this time ... I'm improving code skills
", "votes": 1, "replies": [] } ] }, { "id": 730417, "author_name": "Tayyab A", "author_url": "", "post_date": "2020-01-27T13:19:48.573000", "content": "Good Try Keep it up
", "votes": 2, "replies": [] }, { "id": 728403, "author_name": "Subbu Vidyasekar", "author_url": "", "post_date": "2020-01-24T17:25:57.377000", "content": "Even this is my first competition, I understood the value of Feature engineering. Congrats bro!
", "votes": 2, "replies": [] }, { "id": 727891, "author_name": "LeiZhou", "author_url": "", "post_date": "2020-01-24T06:53:10.970000", "content": "Congratulations! do you use 1000 public test cases to do adversarial test or resampling cases from train data?
", "votes": 2, "replies": [ { "id": 728630, "author_name": "zr", "author_url": "", "post_date": "2020-01-25T01:29:03.703000", "content": "Thanks! We use public 1000 test cases to do adversarial test. 3600 train cases for target 1, and 1000 lb test cases for target 0.
", "votes": 2, "replies": [] } ] }, { "id": 727832, "author_name": "heartkilla", "author_url": "", "post_date": "2020-01-24T05:11:28.143000", "content": "Who would have guessed that the 1st place can be reached with a single model? Very impressive and clean solution. Thank you and congrats on the GM title!
", "votes": 2, "replies": [ { "id": 728634, "author_name": "zr", "author_url": "", "post_date": "2020-01-25T01:47:38.700000", "content": "Thanks!
", "votes": 1, "replies": [] } ] }, { "id": 728670, "author_name": "HELLOWORLD", "author_url": "", "post_date": "2020-01-25T04:06:44.977000", "content": "choice and concentration! its like a magic.
", "votes": 1, "replies": [] }, { "id": 731044, "author_name": "Kuldeep Arya", "author_url": "", "post_date": "2020-01-28T09:40:55.930000", "content": "This is the very useful for beginner and better understanding for the rough ideas you describe for creating features are something we also came up with but we didn't reach that level. Can you give some more specifics on the features, it would really help us. Thank you for sharing.
", "votes": 1, "replies": [] }, { "id": 730302, "author_name": "Nirmal Fernando", "author_url": "", "post_date": "2020-01-27T10:28:04.363000", "content": "Congratulations! One thing I failed to learn is how to get adversarial AUC to 0.5. Could you please explain what you did to get a 0.5 AUC?
", "votes": 1, "replies": [ { "id": 730392, "author_name": "zr", "author_url": "", "post_date": "2020-01-27T12:26:57.220000", "content": "Thanks!\nFor train part, random choose one assessment for each ins_id; for test part, use 1000 test assessments.\nTrain a model to classify train part or test part.\nIn our practice, the model mean 5-fold val auc is about 0.49 ~ 0.51
", "votes": 3, "replies": [] }, { "id": 730654, "author_name": "Nirmal Fernando", "author_url": "", "post_date": "2020-01-27T18:30:33.533000", "content": "I did the something similar but AUC was 0.7+. Does it mean that there are leakages in my features?
", "votes": 0, "replies": [] }, { "id": 730846, "author_name": "zr", "author_url": "", "post_date": "2020-01-28T02:33:11.610000", "content": "you can check feature importance to see which feature caused the leak.
", "votes": 1, "replies": [] } ] }, { "id": 728741, "author_name": "Ace", "author_url": "", "post_date": "2020-01-25T07:14:19.083000", "content": "Impressive feature selection and single model lightgbm solution. Congratulations!
", "votes": 1, "replies": [] }, { "id": 728700, "author_name": "Hi Im Ryo", "author_url": "", "post_date": "2020-01-25T05:28:32.603000", "content": "Congrats! Thank you for sharing your solution!!!
", "votes": 1, "replies": [] }, { "id": 728676, "author_name": "Laevatein", "author_url": "", "post_date": "2020-01-25T04:20:33.460000", "content": "Congrats! Great feature engineering.
", "votes": 1, "replies": [] }, { "id": 728472, "author_name": "Yust", "author_url": "", "post_date": "2020-01-24T18:58:16.693000", "content": "Congratulations！！
", "votes": 1, "replies": [] }, { "id": 728389, "author_name": "Massoud Hosseinali", "author_url": "", "post_date": "2020-01-24T17:12:56.950000", "content": "Congratulations.
", "votes": 1, "replies": [] }, { "id": 727947, "author_name": "SvC", "author_url": "", "post_date": "2020-01-24T08:31:03.860000", "content": "Very nice to see that carefully handcrafted features in combination with a solid model validation strategy still are super important. Thanks for sharing !!!
", "votes": 1, "replies": [] }, { "id": 727801, "author_name": "Sergei Fironov", "author_url": "", "post_date": "2020-01-24T04:25:28.330000", "content": "Congrats! Impressive validation scheme!
", "votes": 1, "replies": [ { "id": 728633, "author_name": "zr", "author_url": "", "post_date": "2020-01-25T01:46:58.550000", "content": "Thanks and congrats to you too, your NN is always amazing, I've learned a lot from your sharing.
", "votes": 1, "replies": [] } ] }, { "id": 729939, "author_name": "James Ingram", "author_url": "", "post_date": "2020-01-26T21:20:04.390000", "content": "Congrats, and thanks for the great solution, feature engineering and validation method!
\n\n\n", "votes": 0, "replies": [ { "id": 729977, "author_name": "zr", "author_url": "", "post_date": "2020-01-26T23:39:48.767000", "content": "btw, there's a small typo on the following item: \n 3. Feature Engineering \n (4.) Video Prop Ratio ... If so, when dose he skip? \n should be: when does he skip?
\n
thanks, the typo has been corrected
", "votes": 1, "replies": [] }, { "id": 730057, "author_name": "James Ingram", "author_url": "", "post_date": "2020-01-27T03:18:19.970000", "content": "aah one more:\n2. Validation Strategy
\n\n2.1 GroupK CV : We use the 5 times * 5-fold GroupK by ***instation_id,***\n\n", "votes": 0, "replies": [] }, { "id": 730387, "author_name": "zr", "author_url": "", "post_date": "2020-01-27T12:20:30.667000", "content": "installation_id
\n
\n 👍 You're the best!
thanks again! corrected 👍 👍
", "votes": 0, "replies": [] } ] }, { "id": 2452542, "author_name": "nebiyu youhannes", "author_url": "", "post_date": "2023-09-23T11:37:51", "content": "congrats 4 years later sorry but not sorry 😁
", "votes": 0, "replies": [] }, { "id": 766656, "author_name": "leandro melo", "author_url": "", "post_date": "2020-03-08T14:22:11.893000", "content": "Parabéns
", "votes": 0, "replies": [] }, { "id": 757702, "author_name": "", "author_url": "", "post_date": "2020-02-27T03:08:23.757000", "content": "great work!
", "votes": 0, "replies": [] }, { "id": 752162, "author_name": "", "author_url": "", "post_date": "2020-02-20T19:20:21.510000", "content": "great
", "votes": 0, "replies": [] }, { "id": 742420, "author_name": "Soju", "author_url": "", "post_date": "2020-02-11T09:35:45.120000", "content": "congrats! great work!
", "votes": 0, "replies": [] }, { "id": 737147, "author_name": "Yuan", "author_url": "", "post_date": "2020-02-05T00:53:07.160000", "content": "Congratz and thx for the sharing! It is really helpful for learning!👍 🤘
", "votes": 0, "replies": [] }, { "id": 735020, "author_name": "NorwayPing", "author_url": "", "post_date": "2020-02-02T11:47:54.513000", "content": "Congratulations!
", "votes": 0, "replies": [] }, { "id": 734885, "author_name": "", "author_url": "", "post_date": "2020-02-02T06:07:08.797000", "content": "Congratulations
", "votes": 0, "replies": [] }, { "id": 734543, "author_name": "Ravi Teja Devineni", "author_url": "", "post_date": "2020-02-01T16:26:04.127000", "content": "It is so useful.
", "votes": 0, "replies": [] }, { "id": 734338, "author_name": "Leonie", "author_url": "", "post_date": "2020-02-01T09:57:53.053000", "content": "Congratulations! Thank you for sharing your approach. It was very interesting to read.
", "votes": 0, "replies": [] }, { "id": 733381, "author_name": "", "author_url": "", "post_date": "2020-01-31T03:52:14.497000", "content": "", "votes": 0, "replies": [] }, { "id": 732605, "author_name": "", "author_url": "", "post_date": "2020-01-30T01:50:21.887000", "content": "", "votes": 0, "replies": [] }, { "id": 732344, "author_name": "", "author_url": "", "post_date": "2020-01-29T17:53:13.760000", "content": "", "votes": 0, "replies": [] }, { "id": 732151, "author_name": "", "author_url": "", "post_date": "2020-01-29T14:35:37.640000", "content": "", "votes": 0, "replies": [] }, { "id": 731927, "author_name": "", "author_url": "", "post_date": "2020-01-29T09:02:58.187000", "content": "", "votes": 0, "replies": [] }, { "id": 731922, "author_name": "", "author_url": "", "post_date": "2020-01-29T08:56:48.843000", "content": "", "votes": 0, "replies": [] }, { "id": 731791, "author_name": "", "author_url": "", "post_date": "2020-01-29T03:56:49.467000", "content": "", "votes": 0, "replies": [] }, { "id": 731125, "author_name": "", "author_url": "", "post_date": "2020-01-28T11:35:43.597000", "content": "", "votes": 0, "replies": [] }, { "id": 730979, "author_name": "", "author_url": "", "post_date": "2020-01-28T07:47:26.347000", "content": "", "votes": 0, "replies": [] }, { "id": 730879, "author_name": "", "author_url": "", "post_date": "2020-01-28T03:46:31.313000", "content": "", "votes": 0, "replies": [ { "id": 732801, "author_name": "", "author_url": "", "post_date": "2020-01-30T09:38:17.330000", "content": "", "votes": 2, "replies": [] }, { "id": 732968, "author_name": "", "author_url": "", "post_date": "2020-01-30T14:00:59.517000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 730878, "author_name": "", "author_url": "", "post_date": "2020-01-28T03:45:17.447000", "content": "", "votes": 0, "replies": [] }, { "id": 730860, "author_name": "", "author_url": "", "post_date": "2020-01-28T02:50:40.580000", "content": "", "votes": 0, "replies": [] }, { "id": 730436, "author_name": "", "author_url": "", "post_date": "2020-01-27T13:33:42.203000", "content": "", "votes": 0, "replies": [ { "id": 730858, "author_name": "", "author_url": "", "post_date": "2020-01-28T02:47:37.060000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 730002, "author_name": "", "author_url": "", "post_date": "2020-01-27T00:43:18.637000", "content": "", "votes": 0, "replies": [ { "id": 730386, "author_name": "", "author_url": "", "post_date": "2020-01-27T12:18:57.413000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 729594, "author_name": "", "author_url": "", "post_date": "2020-01-26T12:30:58.117000", "content": "", "votes": 0, "replies": [] }, { "id": 729582, "author_name": "", "author_url": "", "post_date": "2020-01-26T12:04:06.117000", "content": "", "votes": 0, "replies": [] }, { "id": 729519, "author_name": "", "author_url": "", "post_date": "2020-01-26T10:06:46.547000", "content": "", "votes": 0, "replies": [] }, { "id": 728922, "author_name": "", "author_url": "", "post_date": "2020-01-25T13:31:46.573000", "content": "", "votes": 0, "replies": [] }, { "id": 728853, "author_name": "", "author_url": "", "post_date": "2020-01-25T11:30:16.067000", "content": "", "votes": 0, "replies": [] }, { "id": 728828, "author_name": "", "author_url": "", "post_date": "2020-01-25T10:51:19.263000", "content": "", "votes": 0, "replies": [] }, { "id": 727876, "author_name": "", "author_url": "", "post_date": "2020-01-24T06:35:52.813000", "content": "", "votes": 0, "replies": [ { "id": 728632, "author_name": "", "author_url": "", "post_date": "2020-01-25T01:44:37.357000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 727857, "author_name": "", "author_url": "", "post_date": "2020-01-24T05:52:05.673000", "content": "", "votes": 0, "replies": [] }, { "id": 727826, "author_name": "", "author_url": "", "post_date": "2020-01-24T05:06:09.020000", "content": "", "votes": 0, "replies": [] }, { "id": 727821, "author_name": "", "author_url": "", "post_date": "2020-01-24T05:01:08.873000", "content": "", "votes": 0, "replies": [] }, { "id": 727798, "author_name": "", "author_url": "", "post_date": "2020-01-24T04:23:49.463000", "content": "", "votes": 0, "replies": [] }, { "id": 727796, "author_name": "", "author_url": "", "post_date": "2020-01-24T04:15:50.873000", "content": "", "votes": 0, "replies": [] }, { "id": 727794, "author_name": "", "author_url": "", "post_date": "2020-01-24T04:02:44.863000", "content": "", "votes": 0, "replies": [] }, { "id": 733656, "author_name": "", "author_url": "", "post_date": "2020-01-31T11:43:25.337000", "content": "", "votes": 0, "replies": [] }, { "id": 731666, "author_name": "", "author_url": "", "post_date": "2020-01-28T21:56:37.880000", "content": "", "votes": 0, "replies": [] }, { "id": 730840, "author_name": "", "author_url": "", "post_date": "2020-01-28T02:23:56.770000", "content": "", "votes": 0, "replies": [] }, { "id": 729210, "author_name": "", "author_url": "", "post_date": "2020-01-25T22:24:41.693000", "content": "", "votes": 0, "replies": [] }, { "id": 728779, "author_name": "", "author_url": "", "post_date": "2020-01-25T08:42:24.543000", "content": "", "votes": 0, "replies": [ { "id": 728813, "author_name": "", "author_url": "", "post_date": "2020-01-25T10:09:29.547000", "content": "", "votes": 4, "replies": [] }, { "id": 728840, "author_name": "", "author_url": "", "post_date": "2020-01-25T11:13:07.813000", "content": "", "votes": 0, "replies": [] }, { "id": 728845, "author_name": "", "author_url": "", "post_date": "2020-01-25T11:17:02.473000", "content": "", "votes": 1, "replies": [] }, { "id": 730067, "author_name": "", "author_url": "", "post_date": "2020-01-27T03:58:01.753000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 731948, "author_name": "", "author_url": "", "post_date": "2020-01-29T09:38:04.557000", "content": "", "votes": 1, "replies": [] }, { "id": 737210, "author_name": "", "author_url": "", "post_date": "2020-02-05T03:48:37", "content": "", "votes": 0, "replies": [] }, { "id": 737205, "author_name": "", "author_url": "", "post_date": "2020-02-05T03:39:40.227000", "content": "", "votes": 0, "replies": [] }, { "id": 732941, "author_name": "", "author_url": "", "post_date": "2020-01-30T13:20:29.397000", "content": "", "votes": 0, "replies": [] }, { "id": 732561, "author_name": "", "author_url": "", "post_date": "2020-01-30T00:01:00.540000", "content": "", "votes": 0, "replies": [] }, { "id": 732311, "author_name": "", "author_url": "", "post_date": "2020-01-29T17:08:15.190000", "content": "", "votes": 0, "replies": [] }, { "id": 732224, "author_name": "", "author_url": "", "post_date": "2020-01-29T15:33:07.173000", "content": "", "votes": 0, "replies": [] }, { "id": 731700, "author_name": "", "author_url": "", "post_date": "2020-01-28T23:58:50.607000", "content": "", "votes": 0, "replies": [] }, { "id": 731207, "author_name": "", "author_url": "", "post_date": "2020-01-28T12:45:41.330000", "content": "", "votes": 0, "replies": [] } ], "comment_count": 94, "messages": [ { "id": 727792, "postDate": "2020-01-24T03:59:37.650Z", "content": "Thanks to Booz Allen Hamilton, Kaggle and everyone for this wonderful competition. And also thanks to my teammate @oyxuan . Congratulations to all winners!
\n\nOur selected score is based on a single lightgbm (average on multi-seeds 5 fold).
\n\nThe model score : \nprivate qwk 0.568, public qwk 0.563\ncv weighted qwk 0.591, cv weighted rmse 1.009
\n\nIn the early game , we find that the LB score is unstable and has low correlation with the local cv, so we decide to focus on the local cv score only. We have tried several ideas to make the local cv stable. Below are two validation sets we use:
\n\n2.1 GroupK CV : We use the 5 times * 5-fold GroupK by installation_id, each time with random groupk split seed and random column order. However qwk is still not so stable on our local cv, so we mainly concern the weighted rmse when validating our ideas and ignore qwk. For the weighted loss , the weight is the sample prob for each sample (We use full data, for the test part, we calculate the expectation of the sample prob as weight).
\n\n2.2 Nested CV: Usually, the GroupK cv above works well. When we think the GroupK cv's decision has low confidence (eg. inconsistent with our common sense), we will use another nested set for double check: We simulate the train-test split on the local data : random select 1400 users with full history for the nested training and 2200 users with truncated history for the nested testing. We repeat it for 50~100 times and calculate the mean score for validation.
\n\nMost of our time are spending on feature engineer. We generate around 20,000 features these days, and use the null importance method to select the top 500 features.
\n\nLots of stats (mean/sum/last/std/max/slope) from true attempts ratio, correct true ratio, correct feedback ratio etc. Stats based on same assessment or similar game are highest important (Similar game : we map each game to the corresponding similar assessment, since they are similar task)
We extract features from different parts of the child history data : 1) full history part, 2) last 5/12/48 hour, 3) from last assessment to the current assessment. Since here are some shared devices phenomenon, add different part info may help model.
Event interval features (next event timestamps - current event timestamps) : Stats (mean/last) of event interval groupby event_id / event_code. Several event interval features show high importance.
Video skip prop ratio : clip event interval / clip length provided by organizer. (Does the child skip the video? If so, when does he skip?)
Event data feature : Stats(mean/sum/last) of all numerical args in event data X event_id / event code combination. We get the combination and args type from the specs file. \neg. event_code2030_misses_mean.
Congrats! So feature engineering was key after all!
", "rawMarkdown": "Congrats! So feature engineering was key after all!", "votes": 7, "replies": [ { "id": 729999, "postDate": "2020-01-27T00:38:22.307Z", "content": "I think \"focus on the local cv score only\" is the key
", "rawMarkdown": "I think \"focus on the local cv score only\" is the key ", "votes": 7 } ] }, { "id": 727888, "postDate": "2020-01-24T06:50:01.353Z", "content": "Thanks for sharing your solution! By the way, for those that aren't familiar with the null importance method mentioned, here is a kernel: https://www.kaggle.com/ogrellier/feature-selection-with-null-importances \nLet me know if this is the one you had in mind. Cheers!
", "rawMarkdown": "Thanks for sharing your solution! By the way, for those that aren't familiar with the null importance method mentioned, here is a kernel: https://www.kaggle.com/ogrellier/feature-selection-with-null-importances \nLet me know if this is the one you had in mind. Cheers!\n", "votes": 7, "replies": [ { "id": 727968, "postDate": "2020-01-24T09:17:15.793Z", "content": "Thanks, was wondering what that was
", "rawMarkdown": "Thanks, was wondering what that was", "votes": 1 }, { "id": 728629, "postDate": "2020-01-25T01:26:58.313Z", "content": "Thanks! and thanks for @ogrellier 's great kernel, we use it in the competition and it works well. I've added the link to the summary.
", "rawMarkdown": "Thanks! and thanks for @ogrellier 's great kernel, we use it in the competition and it works well. I've added the link to the summary.", "votes": 2 } ] }, { "id": 729073, "postDate": "2020-01-25T17:36:46.717Z", "content": "It is very impressive to reach that level of performance without many models or other complicated ideas like we have seen in the other gold solutions. This probably means your feature engineering was absolutely on point. We are trying to produce a stronger submission to learn more about the process. The rough ideas you describe for creating features are something we also came up with but we didn't reach that level. Can you give some more specifics on the features, it would really help us. Thank you for sharing, kaggle is amazing.
", "rawMarkdown": "It is very impressive to reach that level of performance without many models or other complicated ideas like we have seen in the other gold solutions. This probably means your feature engineering was absolutely on point. We are trying to produce a stronger submission to learn more about the process. The rough ideas you describe for creating features are something we also came up with but we didn't reach that level. Can you give some more specifics on the features, it would really help us. Thank you for sharing, kaggle is amazing.", "votes": 3, "replies": [ { "id": 730397, "postDate": "2020-01-27T12:41:15.843Z", "content": "Thanks! Here is a feature list we use in our final model, sorted by null importance score, most of features are mentioned in our summary, you can check it for more details.
", "rawMarkdown": "Thanks! Here is a feature list we use in our final model, sorted by null importance score, most of features are mentioned in our summary, you can check it for more details.", "votes": 4 }, { "id": 730600, "postDate": "2020-01-27T17:12:22.447Z", "content": "Great Thanks!
", "rawMarkdown": "Great Thanks!", "votes": 1 } ] }, { "id": 727882, "postDate": "2020-01-24T06:41:46.253Z", "content": "Congratulations, very impressive!\nWill you open source your code?
", "rawMarkdown": "Congratulations, very impressive!\nWill you open source your code?", "votes": 3, "replies": [ { "id": 728637, "postDate": "2020-01-25T01:58:02.140Z", "content": "Thanks. Our code is so messy and we can't to open source this time ... I'm improving code skills
", "rawMarkdown": "Thanks. Our code is so messy and we can't to open source this time ... I'm improving code skills", "votes": 1 } ] }, { "id": 730417, "postDate": "2020-01-27T13:19:48.573Z", "content": "Good Try Keep it up
", "rawMarkdown": "Good Try Keep it up", "votes": 2 }, { "id": 728403, "postDate": "2020-01-24T17:25:57.377Z", "content": "Even this is my first competition, I understood the value of Feature engineering. Congrats bro!
", "rawMarkdown": "Even this is my first competition, I understood the value of Feature engineering. Congrats bro!", "votes": 2 }, { "id": 727891, "postDate": "2020-01-24T06:53:10.970Z", "content": "Congratulations! do you use 1000 public test cases to do adversarial test or resampling cases from train data?
", "rawMarkdown": "Congratulations! do you use 1000 public test cases to do adversarial test or resampling cases from train data?", "votes": 2, "replies": [ { "id": 728630, "postDate": "2020-01-25T01:29:03.703Z", "content": "Thanks! We use public 1000 test cases to do adversarial test. 3600 train cases for target 1, and 1000 lb test cases for target 0.
", "rawMarkdown": "Thanks! We use public 1000 test cases to do adversarial test. 3600 train cases for target 1, and 1000 lb test cases for target 0.", "votes": 2 } ] }, { "id": 727832, "postDate": "2020-01-24T05:11:28.143Z", "content": "Who would have guessed that the 1st place can be reached with a single model? Very impressive and clean solution. Thank you and congrats on the GM title!
", "rawMarkdown": "Who would have guessed that the 1st place can be reached with a single model? Very impressive and clean solution. Thank you and congrats on the GM title!", "votes": 2, "replies": [ { "id": 728634, "postDate": "2020-01-25T01:47:38.700Z", "content": "Thanks!
", "rawMarkdown": "Thanks!", "votes": 1 } ] }, { "id": 728670, "postDate": "2020-01-25T04:06:44.977Z", "content": "choice and concentration! its like a magic.
", "rawMarkdown": "choice and concentration! its like a magic.", "votes": 1 }, { "id": 731044, "postDate": "2020-01-28T09:40:55.930Z", "content": "This is the very useful for beginner and better understanding for the rough ideas you describe for creating features are something we also came up with but we didn't reach that level. Can you give some more specifics on the features, it would really help us. Thank you for sharing.
", "rawMarkdown": "This is the very useful for beginner and better understanding for the rough ideas you describe for creating features are something we also came up with but we didn't reach that level. Can you give some more specifics on the features, it would really help us. Thank you for sharing.", "votes": 1 }, { "id": 730302, "postDate": "2020-01-27T10:28:04.363Z", "content": "Congratulations! One thing I failed to learn is how to get adversarial AUC to 0.5. Could you please explain what you did to get a 0.5 AUC?
", "rawMarkdown": "Congratulations! One thing I failed to learn is how to get adversarial AUC to 0.5. Could you please explain what you did to get a 0.5 AUC?", "votes": 1, "replies": [ { "id": 730392, "postDate": "2020-01-27T12:26:57.220Z", "content": "Thanks!\nFor train part, random choose one assessment for each ins_id; for test part, use 1000 test assessments.\nTrain a model to classify train part or test part.\nIn our practice, the model mean 5-fold val auc is about 0.49 ~ 0.51
", "rawMarkdown": "Thanks!\nFor train part, random choose one assessment for each ins_id; for test part, use 1000 test assessments.\nTrain a model to classify train part or test part.\nIn our practice, the model mean 5-fold val auc is about 0.49 ~ 0.51", "votes": 3 }, { "id": 730654, "postDate": "2020-01-27T18:30:33.533Z", "content": "I did the something similar but AUC was 0.7+. Does it mean that there are leakages in my features?
", "rawMarkdown": "I did the something similar but AUC was 0.7+. Does it mean that there are leakages in my features?" }, { "id": 730846, "postDate": "2020-01-28T02:33:11.610Z", "content": "you can check feature importance to see which feature caused the leak.
", "rawMarkdown": "you can check feature importance to see which feature caused the leak.", "votes": 1 } ] }, { "id": 728741, "postDate": "2020-01-25T07:14:19.083Z", "content": "Impressive feature selection and single model lightgbm solution. Congratulations!
", "rawMarkdown": "Impressive feature selection and single model lightgbm solution. Congratulations!", "votes": 1 }, { "id": 728700, "postDate": "2020-01-25T05:28:32.603Z", "content": "Congrats! Thank you for sharing your solution!!!
", "rawMarkdown": "Congrats! Thank you for sharing your solution!!!", "votes": 1 }, { "id": 728676, "postDate": "2020-01-25T04:20:33.460Z", "content": "Congrats! Great feature engineering.
", "rawMarkdown": "Congrats! Great feature engineering.", "votes": 1 }, { "id": 728472, "postDate": "2020-01-24T18:58:16.693Z", "content": "Congratulations！！
", "rawMarkdown": "Congratulations！！", "votes": 1 }, { "id": 728389, "postDate": "2020-01-24T17:12:56.950Z", "content": "Congratulations.
", "rawMarkdown": "Congratulations.", "votes": 1 }, { "id": 727947, "postDate": "2020-01-24T08:31:03.860Z", "content": "Very nice to see that carefully handcrafted features in combination with a solid model validation strategy still are super important. Thanks for sharing !!!
", "rawMarkdown": "Very nice to see that carefully handcrafted features in combination with a solid model validation strategy still are super important. Thanks for sharing !!!", "votes": 1 }, { "id": 727801, "postDate": "2020-01-24T04:25:28.330Z", "content": "Congrats! Impressive validation scheme!
", "rawMarkdown": "Congrats! Impressive validation scheme!", "votes": 1, "replies": [ { "id": 728633, "postDate": "2020-01-25T01:46:58.550Z", "content": "Thanks and congrats to you too, your NN is always amazing, I've learned a lot from your sharing.
", "rawMarkdown": "Thanks and congrats to you too, your NN is always amazing, I've learned a lot from your sharing.", "votes": 1 } ] }, { "id": 729939, "postDate": "2020-01-26T21:20:04.390Z", "content": "Congrats, and thanks for the great solution, feature engineering and validation method!
\n\n\n", "rawMarkdown": "Congrats, and thanks for the great solution, feature engineering and validation method! \n\n> btw, there's a small typo on the following item: \n3. Feature Engineering \n (4.) Video Prop Ratio ... If so, when dose he skip? \nshould be: *when does he skip?*", "replies": [ { "id": 729977, "postDate": "2020-01-26T23:39:48.767Z", "content": "btw, there's a small typo on the following item: \n 3. Feature Engineering \n (4.) Video Prop Ratio ... If so, when dose he skip? \n should be: when does he skip?
\n
thanks, the typo has been corrected
", "rawMarkdown": "thanks, the typo has been corrected", "votes": 1 }, { "id": 730057, "postDate": "2020-01-27T03:18:19.970Z", "content": "aah one more:\n2. Validation Strategy
\n\n2.1 GroupK CV : We use the 5 times * 5-fold GroupK by ***instation_id,***\n\n", "rawMarkdown": "aah one more:\n2. Validation Strategy\n\n 2.1 GroupK CV : We use the 5 times * 5-fold GroupK by ***instation_id,***\n\n > **installation_id** \n👍 You're the best!" }, { "id": 730387, "postDate": "2020-01-27T12:20:30.667Z", "content": "installation_id
\n
\n 👍 You're the best!
thanks again! corrected 👍 👍
", "rawMarkdown": "thanks again! corrected 👍 👍 " } ] }, { "id": 2452542, "postDate": "2023-09-23T11:37:51Z", "content": "congrats 4 years later sorry but not sorry 😁
", "rawMarkdown": "congrats 4 years later sorry but not sorry 😁" }, { "id": 766656, "postDate": "2020-03-08T14:22:11.893Z", "content": "Parabéns
", "rawMarkdown": "Parabéns " }, { "id": 757702, "postDate": "2020-02-27T03:08:23.757Z", "content": "great work!
", "rawMarkdown": " great work!" }, { "id": 752162, "postDate": "2020-02-20T19:20:21.510Z", "content": "great
", "rawMarkdown": "great" }, { "id": 742420, "postDate": "2020-02-11T09:35:45.120Z", "content": "congrats! great work!
", "rawMarkdown": "congrats! great work!" }, { "id": 737147, "postDate": "2020-02-05T00:53:07.160Z", "content": "Congratz and thx for the sharing! It is really helpful for learning!👍 🤘
", "rawMarkdown": "Congratz and thx for the sharing! It is really helpful for learning!👍 🤘 " }, { "id": 735020, "postDate": "2020-02-02T11:47:54.513Z", "content": "Congratulations!
", "rawMarkdown": "Congratulations!" }, { "id": 734885, "postDate": "2020-02-02T06:07:08.797Z", "content": "Congratulations
", "rawMarkdown": "Congratulations" }, { "id": 734543, "postDate": "2020-02-01T16:26:04.127Z", "content": "It is so useful.
", "rawMarkdown": "It is so useful." }, { "id": 734338, "postDate": "2020-02-01T09:57:53.053Z", "content": "Congratulations! Thank you for sharing your approach. It was very interesting to read.
", "rawMarkdown": "Congratulations! Thank you for sharing your approach. It was very interesting to read." }, { "id": 733381, "postDate": "2020-01-31T03:52:14.497Z", "content": "xuedaole
", "rawMarkdown": "xuedaole" }, { "id": 732605, "postDate": "2020-01-30T01:50:21.887Z", "content": "Congraturations!\nThat's very helpful for me.\nThank you for shareing.
", "rawMarkdown": "Congraturations!\nThat's very helpful for me.\nThank you for shareing." }, { "id": 732344, "postDate": "2020-01-29T17:53:13.760Z", "content": "Good study!!
", "rawMarkdown": "Good study!!" }, { "id": 732151, "postDate": "2020-01-29T14:35:37.640Z", "content": "That's very inspiring, thank you for sharing 👍
", "rawMarkdown": "That's very inspiring, thank you for sharing 👍 " }, { "id": 731927, "postDate": "2020-01-29T09:02:58.187Z", "content": "Congratulations, thanks for the detailed explanation.
", "rawMarkdown": "Congratulations, thanks for the detailed explanation." }, { "id": 731922, "postDate": "2020-01-29T08:56:48.843Z", "content": "Congratulations! Interesting validation scheme! Many good things to learn!
", "rawMarkdown": "Congratulations! Interesting validation scheme! Many good things to learn! " }, { "id": 731791, "postDate": "2020-01-29T03:56:49.467Z", "content": "That was a very detailed and progressive solution to the problem! Thanks for sharing!
", "rawMarkdown": "That was a very detailed and progressive solution to the problem! Thanks for sharing!" }, { "id": 731125, "postDate": "2020-01-28T11:35:43.597Z", "content": "Congratulations..!!
", "rawMarkdown": "Congratulations..!!" }, { "id": 730979, "postDate": "2020-01-28T07:47:26.347Z", "content": "that's great
", "rawMarkdown": "that's great " }, { "id": 730879, "postDate": "2020-01-28T03:46:31.313Z", "content": "Congratulations and thank you for your sharing.\nHow did you experiment to see whether or not a new feature is useful?\nI encountered the problem of overfitting for KFold and LightGBM's seed.\n(One feature improves a loss in one seed. But using another seed, that feature worsens a loss.)\nHow did you deal with it?
", "rawMarkdown": "Congratulations and thank you for your sharing.\nHow did you experiment to see whether or not a new feature is useful?\nI encountered the problem of overfitting for KFold and LightGBM's seed.\n(One feature improves a loss in one seed. But using another seed, that feature worsens a loss.)\nHow did you deal with it?", "replies": [ { "id": 732801, "postDate": "2020-01-30T09:38:17.330Z", "content": "Thanks!\nI have asked similar question on the kaggle forum, lol. you can check the link\nTo increase confidence of cv improvement, this is what I do now:\n1) make sure most of folds are improving. \n2) if we are not confident of cv improvement, use another seed for double check.
", "rawMarkdown": "Thanks!\nI have asked similar question on the kaggle forum, lol. you can check the [link](https://www.kaggle.com/c/home-credit-default-risk/discussion/64596#378926)\nTo increase confidence of cv improvement, this is what I do now:\n1) make sure most of folds are improving. \n2) if we are not confident of cv improvement, use another seed for double check.", "votes": 2 }, { "id": 732968, "postDate": "2020-01-30T14:00:59.517Z", "content": "Your answer and link are very useful for me !!\nThank you very much.
", "rawMarkdown": "Your answer and link are very useful for me !!\nThank you very much.", "votes": 1 } ] }, { "id": 730878, "postDate": "2020-01-28T03:45:17.447Z", "content": "👍
", "rawMarkdown": "👍 " }, { "id": 730860, "postDate": "2020-01-28T02:50:40.580Z", "content": "Congrats! Your teams feature engineering strategy is very impressive with the level of detail involved.
", "rawMarkdown": "Congrats! Your teams feature engineering strategy is very impressive with the level of detail involved. " }, { "id": 730436, "postDate": "2020-01-27T13:33:42.203Z", "content": "What is local cv and global cv?
", "rawMarkdown": "What is local cv and global cv?", "replies": [ { "id": 730858, "postDate": "2020-01-28T02:47:37.060Z", "content": "\"local cv\", I mean cross-validation score we did in local, correspond to Public score and Private score.
", "rawMarkdown": "\"local cv\", I mean cross-validation score we did in local, correspond to Public score and Private score." } ] }, { "id": 730002, "postDate": "2020-01-27T00:43:18.637Z", "content": "Congrats!
\n\nIs \"GroupK CV\" and \"Nested CV\" very correlated?
", "rawMarkdown": "Congrats!\n\nIs \"GroupK CV\" and \"Nested CV\" very correlated?\n", "replies": [ { "id": 730386, "postDate": "2020-01-27T12:18:57.413Z", "content": "Thanks! \"GroupK CV\" and \"Nested CV\" is very correlated.
", "rawMarkdown": "Thanks! \"GroupK CV\" and \"Nested CV\" is very correlated.", "votes": 1 } ] }, { "id": 729594, "postDate": "2020-01-26T12:30:58.117Z", "content": "Congratulations!
", "rawMarkdown": "Congratulations!" }, { "id": 729582, "postDate": "2020-01-26T12:04:06.117Z", "content": "Congrats! Nice solution and great feature engineering!!!
", "rawMarkdown": "Congrats! Nice solution and great feature engineering!!!" }, { "id": 729519, "postDate": "2020-01-26T10:06:46.547Z", "content": "Congratulations.\nThat is very helpful.
", "rawMarkdown": "Congratulations.\nThat is very helpful." }, { "id": 728922, "postDate": "2020-01-25T13:31:46.573Z", "content": "Great work, Congrats!
", "rawMarkdown": "Great work, Congrats!" }, { "id": 728853, "postDate": "2020-01-25T11:30:16.067Z", "content": "Congratulations. Are you going to opensource your solution?
", "rawMarkdown": "Congratulations. Are you going to opensource your solution?" }, { "id": 728828, "postDate": "2020-01-25T10:51:19.263Z", "content": "This is very detailed. Thanks a lot and congratulations!
", "rawMarkdown": "This is very detailed. Thanks a lot and congratulations!" }, { "id": 727876, "postDate": "2020-01-24T06:35:52.813Z", "content": "Thank you for the write up, congratulations. So many ideas I hadn't thought of.\nI have a few questions.\n- Can you tell me how you figured out which game is related to which assessment? \n- And how did you map it to an assessment?\n-Also how is the assessment title information fed to the model? Is it one-hot encoding? Or is there a better way.
\n\nA general question, not necessarily related to your particular solution: Is splitting the model into 5 (one for each assessment) a bad idea, I thought that is what everyone would do, but I don't see that in the write ups.
\n\nThanks again.
", "rawMarkdown": "Thank you for the write up, congratulations. So many ideas I hadn't thought of.\nI have a few questions.\n- Can you tell me how you figured out which game is related to which assessment? \n- And how did you map it to an assessment?\n-Also how is the assessment title information fed to the model? Is it one-hot encoding? Or is there a better way.\n\nA general question, not necessarily related to your particular solution: Is splitting the model into 5 (one for each assessment) a bad idea, I thought that is what everyone would do, but I don't see that in the write ups.\n\nThanks again.", "replies": [ { "id": 728632, "postDate": "2020-01-25T01:44:37.357Z", "content": "1) We use the media order to decide similar games. Several games are set before each assessment, and we think these games are designed to practice similar skill for child.\n2) Lable encoding assess title and set cat params in lightgbm or catboost. you can check lightgbm category support\n3) Splitting the model into 5 group by assessment title is a idea worth try. It perhaps provides diversity for model ensemble. We try similar idea in past competition Avito Demand Prediction Challenge and it works well.
", "rawMarkdown": "1) We use the media order to decide similar games. Several games are set before each assessment, and we think these games are designed to practice similar skill for child.\n2) Lable encoding assess title and set cat params in lightgbm or catboost. you can check [lightgbm category support](https://lightgbm.readthedocs.io/en/latest/Advanced-Topics.html#categorical-feature-support)\n3) Splitting the model into 5 group by assessment title is a idea worth try. It perhaps provides diversity for model ensemble. We try similar idea in past competition *Avito Demand Prediction Challenge* and it works well.\n", "votes": 2 } ] }, { "id": 727857, "postDate": "2020-01-24T05:52:05.673Z", "content": "Wow, I can't believe the winner is only using LightGBM with Catboost!
", "rawMarkdown": "Wow, I can't believe the winner is only using LightGBM with Catboost!" }, { "id": 727826, "postDate": "2020-01-24T05:06:09.020Z", "content": "Congratz
", "rawMarkdown": "Congratz" }, { "id": 727821, "postDate": "2020-01-24T05:01:08.873Z", "content": "Congratulations, well done especially for those who are still students.
", "rawMarkdown": "Congratulations, well done especially for those who are still students." }, { "id": 727798, "postDate": "2020-01-24T04:23:49.463Z", "content": "Congrats! Great work
", "rawMarkdown": "Congrats! Great work" }, { "id": 727796, "postDate": "2020-01-24T04:15:50.873Z", "content": "Congratulations!👍
", "rawMarkdown": "Congratulations!👍 " }, { "id": 727794, "postDate": "2020-01-24T04:02:44.863Z", "content": "Congratulations
", "rawMarkdown": "Congratulations " }, { "id": 733656, "postDate": "2020-01-31T11:43:25.337Z", "rawMarkdown": "", "isDeleted": true }, { "id": 731666, "postDate": "2020-01-28T21:56:37.880Z", "rawMarkdown": "", "isDeleted": true }, { "id": 730840, "postDate": "2020-01-28T02:23:56.770Z", "rawMarkdown": "", "isDeleted": true }, { "id": 729210, "postDate": "2020-01-25T22:24:41.693Z", "rawMarkdown": "", "isDeleted": true }, { "id": 728779, "postDate": "2020-01-25T08:42:24.543Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 728813, "postDate": "2020-01-25T10:09:29.547Z", "content": "Suppose one test installation_id have total n assessments, random choose one.
\n\n$$\n\\begin{align}\nE(\\text{number of assessments we observed in test}) = \\sum_{i=1}^n\\frac{i}{n} = \\frac{1+n}{2}\n\\end{align}\n$$\n$$\n\\begin{align}\n\\text{weight for test} = \\frac{1}{2 * \\text{number of assessments we observed in test} - 1}\n\\end{align}\n$$
", "rawMarkdown": "Suppose one test installation_id have total *n* assessments, random choose one.\n\n$$\n\\begin{align}\nE(\\text{number of assessments we observed in test}) = \\sum_{i=1}^n\\frac{i}{n} = \\frac{1+n}{2}\n\\end{align}\n$$\n$$\n\\begin{align}\n\\text{weight for test} = \\frac{1}{2 * \\text{number of assessments we observed in test} - 1}\n\\end{align}\n$$ ", "votes": 4 }, { "id": 728840, "postDate": "2020-01-25T11:13:07.813Z", "rawMarkdown": "", "isDeleted": true }, { "id": 728845, "postDate": "2020-01-25T11:17:02.473Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 730067, "postDate": "2020-01-27T03:58:01.753Z", "content": "Good observation, seems you nailed it.
\n\n\n", "rawMarkdown": "Good observation, seems you nailed it. \n\n> btw: *we observerd *\n *observed*", "votes": 1 } ] }, { "id": 731948, "postDate": "2020-01-29T09:38:04.557Z", "content": "btw: *we observerd *\n observed
\n
Thank you! very useful :)
", "rawMarkdown": "Thank you! very useful :)", "votes": 1 }, { "id": 737210, "postDate": "2020-02-05T03:48:37Z", "content": "Congratulations and thanks for sharing
", "rawMarkdown": "Congratulations and thanks for sharing" }, { "id": 737205, "postDate": "2020-02-05T03:39:40.227Z", "content": "Usefull, thank you 💪
", "rawMarkdown": "Usefull, thank you 💪 " }, { "id": 732941, "postDate": "2020-01-30T13:20:29.397Z", "content": "Thanks for sharing 👌
", "rawMarkdown": "Thanks for sharing 👌 " }, { "id": 732561, "postDate": "2020-01-30T00:01:00.540Z", "content": "Congrats and thanks for sharing!
", "rawMarkdown": "Congrats and thanks for sharing!" }, { "id": 732311, "postDate": "2020-01-29T17:08:15.190Z", "content": "thanks for sharing.
", "rawMarkdown": "thanks for sharing." }, { "id": 732224, "postDate": "2020-01-29T15:33:07.173Z", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!" }, { "id": 731700, "postDate": "2020-01-28T23:58:50.607Z", "content": "Congratulations! Thanks for sharing
", "rawMarkdown": "Congratulations! Thanks for sharing" }, { "id": 731207, "postDate": "2020-01-28T12:45:41.330Z", "content": "Thanks for sharing.
", "rawMarkdown": "Thanks for sharing. " } ], "complete": true, "source": "live", "fetched_at": "2026-10-23T06:54:22Z", "competition_id": 16531, "competition_slug": "data-science-bowl-2019", "team_id": 3855413, "solution_url": "/competitions/data-science-bowl-2019/writeups/zr-oyx-1st-place-solution" }