![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1305588%2F03a952505303120659eb4ec830ad0ab0%2Ftruncated_oof_prediction_average.png?generation=1579774962577581&alt=media\")
First, thanks kaggle team for a exciting competition, and congratulations to all winning teams and good result teams.\nI joined this competition solely, so it was hard but very interesting competition.
\n\n1 year ago, I have experienced very big shake down(2th => about 1500th) at Microsoft Malware Competition.\nhttps://www.kaggle.com/c/microsoft-malware-prediction/discussion/83950
\n\nFrom this experience, I made effort to validation strategy and public/private analysis.\nAs a result I got my first Gold Medal by shake up.
\n\nHere is my solution. (I am sorry for my poor English.)
\n\n10th(solo Gold) / 3523
\n\nI think public dataset is not good distribution for validation because there are only 1000 records.\nI calculated by 1000 times the average of train oof prediction which is truncated and randomly sampling 1000 rows.\nThe histgram is as follows.
\n\n
From this histgram, It seems that the public dataset is rare case.
\n\nTherefore I trust CV (ignore LB) and use threshold of CV best (explain detail in next).
\n\nFinaly I used constant threshold [1.04, 1.76, 2.18].\nThis threshold is calculated by OptimizedRounder's threshold average in 500 truncated oof validation.
\n\nI tryed many methods, but I believed maximum threshold in local CV prediction is most reliable.
\n\nSome public kernels decided threshold by target distribution.\nIn my experiment, the method is good for public LB than other methods, but I think this method is overfitting to Public LB because the distribution is not equal to the truncated target distribution and not best distribution for QWK.
\n\nI made 3000~5000 features overall, but I think there are no magic features.\n(Finaly I used about 300 features.)
\n\nGood features for me is as follows.
\n\nTo evaluate features effect in truncated validation, I use LGB feature importance by truncated training data.\nIn each fold, I make 50 truncated datasets, and change dataset per 5 iteration by using lightgbm init_model params.
\n\nI use top 300 features (the number is feeling).
\n\nCongrats for your first solo gold and thank you for sharing 🎉
", "votes": 1, "replies": [] }, { "id": 727287, "author_name": "Massoud Hosseinali", "author_url": "", "post_date": "2020-01-23T15:56:01.230000", "content": "Congrats Pao.
", "votes": 1, "replies": [] }, { "id": 727110, "author_name": "IGreen", "author_url": "", "post_date": "2020-01-23T13:17:58.833000", "content": "Congratulations! Thank you so much for sharing your solutions.
", "votes": 1, "replies": [] }, { "id": 727082, "author_name": "zakopuro", "author_url": "", "post_date": "2020-01-23T12:54:37.430000", "content": "Congratulations!
", "votes": 1, "replies": [] }, { "id": 726971, "author_name": "Ailurophile", "author_url": "", "post_date": "2020-01-23T10:46:32.770000", "content": "Congratulations\nGreat Write-Up\nThanks for Sharing your Approach & Insights!! @go5kuramubon
", "votes": 1, "replies": [] }, { "id": 726959, "author_name": "Hanjoon Choe", "author_url": "", "post_date": "2020-01-23T10:39:58.897000", "content": "Congratulations!
", "votes": 1, "replies": [] }, { "id": 726955, "author_name": "davidlee0314", "author_url": "", "post_date": "2020-01-23T10:36:46.767000", "content": "Congratulations! Your features seem well preprocessed
", "votes": 1, "replies": [] }, { "id": 726952, "author_name": "T.Ito", "author_url": "", "post_date": "2020-01-23T10:35:08.367000", "content": "Congratulations and thanks for sharing your solution!!!!!
", "votes": 1, "replies": [] }, { "id": 727862, "author_name": "umut", "author_url": "", "post_date": "2020-01-24T06:07:32.203000", "content": "One more question on the topic of weights in xgboost. I added the completed assessments in test.csv to the training data and thinking that those are better data than train (because they are same IDs as the users for which we need to predict, I put a larger weight for them, I tried putting weight=1 for train.csv rows and weight=3 for test.csv rows but my score decreased. Can anyone comment why? Is there an obvious flaw with this line of thinking? Maybe 3 was too hight, would it have made sense to try some smaller numbers like 1.5?
", "votes": 0, "replies": [ { "id": 729991, "author_name": "pao", "author_url": "", "post_date": "2020-01-27T00:19:06.387000", "content": "I don't know the best weigh but I think we should keep the test weight lower because the count of assessment in test data for train is lower than test for LB.\nIt makes the training data distribution different from test for LB though increasing train\ndata is good for model accuracy.
", "votes": 0, "replies": [] } ] }, { "id": 727860, "author_name": "umut", "author_url": "", "post_date": "2020-01-24T06:02:39.207000", "content": "I did a similar test as you did in Public vs. Private. My conclusion was different, I thought that the IDs in test.csv are more successful so I did this: I computed probability of getting a 3 in Cauldron Filler in train data and in test data, then I took the ratios. Let's say ratio is 1.2, that means the IDs in test.csv were x1.2 better so I put that as the weight column (xgboost allows for a weight parameter) for all rows in train where title==Cauldron Filler and accuracy_group==3. I did this for all accuracy groups. So even if test IDs are more or less successful compared to train, this way I thought the train set would be calibrated somewhat to the testing data. This increased my CV somewhat but my score was already so low that I wasn't really paying too much attention to small bumps in my score so I am not sure.\nCan you or anyone else chime in on this. Is this a logical thing to try? And is it correct use of xgboost weight parameter?\nThanks
", "votes": 0, "replies": [ { "id": 729990, "author_name": "pao", "author_url": "", "post_date": "2020-01-27T00:10:37.390000", "content": "I think this method makes overfitting because we cannot know the distribution of private test.\nMoreover I think the difficulty of assessment is same between train and test.
", "votes": 1, "replies": [] } ] }, { "id": 727854, "author_name": "umut", "author_url": "", "post_date": "2020-01-24T05:49:11.793000", "content": "Hello, thank you for the write up, beginner questions here. I am trying to understand how you did validation and what is truncated CV in general.\n1)What is the data that you do the GroupKFold on? Is it on all the rows where accuracy_group exists, or the entire 11.3 million rows of train.csv ? \nI checked there are 3614 installation_ids with at least 1 assessment done. So I think you get 361 ids per group\n2)then why do you split it to 51 folds? Why not ask for 510 folds (or 501) in the first place?\n3) for each fold, since 1 set is just early stopping, does that mean you run cv on the other 50 sets which means the model is trained with 49 sets, and error is computed on the 1 remaining set, and all 50 errors are averaged?
", "votes": 0, "replies": [ { "id": 729985, "author_name": "pao", "author_url": "", "post_date": "2020-01-27T00:01:42.460000", "content": "Sorry for late.
\n\n\n", "votes": 1, "replies": [] } ] }, { "id": 727267, "author_name": "samshipengs", "author_url": "", "post_date": "2020-01-23T15:33:15.877000", "content": "1\n Group on installation_id.
\n \n2\n I don't use 51 fold. Only 10fold.\n From the one validation set in 10 fold, I make 51 sets by selecting assessment randomly per installation_id.
\n \n3\n I use only 10 fold.\n Per 1 fold, 1 set for early stopping and 50 set for validation score.
\n
thanks for sharing and congrats! \n> In each fold, I make 50 truncated datasets, and change dataset per 5 iteration by using lightgbm init_model params.
\n\nthis is pretty interesting and its my first time seen this kind of approach. Have you compared this with just train one lgb on the combination of these 50 truncated datasets? Is there a good motivation for doing this 5 iteration training each time with different truncated datasets? It sort makes sense to me in neural nets (i.e. random sampling of shuffled training data in batch training), where the nn weights carries information, but its hard to imagine how it works for lgb, where the training is made based on splits. Does it keep building leafs or it could replace the spliting criteria from init_model completely? (EDIT: now i think about, it seems like it just keeps building base trees but with new data)
Did you still apply early stopping during training, or the training is done when all the 50x5 iteration is finished?
", "votes": 0, "replies": [ { "id": 728604, "author_name": "pao", "author_url": "", "post_date": "2020-01-25T00:38:39.680000", "content": "Thanks for a good question.
\n\n> Have you compared this with just train one lgb on the combination of these 50 truncated datasets? Is there a good motivation for doing this 5 iteration training each time with different truncated datasets?
\n\nI compared the two model. In fact, the validation score is almost same, and 50 truncated model takes more time to learning. \nHowever my motivation of truncated model is to evaluate feature importance correctly.\nThe distribution of all data is different from that of truncated data.\nIt means that the feature importance of training with all data is a little different from the feature importance which we want to check.
\n\n> Does it keep building leafs or it could replace the spliting criteria from init_model completely?
\n\nIn lgb, a decision tree is added to the model by each iteration, and if use init_model, the added trees keeps and add new tree to the init_model trees.
", "votes": 1, "replies": [] } ] }, { "id": 730053, "author_name": "", "author_url": "", "post_date": "2020-01-27T03:17:13.940000", "content": "", "votes": 3, "replies": [] }, { "id": 727500, "author_name": "", "author_url": "", "post_date": "2020-01-23T18:47:59.927000", "content": "Congratulations Thanks for sharing !
", "votes": 6, "replies": [] } ], "comment_count": 18, "messages": [ { "id": 726942, "postDate": "2020-01-23T10:25:16.653Z", "content": "First, thanks kaggle team for a exciting competition, and congratulations to all winning teams and good result teams.\nI joined this competition solely, so it was hard but very interesting competition.
\n\n1 year ago, I have experienced very big shake down(2th => about 1500th) at Microsoft Malware Competition.\nhttps://www.kaggle.com/c/microsoft-malware-prediction/discussion/83950
\n\nFrom this experience, I made effort to validation strategy and public/private analysis.\nAs a result I got my first Gold Medal by shake up.
\n\nHere is my solution. (I am sorry for my poor English.)
\n\n10th(solo Gold) / 3523
\n\nI think public dataset is not good distribution for validation because there are only 1000 records.\nI calculated by 1000 times the average of train oof prediction which is truncated and randomly sampling 1000 rows.\nThe histgram is as follows.
\n\n
From this histgram, It seems that the public dataset is rare case.
\n\nTherefore I trust CV (ignore LB) and use threshold of CV best (explain detail in next).
\n\nFinaly I used constant threshold [1.04, 1.76, 2.18].\nThis threshold is calculated by OptimizedRounder's threshold average in 500 truncated oof validation.
\n\nI tryed many methods, but I believed maximum threshold in local CV prediction is most reliable.
\n\nSome public kernels decided threshold by target distribution.\nIn my experiment, the method is good for public LB than other methods, but I think this method is overfitting to Public LB because the distribution is not equal to the truncated target distribution and not best distribution for QWK.
\n\nI made 3000~5000 features overall, but I think there are no magic features.\n(Finaly I used about 300 features.)
\n\nGood features for me is as follows.
\n\nTo evaluate features effect in truncated validation, I use LGB feature importance by truncated training data.\nIn each fold, I make 50 truncated datasets, and change dataset per 5 iteration by using lightgbm init_model params.
\n\nI use top 300 features (the number is feeling).
\n\nCongrats for your first solo gold and thank you for sharing 🎉
", "rawMarkdown": "Congrats for your first solo gold and thank you for sharing 🎉 ", "votes": 1 }, { "id": 727287, "postDate": "2020-01-23T15:56:01.230Z", "content": "Congrats Pao.
", "rawMarkdown": "Congrats Pao.", "votes": 1 }, { "id": 727110, "postDate": "2020-01-23T13:17:58.833Z", "content": "Congratulations! Thank you so much for sharing your solutions.
", "rawMarkdown": "Congratulations! Thank you so much for sharing your solutions.", "votes": 1 }, { "id": 727082, "postDate": "2020-01-23T12:54:37.430Z", "content": "Congratulations!
", "rawMarkdown": "Congratulations!", "votes": 1 }, { "id": 726971, "postDate": "2020-01-23T10:46:32.770Z", "content": "Congratulations\nGreat Write-Up\nThanks for Sharing your Approach & Insights!! @go5kuramubon
", "rawMarkdown": "Congratulations\nGreat Write-Up\nThanks for Sharing your Approach & Insights!! @go5kuramubon ", "votes": 1 }, { "id": 726959, "postDate": "2020-01-23T10:39:58.897Z", "content": "Congratulations!
", "rawMarkdown": "Congratulations!", "votes": 1 }, { "id": 726955, "postDate": "2020-01-23T10:36:46.767Z", "content": "Congratulations! Your features seem well preprocessed
", "rawMarkdown": "Congratulations! Your features seem well preprocessed", "votes": 1 }, { "id": 726952, "postDate": "2020-01-23T10:35:08.367Z", "content": "Congratulations and thanks for sharing your solution!!!!!
", "rawMarkdown": "Congratulations and thanks for sharing your solution!!!!!", "votes": 1 }, { "id": 727862, "postDate": "2020-01-24T06:07:32.203Z", "content": "One more question on the topic of weights in xgboost. I added the completed assessments in test.csv to the training data and thinking that those are better data than train (because they are same IDs as the users for which we need to predict, I put a larger weight for them, I tried putting weight=1 for train.csv rows and weight=3 for test.csv rows but my score decreased. Can anyone comment why? Is there an obvious flaw with this line of thinking? Maybe 3 was too hight, would it have made sense to try some smaller numbers like 1.5?
", "rawMarkdown": "One more question on the topic of weights in xgboost. I added the completed assessments in test.csv to the training data and thinking that those are better data than train (because they are same IDs as the users for which we need to predict, I put a larger weight for them, I tried putting weight=1 for train.csv rows and weight=3 for test.csv rows but my score decreased. Can anyone comment why? Is there an obvious flaw with this line of thinking? Maybe 3 was too hight, would it have made sense to try some smaller numbers like 1.5?", "replies": [ { "id": 729991, "postDate": "2020-01-27T00:19:06.387Z", "content": "I don't know the best weigh but I think we should keep the test weight lower because the count of assessment in test data for train is lower than test for LB.\nIt makes the training data distribution different from test for LB though increasing train\ndata is good for model accuracy.
", "rawMarkdown": "I don't know the best weigh but I think we should keep the test weight lower because the count of assessment in test data for train is lower than test for LB.\nIt makes the training data distribution different from test for LB though increasing train\ndata is good for model accuracy." } ] }, { "id": 727860, "postDate": "2020-01-24T06:02:39.207Z", "content": "I did a similar test as you did in Public vs. Private. My conclusion was different, I thought that the IDs in test.csv are more successful so I did this: I computed probability of getting a 3 in Cauldron Filler in train data and in test data, then I took the ratios. Let's say ratio is 1.2, that means the IDs in test.csv were x1.2 better so I put that as the weight column (xgboost allows for a weight parameter) for all rows in train where title==Cauldron Filler and accuracy_group==3. I did this for all accuracy groups. So even if test IDs are more or less successful compared to train, this way I thought the train set would be calibrated somewhat to the testing data. This increased my CV somewhat but my score was already so low that I wasn't really paying too much attention to small bumps in my score so I am not sure.\nCan you or anyone else chime in on this. Is this a logical thing to try? And is it correct use of xgboost weight parameter?\nThanks
", "rawMarkdown": "I did a similar test as you did in Public vs. Private. My conclusion was different, I thought that the IDs in test.csv are more successful so I did this: I computed probability of getting a 3 in Cauldron Filler in train data and in test data, then I took the ratios. Let's say ratio is 1.2, that means the IDs in test.csv were x1.2 better so I put that as the weight column (xgboost allows for a weight parameter) for all rows in train where title==Cauldron Filler and accuracy_group==3. I did this for all accuracy groups. So even if test IDs are more or less successful compared to train, this way I thought the train set would be calibrated somewhat to the testing data. This increased my CV somewhat but my score was already so low that I wasn't really paying too much attention to small bumps in my score so I am not sure.\nCan you or anyone else chime in on this. Is this a logical thing to try? And is it correct use of xgboost weight parameter?\nThanks", "replies": [ { "id": 729990, "postDate": "2020-01-27T00:10:37.390Z", "content": "I think this method makes overfitting because we cannot know the distribution of private test.\nMoreover I think the difficulty of assessment is same between train and test.
", "rawMarkdown": "I think this method makes overfitting because we cannot know the distribution of private test.\nMoreover I think the difficulty of assessment is same between train and test.", "votes": 1 } ] }, { "id": 727854, "postDate": "2020-01-24T05:49:11.793Z", "content": "Hello, thank you for the write up, beginner questions here. I am trying to understand how you did validation and what is truncated CV in general.\n1)What is the data that you do the GroupKFold on? Is it on all the rows where accuracy_group exists, or the entire 11.3 million rows of train.csv ? \nI checked there are 3614 installation_ids with at least 1 assessment done. So I think you get 361 ids per group\n2)then why do you split it to 51 folds? Why not ask for 510 folds (or 501) in the first place?\n3) for each fold, since 1 set is just early stopping, does that mean you run cv on the other 50 sets which means the model is trained with 49 sets, and error is computed on the 1 remaining set, and all 50 errors are averaged?
", "rawMarkdown": "Hello, thank you for the write up, beginner questions here. I am trying to understand how you did validation and what is truncated CV in general.\n1)What is the data that you do the GroupKFold on? Is it on all the rows where accuracy_group exists, or the entire 11.3 million rows of train.csv ? \nI checked there are 3614 installation_ids with at least 1 assessment done. So I think you get 361 ids per group\n2)then why do you split it to 51 folds? Why not ask for 510 folds (or 501) in the first place?\n3) for each fold, since 1 set is just early stopping, does that mean you run cv on the other 50 sets which means the model is trained with 49 sets, and error is computed on the 1 remaining set, and all 50 errors are averaged?\n", "replies": [ { "id": 729985, "postDate": "2020-01-27T00:01:42.460Z", "content": "Sorry for late.
\n\n\n", "rawMarkdown": "Sorry for late.\n> 1\nGroup on installation_id.\n\n> 2\nI don't use 51 fold. Only 10fold.\nFrom the one validation set in 10 fold, I make 51 sets by selecting assessment randomly per installation_id.\n\n> 3\nI use only 10 fold.\nPer 1 fold, 1 set for early stopping and 50 set for validation score.", "votes": 1 } ] }, { "id": 727267, "postDate": "2020-01-23T15:33:15.877Z", "content": "1\n Group on installation_id.
\n \n2\n I don't use 51 fold. Only 10fold.\n From the one validation set in 10 fold, I make 51 sets by selecting assessment randomly per installation_id.
\n \n3\n I use only 10 fold.\n Per 1 fold, 1 set for early stopping and 50 set for validation score.
\n
thanks for sharing and congrats! \n> In each fold, I make 50 truncated datasets, and change dataset per 5 iteration by using lightgbm init_model params.
\n\nthis is pretty interesting and its my first time seen this kind of approach. Have you compared this with just train one lgb on the combination of these 50 truncated datasets? Is there a good motivation for doing this 5 iteration training each time with different truncated datasets? It sort makes sense to me in neural nets (i.e. random sampling of shuffled training data in batch training), where the nn weights carries information, but its hard to imagine how it works for lgb, where the training is made based on splits. Does it keep building leafs or it could replace the spliting criteria from init_model completely? (EDIT: now i think about, it seems like it just keeps building base trees but with new data)
Did you still apply early stopping during training, or the training is done when all the 50x5 iteration is finished?
", "rawMarkdown": "thanks for sharing and congrats! \n> In each fold, I make 50 truncated datasets, and change dataset per 5 iteration by using lightgbm init_model params.\n\nthis is pretty interesting and its my first time seen this kind of approach. Have you compared this with just train one lgb on the combination of these 50 truncated datasets? Is there a good motivation for doing this 5 iteration training each time with different truncated datasets? It sort makes sense to me in neural nets (i.e. random sampling of shuffled training data in batch training), where the nn weights carries information, but its hard to imagine how it works for lgb, where the training is made based on splits. Does it keep building leafs or it could replace the spliting criteria from `init_model` completely? (EDIT: now i think about, it seems like it just keeps building base trees but with new data)\n\nDid you still apply early stopping during training, or the training is done when all the 50x5 iteration is finished?", "replies": [ { "id": 728604, "postDate": "2020-01-25T00:38:39.680Z", "content": "Thanks for a good question.
\n\n> Have you compared this with just train one lgb on the combination of these 50 truncated datasets? Is there a good motivation for doing this 5 iteration training each time with different truncated datasets?
\n\nI compared the two model. In fact, the validation score is almost same, and 50 truncated model takes more time to learning. \nHowever my motivation of truncated model is to evaluate feature importance correctly.\nThe distribution of all data is different from that of truncated data.\nIt means that the feature importance of training with all data is a little different from the feature importance which we want to check.
\n\n> Does it keep building leafs or it could replace the spliting criteria from init_model completely?
\n\nIn lgb, a decision tree is added to the model by each iteration, and if use init_model, the added trees keeps and add new tree to the init_model trees.
", "rawMarkdown": "Thanks for a good question.\n\n> Have you compared this with just train one lgb on the combination of these 50 truncated datasets? Is there a good motivation for doing this 5 iteration training each time with different truncated datasets?\n\nI compared the two model. In fact, the validation score is almost same, and 50 truncated model takes more time to learning. \nHowever my motivation of truncated model is to evaluate feature importance correctly.\nThe distribution of all data is different from that of truncated data.\nIt means that the feature importance of training with all data is a little different from the feature importance which we want to check.\n\n\n> Does it keep building leafs or it could replace the spliting criteria from init_model completely?\n\nIn lgb, a decision tree is added to the model by each iteration, and if use init_model, the added trees keeps and add new tree to the init_model trees.", "votes": 1 } ] }, { "id": 730053, "postDate": "2020-01-27T03:17:13.940Z", "rawMarkdown": "", "votes": 3, "isDeleted": true }, { "id": 727500, "postDate": "2020-01-23T18:47:59.927Z", "content": "Congratulations Thanks for sharing !
", "rawMarkdown": "Congratulations Thanks for sharing !", "votes": 6 } ], "complete": true, "source": "live", "fetched_at": "2026-10-22T10:52:18Z", "competition_id": 16531, "competition_slug": "data-science-bowl-2019", "team_id": 3924421, "solution_url": "/competitions/data-science-bowl-2019/writeups/pao-10th-place-solution" }