So, i was training my model with large number of estimators (around 1000-1500) as that could make the eval score on validation set near 0.70 . But that submissions were getting score around 1.3-1.4 on public leaderboard. Then i decided not to train the model too much (around 500 n_estimators and learning rate 0.1 ) and also made some features categorical , which made sense . But when i submitted it, the public score came down to 1.12 with a boom ! I didn't expect that much overfitting in my model, as the validation split could still be decreased a lot.
", "messages": [ { "id": 654919, "postDate": "2019-10-22T13:42:40.810Z", "content": "So, i was training my model with large number of estimators (around 1000-1500) as that could make the eval score on validation set near 0.70 . But that submissions were getting score around 1.3-1.4 on public leaderboard. Then i decided not to train the model too much (around 500 n_estimators and learning rate 0.1 ) and also made some features categorical , which made sense . But when i submitted it, the public score came down to 1.12 with a boom ! I didn't expect that much overfitting in my model, as the validation split could still be decreased a lot.
", "rawMarkdown": "So, i was training my model with large number of estimators (around 1000-1500) as that could make the eval score on validation set near 0.70 . But that submissions were getting score around 1.3-1.4 on public leaderboard. Then i decided not to train the model too much (around 500 n_estimators and learning rate 0.1 ) and also made some features categorical , which made sense . But when i submitted it, the public score came down to 1.12 with a boom ! I didn't expect that much overfitting in my model, as the validation split could still be decreased a lot.", "votes": 16 }, { "id": 655240, "postDate": "2019-10-22T21:01:22.403Z", "content": "Your validation split is leaking information - so longer epochs the leak lets the eval score get better, but the model is over fitting and would have early stopped with a not leaking validation set.
\n\nI think the solution is to change the split - not reduce epochs.
", "rawMarkdown": "Your validation split is leaking information - so longer epochs the leak lets the eval score get better, but the model is over fitting and would have early stopped with a not leaking validation set. \n\nI think the solution is to change the split - not reduce epochs.", "votes": 5, "replies": [ { "id": 655453, "postDate": "2019-10-23T04:21:07.853Z", "content": "You are right , cause i see the val_score always keeps decreasing no matter many epochs i take. That is indeed possible when there is some leak , only i still haven't figure out a gd solution to resolve it.
", "rawMarkdown": "You are right , cause i see the val_score always keeps decreasing no matter many epochs i take. That is indeed possible when there is some leak , only i still haven't figure out a gd solution to resolve it. ", "votes": 1 } ] }, { "id": 655901, "postDate": "2019-10-23T16:29:46.263Z", "content": "@khairulislam is your score 1.12 a single model score?
", "rawMarkdown": "@khairulislam is your score 1.12 a single model score?", "votes": 1, "replies": [ { "id": 655907, "postDate": "2019-10-23T16:45:14.930Z", "content": "@themonologue i took mean of the predictions from models trained during cross fold validation.
", "rawMarkdown": "@themonologue i took mean of the predictions from models trained during cross fold validation. ", "votes": 2 }, { "id": 655916, "postDate": "2019-10-23T17:06:13.300Z", "content": "@khairulislam I mean did you use only one model? Only LGBM?
", "rawMarkdown": "@khairulislam I mean did you use only one model? Only LGBM?" }, { "id": 655928, "postDate": "2019-10-23T17:20:48.487Z", "content": "@themonologue Yes. Still haven't managed time to work of blending models.
", "rawMarkdown": "@themonologue Yes. Still haven't managed time to work of blending models.", "votes": 1 }, { "id": 655954, "postDate": "2019-10-23T17:58:26.070Z", "content": "@khairulislam Okay,thanks. All the best.
", "rawMarkdown": "@khairulislam Okay,thanks. All the best." } ] }, { "id": 655714, "postDate": "2019-10-23T12:19:35.727Z", "content": "Thank for your comment, I was using too many estimators around 5000 getting an score of 1.30, I will try to train with less estimators, how was your eval score with 500 estimators?
\n\nThanks for sharing Khairul.
", "rawMarkdown": "Thank for your comment, I was using too many estimators around 5000 getting an score of 1.30, I will try to train with less estimators, how was your eval score with 500 estimators?\n\nThanks for sharing Khairul.", "votes": 2, "replies": [ { "id": 655774, "postDate": "2019-10-23T13:40:15.793Z", "content": "it was around 0.88. For 400-500 estimators and n-fold = 5
", "rawMarkdown": "it was around 0.88. For 400-500 estimators and n-fold = 5", "votes": 1 }, { "id": 655942, "postDate": "2019-10-23T17:38:47.460Z", "content": "I am not using KFold yet, but I managed to get 1.27 which is kind of a bit improvement, I will keep tweaking and probably will go down to 1.20, thanks a lot.
\n\nI mean in my final submission, validation score was around that too.
", "rawMarkdown": "I am not using KFold yet, but I managed to get 1.27 which is kind of a bit improvement, I will keep tweaking and probably will go down to 1.20, thanks a lot.\n\nI mean in my final submission, validation score was around that too.", "votes": 1 } ] }, { "id": 655282, "postDate": "2019-10-22T21:47:34.623Z", "content": "Interesting! What was your val score after the changes? How did you select your validation set, train_test_split?
\n\nI personally tried TimeseriesSplit and StratifiedKFold on building IDs keeping the remaining settings the same. \nTill now, I got better results with the stratification, so I guess it's really important to maintain the same buildings in training and validation sets.
", "rawMarkdown": "Interesting! What was your val score after the changes? How did you select your validation set, train_test_split?\n\nI personally tried TimeseriesSplit and StratifiedKFold on building IDs keeping the remaining settings the same. \nTill now, I got better results with the stratification, so I guess it's really important to maintain the same buildings in training and validation sets.", "votes": 2, "replies": [ { "id": 655290, "postDate": "2019-10-22T21:58:57.207Z", "content": "Not sure that's its possible to keep leaks out. I am trying to have different buildings in the train vs validation, but other buildings for the same site can leak weather info.
\n\nRight now my brain is fried - can't seem to get the right number of rows in train and test data sets with my merges. All my work up to today has been done with bad merge so not sure I can believe anything I know. But once I get the merges fixed will try some other splits.
\n\nSeems like will need to make a judgement call - what worries me more - unbalanced buildings or memory leaking?
\n\nUsing a time split my catboost model ran for 250,000 epochs and still wanted to keep going :)
\n\nWhen my model never wants to early stop one of the things that has often been the case was a leak.
", "rawMarkdown": "Not sure that's its possible to keep leaks out. I am trying to have different buildings in the train vs validation, but other buildings for the same site can leak weather info.\n\nRight now my brain is fried - can't seem to get the right number of rows in train and test data sets with my merges. All my work up to today has been done with bad merge so not sure I can believe anything I know. But once I get the merges fixed will try some other splits. \n\nSeems like will need to make a judgement call - what worries me more - unbalanced buildings or memory leaking?\n\nUsing a time split my catboost model ran for 250,000 epochs and still wanted to keep going :)\n\nWhen my model never wants to early stop one of the things that has often been the case was a leak.", "votes": 3 }, { "id": 655450, "postDate": "2019-10-23T04:18:24.410Z", "content": "I used StratifiedKFold on building IDs. My validation scores were around 0.88.
", "rawMarkdown": "I used StratifiedKFold on building IDs. My validation scores were around 0.88.", "votes": 2 }, { "id": 655455, "postDate": "2019-10-23T04:22:38.290Z", "content": "\n", "rawMarkdown": "> Using a time split my catboost model ran for 250,000 epochs and still wanted to keep going :)\nI am facing the same issue ☹️ . Only i don't run more than 500 epochs, as it overfits afterwards." }, { "id": 655829, "postDate": "2019-10-23T14:49:09.987Z", "content": "Using a time split my catboost model ran for 250,000 epochs and still wanted to keep going :)\n I am facing the same issue ☹️ . Only i don't run more than 500 epochs, as it overfits afterwards.
\n
Thanks for the reply @khairulislam. When you say 'made some feature categorical', do you mean newly created features or just passing some of the starting features as categorical to the model?
", "rawMarkdown": "Thanks for the reply @khairulislam. When you say 'made some feature categorical', do you mean newly created features or just passing some of the starting features as categorical to the model? " }, { "id": 655846, "postDate": "2019-10-23T15:14:29.920Z", "content": "@raimonds1993 Some of the starting features like primary_usage, meter etc. I did make 2/3 new features, but they weren't categorical.
", "rawMarkdown": "@raimonds1993 Some of the starting features like primary_usage, meter etc. I did make 2/3 new features, but they weren't categorical.", "votes": 1 } ] }, { "id": 655221, "postDate": "2019-10-22T20:33:02.377Z", "content": "The problem is that your validation set might have a very different distribution than the test set - so your models behave differently.
", "rawMarkdown": "The problem is that your validation set might have a very different distribution than the test set - so your models behave differently.", "votes": 2, "replies": [ { "id": 655456, "postDate": "2019-10-23T04:24:41.807Z", "content": "Yes. As we know train data is from year 2016 and test data is from 2017 and 2018 . May be some time related features or weather change due to time causing huge difference .
", "rawMarkdown": "Yes. As we know train data is from year 2016 and test data is from 2017 and 2018 . May be some time related features or weather change due to time causing huge difference . " } ] }, { "id": 655184, "postDate": "2019-10-22T19:16:07.693Z", "content": "Overfitting is Allways a big issue here on Kaggle.
", "rawMarkdown": "Overfitting is Allways a big issue here on Kaggle.", "votes": 2 }, { "id": 658217, "postDate": "2019-10-25T19:43:24.877Z", "content": "hmmm i guess because of the split, the varying between the public and private lb sometimes scares me.
", "rawMarkdown": "hmmm i guess because of the split, the varying between the public and private lb sometimes scares me." }, { "id": 656079, "postDate": "2019-10-23T22:10:40.473Z", "content": "@khairulislam thank you for your post! Are you training on whole train dataset? I get a score of 1.46 with just 5% of the train dataset
", "rawMarkdown": "@khairulislam thank you for your post! Are you training on whole train dataset? I get a score of 1.46 with just 5% of the train dataset", "replies": [ { "id": 656246, "postDate": "2019-10-24T03:42:37.550Z", "content": "Ah yes , I am training on whole train dataset.
\n\n\n", "rawMarkdown": "Ah yes , I am training on whole train dataset. \n> I get a score of 1.46\nThat's interesting. I haven't seen anything like that so far 😲 ", "votes": 1 } ] }, { "id": 655458, "postDate": "2019-10-23T04:29:13.863Z", "rawMarkdown": "", "isDeleted": true }, { "id": 656057, "postDate": "2019-10-23T21:24:18.677Z", "content": "I get a score of 1.46\n That's interesting. I haven't seen anything like that so far 😲
\n
Nice work! Thank you.
", "rawMarkdown": "Nice work! Thank you." } ], "comments": [ { "id": 655240, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2019-10-22T21:01:22.403000", "content": "Your validation split is leaking information - so longer epochs the leak lets the eval score get better, but the model is over fitting and would have early stopped with a not leaking validation set.
\n\nI think the solution is to change the split - not reduce epochs.
", "votes": 5, "replies": [ { "id": 655453, "author_name": "Md Khairul Islam", "author_url": "", "post_date": "2019-10-23T04:21:07.853000", "content": "You are right , cause i see the val_score always keeps decreasing no matter many epochs i take. That is indeed possible when there is some leak , only i still haven't figure out a gd solution to resolve it.
", "votes": 1, "replies": [] } ] }, { "id": 655901, "author_name": "Manraj Singh", "author_url": "", "post_date": "2019-10-23T16:29:46.263000", "content": "@khairulislam is your score 1.12 a single model score?
", "votes": 1, "replies": [ { "id": 655907, "author_name": "Md Khairul Islam", "author_url": "", "post_date": "2019-10-23T16:45:14.930000", "content": "@themonologue i took mean of the predictions from models trained during cross fold validation.
", "votes": 2, "replies": [] }, { "id": 655916, "author_name": "Manraj Singh", "author_url": "", "post_date": "2019-10-23T17:06:13.300000", "content": "@khairulislam I mean did you use only one model? Only LGBM?
", "votes": 0, "replies": [] }, { "id": 655928, "author_name": "Md Khairul Islam", "author_url": "", "post_date": "2019-10-23T17:20:48.487000", "content": "@themonologue Yes. Still haven't managed time to work of blending models.
", "votes": 1, "replies": [] }, { "id": 655954, "author_name": "Manraj Singh", "author_url": "", "post_date": "2019-10-23T17:58:26.070000", "content": "@khairulislam Okay,thanks. All the best.
", "votes": 0, "replies": [] } ] }, { "id": 655714, "author_name": "Ivan de los Santos", "author_url": "", "post_date": "2019-10-23T12:19:35.727000", "content": "Thank for your comment, I was using too many estimators around 5000 getting an score of 1.30, I will try to train with less estimators, how was your eval score with 500 estimators?
\n\nThanks for sharing Khairul.
", "votes": 2, "replies": [ { "id": 655774, "author_name": "Md Khairul Islam", "author_url": "", "post_date": "2019-10-23T13:40:15.793000", "content": "it was around 0.88. For 400-500 estimators and n-fold = 5
", "votes": 1, "replies": [] }, { "id": 655942, "author_name": "Ivan de los Santos", "author_url": "", "post_date": "2019-10-23T17:38:47.460000", "content": "I am not using KFold yet, but I managed to get 1.27 which is kind of a bit improvement, I will keep tweaking and probably will go down to 1.20, thanks a lot.
\n\nI mean in my final submission, validation score was around that too.
", "votes": 1, "replies": [] } ] }, { "id": 655282, "author_name": "Federico Raimondi Cominesi", "author_url": "", "post_date": "2019-10-22T21:47:34.623000", "content": "Interesting! What was your val score after the changes? How did you select your validation set, train_test_split?
\n\nI personally tried TimeseriesSplit and StratifiedKFold on building IDs keeping the remaining settings the same. \nTill now, I got better results with the stratification, so I guess it's really important to maintain the same buildings in training and validation sets.
", "votes": 2, "replies": [ { "id": 655290, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2019-10-22T21:58:57.207000", "content": "Not sure that's its possible to keep leaks out. I am trying to have different buildings in the train vs validation, but other buildings for the same site can leak weather info.
\n\nRight now my brain is fried - can't seem to get the right number of rows in train and test data sets with my merges. All my work up to today has been done with bad merge so not sure I can believe anything I know. But once I get the merges fixed will try some other splits.
\n\nSeems like will need to make a judgement call - what worries me more - unbalanced buildings or memory leaking?
\n\nUsing a time split my catboost model ran for 250,000 epochs and still wanted to keep going :)
\n\nWhen my model never wants to early stop one of the things that has often been the case was a leak.
", "votes": 3, "replies": [] }, { "id": 655450, "author_name": "Md Khairul Islam", "author_url": "", "post_date": "2019-10-23T04:18:24.410000", "content": "I used StratifiedKFold on building IDs. My validation scores were around 0.88.
", "votes": 2, "replies": [] }, { "id": 655455, "author_name": "Md Khairul Islam", "author_url": "", "post_date": "2019-10-23T04:22:38.290000", "content": "\n", "votes": 0, "replies": [] }, { "id": 655829, "author_name": "Federico Raimondi Cominesi", "author_url": "", "post_date": "2019-10-23T14:49:09.987000", "content": "Using a time split my catboost model ran for 250,000 epochs and still wanted to keep going :)\n I am facing the same issue ☹️ . Only i don't run more than 500 epochs, as it overfits afterwards.
\n
Thanks for the reply @khairulislam. When you say 'made some feature categorical', do you mean newly created features or just passing some of the starting features as categorical to the model?
", "votes": 0, "replies": [] }, { "id": 655846, "author_name": "Md Khairul Islam", "author_url": "", "post_date": "2019-10-23T15:14:29.920000", "content": "@raimonds1993 Some of the starting features like primary_usage, meter etc. I did make 2/3 new features, but they weren't categorical.
", "votes": 1, "replies": [] } ] }, { "id": 655221, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2019-10-22T20:33:02.377000", "content": "The problem is that your validation set might have a very different distribution than the test set - so your models behave differently.
", "votes": 2, "replies": [ { "id": 655456, "author_name": "Md Khairul Islam", "author_url": "", "post_date": "2019-10-23T04:24:41.807000", "content": "Yes. As we know train data is from year 2016 and test data is from 2017 and 2018 . May be some time related features or weather change due to time causing huge difference .
", "votes": 0, "replies": [] } ] }, { "id": 655184, "author_name": "mezoganet", "author_url": "", "post_date": "2019-10-22T19:16:07.693000", "content": "Overfitting is Allways a big issue here on Kaggle.
", "votes": 2, "replies": [] }, { "id": 658217, "author_name": "", "author_url": "", "post_date": "2019-10-25T19:43:24.877000", "content": "hmmm i guess because of the split, the varying between the public and private lb sometimes scares me.
", "votes": 0, "replies": [] }, { "id": 656079, "author_name": "Hakan", "author_url": "", "post_date": "2019-10-23T22:10:40.473000", "content": "@khairulislam thank you for your post! Are you training on whole train dataset? I get a score of 1.46 with just 5% of the train dataset
", "votes": 0, "replies": [ { "id": 656246, "author_name": "Md Khairul Islam", "author_url": "", "post_date": "2019-10-24T03:42:37.550000", "content": "Ah yes , I am training on whole train dataset.
\n\n\n", "votes": 1, "replies": [] } ] }, { "id": 655458, "author_name": "", "author_url": "", "post_date": "2019-10-23T04:29:13.863000", "content": "", "votes": 0, "replies": [] }, { "id": 656057, "author_name": "Agastya Kommanamanchi", "author_url": "", "post_date": "2019-10-23T21:24:18.677000", "content": "I get a score of 1.46\n That's interesting. I haven't seen anything like that so far 😲
\n
Nice work! Thank you.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "654919": "So, i was training my model with large number of estimators (around 1000-1500) as that could make the eval score on validation set near 0.70 . But that submissions were getting score around 1.3-1.4 on public leaderboard. Then i decided not to train the model too much (around 500 n_estimators and learning rate 0.1 ) and also made some features categorical , which made sense . But when i submitted it, the public score came down to 1.12 with a boom ! I didn't expect that much overfitting in my model, as the validation split could still be decreased a lot.", "655240": "Your validation split is leaking information - so longer epochs the leak lets the eval score get better, but the model is over fitting and would have early stopped with a not leaking validation set. \n\nI think the solution is to change the split - not reduce epochs.", "655901": "@khairulislam is your score 1.12 a single model score?", "655714": "Thank for your comment, I was using too many estimators around 5000 getting an score of 1.30, I will try to train with less estimators, how was your eval score with 500 estimators?\n\nThanks for sharing Khairul.", "655282": "Interesting! What was your val score after the changes? How did you select your validation set, train_test_split?\n\nI personally tried TimeseriesSplit and StratifiedKFold on building IDs keeping the remaining settings the same. \nTill now, I got better results with the stratification, so I guess it's really important to maintain the same buildings in training and validation sets.", "655221": "The problem is that your validation set might have a very different distribution than the test set - so your models behave differently.", "655184": "Overfitting is Allways a big issue here on Kaggle.", "658217": "hmmm i guess because of the split, the varying between the public and private lb sometimes scares me.", "656079": "@khairulislam thank you for your post! Are you training on whole train dataset? I get a score of 1.46 with just 5% of the train dataset", "655458": "", "656057": "Nice work! Thank you." } }