![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2652652%2Ff5095b21f6ca0a215dd909c595226a2a%2FScreenshot%20from%202019-08-05%2017-22-05.png?generation=1565018712966033&alt=media\")
Hi!
\n\nLooking for inspiration on what to try next, I saw that a lot of kernels were using LightGBM, so I decided to give it a try. I would like to know how you guys deal with overfitting. Specifically, how to detect that it's happening.
\n\nFor the time being, while I try different things out, I have the training data split into a training and a validation set. I thought that keeping an eye on the learning curves would give me a good feeling for the onset of overfitting. However, I get curves like this:
\n\n
...where the validation and training errors basically go hand in hand the whole time. My gut feeling is that after ~200 iterations, the model should start overfitting, but I don't see the validation error increasing. What do you think about this? What other methods do you use to detect overfitting?
", "messages": [ { "id": 592630, "postDate": "2019-08-05T15:33:18.937Z", "content": "Hi!
\n\nLooking for inspiration on what to try next, I saw that a lot of kernels were using LightGBM, so I decided to give it a try. I would like to know how you guys deal with overfitting. Specifically, how to detect that it's happening.
\n\nFor the time being, while I try different things out, I have the training data split into a training and a validation set. I thought that keeping an eye on the learning curves would give me a good feeling for the onset of overfitting. However, I get curves like this:
\n\n
...where the validation and training errors basically go hand in hand the whole time. My gut feeling is that after ~200 iterations, the model should start overfitting, but I don't see the validation error increasing. What do you think about this? What other methods do you use to detect overfitting?
", "rawMarkdown": "Hi!\n\nLooking for inspiration on what to try next, I saw that a lot of kernels were using LightGBM, so I decided to give it a try. I would like to know how you guys deal with overfitting. Specifically, how to detect that it's happening.\n\nFor the time being, while I try different things out, I have the training data split into a training and a validation set. I thought that keeping an eye on the learning curves would give me a good feeling for the onset of overfitting. However, I get curves like this:\n\n\n\n...where the validation and training errors basically go hand in hand the whole time. My gut feeling is that after ~200 iterations, the model should start overfitting, but I don't see the validation error increasing. What do you think about this? What other methods do you use to detect overfitting?", "votes": 4 }, { "id": 592764, "postDate": "2019-08-05T19:39:21.833Z", "content": "If the validation error is still decreasing, then there is an argument to say that you are not overfitting.
", "rawMarkdown": "If the validation error is still decreasing, then there is an argument to say that you are not overfitting.", "votes": 1, "replies": [ { "id": 593105, "postDate": "2019-08-06T07:33:00.197Z", "content": "Yes, that's exactly what puzzles me! By that reasoning, the training is just not finished. But it's been thousands of iterations for very little improvement, compared to the huge drop in the first ~100. That's why I suspect overfitting rather than actual learning. Also, the submissions aren't anywhere near the score predicted by the validation results.
", "rawMarkdown": "Yes, that's exactly what puzzles me! By that reasoning, the training is just not finished. But it's been thousands of iterations for very little improvement, compared to the huge drop in the first ~100. That's why I suspect overfitting rather than actual learning. Also, the submissions aren't anywhere near the score predicted by the validation results." } ] }, { "id": 592645, "postDate": "2019-08-05T15:57:19.237Z", "content": "I am a newb so take it with some caution, but this looks like you are underfitting? try to increase your max_depth or num_leaves?
", "rawMarkdown": "I am a newb so take it with some caution, but this looks like you are underfitting? try to increase your max_depth or num_leaves?", "votes": 1, "replies": [ { "id": 593102, "postDate": "2019-08-06T07:26:59.743Z", "content": "Thanks, @proletheus! I'm also quite new at this, so I will double-check everything (: Why do you think it is underfitting? I would expect the MAE to remain high (> 1) in this case.
\n\nI tried your suggestion, to make sure I wasn't overlooking something. It doesn't seem to help (the score improves a little, but I think the overfitting is just even stronger). Also, during training LightGBM produces several warnings that there are no further splits with positive gain, so I guess that's as precise as it can get.
", "rawMarkdown": "Thanks, @proletheus! I'm also quite new at this, so I will double-check everything (: Why do you think it is underfitting? I would expect the MAE to remain high (> 1) in this case. \n\nI tried your suggestion, to make sure I wasn't overlooking something. It doesn't seem to help (the score improves a little, but I think the overfitting is just even stronger). Also, during training LightGBM produces several warnings that there are no further splits with positive gain, so I guess that's as precise as it can get.", "votes": 1 }, { "id": 593197, "postDate": "2019-08-06T09:45:33.940Z", "content": "but your training and validation error is pretty much the same? I assume overfitting means building a model that does too well on train data but fails to generalize on test/validation data. for yours its the opposite, it performs the same on train/test, which might indicate it is not flexible enough to capture any meaningful relationships?
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3353913%2F84274976ceabab8210f25912b39cdfbf%2FScreenshot_2.png?generation=1565084709900227&alt=media\")
i stole this from https://www.youtube.com/watch?v=VZuKBKd4ck4.
", "rawMarkdown": "but your training and validation error is pretty much the same? I assume overfitting means building a model that does too well on train data but fails to generalize on test/validation data. for yours its the opposite, it performs the same on train/test, which might indicate it is not flexible enough to capture any meaningful relationships?\n\n\n\n\ni stole this from https://www.youtube.com/watch?v=VZuKBKd4ck4. ", "votes": 1 }, { "id": 593535, "postDate": "2019-08-06T18:31:05.993Z", "content": "I thought about it and I think that you are absolutely right. Like I commented here, I was distracted by the very long plateau without error improvement. Thanks!
", "rawMarkdown": "I thought about it and I think that you are absolutely right. Like I commented [here](https://www.kaggle.com/c/champs-scalar-coupling/discussion/102856#593531), I was distracted by the very long plateau without error improvement. Thanks!", "votes": 1 } ] }, { "id": 593270, "postDate": "2019-08-06T11:48:01.767Z", "content": "\"My gut feeling is that after ~200 iterations, the model should start overfitting, but I don't see the validation error increasing.\"
\n\nWhat is this feeling based on? I see no reason to believe that the model should start overfitting after 200 trees. It's common to fit GBMs with way more trees than this.
", "rawMarkdown": "\"My gut feeling is that after ~200 iterations, the model should start overfitting, but I don't see the validation error increasing.\" \n\nWhat is this feeling based on? I see no reason to believe that the model should start overfitting after 200 trees. It's common to fit GBMs with *way* more trees than this.\n\n", "votes": 2, "replies": [ { "id": 593522, "postDate": "2019-08-06T18:12:32.643Z", "content": "Good point. Two main reasons:\n- The sudden change in the learning speed. There is a lot happening in the first 200 iterations, and the next thousands only improve the error marginally.\n- The fact that the LB score is significantly worse than the validation score.
\n\nHowever, reading all the comments is making me change my mind about that interpretation...
", "rawMarkdown": "Good point. Two main reasons:\n- The sudden change in the learning speed. There is a lot happening in the first 200 iterations, and the next thousands only improve the error marginally.\n- The fact that the LB score is significantly worse than the validation score.\n\nHowever, reading all the comments is making me change my mind about that interpretation...", "votes": 1 }, { "id": 593545, "postDate": "2019-08-06T19:06:32.177Z", "content": "LB worse than validation score may be a different issue. Are you using cross-validation? If so, are you using GroupKFold by to ensure that rows for the same molecule all up in the same fold?
", "rawMarkdown": "LB worse than validation score may be a different issue. Are you using cross-validation? If so, are you using GroupKFold by to ensure that rows for the same molecule all up in the same fold?" }, { "id": 593577, "postDate": "2019-08-06T20:33:16.453Z", "content": "Aha, interesting! That makes sense, that probably is another issue. I'm not doing CV at the moment: I wanted to start with a simple train/test split to get a first feeling of how the model behaves. But thanks for mentioning it, I'll keep that in mind if I compare the results in the future!
", "rawMarkdown": "Aha, interesting! That makes sense, that probably is another issue. I'm not doing CV at the moment: I wanted to start with a simple train/test split to get a first feeling of how the model behaves. But thanks for mentioning it, I'll keep that in mind if I compare the results in the future!", "votes": 1 } ] }, { "id": 593330, "postDate": "2019-08-06T13:09:24.373Z", "content": "The random split also goes a long way in detecting the overfitting.\nBayesian Optimization can help reduce it also
", "rawMarkdown": "The random split also goes a long way in detecting the overfitting.\nBayesian Optimization can help reduce it also", "replies": [ { "id": 593580, "postDate": "2019-08-06T20:34:29.130Z", "content": "Hey thanks for the input! I couldn't quite follow, could you elaborate?
", "rawMarkdown": "Hey thanks for the input! I couldn't quite follow, could you elaborate? " } ] }, { "id": 592876, "postDate": "2019-08-05T23:54:55.710Z", "content": "Hi! @lesamu, I will try to use some cross-validation, just to see how the training/validation performs
", "rawMarkdown": "Hi! @lesamu, I will try to use some cross-validation, just to see how the training/validation performs", "replies": [ { "id": 593108, "postDate": "2019-08-06T07:36:12.410Z", "content": "Thanks, I'll try that.
", "rawMarkdown": "Thanks, I'll try that." } ] }, { "id": 593437, "postDate": "2019-08-06T15:48:29.280Z", "rawMarkdown": "", "votes": 1, "isDeleted": true, "replies": [ { "id": 593531, "postDate": "2019-08-06T18:27:47.273Z", "content": "Many thanks for the suggestion! That gives me some insight into what could be \"expected\". Reading the previous comments I decided to give it a try at 20.000 rounds / early stopping after 200. Most groups haven't triggered the early stopping by then! The graphs look like this:
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2652652%2F7856360c173044d337aee7f222bea371%2FScreenshot%20from%202019-08-06%2020-15-03.png?generation=1565115376399356&alt=media\")
Reading the comments, I am more convinced now that this is rather a case of underfitting. Funny! I was just blown away by sudden change in learning speed and the incredibly long plateau with almost no error improvement. Also the fact that the score on the submissions was lower than the validation score made me suspicious.
\n\nIt's nice that you and the others took some time to share their ideas. In the beginning, these comments are super helpful to understand what is going on, it can be quite confusing at times.
", "rawMarkdown": "Many thanks for the suggestion! That gives me some insight into what could be \"expected\". Reading the previous comments I decided to give it a try at 20.000 rounds / early stopping after 200. Most groups haven't triggered the early stopping by then! The graphs look like this:\n\n\n\nReading the comments, I am more convinced now that this is rather a case of underfitting. Funny! I was just blown away by sudden change in learning speed and the incredibly long plateau with almost no error improvement. Also the fact that the score on the submissions was lower than the validation score made me suspicious.\n\nIt's nice that you and the others took some time to share their ideas. In the beginning, these comments are super helpful to understand what is going on, it can be quite confusing at times.", "votes": 2 } ] } ], "comments": [ { "id": 592764, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2019-08-05T19:39:21.833000", "content": "If the validation error is still decreasing, then there is an argument to say that you are not overfitting.
", "votes": 1, "replies": [ { "id": 593105, "author_name": "Samuel", "author_url": "", "post_date": "2019-08-06T07:33:00.197000", "content": "Yes, that's exactly what puzzles me! By that reasoning, the training is just not finished. But it's been thousands of iterations for very little improvement, compared to the huge drop in the first ~100. That's why I suspect overfitting rather than actual learning. Also, the submissions aren't anywhere near the score predicted by the validation results.
", "votes": 0, "replies": [] } ] }, { "id": 592645, "author_name": "Proletheus", "author_url": "", "post_date": "2019-08-05T15:57:19.237000", "content": "I am a newb so take it with some caution, but this looks like you are underfitting? try to increase your max_depth or num_leaves?
", "votes": 1, "replies": [ { "id": 593102, "author_name": "Samuel", "author_url": "", "post_date": "2019-08-06T07:26:59.743000", "content": "Thanks, @proletheus! I'm also quite new at this, so I will double-check everything (: Why do you think it is underfitting? I would expect the MAE to remain high (> 1) in this case.
\n\nI tried your suggestion, to make sure I wasn't overlooking something. It doesn't seem to help (the score improves a little, but I think the overfitting is just even stronger). Also, during training LightGBM produces several warnings that there are no further splits with positive gain, so I guess that's as precise as it can get.
", "votes": 1, "replies": [] }, { "id": 593197, "author_name": "Proletheus", "author_url": "", "post_date": "2019-08-06T09:45:33.940000", "content": "but your training and validation error is pretty much the same? I assume overfitting means building a model that does too well on train data but fails to generalize on test/validation data. for yours its the opposite, it performs the same on train/test, which might indicate it is not flexible enough to capture any meaningful relationships?
\n\n
i stole this from https://www.youtube.com/watch?v=VZuKBKd4ck4.
", "votes": 1, "replies": [] }, { "id": 593535, "author_name": "Samuel", "author_url": "", "post_date": "2019-08-06T18:31:05.993000", "content": "I thought about it and I think that you are absolutely right. Like I commented here, I was distracted by the very long plateau without error improvement. Thanks!
", "votes": 1, "replies": [] } ] }, { "id": 593270, "author_name": "fafafa fafafafa", "author_url": "", "post_date": "2019-08-06T11:48:01.767000", "content": "\"My gut feeling is that after ~200 iterations, the model should start overfitting, but I don't see the validation error increasing.\"
\n\nWhat is this feeling based on? I see no reason to believe that the model should start overfitting after 200 trees. It's common to fit GBMs with way more trees than this.
", "votes": 2, "replies": [ { "id": 593522, "author_name": "Samuel", "author_url": "", "post_date": "2019-08-06T18:12:32.643000", "content": "Good point. Two main reasons:\n- The sudden change in the learning speed. There is a lot happening in the first 200 iterations, and the next thousands only improve the error marginally.\n- The fact that the LB score is significantly worse than the validation score.
\n\nHowever, reading all the comments is making me change my mind about that interpretation...
", "votes": 1, "replies": [] }, { "id": 593545, "author_name": "fafafa fafafafa", "author_url": "", "post_date": "2019-08-06T19:06:32.177000", "content": "LB worse than validation score may be a different issue. Are you using cross-validation? If so, are you using GroupKFold by to ensure that rows for the same molecule all up in the same fold?
", "votes": 0, "replies": [] }, { "id": 593577, "author_name": "Samuel", "author_url": "", "post_date": "2019-08-06T20:33:16.453000", "content": "Aha, interesting! That makes sense, that probably is another issue. I'm not doing CV at the moment: I wanted to start with a simple train/test split to get a first feeling of how the model behaves. But thanks for mentioning it, I'll keep that in mind if I compare the results in the future!
", "votes": 1, "replies": [] } ] }, { "id": 593330, "author_name": "Raheem Nasirudeen", "author_url": "", "post_date": "2019-08-06T13:09:24.373000", "content": "The random split also goes a long way in detecting the overfitting.\nBayesian Optimization can help reduce it also
", "votes": 0, "replies": [ { "id": 593580, "author_name": "Samuel", "author_url": "", "post_date": "2019-08-06T20:34:29.130000", "content": "Hey thanks for the input! I couldn't quite follow, could you elaborate?
", "votes": 0, "replies": [] } ] }, { "id": 592876, "author_name": "C4rl05/V", "author_url": "", "post_date": "2019-08-05T23:54:55.710000", "content": "Hi! @lesamu, I will try to use some cross-validation, just to see how the training/validation performs
", "votes": 0, "replies": [ { "id": 593108, "author_name": "Samuel", "author_url": "", "post_date": "2019-08-06T07:36:12.410000", "content": "Thanks, I'll try that.
", "votes": 0, "replies": [] } ] }, { "id": 593437, "author_name": "", "author_url": "", "post_date": "2019-08-06T15:48:29.280000", "content": "", "votes": 1, "replies": [ { "id": 593531, "author_name": "Samuel", "author_url": "", "post_date": "2019-08-06T18:27:47.273000", "content": "Many thanks for the suggestion! That gives me some insight into what could be \"expected\". Reading the previous comments I decided to give it a try at 20.000 rounds / early stopping after 200. Most groups haven't triggered the early stopping by then! The graphs look like this:
\n\n
Reading the comments, I am more convinced now that this is rather a case of underfitting. Funny! I was just blown away by sudden change in learning speed and the incredibly long plateau with almost no error improvement. Also the fact that the score on the submissions was lower than the validation score made me suspicious.
\n\nIt's nice that you and the others took some time to share their ideas. In the beginning, these comments are super helpful to understand what is going on, it can be quite confusing at times.
", "votes": 2, "replies": [] } ] } ], "raw_markdown_by_id": { "592630": "Hi!\n\nLooking for inspiration on what to try next, I saw that a lot of kernels were using LightGBM, so I decided to give it a try. I would like to know how you guys deal with overfitting. Specifically, how to detect that it's happening.\n\nFor the time being, while I try different things out, I have the training data split into a training and a validation set. I thought that keeping an eye on the learning curves would give me a good feeling for the onset of overfitting. However, I get curves like this:\n\n\n\n...where the validation and training errors basically go hand in hand the whole time. My gut feeling is that after ~200 iterations, the model should start overfitting, but I don't see the validation error increasing. What do you think about this? What other methods do you use to detect overfitting?", "592764": "If the validation error is still decreasing, then there is an argument to say that you are not overfitting.", "592645": "I am a newb so take it with some caution, but this looks like you are underfitting? try to increase your max_depth or num_leaves?", "593270": "\"My gut feeling is that after ~200 iterations, the model should start overfitting, but I don't see the validation error increasing.\" \n\nWhat is this feeling based on? I see no reason to believe that the model should start overfitting after 200 trees. It's common to fit GBMs with *way* more trees than this.\n\n", "593330": "The random split also goes a long way in detecting the overfitting.\nBayesian Optimization can help reduce it also", "592876": "Hi! @lesamu, I will try to use some cross-validation, just to see how the training/validation performs", "593437": "" } }