I'm using LGB and have log like this during training.\n\nTraining until validation scores don't improve for 200 rounds.\n[500] training's l1: 0.849387 valid_1's l1: 0.896046\n[1000] training's l1: 0.732159 valid_1's l1: 0.810691\n[1500] training's l1: 0.663674 valid_1's l1: 0.768826\n[2000] training's l1: 0.613874 valid_1's l1: 0.742451\n\nThe mae loss on training set is smaller than valid set after 1000 iterations.\nDoes that mean the model is overfit when loss on train set smaller than valid set?
But as the iteration grows, the mae of valid set is still falling , which indicates the model at 1000 iteration is still underfitting?
\n\nI'm a little confused here....overfit or underfit.\nIf underfit why mae of train set much better than valid set? If overfit why mae of valid set keep falling?\nThis is a noob question....It would be nice if someone help me out.....
", "messages": [ { "id": "566329", "postDate": "07/02/2019 04:07:23", "content": "I'm using LGB and have log like this during training.\n\nTraining until validation scores don't improve for 200 rounds.\n[500] training's l1: 0.849387 valid_1's l1: 0.896046\n[1000] training's l1: 0.732159 valid_1's l1: 0.810691\n[1500] training's l1: 0.663674 valid_1's l1: 0.768826\n[2000] training's l1: 0.613874 valid_1's l1: 0.742451\n\nThe mae loss on training set is smaller than valid set after 1000 iterations.\nDoes that mean the model is overfit when loss on train set smaller than valid set?
But as the iteration grows, the mae of valid set is still falling , which indicates the model at 1000 iteration is still underfitting?
\n\nI'm a little confused here....overfit or underfit.\nIf underfit why mae of train set much better than valid set? If overfit why mae of valid set keep falling?\nThis is a noob question....It would be nice if someone help me out.....
", "rawMarkdown": "I'm using LGB and have log like this during training.\n```\nTraining until validation scores don't improve for 200 rounds.\n[500]\ttraining's l1: 0.849387\tvalid_1's l1: 0.896046\n[1000]\ttraining's l1: 0.732159\tvalid_1's l1: 0.810691\n[1500]\ttraining's l1: 0.663674\tvalid_1's l1: 0.768826\n[2000]\ttraining's l1: 0.613874\tvalid_1's l1: 0.742451\n```\nThe mae loss on training set is smaller than valid set after 1000 iterations.\nDoes that mean the model is overfit when loss on train set smaller than valid set?\n\nBut as the iteration grows, the mae of valid set is still falling , which indicates the model at 1000 iteration is still underfitting?\n\nI'm a little confused here....overfit or underfit.\nIf underfit why mae of train set much better than valid set? If overfit why mae of valid set keep falling?\nThis is a noob question....It would be nice if someone help me out.....", "votes": null }, { "id": "566404", "postDate": "07/02/2019 06:19:16", "content": "I would say, that what you observe, is totally normal behaviour. The model is just still learning, that is why both errors are decreasing. Train error is usually better than validation error because your model is not learning on validation samples, it is learning on training samples, so it makes sense, that it can predict better for them. If you would be overfitting, then your training error would be decreasing and your validation error increasing. However, lightGBM has a mechanism to stop when it is overfitting and validation error does not improve for X rounds. Actually, in this case, I would increase the number of rounds, because, from this, you don't know how low in error you can get with your model.
", "rawMarkdown": "I would say, that what you observe, is totally normal behaviour. The model is just still learning, that is why both errors are decreasing. Train error is usually better than validation error because your model is not learning on validation samples, it is learning on training samples, so it makes sense, that it can predict better for them. If you would be overfitting, then your training error would be decreasing and your validation error increasing. However, lightGBM has a mechanism to stop when it is overfitting and validation error does not improve for X rounds. Actually, in this case, I would increase the number of rounds, because, from this, you don't know how low in error you can get with your model.", "votes": null }, { "id": "566411", "postDate": "07/02/2019 06:25:31", "content": "Thanks a lot! Your explaination really help!🙏
", "rawMarkdown": "Thanks a lot! Your explaination really help!🙏", "votes": null } ], "comments": [ { "id": 566404, "author_name": "somnambwl", "author_url": "", "post_date": "07/02/2019 06:19:16", "content": "I would say, that what you observe, is totally normal behaviour. The model is just still learning, that is why both errors are decreasing. Train error is usually better than validation error because your model is not learning on validation samples, it is learning on training samples, so it makes sense, that it can predict better for them. If you would be overfitting, then your training error would be decreasing and your validation error increasing. However, lightGBM has a mechanism to stop when it is overfitting and validation error does not improve for X rounds. Actually, in this case, I would increase the number of rounds, because, from this, you don't know how low in error you can get with your model.
", "votes": null, "replies": [ { "id": 566411, "author_name": "littlebeandog2", "author_url": "", "post_date": "07/02/2019 06:25:31", "content": "Thanks a lot! Your explaination really help!🙏
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "566329": "I'm using LGB and have log like this during training.\n```\nTraining until validation scores don't improve for 200 rounds.\n[500]\ttraining's l1: 0.849387\tvalid_1's l1: 0.896046\n[1000]\ttraining's l1: 0.732159\tvalid_1's l1: 0.810691\n[1500]\ttraining's l1: 0.663674\tvalid_1's l1: 0.768826\n[2000]\ttraining's l1: 0.613874\tvalid_1's l1: 0.742451\n```\nThe mae loss on training set is smaller than valid set after 1000 iterations.\nDoes that mean the model is overfit when loss on train set smaller than valid set?\n\nBut as the iteration grows, the mae of valid set is still falling , which indicates the model at 1000 iteration is still underfitting?\n\nI'm a little confused here....overfit or underfit.\nIf underfit why mae of train set much better than valid set? If overfit why mae of valid set keep falling?\nThis is a noob question....It would be nice if someone help me out.....", "566404": "I would say, that what you observe, is totally normal behaviour. The model is just still learning, that is why both errors are decreasing. Train error is usually better than validation error because your model is not learning on validation samples, it is learning on training samples, so it makes sense, that it can predict better for them. If you would be overfitting, then your training error would be decreasing and your validation error increasing. However, lightGBM has a mechanism to stop when it is overfitting and validation error does not improve for X rounds. Actually, in this case, I would increase the number of rounds, because, from this, you don't know how low in error you can get with your model.", "566411": "Thanks a lot! Your explaination really help!🙏" }, "source": "meta" }