My： local cv 0.4 LB 1.4 five kfolds.
\n\nCan't find a good validation yet.
", "messages": [ { "id": "658451", "postDate": "10/26/2019 02:50:35", "content": "My： local cv 0.4 LB 1.4 five kfolds.
\n\nCan't find a good validation yet.
", "rawMarkdown": "My： local cv 0.4 LB 1.4 five kfolds.\n\n\nCan't find a good validation yet.", "votes": null }, { "id": "658498", "postDate": "10/26/2019 04:21:30", "content": "\nMean RMSLE 1.234567876376464 [STD:0.13510706991544344]\nOOF RMSLE 1.241938710936183\n
LightGBM model with 5kfold
", "rawMarkdown": "```\nMean RMSLE 1.234567876376464 [STD:0.13510706991544344]\nOOF RMSLE 1.241938710936183\n```\n\nLightGBM model with 5kfold", "votes": null }, { "id": "658578", "postDate": "10/26/2019 07:47:01", "content": "I get everything from 0.7 to 1.1 in CV but with no relationship to LB. Kind of stuck at the moment!
", "rawMarkdown": "I get everything from 0.7 to 1.1 in CV but with no relationship to LB. Kind of stuck at the moment!", "votes": null }, { "id": "658910", "postDate": "10/26/2019 18:12:08", "content": "LightGBM model with 5kfold. Local CV 0.88, LB 1.12
", "rawMarkdown": "LightGBM model with 5kfold. Local CV 0.88, LB 1.12", "votes": null }, { "id": "659031", "postDate": "10/27/2019 00:49:09", "content": "Can't find it right now but discussion topic or kernel that teaches that there is no real good way to build folds that does not contain leaks. When I come across it again with edit with the link.
\n\nEach method I used to build folds gave me a different local CV for my base data set. Not normally a big issue for me when this happens but the 2 submissions per day is making it a real pain and confusing the heck of out me.
", "rawMarkdown": "Can't find it right now but discussion topic or kernel that teaches that there is no real good way to build folds that does not contain leaks. When I come across it again with edit with the link.\n\nEach method I used to build folds gave me a different local CV for my base data set. Not normally a big issue for me when this happens but the 2 submissions per day is making it a real pain and confusing the heck of out me.", "votes": null }, { "id": "659221", "postDate": "10/27/2019 09:47:03", "content": "\nMean RMSLE 1.0343172484146832 [STD:0.04812219988395794]\nOOF RMSLE 1.0354360992499807\n\nLightGBM with 5 kfold again, lb score 1.1
Sometimes my CV is higher than 0.9 but my LB score is better.
", "rawMarkdown": "Sometimes my CV is higher than 0.9 but my LB score is better.", "votes": null }, { "id": "660426", "postDate": "10/29/2019 05:39:35", "content": "what's your cv stratigy, my local cv got 0.4 but lb 1.3, seems the two have no relationship
", "rawMarkdown": "what's your cv stratigy, my local cv got 0.4 but lb 1.3, seems the two have no relationship", "votes": null }, { "id": "661162", "postDate": "10/30/2019 00:56:18", "content": "Validation: 1.0111\nLB: 1.23
\n\nWhat's your validation technique?
", "rawMarkdown": "Validation: 1.0111\nLB: 1.23\n\nWhat's your validation technique?", "votes": null }, { "id": "661413", "postDate": "10/30/2019 09:09:22", "content": "@gunesevitan How many features do you train for the lgbm 5kfold model?
", "rawMarkdown": "gunesevitan How many features do you train for the lgbm 5kfold model?", "votes": null }, { "id": "661421", "postDate": "10/30/2019 09:24:59", "content": "@dawnnnn 5 regular kfold\n@marquis08 20 features
", "rawMarkdown": "dawnnnn 5 regular kfold\n@marquis08 20 features", "votes": null }, { "id": "661529", "postDate": "10/30/2019 12:38:10", "content": "@khairulislam , how many features do you train for the LightGBM 5kfold model ?
", "rawMarkdown": "khairulislam , how many features do you train for the LightGBM 5kfold model ?", "votes": null }, { "id": "661550", "postDate": "10/30/2019 13:02:31", "content": "@caesarlupum 16. I dropped 'wind_direction', 'sea_level_pressure'
", "rawMarkdown": "caesarlupum 16. I dropped 'wind_direction', 'sea_level_pressure'", "votes": null }, { "id": "662153", "postDate": "10/31/2019 06:29:05", "content": "It's not useful to crosswise compare different CV strategy in this competition.
", "rawMarkdown": "It's not useful to crosswise compare different CV strategy in this competition.", "votes": null }, { "id": "662323", "postDate": "10/31/2019 12:30:24", "content": "Thanks for response @khairulislam ! Have a good day!
", "rawMarkdown": "Thanks for response @khairulislam ! Have a good day!", "votes": null }, { "id": "662510", "postDate": "10/31/2019 16:25:38", "content": "\"Each method I used to build folds gave me a different local CV for my base data set. Not normally a big issue for me when this happens but the 2 submissions per day is making it a real pain and confusing the heck of out me.\"
\n\nI pointed that point too some days ago...
", "rawMarkdown": "\"*Each method I used to build folds gave me a different local CV for my base data set. Not normally a big issue for me when this happens but the **2 submissions per day** is making it a real pain and confusing the heck of out me.*\"\n\nI pointed that point too some days ago...", "votes": null }, { "id": "662541", "postDate": "10/31/2019 17:05:16", "content": "CV 0.84/1.133\nLB 1.10
\n\nModel - lgbm - 5 folds stratified by building and month
\n\nCV score 0.84 comes from training (artificial stops).\nCV 1.133 comes from Features validation 5/2/5 train/skip/test (natural early stoppings)
", "rawMarkdown": "CV 0.84/1.133\nLB 1.10\n\nModel - lgbm - 5 folds stratified by building and month\n\nCV score 0.84 comes from training (artificial stops).\nCV 1.133 comes from Features validation 5/2/5 train/skip/test (natural early stoppings)", "votes": null }, { "id": "662774", "postDate": "11/01/2019 01:34:43", "content": "@daishu Really? I would have thought that this competition is perfect for cross-comparing CV strategies. Because the LB test set has so much data, it's super reliable, so a good strategy is to make lots of submissions and rely on their scores.
\n\nBut this comp only lets you make a few submissions, so instead you need to create a local CV strategy that is highly correlated with the LB. And to make a reliable CV strategy you need to compare lots of different local CV strategies together (I would imagine).
\n\nAm I missing something?
", "rawMarkdown": "daishu Really? I would have thought that this competition is *perfect* for cross-comparing CV strategies. Because the LB test set has so much data, it's super reliable, so a good strategy is to make lots of submissions and rely on their scores.\n\nBut this comp only lets you make a few submissions, so instead you need to create a local CV strategy that is highly correlated with the LB. And to make a reliable CV strategy you need to compare lots of different local CV strategies together (I would imagine).\n\nAm I missing something?", "votes": null }, { "id": "662801", "postDate": "11/01/2019 02:34:38", "content": "> you need to create a local CV strategy that is highly correlated with the LB
\n\nYou are right. But I think the key is how to select feature and split dataset not compare different cv strategy.
\n\nOf course, it's just my opinion. The best way is to test and verify your thoughts, then think about why it can(not) work.
", "rawMarkdown": "> you need to create a local CV strategy that is highly correlated with the LB\n\nYou are right. But I think the key is how to select feature and split dataset not compare different cv strategy. \n\nOf course, it's just my opinion. The best way is to test and verify your thoughts, then think about why it can(not) work.", "votes": null } ], "comments": [ { "id": 658498, "author_name": "gunesevitan", "author_url": "", "post_date": "10/26/2019 04:21:30", "content": "\nMean RMSLE 1.234567876376464 [STD:0.13510706991544344]\nOOF RMSLE 1.241938710936183\n
LightGBM model with 5kfold
", "votes": null, "replies": [ { "id": 659221, "author_name": "gunesevitan", "author_url": "", "post_date": "10/27/2019 09:47:03", "content": "\nMean RMSLE 1.0343172484146832 [STD:0.04812219988395794]\nOOF RMSLE 1.0354360992499807\n\nLightGBM with 5 kfold again, lb score 1.1
what's your cv stratigy, my local cv got 0.4 but lb 1.3, seems the two have no relationship
", "votes": null, "replies": [] }, { "id": 661413, "author_name": "marquis08", "author_url": "", "post_date": "10/30/2019 09:09:22", "content": "@gunesevitan How many features do you train for the lgbm 5kfold model?
", "votes": null, "replies": [] }, { "id": 661421, "author_name": "gunesevitan", "author_url": "", "post_date": "10/30/2019 09:24:59", "content": "@dawnnnn 5 regular kfold\n@marquis08 20 features
", "votes": null, "replies": [] } ] }, { "id": 658578, "author_name": "frjo0083", "author_url": "", "post_date": "10/26/2019 07:47:01", "content": "I get everything from 0.7 to 1.1 in CV but with no relationship to LB. Kind of stuck at the moment!
", "votes": null, "replies": [] }, { "id": 658910, "author_name": "khairulislam", "author_url": "", "post_date": "10/26/2019 18:12:08", "content": "LightGBM model with 5kfold. Local CV 0.88, LB 1.12
", "votes": null, "replies": [ { "id": 661529, "author_name": "caesarlupum", "author_url": "", "post_date": "10/30/2019 12:38:10", "content": "@khairulislam , how many features do you train for the LightGBM 5kfold model ?
", "votes": null, "replies": [] }, { "id": 661550, "author_name": "khairulislam", "author_url": "", "post_date": "10/30/2019 13:02:31", "content": "@caesarlupum 16. I dropped 'wind_direction', 'sea_level_pressure'
", "votes": null, "replies": [] }, { "id": 662323, "author_name": "caesarlupum", "author_url": "", "post_date": "10/31/2019 12:30:24", "content": "Thanks for response @khairulislam ! Have a good day!
", "votes": null, "replies": [] } ] }, { "id": 659031, "author_name": "pcjimmmy", "author_url": "", "post_date": "10/27/2019 00:49:09", "content": "Can't find it right now but discussion topic or kernel that teaches that there is no real good way to build folds that does not contain leaks. When I come across it again with edit with the link.
\n\nEach method I used to build folds gave me a different local CV for my base data set. Not normally a big issue for me when this happens but the 2 submissions per day is making it a real pain and confusing the heck of out me.
", "votes": null, "replies": [ { "id": 662510, "author_name": "pfichou", "author_url": "", "post_date": "10/31/2019 16:25:38", "content": "\"Each method I used to build folds gave me a different local CV for my base data set. Not normally a big issue for me when this happens but the 2 submissions per day is making it a real pain and confusing the heck of out me.\"
\n\nI pointed that point too some days ago...
", "votes": null, "replies": [] } ] }, { "id": 659384, "author_name": "klloke", "author_url": "", "post_date": "10/27/2019 15:00:27", "content": "Sometimes my CV is higher than 0.9 but my LB score is better.
", "votes": null, "replies": [] }, { "id": 661162, "author_name": "caesarlupum", "author_url": "", "post_date": "10/30/2019 00:56:18", "content": "Validation: 1.0111\nLB: 1.23
\n\nWhat's your validation technique?
", "votes": null, "replies": [] }, { "id": 662153, "author_name": "daishu", "author_url": "", "post_date": "10/31/2019 06:29:05", "content": "It's not useful to crosswise compare different CV strategy in this competition.
", "votes": null, "replies": [ { "id": 662774, "author_name": "lewington", "author_url": "", "post_date": "11/01/2019 01:34:43", "content": "@daishu Really? I would have thought that this competition is perfect for cross-comparing CV strategies. Because the LB test set has so much data, it's super reliable, so a good strategy is to make lots of submissions and rely on their scores.
\n\nBut this comp only lets you make a few submissions, so instead you need to create a local CV strategy that is highly correlated with the LB. And to make a reliable CV strategy you need to compare lots of different local CV strategies together (I would imagine).
\n\nAm I missing something?
", "votes": null, "replies": [] }, { "id": 662801, "author_name": "daishu", "author_url": "", "post_date": "11/01/2019 02:34:38", "content": "> you need to create a local CV strategy that is highly correlated with the LB
\n\nYou are right. But I think the key is how to select feature and split dataset not compare different cv strategy.
\n\nOf course, it's just my opinion. The best way is to test and verify your thoughts, then think about why it can(not) work.
", "votes": null, "replies": [] } ] }, { "id": 662541, "author_name": "kyakovlev", "author_url": "", "post_date": "10/31/2019 17:05:16", "content": "CV 0.84/1.133\nLB 1.10
\n\nModel - lgbm - 5 folds stratified by building and month
\n\nCV score 0.84 comes from training (artificial stops).\nCV 1.133 comes from Features validation 5/2/5 train/skip/test (natural early stoppings)
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "658451": "My： local cv 0.4 LB 1.4 five kfolds.\n\n\nCan't find a good validation yet.", "658498": "```\nMean RMSLE 1.234567876376464 [STD:0.13510706991544344]\nOOF RMSLE 1.241938710936183\n```\n\nLightGBM model with 5kfold", "658578": "I get everything from 0.7 to 1.1 in CV but with no relationship to LB. Kind of stuck at the moment!", "658910": "LightGBM model with 5kfold. Local CV 0.88, LB 1.12", "659031": "Can't find it right now but discussion topic or kernel that teaches that there is no real good way to build folds that does not contain leaks. When I come across it again with edit with the link.\n\nEach method I used to build folds gave me a different local CV for my base data set. Not normally a big issue for me when this happens but the 2 submissions per day is making it a real pain and confusing the heck of out me.", "659221": "```\nMean RMSLE 1.0343172484146832 [STD:0.04812219988395794]\nOOF RMSLE 1.0354360992499807\n```\nLightGBM with 5 kfold again, lb score `1.1`", "659384": "Sometimes my CV is higher than 0.9 but my LB score is better.", "660426": "what's your cv stratigy, my local cv got 0.4 but lb 1.3, seems the two have no relationship", "661162": "Validation: 1.0111\nLB: 1.23\n\nWhat's your validation technique?", "661413": "gunesevitan How many features do you train for the lgbm 5kfold model?", "661421": "dawnnnn 5 regular kfold\n@marquis08 20 features", "661529": "khairulislam , how many features do you train for the LightGBM 5kfold model ?", "661550": "caesarlupum 16. I dropped 'wind_direction', 'sea_level_pressure'", "662153": "It's not useful to crosswise compare different CV strategy in this competition.", "662323": "Thanks for response @khairulislam ! Have a good day!", "662510": "\"*Each method I used to build folds gave me a different local CV for my base data set. Not normally a big issue for me when this happens but the **2 submissions per day** is making it a real pain and confusing the heck of out me.*\"\n\nI pointed that point too some days ago...", "662541": "CV 0.84/1.133\nLB 1.10\n\nModel - lgbm - 5 folds stratified by building and month\n\nCV score 0.84 comes from training (artificial stops).\nCV 1.133 comes from Features validation 5/2/5 train/skip/test (natural early stoppings)", "662774": "daishu Really? I would have thought that this competition is *perfect* for cross-comparing CV strategies. Because the LB test set has so much data, it's super reliable, so a good strategy is to make lots of submissions and rely on their scores.\n\nBut this comp only lets you make a few submissions, so instead you need to create a local CV strategy that is highly correlated with the LB. And to make a reliable CV strategy you need to compare lots of different local CV strategies together (I would imagine).\n\nAm I missing something?", "662801": "> you need to create a local CV strategy that is highly correlated with the LB\n\nYou are right. But I think the key is how to select feature and split dataset not compare different cv strategy. \n\nOf course, it's just my opinion. The best way is to test and verify your thoughts, then think about why it can(not) work." }, "source": "meta" }