I was wondering if anybody had a similar experience: I have decided to focus on feature engineering + single model for now (lgbm with 10-fold cross validation). Variation across folds - much higher than the distance between my current position and the top - is somewhat annoying, but not uncommon. What I find more peculiar is that it feels like almost any feature I create, it improves the score in cv, but makes the LB worse. I have tried all kinds of aggregates, including merchants info - and the effect stays the same => I am unable to beat my original attempt scoring 3.709.
\n\nReally curious about experience others are having.
", "messages": [ { "id": "432265", "postDate": "12/03/2018 16:28:03", "content": "I was wondering if anybody had a similar experience: I have decided to focus on feature engineering + single model for now (lgbm with 10-fold cross validation). Variation across folds - much higher than the distance between my current position and the top - is somewhat annoying, but not uncommon. What I find more peculiar is that it feels like almost any feature I create, it improves the score in cv, but makes the LB worse. I have tried all kinds of aggregates, including merchants info - and the effect stays the same => I am unable to beat my original attempt scoring 3.709.
\n\nReally curious about experience others are having.
", "rawMarkdown": "I was wondering if anybody had a similar experience: I have decided to focus on feature engineering + single model for now (lgbm with 10-fold cross validation). Variation across folds - much higher than the distance between my current position and the top - is somewhat annoying, but not uncommon. What I find more peculiar is that it feels like almost any feature I create, it improves the score in cv, but makes the LB worse. I have tried all kinds of aggregates, including merchants info - and the effect stays the same => I am unable to beat my original attempt scoring 3.709. \n\nReally curious about experience others are having.", "votes": null }, { "id": "432278", "postDate": "12/03/2018 16:53:22", "content": "The same experience ...
", "rawMarkdown": "The same experience ...", "votes": null }, { "id": "432280", "postDate": "12/03/2018 16:55:25", "content": "Same here. I kind of feel a little disappointed now.
", "rawMarkdown": "Same here. I kind of feel a little disappointed now.", "votes": null }, { "id": "432323", "postDate": "12/03/2018 18:12:46", "content": "Same here :/
", "rawMarkdown": "Same here :/", "votes": null }, { "id": "432337", "postDate": "12/03/2018 18:27:29", "content": "Same here. New features improve CV but lower LB. I am tired
", "rawMarkdown": "Same here. New features improve CV but lower LB. I am tired", "votes": null }, { "id": "432358", "postDate": "12/03/2018 18:58:04", "content": "Yeah, I'm about to think about giving up. I just added merchants.csv features in, and no LB improvement.
", "rawMarkdown": "Yeah, I'm about to think about giving up. I just added merchants.csv features in, and no LB improvement.", "votes": null }, { "id": "432359", "postDate": "12/03/2018 19:03:50", "content": "like nailing jelly to the wall
", "rawMarkdown": "like nailing jelly to the wall", "votes": null }, { "id": "432383", "postDate": "12/03/2018 19:40:45", "content": "Same experience here.
", "rawMarkdown": "Same experience here.", "votes": null }, { "id": "432385", "postDate": "12/03/2018 19:42:22", "content": "I'd also add that because of the randomness of the features and the target, it's likely the top of the leaderboard will see a massive shuffle on the private leaderboard when the other 70% of the data is used due to overfitting on lucky seeds.
", "rawMarkdown": "I'd also add that because of the randomness of the features and the target, it's likely the top of the leaderboard will see a massive shuffle on the private leaderboard when the other 70% of the data is used due to overfitting on lucky seeds.", "votes": null }, { "id": "432388", "postDate": "12/03/2018 19:44:07", "content": "Definitely.. people seem to have hit upon a seed of 11 for the lightgbm model and 15 for Kfolds.. but that should hopefully change on the private LB
", "rawMarkdown": "Definitely.. people seem to have hit upon a seed of 11 for the lightgbm model and 15 for Kfolds.. but that should hopefully change on the private LB", "votes": null }, { "id": "432462", "postDate": "12/03/2018 22:29:28", "content": "Yep, I'm also getting stuck in this world... >.< Hopefully trusting my CV is the way to go?
", "rawMarkdown": "Yep, I'm also getting stuck in this world... >.< Hopefully trusting my CV is the way to go?", "votes": null }, { "id": "432479", "postDate": "12/03/2018 23:39:24", "content": "One can only hope.
", "rawMarkdown": "One can only hope.", "votes": null }, { "id": "432584", "postDate": "12/04/2018 04:24:21", "content": "From my experience, when cv improves, lb improves (now cv ~ 3.653, lb ~ 3.699, ~80 features).\nP/S : merchants.csv features didn't help my cv at all.
", "rawMarkdown": "From my experience, when cv improves, lb improves (now cv ~ 3.653, lb ~ 3.699, ~80 features).\nP/S : merchants.csv features didn't help my cv at all.", "votes": null }, { "id": "432907", "postDate": "12/04/2018 13:03:19", "content": "@Chau Ngoc Huynh
\n\nSame on my side... Really strange.
", "rawMarkdown": "Chau Ngoc Huynh\n\nSame on my side... Really strange.", "votes": null }, { "id": "433999", "postDate": "12/05/2018 18:46:50", "content": "For me aggregations definitely help to improve both cv ad lb, but manual features (like purchase_amount / date_diff) improve only local cv. It seems that it's time to speak about reliable cv scheme.
", "rawMarkdown": "For me aggregations definitely help to improve both cv ad lb, but manual features (like purchase\\_amount / date\\_diff) improve only local cv. It seems that it's time to speak about reliable cv scheme.", "votes": null }, { "id": "434004", "postDate": "12/05/2018 18:53:20", "content": "Exactly, I've tried 50 different features already. No manually created features work for me.
", "rawMarkdown": "Exactly, I've tried 50 different features already. No manually created features work for me.", "votes": null }, { "id": "441516", "postDate": "12/18/2018 18:20:52", "content": "There are some high correlations in Merchants Data. Perhaps that explains why the features we think will be important - are not important at all
", "rawMarkdown": "There are some high correlations in Merchants Data. Perhaps that explains why the features we think will be important - are not important at all", "votes": null }, { "id": "441742", "postDate": "12/19/2018 02:05:45", "content": "I just joined df_hist with merchant data then did some agg, but got a worse cv score and lb score.
", "rawMarkdown": "I just joined df_hist with merchant data then did some agg, but got a worse cv score and lb score.", "votes": null } ], "comments": [ { "id": 432278, "author_name": "mileta1976", "author_url": "", "post_date": "12/03/2018 16:53:22", "content": "The same experience ...
", "votes": null, "replies": [] }, { "id": 432280, "author_name": "jinjunding", "author_url": "", "post_date": "12/03/2018 16:55:25", "content": "Same here. I kind of feel a little disappointed now.
", "votes": null, "replies": [] }, { "id": 432323, "author_name": "silverstone1903", "author_url": "", "post_date": "12/03/2018 18:12:46", "content": "Same here :/
", "votes": null, "replies": [] }, { "id": 432337, "author_name": "deffro", "author_url": "", "post_date": "12/03/2018 18:27:29", "content": "Same here. New features improve CV but lower LB. I am tired
", "votes": null, "replies": [] }, { "id": 432358, "author_name": "delayedkarma", "author_url": "", "post_date": "12/03/2018 18:58:04", "content": "Yeah, I'm about to think about giving up. I just added merchants.csv features in, and no LB improvement.
", "votes": null, "replies": [] }, { "id": 432359, "author_name": "franzxschmid", "author_url": "", "post_date": "12/03/2018 19:03:50", "content": "like nailing jelly to the wall
", "votes": null, "replies": [] }, { "id": 432383, "author_name": "hampsonw", "author_url": "", "post_date": "12/03/2018 19:40:45", "content": "Same experience here.
", "votes": null, "replies": [] }, { "id": 432385, "author_name": "hampsonw", "author_url": "", "post_date": "12/03/2018 19:42:22", "content": "I'd also add that because of the randomness of the features and the target, it's likely the top of the leaderboard will see a massive shuffle on the private leaderboard when the other 70% of the data is used due to overfitting on lucky seeds.
", "votes": null, "replies": [ { "id": 432388, "author_name": "delayedkarma", "author_url": "", "post_date": "12/03/2018 19:44:07", "content": "Definitely.. people seem to have hit upon a seed of 11 for the lightgbm model and 15 for Kfolds.. but that should hopefully change on the private LB
", "votes": null, "replies": [] } ] }, { "id": 432462, "author_name": "peterhurford", "author_url": "", "post_date": "12/03/2018 22:29:28", "content": "Yep, I'm also getting stuck in this world... >.< Hopefully trusting my CV is the way to go?
", "votes": null, "replies": [ { "id": 432479, "author_name": "delayedkarma", "author_url": "", "post_date": "12/03/2018 23:39:24", "content": "One can only hope.
", "votes": null, "replies": [] } ] }, { "id": 432584, "author_name": "chauhuynh", "author_url": "", "post_date": "12/04/2018 04:24:21", "content": "From my experience, when cv improves, lb improves (now cv ~ 3.653, lb ~ 3.699, ~80 features).\nP/S : merchants.csv features didn't help my cv at all.
", "votes": null, "replies": [ { "id": 432907, "author_name": "pfichou", "author_url": "", "post_date": "12/04/2018 13:03:19", "content": "@Chau Ngoc Huynh
\n\nSame on my side... Really strange.
", "votes": null, "replies": [] } ] }, { "id": 433999, "author_name": "kailex", "author_url": "", "post_date": "12/05/2018 18:46:50", "content": "For me aggregations definitely help to improve both cv ad lb, but manual features (like purchase_amount / date_diff) improve only local cv. It seems that it's time to speak about reliable cv scheme.
", "votes": null, "replies": [ { "id": 434004, "author_name": "jinjunding", "author_url": "", "post_date": "12/05/2018 18:53:20", "content": "Exactly, I've tried 50 different features already. No manually created features work for me.
", "votes": null, "replies": [] } ] }, { "id": 441516, "author_name": "roydatascience", "author_url": "", "post_date": "12/18/2018 18:20:52", "content": "There are some high correlations in Merchants Data. Perhaps that explains why the features we think will be important - are not important at all
", "votes": null, "replies": [ { "id": 441742, "author_name": "codlife", "author_url": "", "post_date": "12/19/2018 02:05:45", "content": "I just joined df_hist with merchant data then did some agg, but got a worse cv score and lb score.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "432265": "I was wondering if anybody had a similar experience: I have decided to focus on feature engineering + single model for now (lgbm with 10-fold cross validation). Variation across folds - much higher than the distance between my current position and the top - is somewhat annoying, but not uncommon. What I find more peculiar is that it feels like almost any feature I create, it improves the score in cv, but makes the LB worse. I have tried all kinds of aggregates, including merchants info - and the effect stays the same => I am unable to beat my original attempt scoring 3.709. \n\nReally curious about experience others are having.", "432278": "The same experience ...", "432280": "Same here. I kind of feel a little disappointed now.", "432323": "Same here :/", "432337": "Same here. New features improve CV but lower LB. I am tired", "432358": "Yeah, I'm about to think about giving up. I just added merchants.csv features in, and no LB improvement.", "432359": "like nailing jelly to the wall", "432383": "Same experience here.", "432385": "I'd also add that because of the randomness of the features and the target, it's likely the top of the leaderboard will see a massive shuffle on the private leaderboard when the other 70% of the data is used due to overfitting on lucky seeds.", "432388": "Definitely.. people seem to have hit upon a seed of 11 for the lightgbm model and 15 for Kfolds.. but that should hopefully change on the private LB", "432462": "Yep, I'm also getting stuck in this world... >.< Hopefully trusting my CV is the way to go?", "432479": "One can only hope.", "432584": "From my experience, when cv improves, lb improves (now cv ~ 3.653, lb ~ 3.699, ~80 features).\nP/S : merchants.csv features didn't help my cv at all.", "432907": "Chau Ngoc Huynh\n\nSame on my side... Really strange.", "433999": "For me aggregations definitely help to improve both cv ad lb, but manual features (like purchase\\_amount / date\\_diff) improve only local cv. It seems that it's time to speak about reliable cv scheme.", "434004": "Exactly, I've tried 50 different features already. No manually created features work for me.", "441516": "There are some high correlations in Merchants Data. Perhaps that explains why the features we think will be important - are not important at all", "441742": "I just joined df_hist with merchant data then did some agg, but got a worse cv score and lb score." }, "source": "meta" }