here are some of my single models and their CV and LB, \nMean CV .......... Std CV > LB\n3.651212 .......... 0.008416202 > 3.692\n3.651752 .......... 0.008498062 > 3.694\n3.651844 .......... 0.008310051 > 3.695\n3.652318 .......... 0.007261579 > 3.696\n3.64724 .......... 0.007691195 > 3.686\n3.6468 .......... 0.007567346 > 3.686\n3.647444 .......... 0.007899799 > 3.685\n3.6477 .......... 0.00752713 > 3.690\n3.647306 .......... 0.00675899 > 3.689\n3.6486 .......... 0.007100887 > 3.688\n3.648186 .......... 0.008152076 > 3.691\n3.646586 .......... 0.007440149 > 3.687\n3.646064 .......... 0.019902746 > 3.686\n3.648488 .......... 0.019731112 > 3.685\nhere are some of my single models and their CV and LB, \nMean CV .......... Std CV > LB\n3.651212 .......... 0.008416202 > 3.692\n3.651752 .......... 0.008498062 > 3.694\n3.651844 .......... 0.008310051 > 3.695\n3.652318 .......... 0.007261579 > 3.696\n3.64724 .......... 0.007691195 > 3.686\n3.6468 .......... 0.007567346 > 3.686\n3.647444 .......... 0.007899799 > 3.685\n3.6477 .......... 0.00752713 > 3.690\n3.647306 .......... 0.00675899 > 3.689\n3.6486 .......... 0.007100887 > 3.688\n3.648186 .......... 0.008152076 > 3.691\n3.646586 .......... 0.007440149 > 3.687\n3.646064 .......... 0.019902746 > 3.686\n3.648488 .......... 0.019731112 > 3.685\nNice work! \nThanks for sharing. \nBecause of the outliers even a few outliers in a specific fold or lb that get better or worse score affect the score pretty significantly.\nBecause CV is based on more data, with seemingly same distribution to test set, i tend to trust it. \nI would estimate that your 3.6460-3.686 is your currently best model
", "rawMarkdown": "Nice work! \nThanks for sharing. \nBecause of the outliers even a few outliers in a specific fold or lb that get better or worse score affect the score pretty significantly.\nBecause CV is based on more data, with seemingly same distribution to test set, i tend to trust it. \nI would estimate that your 3.6460-3.686 is your currently best model", "votes": null }, { "id": "469534", "postDate": "02/11/2019 11:35:26", "content": "Thanks for sharing! Your std seems to be good. How did you do cross validation?
", "rawMarkdown": "Thanks for sharing! Your std seems to be good. How did you do cross validation?", "votes": null }, { "id": "469743", "postDate": "02/11/2019 18:58:03", "content": "StratifiedKFold, did stratification over outliers
", "rawMarkdown": "StratifiedKFold, did stratification over outliers", "votes": null }, { "id": "471659", "postDate": "02/14/2019 18:19:05", "content": "Manoj \nYou have pretty good gap between CV and LB. I was unable to get so close.
\n\nMy trials gave me almost linear dependency (attached).
", "rawMarkdown": "Manoj \nYou have pretty good gap between CV and LB. I was unable to get so close.\n\nMy trials gave me almost linear dependency (attached).", "votes": null }, { "id": "471692", "postDate": "02/14/2019 19:00:52", "content": "initially i had similar problem. In my case i had done poor feature engineering\nthis helped me. see if it works in your case\nhttps://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/77537
", "rawMarkdown": "initially i had similar problem. In my case i had done poor feature engineering\nthis helped me. see if it works in your case\nhttps://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/77537", "votes": null }, { "id": "471941", "postDate": "02/15/2019 05:53:45", "content": "Great that you keep track of your cv and lb results. I didn't keep track of it and realized that after a long time. My current single model cv is 3.64997 and lb is 3.680.
", "rawMarkdown": "Great that you keep track of your cv and lb results. I didn't keep track of it and realized that after a long time. My current single model cv is 3.64997 and lb is 3.680.", "votes": null }, { "id": "471947", "postDate": "02/15/2019 06:09:18", "content": "Great, your gap between CV and LB is nice
", "rawMarkdown": "Great, your gap between CV and LB is nice", "votes": null }, { "id": "472610", "postDate": "02/16/2019 09:50:18", "content": "Hi, Manoj, how many features do you use in your model. I only use agg to get features but LB performance is poor. ToT
", "rawMarkdown": "Hi, Manoj, how many features do you use in your model. I only use agg to get features but LB performance is poor. ToT", "votes": null }, { "id": "472613", "postDate": "02/16/2019 09:52:56", "content": "42
", "rawMarkdown": "42", "votes": null }, { "id": "472617", "postDate": "02/16/2019 10:09:53", "content": "@Manoj: May I ask whether you use Boruta or PIMP for feature selection?
", "rawMarkdown": "Manoj: May I ask whether you use Boruta or PIMP for feature selection?", "votes": null }, { "id": "472622", "postDate": "02/16/2019 10:19:43", "content": "It is done by my teammate @AmirH i had started with 177 features he reduced it to 42 keeping the CV intact.
", "rawMarkdown": "It is done by my teammate @AmirH i had started with 177 features he reduced it to 42 keeping the CV intact.", "votes": null }, { "id": "472634", "postDate": "02/16/2019 10:51:31", "content": "Thanks Manoj. I am starting of with some 2,000 features and reducing it to some 250 features using PIMP (with gain importance) iteratively for my best model. Boruta would reduce significantly more starting from the same initial features resulting in some 54 features for my second best model.
", "rawMarkdown": "Thanks Manoj. I am starting of with some 2,000 features and reducing it to some 250 features using PIMP (with gain importance) iteratively for my best model. Boruta would reduce significantly more starting from the same initial features resulting in some 54 features for my second best model.", "votes": null }, { "id": "472648", "postDate": "02/16/2019 11:36:40", "content": "Interesting and reasonable approach. But in my case it gave opposite result. The gap increased after I removed 4 out of 217 features.
\n\nall features 217 CV 3.66582 LB 3.715\nfeatures excluded by ks_2samp 213 CV 3.66571 LB 3.716
", "rawMarkdown": "Interesting and reasonable approach. But in my case it gave opposite result. The gap increased after I removed 4 out of 217 features.\n\nall features 217 CV 3.66582 LB 3.715\nfeatures excluded by ks_2samp 213 CV 3.66571 LB 3.716", "votes": null } ], "comments": [ { "id": 469244, "author_name": "tpthegreat", "author_url": "", "post_date": "02/10/2019 18:53:58", "content": "Nice work! \nThanks for sharing. \nBecause of the outliers even a few outliers in a specific fold or lb that get better or worse score affect the score pretty significantly.\nBecause CV is based on more data, with seemingly same distribution to test set, i tend to trust it. \nI would estimate that your 3.6460-3.686 is your currently best model
", "votes": null, "replies": [] }, { "id": 469534, "author_name": "salamatburj", "author_url": "", "post_date": "02/11/2019 11:35:26", "content": "Thanks for sharing! Your std seems to be good. How did you do cross validation?
", "votes": null, "replies": [ { "id": 469743, "author_name": "mks2192", "author_url": "", "post_date": "02/11/2019 18:58:03", "content": "StratifiedKFold, did stratification over outliers
", "votes": null, "replies": [] } ] }, { "id": 471659, "author_name": "kmedvedev", "author_url": "", "post_date": "02/14/2019 18:19:05", "content": "Manoj \nYou have pretty good gap between CV and LB. I was unable to get so close.
\n\nMy trials gave me almost linear dependency (attached).
", "votes": null, "replies": [ { "id": 471692, "author_name": "mks2192", "author_url": "", "post_date": "02/14/2019 19:00:52", "content": "initially i had similar problem. In my case i had done poor feature engineering\nthis helped me. see if it works in your case\nhttps://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/77537
", "votes": null, "replies": [] }, { "id": 472648, "author_name": "kmedvedev", "author_url": "", "post_date": "02/16/2019 11:36:40", "content": "Interesting and reasonable approach. But in my case it gave opposite result. The gap increased after I removed 4 out of 217 features.
\n\nall features 217 CV 3.66582 LB 3.715\nfeatures excluded by ks_2samp 213 CV 3.66571 LB 3.716
", "votes": null, "replies": [] } ] }, { "id": 471941, "author_name": "harshit92", "author_url": "", "post_date": "02/15/2019 05:53:45", "content": "Great that you keep track of your cv and lb results. I didn't keep track of it and realized that after a long time. My current single model cv is 3.64997 and lb is 3.680.
", "votes": null, "replies": [ { "id": 471947, "author_name": "mks2192", "author_url": "", "post_date": "02/15/2019 06:09:18", "content": "Great, your gap between CV and LB is nice
", "votes": null, "replies": [] } ] }, { "id": 472610, "author_name": "bu2awgq", "author_url": "", "post_date": "02/16/2019 09:50:18", "content": "Hi, Manoj, how many features do you use in your model. I only use agg to get features but LB performance is poor. ToT
", "votes": null, "replies": [ { "id": 472613, "author_name": "mks2192", "author_url": "", "post_date": "02/16/2019 09:52:56", "content": "42
", "votes": null, "replies": [] }, { "id": 472617, "author_name": "lennartscheiber", "author_url": "", "post_date": "02/16/2019 10:09:53", "content": "@Manoj: May I ask whether you use Boruta or PIMP for feature selection?
", "votes": null, "replies": [] }, { "id": 472622, "author_name": "mks2192", "author_url": "", "post_date": "02/16/2019 10:19:43", "content": "It is done by my teammate @AmirH i had started with 177 features he reduced it to 42 keeping the CV intact.
", "votes": null, "replies": [] }, { "id": 472634, "author_name": "lennartscheiber", "author_url": "", "post_date": "02/16/2019 10:51:31", "content": "Thanks Manoj. I am starting of with some 2,000 features and reducing it to some 250 features using PIMP (with gain importance) iteratively for my best model. Boruta would reduce significantly more starting from the same initial features resulting in some 54 features for my second best model.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "469182": "here are some of my single models and their CV and LB, \n\n----------\n Mean\tCV .......... Std CV \t> LB\n 3.651212\t .......... 0.008416202\t> 3.692\n 3.651752\t .......... 0.008498062\t> 3.694\n 3.651844\t .......... 0.008310051\t> 3.695\n 3.652318\t .......... 0.007261579\t> 3.696\n 3.64724\t .......... 0.007691195\t> 3.686\n 3.6468\t .......... 0.007567346\t> 3.686\n 3.647444\t .......... 0.007899799\t> 3.685\n 3.6477\t .......... 0.00752713\t> 3.690\n 3.647306\t .......... 0.00675899\t> 3.689\n 3.6486\t .......... 0.007100887\t> 3.688\n 3.648186\t .......... 0.008152076\t> 3.691\n 3.646586\t .......... 0.007440149\t> 3.687\n 3.646064\t .......... 0.019902746\t> 3.686\n 3.648488 \t.......... 0.019731112\t> 3.685", "469244": "Nice work! \nThanks for sharing. \nBecause of the outliers even a few outliers in a specific fold or lb that get better or worse score affect the score pretty significantly.\nBecause CV is based on more data, with seemingly same distribution to test set, i tend to trust it. \nI would estimate that your 3.6460-3.686 is your currently best model", "469534": "Thanks for sharing! Your std seems to be good. How did you do cross validation?", "469743": "StratifiedKFold, did stratification over outliers", "471659": "Manoj \nYou have pretty good gap between CV and LB. I was unable to get so close.\n\nMy trials gave me almost linear dependency (attached).", "471692": "initially i had similar problem. In my case i had done poor feature engineering\nthis helped me. see if it works in your case\nhttps://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/77537", "471941": "Great that you keep track of your cv and lb results. I didn't keep track of it and realized that after a long time. My current single model cv is 3.64997 and lb is 3.680.", "471947": "Great, your gap between CV and LB is nice", "472610": "Hi, Manoj, how many features do you use in your model. I only use agg to get features but LB performance is poor. ToT", "472613": "42", "472617": "Manoj: May I ask whether you use Boruta or PIMP for feature selection?", "472622": "It is done by my teammate @AmirH i had started with 177 features he reduced it to 42 keeping the CV intact.", "472634": "Thanks Manoj. I am starting of with some 2,000 features and reducing it to some 250 features using PIMP (with gain importance) iteratively for my best model. Boruta would reduce significantly more starting from the same initial features resulting in some 54 features for my second best model.", "472648": "Interesting and reasonable approach. But in my case it gave opposite result. The gap increased after I removed 4 out of 217 features.\n\nall features 217 CV 3.66582 LB 3.715\nfeatures excluded by ks_2samp 213 CV 3.66571 LB 3.716" }, "source": "meta" }