{ "id": 77537, "title": "Reducing the gap between CV and LB", "url": "/competitions/elo-merchant-category-recommendation/discussion/77537", "author_name": "", "post_date": "2019-01-14T04:29:34.663494400Z", "votes": 88, "comment_count": 31, "views": 0, "content": "

This helped me reducing the GAP

\n\n

LightGBM 5fold CV 3.64593 --> lb 3.695
\nLightGBM 5fold CV 3.64491 --> lb 3.690

\n\n
from scipy.stats import ks_2samp\nlist_p_value =[]\n\nfor i in tqdm(df_train_columns):\n    list_p_value.append(ks_2samp(df_test[i] , df_train[i])[1])\n\nSe = pd.Series(list_p_value, index = df_train_columns).sort_values() \nlist_discarded = list(Se[Se < .1].index)\n
\n\n

i removed features in \"list_discarded\". Although gap is still quite high. fight is on :)

", "messages": [ { "id": "455509", "postDate": "01/14/2019 04:29:34", "content": "

This helped me reducing the GAP

\n\n

LightGBM 5fold CV 3.64593 --> lb 3.695
\nLightGBM 5fold CV 3.64491 --> lb 3.690

\n\n
from scipy.stats import ks_2samp\nlist_p_value =[]\n\nfor i in tqdm(df_train_columns):\n    list_p_value.append(ks_2samp(df_test[i] , df_train[i])[1])\n\nSe = pd.Series(list_p_value, index = df_train_columns).sort_values() \nlist_discarded = list(Se[Se < .1].index)\n
\n\n

i removed features in \"list_discarded\". Although gap is still quite high. fight is on :)

", "rawMarkdown": "This helped me reducing the GAP \n\nLightGBM 5fold CV 3.64593 --> lb 3.695 \nLightGBM 5fold CV 3.64491 --> lb 3.690\n\n from scipy.stats import ks_2samp\n list_p_value =[]\n \n for i in tqdm(df_train_columns):\n list_p_value.append(ks_2samp(df_test[i] , df_train[i])[1])\n \n Se = pd.Series(list_p_value, index = df_train_columns).sort_values() \n list_discarded = list(Se[Se < .1].index)\n\ni removed features in \"list_discarded\". Although gap is still quite high. fight is on :)", "votes": null }, { "id": "455613", "postDate": "01/14/2019 08:47:47", "content": "

You're right. The basic assumption of statistical learning is “Independent and Identically Distributed”

", "rawMarkdown": "You're right. The basic assumption of statistical learning is “Independent and Identically Distributed”", "votes": null }, { "id": "455620", "postDate": "01/14/2019 08:59:53", "content": "

Thank you, I have not heard of ks_2samp - I suck in stat - good knowledge to have in ones' pocket :)

", "rawMarkdown": "Thank you, I have not heard of ks_2samp - I suck in stat - good knowledge to have in ones' pocket :)", "votes": null }, { "id": "455819", "postDate": "01/14/2019 16:43:36", "content": "

In my case , it is quite reasonable.\nCV : 3.671841510192393 LB:3.84

", "rawMarkdown": "In my case , it is quite reasonable.\nCV : 3.671841510192393 LB:3.84", "votes": null }, { "id": "455824", "postDate": "01/14/2019 16:55:50", "content": "

This is perfect. You must have done very good feature engineering :)\nIs this your single model ?

", "rawMarkdown": "This is perfect. You must have done very good feature engineering :)\nIs this your single model ?", "votes": null }, { "id": "455837", "postDate": "01/14/2019 17:12:14", "content": "

Yah. Single model LGBM(5 fold). only aggregate features.

", "rawMarkdown": "Yah. Single model LGBM(5 fold). only aggregate features.", "votes": null }, { "id": "455843", "postDate": "01/14/2019 17:18:45", "content": "

Awesome !!

", "rawMarkdown": "Awesome !!", "votes": null }, { "id": "455999", "postDate": "01/14/2019 23:18:45", "content": "

Thanks for sharing! Can we use it with other feature selection algorithms?

", "rawMarkdown": "Thanks for sharing! Can we use it with other feature selection algorithms?", "votes": null }, { "id": "456025", "postDate": "01/15/2019 01:23:41", "content": "

U can choice a more better select algorithm to improve

", "rawMarkdown": "U can choice a more better select algorithm to improve", "votes": null }, { "id": "456077", "postDate": "01/15/2019 04:51:21", "content": "

Yes

", "rawMarkdown": "Yes", "votes": null }, { "id": "456839", "postDate": "01/16/2019 16:25:50", "content": "

Can you suggest a better algorithm?

", "rawMarkdown": "Can you suggest a better algorithm?", "votes": null }, { "id": "458625", "postDate": "01/20/2019 04:40:59", "content": "

Awesome LB and gap! May I ask how many features you use for your best single model?

", "rawMarkdown": "Awesome LB and gap! May I ask how many features you use for your best single model?", "votes": null }, { "id": "459095", "postDate": "01/21/2019 07:09:43", "content": "

Hi, thanks for sharing. Did you just drop features in \"list_discarded\" and retrain lgb and then get 0.005 improvement?\nI tried the same way, it seems that no improvement for me, really sad ..\nBTW, how many features are you using and how many features have been dropped using this way?

", "rawMarkdown": "Hi, thanks for sharing. Did you just drop features in \"list_discarded\" and retrain lgb and then get 0.005 improvement?\nI tried the same way, it seems that no improvement for me, really sad ..\nBTW, how many features are you using and how many features have been dropped using this way?", "votes": null }, { "id": "459098", "postDate": "01/21/2019 07:18:52", "content": "

such as bruta

", "rawMarkdown": "such as bruta", "votes": null }, { "id": "459109", "postDate": "01/21/2019 07:40:50", "content": "

Hi Neo,\nYou are right we can use boruta to select the variable which are important for fitting the train data best

\n\n

What above i am trying to say is little different.\nThere may be variable which may be very important in the training of the model on the train data and decreasing the local CV, but its quite possible that variable might not be uni-formally represented in the test data (ie LB)

\n\n

So the above feature selection is prior to modeling

\n\n

We can always use boruta on top of that

", "rawMarkdown": "Hi Neo,\nYou are right we can use boruta to select the variable which are important for fitting the train data best\n\nWhat above i am trying to say is little different.\nThere may be variable which may be very important in the training of the model on the train data and decreasing the local CV, but its quite possible that variable might not be uni-formally represented in the test data (ie LB)\n\nSo the above feature selection is prior to modeling \n\nWe can always use boruta on top of that", "votes": null }, { "id": "459272", "postDate": "01/21/2019 13:37:03", "content": "

Hi zjucor,\n I dropped around 20+ features. Actually i had done really bad feature engineering to begin with and i ended up dropping one of my very important feature which was showing in train and was badly represented in test data that gave me the major boost.

", "rawMarkdown": "Hi zjucor,\n I dropped around 20+ features. Actually i had done really bad feature engineering to begin with and i ended up dropping one of my very important feature which was showing in train and was badly represented in test data that gave me the major boost.", "votes": null }, { "id": "459279", "postDate": "01/21/2019 13:48:49", "content": "

thanks for sharing

", "rawMarkdown": "thanks for sharing", "votes": null }, { "id": "459577", "postDate": "01/22/2019 03:00:11", "content": "

Hi Manoj , agree with u

", "rawMarkdown": "Hi Manoj , agree with u", "votes": null }, { "id": "459586", "postDate": "01/22/2019 03:38:07", "content": "

around 141.

", "rawMarkdown": "around 141.", "votes": null }, { "id": "459588", "postDate": "01/22/2019 03:39:28", "content": "

@Neo\nare u successful with any kind of feature selection?

", "rawMarkdown": "Neo\nare u successful with any kind of feature selection?", "votes": null }, { "id": "459603", "postDate": "01/22/2019 04:15:24", "content": "

I‘m not going to reply because you just asking but never answered me.

", "rawMarkdown": "I‘m not going to reply because you just asking but never answered me.", "votes": null }, { "id": "459612", "postDate": "01/22/2019 04:49:53", "content": "

It was my bad that i completely overlooked your request as well as your message. My apologies.

", "rawMarkdown": "It was my bad that i completely overlooked your request as well as your message. My apologies.", "votes": null }, { "id": "459614", "postDate": "01/22/2019 04:51:44", "content": "

nothing serious

", "rawMarkdown": "nothing serious", "votes": null }, { "id": "475676", "postDate": "02/21/2019 04:02:35", "content": "

I assume you are computing the Kolmogorov-Smirnov statistic on 2 samples to test for the null hypothesis that 2 independent samples are drawn from the same continuous distribution. The statistic test computes the KS statistic and pvalue. If the p value is less than .1 the null hypothesis is rejected. In this case you are sampling values from corresponding columns in test and train data frame and storing their pvalues in Se. If the p value is less than .1 the null hypothesis is rejected. Columns with pvalues less than 0.1 are added to the list_discarded.

\n\n

Great information . Thanks

", "rawMarkdown": "I assume you are computing the Kolmogorov-Smirnov statistic on 2 samples to test for the null hypothesis that 2 independent samples are drawn from the same continuous distribution. The statistic test computes the KS statistic and pvalue. If the p value is less than .1 the null hypothesis is rejected. In this case you are sampling values from corresponding columns in test and train data frame and storing their pvalues in Se. If the p value is less than .1 the null hypothesis is rejected. Columns with pvalues less than 0.1 are added to the list_discarded.\n\nGreat information . Thanks", "votes": null }, { "id": "475681", "postDate": "02/21/2019 04:12:23", "content": "

you are right, usually instead of .1 we keeps 0.05

", "rawMarkdown": "you are right, usually instead of .1 we keeps 0.05", "votes": null }, { "id": "480014", "postDate": "02/27/2019 17:22:03", "content": "

I wish I had seen this before! Great sharing.

", "rawMarkdown": "I wish I had seen this before! Great sharing.", "votes": null }, { "id": "480048", "postDate": "02/27/2019 18:10:21", "content": "

it didn't help me.

", "rawMarkdown": "it didn't help me.", "votes": null }, { "id": "480088", "postDate": "02/27/2019 18:59:40", "content": "

Sir, engaging with the kaggle community i learned from you. thank you for being source of inspiration :) @cpmpml

", "rawMarkdown": "Sir, engaging with the kaggle community i learned from you. thank you for being source of inspiration :) @cpmpml", "votes": null }, { "id": "480091", "postDate": "02/27/2019 19:00:35", "content": "

Asish ( @roydatascience ), we are in the business of estimation, nothing is guaranteed ;)

", "rawMarkdown": "Asish ( @roydatascience ), we are in the business of estimation, nothing is guaranteed ;)", "votes": null }, { "id": "534903", "postDate": "05/22/2019 02:22:36", "content": "

Hi,

\n\n

Thanks for sharing this idea here. I know I join this discussion late. But recently I was trying to pick up different methods of feature selection. I came across this post and I just want to ask you one question here.

\n\n

I understand what ks test is doing. But regarding selecting the features to discard, is the idea that if one feature's distribution is different in train and test set, that suggests train and test sets are from different distribution, which means this feature is not representative and we should discard this?

\n\n

Thanks in advance.

", "rawMarkdown": "Hi,\n\nThanks for sharing this idea here. I know I join this discussion late. But recently I was trying to pick up different methods of feature selection. I came across this post and I just want to ask you one question here.\n\nI understand what ks test is doing. But regarding selecting the features to discard, is the idea that if one feature's distribution is different in train and test set, that suggests train and test sets are from different distribution, which means this feature is not representative and we should discard this?\n\nThanks in advance.", "votes": null }, { "id": "534930", "postDate": "05/22/2019 04:48:15", "content": "

yes thats what i meant.

", "rawMarkdown": "yes thats what i meant.", "votes": null }, { "id": "535463", "postDate": "05/23/2019 01:22:09", "content": "

Thanks a lot!

", "rawMarkdown": "Thanks a lot!", "votes": null } ], "comments": [ { "id": 455613, "author_name": "daishu", "author_url": "", "post_date": "01/14/2019 08:47:47", "content": "

You're right. The basic assumption of statistical learning is “Independent and Identically Distributed”

", "votes": null, "replies": [] }, { "id": 455620, "author_name": "yifanxie", "author_url": "", "post_date": "01/14/2019 08:59:53", "content": "

Thank you, I have not heard of ks_2samp - I suck in stat - good knowledge to have in ones' pocket :)

", "votes": null, "replies": [] }, { "id": 455819, "author_name": "subikashpal", "author_url": "", "post_date": "01/14/2019 16:43:36", "content": "

In my case , it is quite reasonable.\nCV : 3.671841510192393 LB:3.84

", "votes": null, "replies": [ { "id": 455824, "author_name": "mks2192", "author_url": "", "post_date": "01/14/2019 16:55:50", "content": "

This is perfect. You must have done very good feature engineering :)\nIs this your single model ?

", "votes": null, "replies": [] }, { "id": 455837, "author_name": "subikashpal", "author_url": "", "post_date": "01/14/2019 17:12:14", "content": "

Yah. Single model LGBM(5 fold). only aggregate features.

", "votes": null, "replies": [] }, { "id": 455843, "author_name": "mks2192", "author_url": "", "post_date": "01/14/2019 17:18:45", "content": "

Awesome !!

", "votes": null, "replies": [] }, { "id": 458625, "author_name": "billxubf", "author_url": "", "post_date": "01/20/2019 04:40:59", "content": "

Awesome LB and gap! May I ask how many features you use for your best single model?

", "votes": null, "replies": [] }, { "id": 459586, "author_name": "subikashpal", "author_url": "", "post_date": "01/22/2019 03:38:07", "content": "

around 141.

", "votes": null, "replies": [] } ] }, { "id": 455999, "author_name": "luuunh", "author_url": "", "post_date": "01/14/2019 23:18:45", "content": "

Thanks for sharing! Can we use it with other feature selection algorithms?

", "votes": null, "replies": [ { "id": 456077, "author_name": "mks2192", "author_url": "", "post_date": "01/15/2019 04:51:21", "content": "

Yes

", "votes": null, "replies": [] } ] }, { "id": 456025, "author_name": "laplaceplanet", "author_url": "", "post_date": "01/15/2019 01:23:41", "content": "

U can choice a more better select algorithm to improve

", "votes": null, "replies": [ { "id": 456839, "author_name": "roydatascience", "author_url": "", "post_date": "01/16/2019 16:25:50", "content": "

Can you suggest a better algorithm?

", "votes": null, "replies": [] }, { "id": 459098, "author_name": "laplaceplanet", "author_url": "", "post_date": "01/21/2019 07:18:52", "content": "

such as bruta

", "votes": null, "replies": [] }, { "id": 459109, "author_name": "mks2192", "author_url": "", "post_date": "01/21/2019 07:40:50", "content": "

Hi Neo,\nYou are right we can use boruta to select the variable which are important for fitting the train data best

\n\n

What above i am trying to say is little different.\nThere may be variable which may be very important in the training of the model on the train data and decreasing the local CV, but its quite possible that variable might not be uni-formally represented in the test data (ie LB)

\n\n

So the above feature selection is prior to modeling

\n\n

We can always use boruta on top of that

", "votes": null, "replies": [] }, { "id": 459577, "author_name": "laplaceplanet", "author_url": "", "post_date": "01/22/2019 03:00:11", "content": "

Hi Manoj , agree with u

", "votes": null, "replies": [] }, { "id": 459588, "author_name": "subikashpal", "author_url": "", "post_date": "01/22/2019 03:39:28", "content": "

@Neo\nare u successful with any kind of feature selection?

", "votes": null, "replies": [] }, { "id": 459603, "author_name": "laplaceplanet", "author_url": "", "post_date": "01/22/2019 04:15:24", "content": "

I‘m not going to reply because you just asking but never answered me.

", "votes": null, "replies": [] }, { "id": 459612, "author_name": "subikashpal", "author_url": "", "post_date": "01/22/2019 04:49:53", "content": "

It was my bad that i completely overlooked your request as well as your message. My apologies.

", "votes": null, "replies": [] }, { "id": 459614, "author_name": "laplaceplanet", "author_url": "", "post_date": "01/22/2019 04:51:44", "content": "

nothing serious

", "votes": null, "replies": [] } ] }, { "id": 459095, "author_name": "zjucor", "author_url": "", "post_date": "01/21/2019 07:09:43", "content": "

Hi, thanks for sharing. Did you just drop features in \"list_discarded\" and retrain lgb and then get 0.005 improvement?\nI tried the same way, it seems that no improvement for me, really sad ..\nBTW, how many features are you using and how many features have been dropped using this way?

", "votes": null, "replies": [ { "id": 459272, "author_name": "mks2192", "author_url": "", "post_date": "01/21/2019 13:37:03", "content": "

Hi zjucor,\n I dropped around 20+ features. Actually i had done really bad feature engineering to begin with and i ended up dropping one of my very important feature which was showing in train and was badly represented in test data that gave me the major boost.

", "votes": null, "replies": [] }, { "id": 459279, "author_name": "zjucor", "author_url": "", "post_date": "01/21/2019 13:48:49", "content": "

thanks for sharing

", "votes": null, "replies": [] } ] }, { "id": 475676, "author_name": "rajeshcv", "author_url": "", "post_date": "02/21/2019 04:02:35", "content": "

I assume you are computing the Kolmogorov-Smirnov statistic on 2 samples to test for the null hypothesis that 2 independent samples are drawn from the same continuous distribution. The statistic test computes the KS statistic and pvalue. If the p value is less than .1 the null hypothesis is rejected. In this case you are sampling values from corresponding columns in test and train data frame and storing their pvalues in Se. If the p value is less than .1 the null hypothesis is rejected. Columns with pvalues less than 0.1 are added to the list_discarded.

\n\n

Great information . Thanks

", "votes": null, "replies": [ { "id": 475681, "author_name": "mks2192", "author_url": "", "post_date": "02/21/2019 04:12:23", "content": "

you are right, usually instead of .1 we keeps 0.05

", "votes": null, "replies": [] } ] }, { "id": 480014, "author_name": "cpmpml", "author_url": "", "post_date": "02/27/2019 17:22:03", "content": "

I wish I had seen this before! Great sharing.

", "votes": null, "replies": [ { "id": 480048, "author_name": "roydatascience", "author_url": "", "post_date": "02/27/2019 18:10:21", "content": "

it didn't help me.

", "votes": null, "replies": [] }, { "id": 480088, "author_name": "mks2192", "author_url": "", "post_date": "02/27/2019 18:59:40", "content": "

Sir, engaging with the kaggle community i learned from you. thank you for being source of inspiration :) @cpmpml

", "votes": null, "replies": [] }, { "id": 480091, "author_name": "mks2192", "author_url": "", "post_date": "02/27/2019 19:00:35", "content": "

Asish ( @roydatascience ), we are in the business of estimation, nothing is guaranteed ;)

", "votes": null, "replies": [] } ] }, { "id": 534903, "author_name": "dondon2019", "author_url": "", "post_date": "05/22/2019 02:22:36", "content": "

Hi,

\n\n

Thanks for sharing this idea here. I know I join this discussion late. But recently I was trying to pick up different methods of feature selection. I came across this post and I just want to ask you one question here.

\n\n

I understand what ks test is doing. But regarding selecting the features to discard, is the idea that if one feature's distribution is different in train and test set, that suggests train and test sets are from different distribution, which means this feature is not representative and we should discard this?

\n\n

Thanks in advance.

", "votes": null, "replies": [ { "id": 534930, "author_name": "mks2192", "author_url": "", "post_date": "05/22/2019 04:48:15", "content": "

yes thats what i meant.

", "votes": null, "replies": [] }, { "id": 535463, "author_name": "dondon2019", "author_url": "", "post_date": "05/23/2019 01:22:09", "content": "

Thanks a lot!

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "455509": "This helped me reducing the GAP \n\nLightGBM 5fold CV 3.64593 --> lb 3.695 \nLightGBM 5fold CV 3.64491 --> lb 3.690\n\n from scipy.stats import ks_2samp\n list_p_value =[]\n \n for i in tqdm(df_train_columns):\n list_p_value.append(ks_2samp(df_test[i] , df_train[i])[1])\n \n Se = pd.Series(list_p_value, index = df_train_columns).sort_values() \n list_discarded = list(Se[Se < .1].index)\n\ni removed features in \"list_discarded\". Although gap is still quite high. fight is on :)", "455613": "You're right. The basic assumption of statistical learning is “Independent and Identically Distributed”", "455620": "Thank you, I have not heard of ks_2samp - I suck in stat - good knowledge to have in ones' pocket :)", "455819": "In my case , it is quite reasonable.\nCV : 3.671841510192393 LB:3.84", "455824": "This is perfect. You must have done very good feature engineering :)\nIs this your single model ?", "455837": "Yah. Single model LGBM(5 fold). only aggregate features.", "455843": "Awesome !!", "455999": "Thanks for sharing! Can we use it with other feature selection algorithms?", "456025": "U can choice a more better select algorithm to improve", "456077": "Yes", "456839": "Can you suggest a better algorithm?", "458625": "Awesome LB and gap! May I ask how many features you use for your best single model?", "459095": "Hi, thanks for sharing. Did you just drop features in \"list_discarded\" and retrain lgb and then get 0.005 improvement?\nI tried the same way, it seems that no improvement for me, really sad ..\nBTW, how many features are you using and how many features have been dropped using this way?", "459098": "such as bruta", "459109": "Hi Neo,\nYou are right we can use boruta to select the variable which are important for fitting the train data best\n\nWhat above i am trying to say is little different.\nThere may be variable which may be very important in the training of the model on the train data and decreasing the local CV, but its quite possible that variable might not be uni-formally represented in the test data (ie LB)\n\nSo the above feature selection is prior to modeling \n\nWe can always use boruta on top of that", "459272": "Hi zjucor,\n I dropped around 20+ features. Actually i had done really bad feature engineering to begin with and i ended up dropping one of my very important feature which was showing in train and was badly represented in test data that gave me the major boost.", "459279": "thanks for sharing", "459577": "Hi Manoj , agree with u", "459586": "around 141.", "459588": "Neo\nare u successful with any kind of feature selection?", "459603": "I‘m not going to reply because you just asking but never answered me.", "459612": "It was my bad that i completely overlooked your request as well as your message. My apologies.", "459614": "nothing serious", "475676": "I assume you are computing the Kolmogorov-Smirnov statistic on 2 samples to test for the null hypothesis that 2 independent samples are drawn from the same continuous distribution. The statistic test computes the KS statistic and pvalue. If the p value is less than .1 the null hypothesis is rejected. In this case you are sampling values from corresponding columns in test and train data frame and storing their pvalues in Se. If the p value is less than .1 the null hypothesis is rejected. Columns with pvalues less than 0.1 are added to the list_discarded.\n\nGreat information . Thanks", "475681": "you are right, usually instead of .1 we keeps 0.05", "480014": "I wish I had seen this before! Great sharing.", "480048": "it didn't help me.", "480088": "Sir, engaging with the kaggle community i learned from you. thank you for being source of inspiration :) @cpmpml", "480091": "Asish ( @roydatascience ), we are in the business of estimation, nothing is guaranteed ;)", "534903": "Hi,\n\nThanks for sharing this idea here. I know I join this discussion late. But recently I was trying to pick up different methods of feature selection. I came across this post and I just want to ask you one question here.\n\nI understand what ks test is doing. But regarding selecting the features to discard, is the idea that if one feature's distribution is different in train and test set, that suggests train and test sets are from different distribution, which means this feature is not representative and we should discard this?\n\nThanks in advance.", "534930": "yes thats what i meant.", "535463": "Thanks a lot!" }, "source": "meta" }