Thanks to ELO for hosting such an interesting competition, and congratulations to all the top teams.\nI will write a brief memo.
\n\nBlending some lgb with different features\nPrivate : 3.600\nPublic : 3.681
\n\n12 fold lgb with 300 features\nPrivate : 3.605\nPublic : 3.688\nCV : 3.634\n・Best cv model has best private score.\n・I didn't any post processing.
\n\nAggregate one-hot-encoded merchant_id.\nI had to select some merchant_id because there was no sufficient RAM.
\n\ntrain['outlier'] = (train['target'] < -33).astype(int)\nhist = hist.drop_duplicates(['card_id','merchant_id'])\nhist = hist.merge(train)\nhist_outlier = hist.groupby('merchant_id')['outlier'].agg([len, sum, np.mean])\nNow, you can see some merchant_id have large value.\nSo you can select them.\nFor exsample.\n\nhist_outlier[(hist_outlier['len']>500) & (hist_outlier['mean']>0.02)]\n
-- Modiling with NN, RNN, xgboost, catboost, FFM\n-- Directly target encoding\n-- Feature decomposition(TrancatedSVD, LDA)\n-- Feature selection with boruta\n-- Stacking models by logistic regression, lightgbm
\n\nI found out later that it was effective to predict target for each transaction row.\n1. Merge hist(or new) transaction data and train data.\n2. Build lightgbm model to predict target of each row.\n3. Aggregate predictions by card_id. \n\ndf.groupby('card_id')['predictions'].agg([min, np.mean, np.median, max, sum, np.std]))\n\nThese features improve Private 0.003 but make worse Public 0.003.\nSo I could't use these features in final submission.\n
\nThat's all. Thank you for your reading.
Thanks to ELO for hosting such an interesting competition, and congratulations to all the top teams.\nI will write a brief memo.
\n\nBlending some lgb with different features\nPrivate : 3.600\nPublic : 3.681
\n\n12 fold lgb with 300 features\nPrivate : 3.605\nPublic : 3.688\nCV : 3.634\n・Best cv model has best private score.\n・I didn't any post processing.
\n\nAggregate one-hot-encoded merchant_id.\nI had to select some merchant_id because there was no sufficient RAM.
\n\ntrain['outlier'] = (train['target'] < -33).astype(int)\nhist = hist.drop_duplicates(['card_id','merchant_id'])\nhist = hist.merge(train)\nhist_outlier = hist.groupby('merchant_id')['outlier'].agg([len, sum, np.mean])\nNow, you can see some merchant_id have large value.\nSo you can select them.\nFor exsample.\n\nhist_outlier[(hist_outlier['len']>500) & (hist_outlier['mean']>0.02)]\n
-- Modiling with NN, RNN, xgboost, catboost, FFM\n-- Directly target encoding\n-- Feature decomposition(TrancatedSVD, LDA)\n-- Feature selection with boruta\n-- Stacking models by logistic regression, lightgbm
\n\nI found out later that it was effective to predict target for each transaction row.\n1. Merge hist(or new) transaction data and train data.\n2. Build lightgbm model to predict target of each row.\n3. Aggregate predictions by card_id. \n\ndf.groupby('card_id')['predictions'].agg([min, np.mean, np.median, max, sum, np.std]))\n\nThese features improve Private 0.003 but make worse Public 0.003.\nSo I could't use these features in final submission.\n
\nThat's all. Thank you for your reading.
Congratulations @hmdhmd
", "rawMarkdown": "Congratulations @hmdhmd", "votes": null }, { "id": "480168", "postDate": "02/27/2019 21:30:40", "content": "Congratulations @hmdhmd ! I agree with you concerning the merchant_id. I think having either a one-hot-encoder or the count of transactions for each merchant per card_id would have been interesting and might have improved results. But like you I didn't have enough RAM. At the beginning I was thinking of taking the favorite merchant for each card_id and get the average transaction amount. But it was a good idea to select only a few of them.
", "rawMarkdown": "Congratulations @hmdhmd ! I agree with you concerning the merchant_id. I think having either a one-hot-encoder or the count of transactions for each merchant per card_id would have been interesting and might have improved results. But like you I didn't have enough RAM. At the beginning I was thinking of taking the favorite merchant for each card_id and get the average transaction amount. But it was a good idea to select only a few of them.", "votes": null }, { "id": "480244", "postDate": "02/28/2019 01:17:15", "content": "Thank you for the reply!\nI agree with you.\nI think merchantIds are important and there are a lot of way to handling them.
", "rawMarkdown": "Thank you for the reply!\nI agree with you.\nI think merchantIds are important and there are a lot of way to handling them.", "votes": null }, { "id": "480245", "postDate": "02/28/2019 01:17:24", "content": "Thanks a lot!
", "rawMarkdown": "Thanks a lot!", "votes": null }, { "id": "480309", "postDate": "02/28/2019 03:20:00", "content": "thanks!! \nyou mean that test set' features are made by predictions using model right?\nex)\ntrain.groupby('card_id')['predictions'].agg([min, np.mean, np.median, max, sum, np.std]))\ntest.groupby('card_id')['predictions'].agg([min, np.mean, np.median, max, sum, np.std]))
", "rawMarkdown": "thanks!! \nyou mean that test set' features are made by predictions using model right?\nex)\ntrain.groupby('card_id')['predictions'].agg([min, np.mean, np.median, max, sum, np.std]))\ntest.groupby('card_id')['predictions'].agg([min, np.mean, np.median, max, sum, np.std]))", "votes": null }, { "id": "480352", "postDate": "02/28/2019 04:48:44", "content": "Thank you for your reply!\nYes, you are right.
\n\ntrain-set:\ntrain.merge(transaction_data, how='left')
\n\ntest-set:\ntest.merge(transaction_data, how='left')
\n\nYou can build model in ordinary way.\n(cross-validation ~ prediction)
\n\nI created feature dataset.\nIf you want, you can try them.\nhttps://www.kaggle.com/hmdhmd/elo-features
", "rawMarkdown": "Thank you for your reply!\nYes, you are right.\n\ntrain-set:\ntrain.merge(transaction_data, how='left')\n\ntest-set:\ntest.merge(transaction_data, how='left')\n\nYou can build model in ordinary way.\n(cross-validation ~ prediction)\n\nI created feature dataset.\nIf you want, you can try them.\nhttps://www.kaggle.com/hmdhmd/elo-features", "votes": null }, { "id": "480382", "postDate": "02/28/2019 05:41:17", "content": "but there are unique card_id in train dataframe, so how to do aggregation?
", "rawMarkdown": "but there are unique card_id in train dataframe, so how to do aggregation?", "votes": null }, { "id": "480406", "postDate": "02/28/2019 06:22:31", "content": "Thank you for the reply!
\n\ntrain = train.merge(new-transaction-data, how='left')\ntest = test.merge(new-transaction-data, how='left')\nlen(train) -> 1241616\nlen(test) -> 756954
\n\nThese data have not unique card_id.\nYou can get 1241616 train-predict-values and 756954 test-predict-values.
\n\ntrain['predictions'] = train-predict-values\ntest['predictions'] = test-predict-values
\n\nYou can aggregate 'predictions' by card_id.
", "rawMarkdown": "Thank you for the reply!\n\ntrain = train.merge(new-transaction-data, how='left')\ntest = test.merge(new-transaction-data, how='left')\nlen(train) -> 1241616\nlen(test) -> 756954\n\nThese data have not unique card_id.\nYou can get 1241616 train-predict-values and 756954 test-predict-values.\n\ntrain['predictions'] = train-predict-values\ntest['predictions'] = test-predict-values\n\nYou can aggregate 'predictions' by card_id.", "votes": null }, { "id": "480421", "postDate": "02/28/2019 06:39:25", "content": "Thanks for your kind reply. But I still have doubt. So what is the on arguments of pd.merge functions? and if merge by left, length of the result dataframe is equal to that of the left, in that case, len(train) == 201917 len(test) == 123623
I’m sorry, but I can't reproduce your problem.\non=card_id\nIn my kaggle kernel, this code works.\nhttps://i.imgur.com/qmIe7ut.png
", "rawMarkdown": "I’m sorry, but I can't reproduce your problem.\non=card_id\nIn my kaggle kernel, this code works.\nhttps://i.imgur.com/qmIe7ut.png", "votes": null }, { "id": "480535", "postDate": "02/28/2019 10:03:40", "content": "Congrats @hmdhmd and thanks for sharing your solution overview. It is interesting that xgboost did not work for you but for me, my best single model is an xgboost.
", "rawMarkdown": "Congrats @hmdhmd and thanks for sharing your solution overview. It is interesting that xgboost did not work for you but for me, my [best single model][1] is an xgboost.\n\n\n [1]: https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/82052", "votes": null }, { "id": "480569", "postDate": "02/28/2019 11:12:50", "content": "Thank you!\nYour discussion of feature selection is useful for me.
\n\nThere are several possible reasons in difference between xgb and lgb.\nParameters, used features...etc\nBut in my case, xgboost always make score worse, private 0.004, public 0.003 ;)
", "rawMarkdown": "Thank you!\nYour discussion of feature selection is useful for me.\n\nThere are several possible reasons in difference between xgb and lgb.\nParameters, used features...etc\nBut in my case, xgboost always make score worse, private 0.004, public 0.003 ;)", "votes": null }, { "id": "480584", "postDate": "02/28/2019 11:36:24", "content": "@hmdhmd, I am glad you found my post useful.
\n\nIn your case it may be that you did not tune your xgboost parameters well since you have good features going by your lgb result. In my case even after tuning my lgbm parameters very well, its performance compared to my xgb model was always about 0.003 worse on the public LB. But the lgbm CV is so much lower than that of the xgb model. I was also worried about the the large 0.04 CV/LB gap. However, I just went back and checked my submissions and discovered my lgbm model is actually better by 0.001 on the private LB. So thanks to your post I now know my lgbm model is slightly better:-)
", "rawMarkdown": "hmdhmd, I am glad you found [my post][1] useful.\n\nIn your case it may be that you did not tune your xgboost parameters well since you have good features going by your lgb result. In my case even after tuning my lgbm parameters very well, its performance compared to my xgb model was always about 0.003 worse on the public LB. But the lgbm CV is so much lower than that of the xgb model. I was also worried about the the large 0.04 CV/LB gap. However, I just went back and checked my submissions and discovered my lgbm model is actually better by 0.001 on the private LB. So thanks to your post I now know my lgbm model is slightly better:-)\n\n\n [1]: https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/82052", "votes": null }, { "id": "480617", "postDate": "02/28/2019 12:35:00", "content": "Yes, my xgboost parameters may not have been good.\nI was not able to spend much time tuning parameters.
\n\nThis competition was difficult because CV and LB did not work together...\nNext time, I would like to participate in the competition which's CV and LB work together ;)
", "rawMarkdown": "Yes, my xgboost parameters may not have been good.\nI was not able to spend much time tuning parameters.\n\nThis competition was difficult because CV and LB did not work together...\nNext time, I would like to participate in the competition which's CV and LB work together ;)", "votes": null }, { "id": "480628", "postDate": "02/28/2019 13:02:25", "content": "I agree that this competition is difficult but I kind of enjoyed the challenges it posed. It helps one with the learning new ways of doing things for when you encounter such problems in real world ML/DS projects at work.
", "rawMarkdown": "I agree that this competition is difficult but I kind of enjoyed the challenges it posed. It helps one with the learning new ways of doing things for when you encounter such problems in real world ML/DS projects at work.", "votes": null } ], "comments": [ { "id": 480055, "author_name": "karthik7395", "author_url": "", "post_date": "02/27/2019 18:23:13", "content": "Congratulations @hmdhmd
", "votes": null, "replies": [ { "id": 480245, "author_name": "hmdhmd", "author_url": "", "post_date": "02/28/2019 01:17:24", "content": "Thanks a lot!
", "votes": null, "replies": [] } ] }, { "id": 480168, "author_name": "ypadawan", "author_url": "", "post_date": "02/27/2019 21:30:40", "content": "Congratulations @hmdhmd ! I agree with you concerning the merchant_id. I think having either a one-hot-encoder or the count of transactions for each merchant per card_id would have been interesting and might have improved results. But like you I didn't have enough RAM. At the beginning I was thinking of taking the favorite merchant for each card_id and get the average transaction amount. But it was a good idea to select only a few of them.
", "votes": null, "replies": [ { "id": 480244, "author_name": "hmdhmd", "author_url": "", "post_date": "02/28/2019 01:17:15", "content": "Thank you for the reply!\nI agree with you.\nI think merchantIds are important and there are a lot of way to handling them.
", "votes": null, "replies": [] } ] }, { "id": 480309, "author_name": "mokmushroom", "author_url": "", "post_date": "02/28/2019 03:20:00", "content": "thanks!! \nyou mean that test set' features are made by predictions using model right?\nex)\ntrain.groupby('card_id')['predictions'].agg([min, np.mean, np.median, max, sum, np.std]))\ntest.groupby('card_id')['predictions'].agg([min, np.mean, np.median, max, sum, np.std]))
", "votes": null, "replies": [ { "id": 480352, "author_name": "hmdhmd", "author_url": "", "post_date": "02/28/2019 04:48:44", "content": "Thank you for your reply!\nYes, you are right.
\n\ntrain-set:\ntrain.merge(transaction_data, how='left')
\n\ntest-set:\ntest.merge(transaction_data, how='left')
\n\nYou can build model in ordinary way.\n(cross-validation ~ prediction)
\n\nI created feature dataset.\nIf you want, you can try them.\nhttps://www.kaggle.com/hmdhmd/elo-features
", "votes": null, "replies": [] }, { "id": 480382, "author_name": "liewmanchoi", "author_url": "", "post_date": "02/28/2019 05:41:17", "content": "but there are unique card_id in train dataframe, so how to do aggregation?
", "votes": null, "replies": [] }, { "id": 480406, "author_name": "hmdhmd", "author_url": "", "post_date": "02/28/2019 06:22:31", "content": "Thank you for the reply!
\n\ntrain = train.merge(new-transaction-data, how='left')\ntest = test.merge(new-transaction-data, how='left')\nlen(train) -> 1241616\nlen(test) -> 756954
\n\nThese data have not unique card_id.\nYou can get 1241616 train-predict-values and 756954 test-predict-values.
\n\ntrain['predictions'] = train-predict-values\ntest['predictions'] = test-predict-values
\n\nYou can aggregate 'predictions' by card_id.
", "votes": null, "replies": [] }, { "id": 480421, "author_name": "liewmanchoi", "author_url": "", "post_date": "02/28/2019 06:39:25", "content": "Thanks for your kind reply. But I still have doubt. So what is the on arguments of pd.merge functions? and if merge by left, length of the result dataframe is equal to that of the left, in that case, len(train) == 201917 len(test) == 123623
I’m sorry, but I can't reproduce your problem.\non=card_id\nIn my kaggle kernel, this code works.\nhttps://i.imgur.com/qmIe7ut.png
", "votes": null, "replies": [] } ] }, { "id": 480535, "author_name": "sheriytm", "author_url": "", "post_date": "02/28/2019 10:03:40", "content": "Congrats @hmdhmd and thanks for sharing your solution overview. It is interesting that xgboost did not work for you but for me, my best single model is an xgboost.
", "votes": null, "replies": [ { "id": 480569, "author_name": "hmdhmd", "author_url": "", "post_date": "02/28/2019 11:12:50", "content": "Thank you!\nYour discussion of feature selection is useful for me.
\n\nThere are several possible reasons in difference between xgb and lgb.\nParameters, used features...etc\nBut in my case, xgboost always make score worse, private 0.004, public 0.003 ;)
", "votes": null, "replies": [] }, { "id": 480584, "author_name": "sheriytm", "author_url": "", "post_date": "02/28/2019 11:36:24", "content": "@hmdhmd, I am glad you found my post useful.
\n\nIn your case it may be that you did not tune your xgboost parameters well since you have good features going by your lgb result. In my case even after tuning my lgbm parameters very well, its performance compared to my xgb model was always about 0.003 worse on the public LB. But the lgbm CV is so much lower than that of the xgb model. I was also worried about the the large 0.04 CV/LB gap. However, I just went back and checked my submissions and discovered my lgbm model is actually better by 0.001 on the private LB. So thanks to your post I now know my lgbm model is slightly better:-)
", "votes": null, "replies": [] }, { "id": 480617, "author_name": "hmdhmd", "author_url": "", "post_date": "02/28/2019 12:35:00", "content": "Yes, my xgboost parameters may not have been good.\nI was not able to spend much time tuning parameters.
\n\nThis competition was difficult because CV and LB did not work together...\nNext time, I would like to participate in the competition which's CV and LB work together ;)
", "votes": null, "replies": [] }, { "id": 480628, "author_name": "sheriytm", "author_url": "", "post_date": "02/28/2019 13:02:25", "content": "I agree that this competition is difficult but I kind of enjoyed the challenges it posed. It helps one with the learning new ways of doing things for when you encounter such problems in real world ML/DS projects at work.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "480022": "Thanks to ELO for hosting such an interesting competition, and congratulations to all the top teams.\nI will write a brief memo.\n# **Final Submission**\nBlending some lgb with different features\nPrivate : 3.600\nPublic : 3.681\n# **Best Single Model**\n12 fold lgb with 300 features\nPrivate : 3.605\nPublic : 3.688\nCV : 3.634\n・Best cv model has best private score.\n・I didn't any post processing.\n# **Useful**\nAggregate one-hot-encoded merchant_id.\nI had to select some merchant_id because there was no sufficient RAM.\n\n train['outlier'] = (train['target'] < -33).astype(int)\n hist = hist.drop_duplicates(['card_id','merchant_id'])\n hist = hist.merge(train)\n hist_outlier = hist.groupby('merchant_id')['outlier'].agg([len, sum, np.mean])\n\nNow, you can see some merchant_id have large value.\nSo you can select them.\nFor exsample.\n```\nhist_outlier[(hist_outlier['len']>500) & (hist_outlier['mean']>0.02)]\n```\n\n#**Didn't work**\n-- Modiling with NN, RNN, xgboost, catboost, FFM\n-- Directly target encoding\n-- Feature decomposition(TrancatedSVD, LDA)\n-- Feature selection with boruta\n-- Stacking models by logistic regression, lightgbm\n\n#**Addition**\n I found out later that it was effective to predict target for each transaction row.\n1. Merge hist(or new) transaction data and train data.\n2. Build lightgbm model to predict target of each row.\n3. Aggregate predictions by card_id. \n```\ndf.groupby('card_id')['predictions'].agg([min, np.mean, np.median, max, sum, np.std]))\n```\nThese features improve Private 0.003 but make worse Public 0.003.\nSo I could't use these features in final submission.\n![https://i.imgur.com/qmIe7ut.png](\"https://i.imgur.com/qmIe7ut.png\"){kind=link}