Hi everyone, congrats all medal winners - you all well deserve the rewards for this exciting competition!!!
\n\nI’d also like to thank my teammate rcarson. Sorry buddy we didn’t do very well this time but let’s makenit better next time!!
\n\nWhile there's nothing to boast as a 55th, I do think there could be something in my solution/experience that might interest you. So there you go:
\n\nMost of the features were derived from Chau Huynh's 3.699 public LB kernel (https://www.kaggle.com/chauhuynh/my-first-kernel-3-699) with minor modifications. I had tried quite a lot of feature engineerings but only a few ended up being helpful including:
\n\nI used primarily LightGBM (should have spent more time on NN using various embeddings but didn't get a chance to implement due to busy schedule (laziness indeed LOL).
\n\nInspired by @Yifan Xie's post (https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/75935), I started exploring feature selections and after playing different strategies I ended up with RFE + LightGBM (random forest booster) which gave a decent boost about 0.006+.
\n\nI then developed a bit crazy (dummy) stackings using similar RFE framework but different parameters and the number of features selected are between 60-100 and each ended up with comparable results. However, stacking of them didn't make any advances on LB.
\n\nJust like most of top, and probably bottom rankers - and I'm sorry for the latter - I applied post-processing using a 3-models solution originally proposed by 李胃疼疼疼 (https://www.kaggle.com/waitingli/combining-your-model-with-a-model-without-outlier). Given that the original solution didn't seem to be able to allow us to perform CV so I modified it a bit particularly on the model without outliers: instead of predicting on test data only I made it generate out-of-fold predictions on training data too, which was quite similar to stacking with a small trick which allows it to make predictions on \"all\" training data rather than \"without outliers\" training data.
\n\nThis allows me to test out the best threshold instead of having to guess. In addition, instead of using predictions from the model trained with all data for those samples with p(outlier)> threshold, I used the average of predictions from both models (all and without outliers), which seemed to be more stable in the end. (0.003+)
\n\nI had to confess that my final submissions were blended with some of the outputs from public kernels, which gave me a boost of 0.001~.
\n\nI have been enjoying the competitions and shared the same self-doubting and frustrating moments as many of you may have had. In the end, it's not only the PB position, nor the medal, but these hard working moments made us stronger than what we were at the beginning of this competition.
\n\nCheers,
\n\nChris,
", "messages": [ { "id": "479235", "postDate": "02/27/2019 03:05:42", "content": "Hi everyone, congrats all medal winners - you all well deserve the rewards for this exciting competition!!!
\n\nI’d also like to thank my teammate rcarson. Sorry buddy we didn’t do very well this time but let’s makenit better next time!!
\n\nWhile there's nothing to boast as a 55th, I do think there could be something in my solution/experience that might interest you. So there you go:
\n\nMost of the features were derived from Chau Huynh's 3.699 public LB kernel (https://www.kaggle.com/chauhuynh/my-first-kernel-3-699) with minor modifications. I had tried quite a lot of feature engineerings but only a few ended up being helpful including:
\n\nI used primarily LightGBM (should have spent more time on NN using various embeddings but didn't get a chance to implement due to busy schedule (laziness indeed LOL).
\n\nInspired by @Yifan Xie's post (https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/75935), I started exploring feature selections and after playing different strategies I ended up with RFE + LightGBM (random forest booster) which gave a decent boost about 0.006+.
\n\nI then developed a bit crazy (dummy) stackings using similar RFE framework but different parameters and the number of features selected are between 60-100 and each ended up with comparable results. However, stacking of them didn't make any advances on LB.
\n\nJust like most of top, and probably bottom rankers - and I'm sorry for the latter - I applied post-processing using a 3-models solution originally proposed by 李胃疼疼疼 (https://www.kaggle.com/waitingli/combining-your-model-with-a-model-without-outlier). Given that the original solution didn't seem to be able to allow us to perform CV so I modified it a bit particularly on the model without outliers: instead of predicting on test data only I made it generate out-of-fold predictions on training data too, which was quite similar to stacking with a small trick which allows it to make predictions on \"all\" training data rather than \"without outliers\" training data.
\n\nThis allows me to test out the best threshold instead of having to guess. In addition, instead of using predictions from the model trained with all data for those samples with p(outlier)> threshold, I used the average of predictions from both models (all and without outliers), which seemed to be more stable in the end. (0.003+)
\n\nI had to confess that my final submissions were blended with some of the outputs from public kernels, which gave me a boost of 0.001~.
\n\nI have been enjoying the competitions and shared the same self-doubting and frustrating moments as many of you may have had. In the end, it's not only the PB position, nor the medal, but these hard working moments made us stronger than what we were at the beginning of this competition.
\n\nCheers,
\n\nChris,
", "rawMarkdown": "Hi everyone, congrats all medal winners - you all well deserve the rewards for this exciting competition!!!\n\n I’d also like to thank my teammate rcarson. Sorry buddy we didn’t do very well this time but let’s makenit better next time!!\n\nWhile there's nothing to boast as a 55th, I do think there could be something in my solution/experience that might interest you. So there you go:\n\n# Feature engineering\n\nMost of the features were derived from Chau Huynh's 3.699 public LB kernel (https://www.kaggle.com/chauhuynh/my-first-kernel-3-699) with minor modifications. I had tried quite a lot of feature engineerings but only a few ended up being helpful including:\n\n1. Transaction counts by month lag. This is probably, as many of you\n may have been aware, the most significant engineered feature.\n (0.005+) \n2. Ridge stackings on tf-idf of a variety of entities count\n by card id (historical transactions only). e.g. merchant id,\n merchant category id, and combinations of them. In fact, adding those features directly to the model didn't seem to help, nor more complicated embeddings such as word2vec and fasttext. However, adding an additional ridge stacking layer on top of those sparse features yielded considerable improvements, both on public board and private board. (0.002+).\n3. Similarities calculated between historical and new transactions using word2vec and fasttext gave\n minor improvements as well. (0.001+)\n\n# Modeling\n\nI used primarily LightGBM (should have spent more time on NN using various embeddings but didn't get a chance to implement due to busy schedule (laziness indeed LOL).\n\n# Feature selection\nInspired by @Yifan Xie's post (https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/75935), I started exploring feature selections and after playing different strategies I ended up with RFE + LightGBM (random forest booster) which gave a decent boost about 0.006+.\n\nI then developed a bit crazy (dummy) stackings using similar RFE framework but different parameters and the number of features selected are between 60-100 and each ended up with comparable results. However, stacking of them didn't make any advances on LB.\n\n# Postprocessing\nJust like most of top, and probably bottom rankers - and I'm sorry for the latter - I applied post-processing using a 3-models solution originally proposed by 李胃疼疼疼 (https://www.kaggle.com/waitingli/combining-your-model-with-a-model-without-outlier). Given that the original solution didn't seem to be able to allow us to perform CV so I modified it a bit particularly on the model without outliers: instead of predicting on test data only I made it generate out-of-fold predictions on training data too, which was quite similar to stacking with a small trick which allows it to make predictions on \"all\" training data rather than \"without outliers\" training data.\n\nThis allows me to test out the best threshold instead of having to guess. In addition, instead of using predictions from the model trained with all data for those samples with p(outlier)> threshold, I used the average of predictions from both models (all and without outliers), which seemed to be more stable in the end. (0.003+)\n\n# Blending\n\nI had to confess that my final submissions were blended with some of the outputs from public kernels, which gave me a boost of 0.001~.\n\nI have been enjoying the competitions and shared the same self-doubting and frustrating moments as many of you may have had. In the end, it's not only the PB position, nor the medal, but **these hard working moments** made us stronger than what we were at the beginning of this competition.\n\nCheers,\n\nChris,", "votes": null }, { "id": "479348", "postDate": "02/27/2019 04:51:41", "content": "Congratulations @ChrisCC !\nI too went through the same methodology but alas! I failed to trust my cv. Slipped Public Lb 190 to Private LB 2700+ \nBut as you said, hard work prevails.
", "rawMarkdown": "Congratulations @ChrisCC !\nI too went through the same methodology but alas! I failed to trust my cv. Slipped Public Lb 190 to Private LB 2700+ \nBut as you said, hard work prevails.", "votes": null }, { "id": "479349", "postDate": "02/27/2019 04:51:46", "content": "Thank you for post! Congraz!
", "rawMarkdown": "Thank you for post! Congraz!", "votes": null }, { "id": "479380", "postDate": "02/27/2019 05:24:36", "content": "Congrats to you on your medal !!
", "rawMarkdown": "Congrats to you on your medal !!", "votes": null }, { "id": "479500", "postDate": "02/27/2019 07:44:54", "content": "Congratulations @ChrisCC
", "rawMarkdown": "Congratulations @ChrisCC", "votes": null }, { "id": "479652", "postDate": "02/27/2019 10:10:08", "content": "73 on public and 2361 on private :c
", "rawMarkdown": "73 on public and 2361 on private :c", "votes": null }, { "id": "479654", "postDate": "02/27/2019 10:12:32", "content": "Congratulations, and what is adding an additional ridge stacking layer on top of those sparse features? Taking the output as the input of lgb? and how to do ridge stacking on tf-idf?
Thanks for sharing,\nGood job!!
", "rawMarkdown": "Thanks for sharing,\nGood job!!", "votes": null } ], "comments": [ { "id": 479348, "author_name": "rajraviprajapat", "author_url": "", "post_date": "02/27/2019 04:51:41", "content": "Congratulations @ChrisCC !\nI too went through the same methodology but alas! I failed to trust my cv. Slipped Public Lb 190 to Private LB 2700+ \nBut as you said, hard work prevails.
", "votes": null, "replies": [ { "id": 479652, "author_name": "l0glikelihood", "author_url": "", "post_date": "02/27/2019 10:10:08", "content": "73 on public and 2361 on private :c
", "votes": null, "replies": [] } ] }, { "id": 479349, "author_name": "rooshroosh", "author_url": "", "post_date": "02/27/2019 04:51:46", "content": "Thank you for post! Congraz!
", "votes": null, "replies": [] }, { "id": 479380, "author_name": "viswanathravindran", "author_url": "", "post_date": "02/27/2019 05:24:36", "content": "Congrats to you on your medal !!
", "votes": null, "replies": [] }, { "id": 479500, "author_name": "karthik7395", "author_url": "", "post_date": "02/27/2019 07:44:54", "content": "Congratulations @ChrisCC
", "votes": null, "replies": [] }, { "id": 479654, "author_name": "liewmanchoi", "author_url": "", "post_date": "02/27/2019 10:12:32", "content": "Congratulations, and what is adding an additional ridge stacking layer on top of those sparse features? Taking the output as the input of lgb? and how to do ridge stacking on tf-idf?
Thanks for sharing,\nGood job!!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "479235": "Hi everyone, congrats all medal winners - you all well deserve the rewards for this exciting competition!!!\n\n I’d also like to thank my teammate rcarson. Sorry buddy we didn’t do very well this time but let’s makenit better next time!!\n\nWhile there's nothing to boast as a 55th, I do think there could be something in my solution/experience that might interest you. So there you go:\n\n# Feature engineering\n\nMost of the features were derived from Chau Huynh's 3.699 public LB kernel (https://www.kaggle.com/chauhuynh/my-first-kernel-3-699) with minor modifications. I had tried quite a lot of feature engineerings but only a few ended up being helpful including:\n\n1. Transaction counts by month lag. This is probably, as many of you\n may have been aware, the most significant engineered feature.\n (0.005+) \n2. Ridge stackings on tf-idf of a variety of entities count\n by card id (historical transactions only). e.g. merchant id,\n merchant category id, and combinations of them. In fact, adding those features directly to the model didn't seem to help, nor more complicated embeddings such as word2vec and fasttext. However, adding an additional ridge stacking layer on top of those sparse features yielded considerable improvements, both on public board and private board. (0.002+).\n3. Similarities calculated between historical and new transactions using word2vec and fasttext gave\n minor improvements as well. (0.001+)\n\n# Modeling\n\nI used primarily LightGBM (should have spent more time on NN using various embeddings but didn't get a chance to implement due to busy schedule (laziness indeed LOL).\n\n# Feature selection\nInspired by @Yifan Xie's post (https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/75935), I started exploring feature selections and after playing different strategies I ended up with RFE + LightGBM (random forest booster) which gave a decent boost about 0.006+.\n\nI then developed a bit crazy (dummy) stackings using similar RFE framework but different parameters and the number of features selected are between 60-100 and each ended up with comparable results. However, stacking of them didn't make any advances on LB.\n\n# Postprocessing\nJust like most of top, and probably bottom rankers - and I'm sorry for the latter - I applied post-processing using a 3-models solution originally proposed by 李胃疼疼疼 (https://www.kaggle.com/waitingli/combining-your-model-with-a-model-without-outlier). Given that the original solution didn't seem to be able to allow us to perform CV so I modified it a bit particularly on the model without outliers: instead of predicting on test data only I made it generate out-of-fold predictions on training data too, which was quite similar to stacking with a small trick which allows it to make predictions on \"all\" training data rather than \"without outliers\" training data.\n\nThis allows me to test out the best threshold instead of having to guess. In addition, instead of using predictions from the model trained with all data for those samples with p(outlier)> threshold, I used the average of predictions from both models (all and without outliers), which seemed to be more stable in the end. (0.003+)\n\n# Blending\n\nI had to confess that my final submissions were blended with some of the outputs from public kernels, which gave me a boost of 0.001~.\n\nI have been enjoying the competitions and shared the same self-doubting and frustrating moments as many of you may have had. In the end, it's not only the PB position, nor the medal, but **these hard working moments** made us stronger than what we were at the beginning of this competition.\n\nCheers,\n\nChris,", "479348": "Congratulations @ChrisCC !\nI too went through the same methodology but alas! I failed to trust my cv. Slipped Public Lb 190 to Private LB 2700+ \nBut as you said, hard work prevails.", "479349": "Thank you for post! Congraz!", "479380": "Congrats to you on your medal !!", "479500": "Congratulations @ChrisCC", "479652": "73 on public and 2361 on private :c", "479654": "Congratulations, and what is `adding an additional ridge stacking layer on top of those sparse features `? Taking the output as the input of lgb? and how to do ridge stacking on tf-idf?", "479690": "Thanks for sharing,\nGood job!!" }, "source": "existing_discussion", "complete": true, "competition_id": 10445, "competition_slug": "elo-merchant-category-recommendation", "team_id": 2441547, "solution_url": "/competitions/elo-merchant-category-recommendation/writeups/rapids-ai-55th-solution-sharing" }