{
  "id": 82127,
  "title": "11th place solution",
  "url": "/competitions/elo-merchant-category-recommendation/writeups/stack-it-all-11th-place-solution",
  "author_name": "",
  "post_date": "2019-02-27T10:58:29.923782500Z",
  "votes": 52,
  "comment_count": 20,
  "views": 0,
  "content": "<p>This is the 11th place solution writeup. </p>\n\n<p>Hello, I would like to congratulate everyone for the efforts they put into this competition. And ELO for hosting such a great competition with interesting data and hair pulling outliers :D </p>\n\n<p>I would like to give special thanks to my teammates <a href=\"https://www.kaggle.com/kangzhang\">@woo</a> <a href=\"https://www.kaggle.com/nabilblk\">@Nabil</a> <a href=\"https://www.kaggle.com/a763337092\">@Lindada</a> for their skills, perseverance and hardwork. </p>\n\n<p>The huge gap between CV and LB did not make it easy for us (and for everyone I imagine). Those that persevered until the end would perfectly understand.  </p>\n\n<p>Congrats for all the medalists, you did a tremendous job :D </p>\n\n<h2><strong>SUMMARY</strong></h2>\n\n<p>So basically we were two separate teams that merged a couple of hours before the deadline. Each had had completed a tremendous amount of work, \nand after almost running out of new ideas, a merge was deemed necessary (and fruitful).   </p>\n\n<h2><strong>FEATURE ENGINEERING</strong></h2>\n\n<p>Like most public kernels, we constructed aggregation features. Here's a list of the our strongest ones: </p>\n\n<pre><code>1. I refer the Kaggle Rank System Compute Formula（link:[https://www.kaggle.com/progression][4])\n    df_data['duration_sqrt_counts'] = df_data['durations']/sqrt(df_data['card_id_counts'])\n    df_data['duration_log1p_counts'] = df_data['durations']/log1p(df_data['card_id_counts'])\n    df_data['duration_counts'] = df_data['durations']/df_data['card_id_counts']\n\n2. Categorical features: frequence, Maxfrequence, MaxfrequenceRatio\n\n3. card_id/merchant_id/mechant_category_id/city_id (visit sequence to sequence embedding)\n\n4. purchase_amount:hist/new\n\n5. features interactions between hist/new\n            df['purchase_amount_ratio_v3'] =                              df['new_purchase_amount_max']/df['hist_purchase_amount_sum']\n            df['purchase_amount_diff_v1'] = df['new_purchase_amount_sum']-df['hist_purchase_amount_sum']\n            df['purchase_amount_diff_v2'] = df['new_purchase_amount_mean']-df['hist_purchase_amount_mean']\n            df['purchase_amount_diff_v3'] = df['new_purchase_amount_max']-df['hist_purchase_amount_max']\n            df['purchase_amount_diff_v4'] = df['new_purchase_amount_min']-df['hist_purchase_amount_min']\n            df['pa_mlag_ratio'] = df['new_purchase_amount_sum']/(df['month_lag_mean'] - 1)\n            df['pa_new_hist_ratio'] = df['new_purchase_amount_sum']/(df['hist_purchase_amount_sum'])\n            df['pa_new_hist_mean_ratio'] = df['new_purchase_amount_mean']/(df['hist_purchase_amount_mean'] )\n            df['pa_new_hist_min_ratio'] = df['new_purchase_amount_min']/(df['hist_purchase_amount_min'] )\n            df['pa_new_hist_max_ratio'] = df['new_purchase_amount_max']/(df['hist_purchase_amount_max'] )\n</code></pre>\n\n<p>We had two separate feature sets. One with +1000 features and another one with +200 features</p>\n\n<p>At this point our best models scored around: CV 3.642X and LB: 3.688 and CV 3.644X and LB: 3.686</p>\n\n<p>After that, we took the correlation matrix of the +200 feature set and paired each feature with the feature it's the least correlated with. \nThen we applied a bunch of aggregations on each pair and it resulted in pretty strong features. </p>\n\n<p>So we ended up with two feature sets with +1000 features each. </p>\n\n<h2><strong>FEATURE SELECTION</strong></h2>\n\n<p>For feature selection we did some manual feature selection based on the features importance feedback we got from lgb. Then we used this (simple yet effective) <a href=\"https://towardsdatascience.com/a-feature-selection-tool-for-machine-learning-in-python-b64dd23710f0\">method here</a> for some further filtering. What it basically does is remove:</p>\n\n<ul>\n<li>Features with a high percentage of missing values according to a\nthreshold    </li>\n<li>Collinear (highly correlated) features   </li>\n<li>Features with zero importance in a tree-based model  </li>\n<li>Features with low importance</li>\n<li>Features with a single unique value</li>\n</ul>\n\n<p>The CV score got better in both feature sets. </p>\n\n<p>After this, our best models scored around CV:3.639X LB:3.682</p>\n\n<h2><strong>MODELS</strong></h2>\n\n<p>We used different models for training: LightGBM / Catboost / Xgboost / H2oRF / H2oGBM. \nWe tried a couple of NN architectures but it didn't work out for us (Check the last section)</p>\n\n<h2><strong>STACKING</strong></h2>\n\n<p>We stacked around 32 models using bayesian regression. Our models were well varied that it yielded a score of CV:3.630X LB :3.675</p>\n\n<h2><strong>POST PROCESSING</strong></h2>\n\n<p>During the last day, we focused on doing some post processing and this is how we cherrypicked our outliers.</p>\n\n<p>we carefully designed a Classifying module that combined four different classifers.</p>\n\n<p>We picked the top 100 outliers predicted from every classifier and then intersected the four sets. This resulted in 21 final outliers. </p>\n\n<p>This gave us a devilish boost on LB. We went from 3.675 to 3.666 (See what I did there?)</p>\n\n<h2><strong>SUBMISSIONS</strong></h2>\n\n<p>Finally we chose:</p>\n\n<ul>\n<li>A model without post processing (Our best stacking sub): CV:3.63019 LB:3.675 and Private LB: 3.599</li>\n<li>A model with post processing: LB:3.666 and Private:3.599 </li>\n</ul>\n\n<h2><strong>STUFF THAT DID NOT WORK</strong></h2>\n\n<p>Of course these last two months were not all roses and rainbows. We pulled our hair trying a lot of things and we failed miserably.</p>\n\n<p>Here are the bloopers of our participation :D : </p>\n\n<ul>\n<li>NN. We tried designing different architectures with the main focus on having a simple NN with heavy regularization (BatchNorm and Strong Dropout)</li>\n<li>In the middle of the competition, we tried tackling the outliers detection as an anomaly detection problem using AutoEncoders trained only on the non outliers data </li>\n<li>We tried PCA for more features. And it didn't work</li>\n<li>We tried TSNE. It didn't work  </li>\n<li>We tried FM and FFM. It did not work</li>\n<li>We tried isolation forest. Nope. Did not work. </li>\n<li>We had a Ridge-based pairwise ranker that we intended to use for outliers detection but it didn't match with the approach we had.</li>\n<li>We tried a lot of weak models in the hope of adding diversity (simple tree-based, linear, svm, etc.). And guess what? It did not work.</li>\n</ul>\n\n<p>Anyway, that was a rough wrap up of all the things we did. </p>\n\n<p>Thank you for sparing the time to read this. </p>\n\n<p>And remember to always trust CV :D </p>",
  "messages": [
    {
      "id": "479703",
      "postDate": "02/27/2019 10:58:29",
      "content": "<p>This is the 11th place solution writeup. </p>\n\n<p>Hello, I would like to congratulate everyone for the efforts they put into this competition. And ELO for hosting such a great competition with interesting data and hair pulling outliers :D </p>\n\n<p>I would like to give special thanks to my teammates <a href=\"https://www.kaggle.com/kangzhang\">@woo</a> <a href=\"https://www.kaggle.com/nabilblk\">@Nabil</a> <a href=\"https://www.kaggle.com/a763337092\">@Lindada</a> for their skills, perseverance and hardwork. </p>\n\n<p>The huge gap between CV and LB did not make it easy for us (and for everyone I imagine). Those that persevered until the end would perfectly understand.  </p>\n\n<p>Congrats for all the medalists, you did a tremendous job :D </p>\n\n<h2><strong>SUMMARY</strong></h2>\n\n<p>So basically we were two separate teams that merged a couple of hours before the deadline. Each had had completed a tremendous amount of work, \nand after almost running out of new ideas, a merge was deemed necessary (and fruitful).   </p>\n\n<h2><strong>FEATURE ENGINEERING</strong></h2>\n\n<p>Like most public kernels, we constructed aggregation features. Here's a list of the our strongest ones: </p>\n\n<pre><code>1. I refer the Kaggle Rank System Compute Formula（link:[https://www.kaggle.com/progression][4])\n    df_data['duration_sqrt_counts'] = df_data['durations']/sqrt(df_data['card_id_counts'])\n    df_data['duration_log1p_counts'] = df_data['durations']/log1p(df_data['card_id_counts'])\n    df_data['duration_counts'] = df_data['durations']/df_data['card_id_counts']\n\n2. Categorical features: frequence, Maxfrequence, MaxfrequenceRatio\n\n3. card_id/merchant_id/mechant_category_id/city_id (visit sequence to sequence embedding)\n\n4. purchase_amount:hist/new\n\n5. features interactions between hist/new\n            df['purchase_amount_ratio_v3'] =                              df['new_purchase_amount_max']/df['hist_purchase_amount_sum']\n            df['purchase_amount_diff_v1'] = df['new_purchase_amount_sum']-df['hist_purchase_amount_sum']\n            df['purchase_amount_diff_v2'] = df['new_purchase_amount_mean']-df['hist_purchase_amount_mean']\n            df['purchase_amount_diff_v3'] = df['new_purchase_amount_max']-df['hist_purchase_amount_max']\n            df['purchase_amount_diff_v4'] = df['new_purchase_amount_min']-df['hist_purchase_amount_min']\n            df['pa_mlag_ratio'] = df['new_purchase_amount_sum']/(df['month_lag_mean'] - 1)\n            df['pa_new_hist_ratio'] = df['new_purchase_amount_sum']/(df['hist_purchase_amount_sum'])\n            df['pa_new_hist_mean_ratio'] = df['new_purchase_amount_mean']/(df['hist_purchase_amount_mean'] )\n            df['pa_new_hist_min_ratio'] = df['new_purchase_amount_min']/(df['hist_purchase_amount_min'] )\n            df['pa_new_hist_max_ratio'] = df['new_purchase_amount_max']/(df['hist_purchase_amount_max'] )\n</code></pre>\n\n<p>We had two separate feature sets. One with +1000 features and another one with +200 features</p>\n\n<p>At this point our best models scored around: CV 3.642X and LB: 3.688 and CV 3.644X and LB: 3.686</p>\n\n<p>After that, we took the correlation matrix of the +200 feature set and paired each feature with the feature it's the least correlated with. \nThen we applied a bunch of aggregations on each pair and it resulted in pretty strong features. </p>\n\n<p>So we ended up with two feature sets with +1000 features each. </p>\n\n<h2><strong>FEATURE SELECTION</strong></h2>\n\n<p>For feature selection we did some manual feature selection based on the features importance feedback we got from lgb. Then we used this (simple yet effective) <a href=\"https://towardsdatascience.com/a-feature-selection-tool-for-machine-learning-in-python-b64dd23710f0\">method here</a> for some further filtering. What it basically does is remove:</p>\n\n<ul>\n<li>Features with a high percentage of missing values according to a\nthreshold    </li>\n<li>Collinear (highly correlated) features   </li>\n<li>Features with zero importance in a tree-based model  </li>\n<li>Features with low importance</li>\n<li>Features with a single unique value</li>\n</ul>\n\n<p>The CV score got better in both feature sets. </p>\n\n<p>After this, our best models scored around CV:3.639X LB:3.682</p>\n\n<h2><strong>MODELS</strong></h2>\n\n<p>We used different models for training: LightGBM / Catboost / Xgboost / H2oRF / H2oGBM. \nWe tried a couple of NN architectures but it didn't work out for us (Check the last section)</p>\n\n<h2><strong>STACKING</strong></h2>\n\n<p>We stacked around 32 models using bayesian regression. Our models were well varied that it yielded a score of CV:3.630X LB :3.675</p>\n\n<h2><strong>POST PROCESSING</strong></h2>\n\n<p>During the last day, we focused on doing some post processing and this is how we cherrypicked our outliers.</p>\n\n<p>we carefully designed a Classifying module that combined four different classifers.</p>\n\n<p>We picked the top 100 outliers predicted from every classifier and then intersected the four sets. This resulted in 21 final outliers. </p>\n\n<p>This gave us a devilish boost on LB. We went from 3.675 to 3.666 (See what I did there?)</p>\n\n<h2><strong>SUBMISSIONS</strong></h2>\n\n<p>Finally we chose:</p>\n\n<ul>\n<li>A model without post processing (Our best stacking sub): CV:3.63019 LB:3.675 and Private LB: 3.599</li>\n<li>A model with post processing: LB:3.666 and Private:3.599 </li>\n</ul>\n\n<h2><strong>STUFF THAT DID NOT WORK</strong></h2>\n\n<p>Of course these last two months were not all roses and rainbows. We pulled our hair trying a lot of things and we failed miserably.</p>\n\n<p>Here are the bloopers of our participation :D : </p>\n\n<ul>\n<li>NN. We tried designing different architectures with the main focus on having a simple NN with heavy regularization (BatchNorm and Strong Dropout)</li>\n<li>In the middle of the competition, we tried tackling the outliers detection as an anomaly detection problem using AutoEncoders trained only on the non outliers data </li>\n<li>We tried PCA for more features. And it didn't work</li>\n<li>We tried TSNE. It didn't work  </li>\n<li>We tried FM and FFM. It did not work</li>\n<li>We tried isolation forest. Nope. Did not work. </li>\n<li>We had a Ridge-based pairwise ranker that we intended to use for outliers detection but it didn't match with the approach we had.</li>\n<li>We tried a lot of weak models in the hope of adding diversity (simple tree-based, linear, svm, etc.). And guess what? It did not work.</li>\n</ul>\n\n<p>Anyway, that was a rough wrap up of all the things we did. </p>\n\n<p>Thank you for sparing the time to read this. </p>\n\n<p>And remember to always trust CV :D </p>",
      "rawMarkdown": "This is the 11th place solution writeup. \n\nHello, I would like to congratulate everyone for the efforts they put into this competition. And ELO for hosting such a great competition with interesting data and hair pulling outliers :D \n\nI would like to give special thanks to my teammates [@woo][1] [@Nabil][2] [@Lindada][3] for their skills, perseverance and hardwork. \n\nThe huge gap between CV and LB did not make it easy for us (and for everyone I imagine). Those that persevered until the end would perfectly understand.  \n\nCongrats for all the medalists, you did a tremendous job :D \n\n**SUMMARY**\n-----------\n\nSo basically we were two separate teams that merged a couple of hours before the deadline. Each had had completed a tremendous amount of work, \nand after almost running out of new ideas, a merge was deemed necessary (and fruitful).   \n\n\n**FEATURE ENGINEERING**\n-----------------------\n\nLike most public kernels, we constructed aggregation features. Here's a list of the our strongest ones: \n\n\t1. I refer the Kaggle Rank System Compute Formula（link:[https://www.kaggle.com/progression][4])\n\t\tdf_data['duration_sqrt_counts'] = df_data['durations']/sqrt(df_data['card_id_counts'])\n\t\tdf_data['duration_log1p_counts'] = df_data['durations']/log1p(df_data['card_id_counts'])\n\t\tdf_data['duration_counts'] = df_data['durations']/df_data['card_id_counts']\n\n\t2. Categorical features: frequence, Maxfrequence, MaxfrequenceRatio\n\n\t3. card_id/merchant_id/mechant_category_id/city_id (visit sequence to sequence embedding)\n\n\t4. purchase_amount:hist/new\n\n\t5. features interactions between hist/new\n                df['purchase_amount_ratio_v3'] =                              df['new_purchase_amount_max']/df['hist_purchase_amount_sum']\n                df['purchase_amount_diff_v1'] = df['new_purchase_amount_sum']-df['hist_purchase_amount_sum']\n                df['purchase_amount_diff_v2'] = df['new_purchase_amount_mean']-df['hist_purchase_amount_mean']\n                df['purchase_amount_diff_v3'] = df['new_purchase_amount_max']-df['hist_purchase_amount_max']\n                df['purchase_amount_diff_v4'] = df['new_purchase_amount_min']-df['hist_purchase_amount_min']\n                df['pa_mlag_ratio'] = df['new_purchase_amount_sum']/(df['month_lag_mean'] - 1)\n                df['pa_new_hist_ratio'] = df['new_purchase_amount_sum']/(df['hist_purchase_amount_sum'])\n                df['pa_new_hist_mean_ratio'] = df['new_purchase_amount_mean']/(df['hist_purchase_amount_mean'] )\n                df['pa_new_hist_min_ratio'] = df['new_purchase_amount_min']/(df['hist_purchase_amount_min'] )\n                df['pa_new_hist_max_ratio'] = df['new_purchase_amount_max']/(df['hist_purchase_amount_max'] )\n\nWe had two separate feature sets. One with +1000 features and another one with +200 features\n\nAt this point our best models scored around: CV 3.642X and LB: 3.688 and CV 3.644X and LB: 3.686\n\nAfter that, we took the correlation matrix of the +200 feature set and paired each feature with the feature it's the least correlated with. \nThen we applied a bunch of aggregations on each pair and it resulted in pretty strong features. \n\nSo we ended up with two feature sets with +1000 features each. \n\n**FEATURE SELECTION**\n---------------------\n\nFor feature selection we did some manual feature selection based on the features importance feedback we got from lgb. Then we used this (simple yet effective) [method here][4] for some further filtering. What it basically does is remove:\n\n - Features with a high percentage of missing values according to a\n   threshold \t\n - Collinear (highly correlated) features \t\n - Features with zero importance in a tree-based model \t\n - Features with low importance\n - Features with a single unique value\n\nThe CV score got better in both feature sets. \n\nAfter this, our best models scored around CV:3.639X LB:3.682\n\n**MODELS**\n----------\n\nWe used different models for training: LightGBM / Catboost / Xgboost / H2oRF / H2oGBM. \nWe tried a couple of NN architectures but it didn't work out for us (Check the last section)\n\n**STACKING**\n------------\n\nWe stacked around 32 models using bayesian regression. Our models were well varied that it yielded a score of CV:3.630X LB :3.675\n\n**POST PROCESSING**\n-------------------\n\nDuring the last day, we focused on doing some post processing and this is how we cherrypicked our outliers.\n\nwe carefully designed a Classifying module that combined four different classifers.\n\nWe picked the top 100 outliers predicted from every classifier and then intersected the four sets. This resulted in 21 final outliers. \n\nThis gave us a devilish boost on LB. We went from 3.675 to 3.666 (See what I did there?)\n\n**SUBMISSIONS**\n---------------\n\nFinally we chose:\n\n - A model without post processing (Our best stacking sub): CV:3.63019 LB:3.675 and Private LB: 3.599\n - A model with post processing: LB:3.666 and Private:3.599 \n\n**STUFF THAT DID NOT WORK**\n---------------------------\n\nOf course these last two months were not all roses and rainbows. We pulled our hair trying a lot of things and we failed miserably.\n\nHere are the bloopers of our participation :D : \n\t\n\n - NN. We tried designing different architectures with the main focus on having a simple NN with heavy regularization (BatchNorm and Strong Dropout)\n - In the middle of the competition, we tried tackling the outliers detection as an anomaly detection problem using AutoEncoders trained only on the non outliers data \n - We tried PCA for more features. And it didn't work\n - We tried TSNE. It didn't work  \n - We tried FM and FFM. It did not work\n - We tried isolation forest. Nope. Did not work. \n - We had a Ridge-based pairwise ranker that we intended to use for outliers detection but it didn't match with the approach we had.\n - We tried a lot of weak models in the hope of adding diversity (simple tree-based, linear, svm, etc.). And guess what? It did not work.\n\nAnyway, that was a rough wrap up of all the things we did. \n\nThank you for sparing the time to read this. \n\nAnd remember to always trust CV :D \n\n  [1]: https://www.kaggle.com/kangzhang\n  [2]: https://www.kaggle.com/nabilblk\n  [3]: https://www.kaggle.com/a763337092\n  [4]: https://towardsdatascience.com/a-feature-selection-tool-for-machine-learning-in-python-b64dd23710f0\n  [5]: https://www.kaggle.com/progression",
      "votes": null
    },
    {
      "id": "479719",
      "postDate": "02/27/2019 11:11:20",
      "content": "<p>Cooperate happily</p>",
      "rawMarkdown": "Cooperate happily",
      "votes": null
    },
    {
      "id": "479729",
      "postDate": "02/27/2019 11:19:21",
      "content": "<p>Congratulation for your work ! </p>",
      "rawMarkdown": "Congratulation for your work !",
      "votes": null
    },
    {
      "id": "479745",
      "postDate": "02/27/2019 11:43:27",
      "content": "<p>Congratulations <a href=\"/sc0d3r\">@sc0d3r</a>.</p>",
      "rawMarkdown": "Congratulations @sc0d3r.",
      "votes": null
    },
    {
      "id": "479750",
      "postDate": "02/27/2019 11:46:13",
      "content": "<p>Congratulation~~~</p>",
      "rawMarkdown": "Congratulation~~~",
      "votes": null
    },
    {
      "id": "479799",
      "postDate": "02/27/2019 12:39:12",
      "content": "<p>Congrats !</p>",
      "rawMarkdown": "Congrats !",
      "votes": null
    },
    {
      "id": "479823",
      "postDate": "02/27/2019 13:06:35",
      "content": "<p>Congrats <a href=\"/sc0d3r\">@sc0d3r</a> and team. Thanks for sharing your solution overview.</p>",
      "rawMarkdown": "Congrats @sc0d3r and team. Thanks for sharing your solution overview.",
      "votes": null
    },
    {
      "id": "479827",
      "postDate": "02/27/2019 13:19:26",
      "content": "<p>Congratulations on achieving such a good rank!</p>\n\n<p>I was curious on the part where you talked about combining correlated features with least correlated features. Could you describe more on this?</p>",
      "rawMarkdown": "Congratulations on achieving such a good rank!\n\nI was curious on the part where you talked about combining correlated features with least correlated features. Could you describe more on this?",
      "votes": null
    },
    {
      "id": "479894",
      "postDate": "02/27/2019 14:42:24",
      "content": "<p>Congrats!!</p>\n\n<p>Thanks for sharing.</p>",
      "rawMarkdown": "Congrats!!\n\nThanks for sharing.",
      "votes": null
    },
    {
      "id": "479905",
      "postDate": "02/27/2019 14:52:24",
      "content": "<p>Congrats. What a brilliant solution to learn from. </p>",
      "rawMarkdown": "Congrats. What a brilliant solution to learn from.",
      "votes": null
    },
    {
      "id": "479969",
      "postDate": "02/27/2019 16:22:14",
      "content": "<p>Congratulation zakaria and thank you for sharing, especially <code>FeatureSelector</code>. I'll try it!</p>",
      "rawMarkdown": "Congratulation zakaria and thank you for sharing, especially `FeatureSelector`. I'll try it!",
      "votes": null
    },
    {
      "id": "479984",
      "postDate": "02/27/2019 16:40:58",
      "content": "<p>Excellent work and congrats. How many fold did you choose for the single model? I found that by choose different number of folds, the CV score vary a lot (5-10cv, 3.648x-3=644x CV score).</p>",
      "rawMarkdown": "Excellent work and congrats. How many fold did you choose for the single model? I found that by choose different number of folds, the CV score vary a lot (5-10cv, 3.648x-3=644x CV score).",
      "votes": null
    },
    {
      "id": "479991",
      "postDate": "02/27/2019 16:52:24",
      "content": "<p>Congratulation！Thanks for sharing.</p>",
      "rawMarkdown": "Congratulation！Thanks for sharing.",
      "votes": null
    },
    {
      "id": "480298",
      "postDate": "02/28/2019 02:40:56",
      "content": "<p>Congrats. Nice works were done by you guys.</p>",
      "rawMarkdown": "Congrats. Nice works were done by you guys.",
      "votes": null
    },
    {
      "id": "480549",
      "postDate": "02/28/2019 10:21:05",
      "content": "<p>I did the same post processing method - let different models vote for outliers. It really worked for public LB. But it was bad for private LB.</p>",
      "rawMarkdown": "I did the same post processing method - let different models vote for outliers. It really worked for public LB. But it was bad for private LB.",
      "votes": null
    },
    {
      "id": "480637",
      "postDate": "02/28/2019 13:19:53",
      "content": "<p>Congrats Zakaria!</p>",
      "rawMarkdown": "Congrats Zakaria!",
      "votes": null
    },
    {
      "id": "480785",
      "postDate": "02/28/2019 17:03:32",
      "content": "<p>Congrats ans tks for sharing</p>",
      "rawMarkdown": "Congrats ans tks for sharing",
      "votes": null
    },
    {
      "id": "482073",
      "postDate": "03/02/2019 09:28:39",
      "content": "<p>That's an awesome summary and treasure trove of many things to try on future competitions. Thanks for sharing the great insights. :)</p>",
      "rawMarkdown": "That's an awesome summary and treasure trove of many things to try on future competitions. Thanks for sharing the great insights. :)",
      "votes": null
    },
    {
      "id": "482274",
      "postDate": "03/02/2019 16:09:42",
      "content": "<p>Congrats! Nice Work!</p>",
      "rawMarkdown": "Congrats! Nice Work!",
      "votes": null
    },
    {
      "id": "502682",
      "postDate": "03/28/2019 23:07:36",
      "content": "<p>Congratulations and thank you for sharing! May I ask where in the pipeline the outlier classifiers were applied? Is it before the 23 models, after or in some other manner?</p>",
      "rawMarkdown": "Congratulations and thank you for sharing! May I ask where in the pipeline the outlier classifiers were applied? Is it before the 23 models, after or in some other manner?",
      "votes": null
    },
    {
      "id": "739466",
      "postDate": "02/07/2020 21:07:42",
      "content": "<p><a href=\"/sc0d3r\">@sc0d3r</a>, <a href=\"/nabilblk\">@nabilblk</a>, <a href=\"/a763337092\">@a763337092</a>, <a href=\"/kangzhang\">@kangzhang</a>, congratulations! Thanks for sharing your approaches and magic! I added your post to my collection of the best kernels and posts of winners of Kaggle Prize Competitions: [Data Science for tabular data: Advanced Techniques] (<a href=\"https://www.kaggle.com/vbmokin/data-science-for-tabular-data-advanced-techniques\">https://www.kaggle.com/vbmokin/data-science-for-tabular-data-advanced-techniques</a>)</p>",
      "rawMarkdown": "sc0d3r, @nabilblk, @a763337092, @kangzhang, congratulations! Thanks for sharing your approaches and magic! I added your post to my collection of the best kernels and posts of winners of Kaggle Prize Competitions: [Data Science for tabular data: Advanced Techniques] (https://www.kaggle.com/vbmokin/data-science-for-tabular-data-advanced-techniques)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 479719,
      "author_name": "kangzhang",
      "author_url": "",
      "post_date": "02/27/2019 11:11:20",
      "content": "<p>Cooperate happily</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 479729,
      "author_name": "francoisdubois",
      "author_url": "",
      "post_date": "02/27/2019 11:19:21",
      "content": "<p>Congratulation for your work ! </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 479745,
      "author_name": "karthik7395",
      "author_url": "",
      "post_date": "02/27/2019 11:43:27",
      "content": "<p>Congratulations <a href=\"/sc0d3r\">@sc0d3r</a>.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 479750,
      "author_name": "peiyuan2019",
      "author_url": "",
      "post_date": "02/27/2019 11:46:13",
      "content": "<p>Congratulation~~~</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 479799,
      "author_name": "chanchalkumarmaji",
      "author_url": "",
      "post_date": "02/27/2019 12:39:12",
      "content": "<p>Congrats !</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 479823,
      "author_name": "sheriytm",
      "author_url": "",
      "post_date": "02/27/2019 13:06:35",
      "content": "<p>Congrats <a href=\"/sc0d3r\">@sc0d3r</a> and team. Thanks for sharing your solution overview.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 479827,
      "author_name": "salilm23",
      "author_url": "",
      "post_date": "02/27/2019 13:19:26",
      "content": "<p>Congratulations on achieving such a good rank!</p>\n\n<p>I was curious on the part where you talked about combining correlated features with least correlated features. Could you describe more on this?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 479894,
      "author_name": "sreeedevi",
      "author_url": "",
      "post_date": "02/27/2019 14:42:24",
      "content": "<p>Congrats!!</p>\n\n<p>Thanks for sharing.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 479905,
      "author_name": "pukkinming",
      "author_url": "",
      "post_date": "02/27/2019 14:52:24",
      "content": "<p>Congrats. What a brilliant solution to learn from. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 479969,
      "author_name": "toshikazuwatanabe",
      "author_url": "",
      "post_date": "02/27/2019 16:22:14",
      "content": "<p>Congratulation zakaria and thank you for sharing, especially <code>FeatureSelector</code>. I'll try it!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 479984,
      "author_name": "x8x8d3n9",
      "author_url": "",
      "post_date": "02/27/2019 16:40:58",
      "content": "<p>Excellent work and congrats. How many fold did you choose for the single model? I found that by choose different number of folds, the CV score vary a lot (5-10cv, 3.648x-3=644x CV score).</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 479991,
      "author_name": "liangruiqi",
      "author_url": "",
      "post_date": "02/27/2019 16:52:24",
      "content": "<p>Congratulation！Thanks for sharing.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 480298,
      "author_name": "kingychiu",
      "author_url": "",
      "post_date": "02/28/2019 02:40:56",
      "content": "<p>Congrats. Nice works were done by you guys.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 480549,
      "author_name": "xiaojiu1414",
      "author_url": "",
      "post_date": "02/28/2019 10:21:05",
      "content": "<p>I did the same post processing method - let different models vote for outliers. It really worked for public LB. But it was bad for private LB.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 480637,
      "author_name": "ishivinal",
      "author_url": "",
      "post_date": "02/28/2019 13:19:53",
      "content": "<p>Congrats Zakaria!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 480785,
      "author_name": "wmyymw",
      "author_url": "",
      "post_date": "02/28/2019 17:03:32",
      "content": "<p>Congrats ans tks for sharing</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 482073,
      "author_name": "yassinealouini",
      "author_url": "",
      "post_date": "03/02/2019 09:28:39",
      "content": "<p>That's an awesome summary and treasure trove of many things to try on future competitions. Thanks for sharing the great insights. :)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 482274,
      "author_name": "lmeazzini",
      "author_url": "",
      "post_date": "03/02/2019 16:09:42",
      "content": "<p>Congrats! Nice Work!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 502682,
      "author_name": "willatkagglecomp",
      "author_url": "",
      "post_date": "03/28/2019 23:07:36",
      "content": "<p>Congratulations and thank you for sharing! May I ask where in the pipeline the outlier classifiers were applied? Is it before the 23 models, after or in some other manner?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 739466,
      "author_name": "vbmokin",
      "author_url": "",
      "post_date": "02/07/2020 21:07:42",
      "content": "<p><a href=\"/sc0d3r\">@sc0d3r</a>, <a href=\"/nabilblk\">@nabilblk</a>, <a href=\"/a763337092\">@a763337092</a>, <a href=\"/kangzhang\">@kangzhang</a>, congratulations! Thanks for sharing your approaches and magic! I added your post to my collection of the best kernels and posts of winners of Kaggle Prize Competitions: [Data Science for tabular data: Advanced Techniques] (<a href=\"https://www.kaggle.com/vbmokin/data-science-for-tabular-data-advanced-techniques\">https://www.kaggle.com/vbmokin/data-science-for-tabular-data-advanced-techniques</a>)</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "479703": "This is the 11th place solution writeup. \n\nHello, I would like to congratulate everyone for the efforts they put into this competition. And ELO for hosting such a great competition with interesting data and hair pulling outliers :D \n\nI would like to give special thanks to my teammates [@woo][1] [@Nabil][2] [@Lindada][3] for their skills, perseverance and hardwork. \n\nThe huge gap between CV and LB did not make it easy for us (and for everyone I imagine). Those that persevered until the end would perfectly understand.  \n\nCongrats for all the medalists, you did a tremendous job :D \n\n**SUMMARY**\n-----------\n\nSo basically we were two separate teams that merged a couple of hours before the deadline. Each had had completed a tremendous amount of work, \nand after almost running out of new ideas, a merge was deemed necessary (and fruitful).   \n\n\n**FEATURE ENGINEERING**\n-----------------------\n\nLike most public kernels, we constructed aggregation features. Here's a list of the our strongest ones: \n\n\t1. I refer the Kaggle Rank System Compute Formula（link:[https://www.kaggle.com/progression][4])\n\t\tdf_data['duration_sqrt_counts'] = df_data['durations']/sqrt(df_data['card_id_counts'])\n\t\tdf_data['duration_log1p_counts'] = df_data['durations']/log1p(df_data['card_id_counts'])\n\t\tdf_data['duration_counts'] = df_data['durations']/df_data['card_id_counts']\n\n\t2. Categorical features: frequence, Maxfrequence, MaxfrequenceRatio\n\n\t3. card_id/merchant_id/mechant_category_id/city_id (visit sequence to sequence embedding)\n\n\t4. purchase_amount:hist/new\n\n\t5. features interactions between hist/new\n                df['purchase_amount_ratio_v3'] =                              df['new_purchase_amount_max']/df['hist_purchase_amount_sum']\n                df['purchase_amount_diff_v1'] = df['new_purchase_amount_sum']-df['hist_purchase_amount_sum']\n                df['purchase_amount_diff_v2'] = df['new_purchase_amount_mean']-df['hist_purchase_amount_mean']\n                df['purchase_amount_diff_v3'] = df['new_purchase_amount_max']-df['hist_purchase_amount_max']\n                df['purchase_amount_diff_v4'] = df['new_purchase_amount_min']-df['hist_purchase_amount_min']\n                df['pa_mlag_ratio'] = df['new_purchase_amount_sum']/(df['month_lag_mean'] - 1)\n                df['pa_new_hist_ratio'] = df['new_purchase_amount_sum']/(df['hist_purchase_amount_sum'])\n                df['pa_new_hist_mean_ratio'] = df['new_purchase_amount_mean']/(df['hist_purchase_amount_mean'] )\n                df['pa_new_hist_min_ratio'] = df['new_purchase_amount_min']/(df['hist_purchase_amount_min'] )\n                df['pa_new_hist_max_ratio'] = df['new_purchase_amount_max']/(df['hist_purchase_amount_max'] )\n\nWe had two separate feature sets. One with +1000 features and another one with +200 features\n\nAt this point our best models scored around: CV 3.642X and LB: 3.688 and CV 3.644X and LB: 3.686\n\nAfter that, we took the correlation matrix of the +200 feature set and paired each feature with the feature it's the least correlated with. \nThen we applied a bunch of aggregations on each pair and it resulted in pretty strong features. \n\nSo we ended up with two feature sets with +1000 features each. \n\n**FEATURE SELECTION**\n---------------------\n\nFor feature selection we did some manual feature selection based on the features importance feedback we got from lgb. Then we used this (simple yet effective) [method here][4] for some further filtering. What it basically does is remove:\n\n - Features with a high percentage of missing values according to a\n   threshold \t\n - Collinear (highly correlated) features \t\n - Features with zero importance in a tree-based model \t\n - Features with low importance\n - Features with a single unique value\n\nThe CV score got better in both feature sets. \n\nAfter this, our best models scored around CV:3.639X LB:3.682\n\n**MODELS**\n----------\n\nWe used different models for training: LightGBM / Catboost / Xgboost / H2oRF / H2oGBM. \nWe tried a couple of NN architectures but it didn't work out for us (Check the last section)\n\n**STACKING**\n------------\n\nWe stacked around 32 models using bayesian regression. Our models were well varied that it yielded a score of CV:3.630X LB :3.675\n\n**POST PROCESSING**\n-------------------\n\nDuring the last day, we focused on doing some post processing and this is how we cherrypicked our outliers.\n\nwe carefully designed a Classifying module that combined four different classifers.\n\nWe picked the top 100 outliers predicted from every classifier and then intersected the four sets. This resulted in 21 final outliers. \n\nThis gave us a devilish boost on LB. We went from 3.675 to 3.666 (See what I did there?)\n\n**SUBMISSIONS**\n---------------\n\nFinally we chose:\n\n - A model without post processing (Our best stacking sub): CV:3.63019 LB:3.675 and Private LB: 3.599\n - A model with post processing: LB:3.666 and Private:3.599 \n\n**STUFF THAT DID NOT WORK**\n---------------------------\n\nOf course these last two months were not all roses and rainbows. We pulled our hair trying a lot of things and we failed miserably.\n\nHere are the bloopers of our participation :D : \n\t\n\n - NN. We tried designing different architectures with the main focus on having a simple NN with heavy regularization (BatchNorm and Strong Dropout)\n - In the middle of the competition, we tried tackling the outliers detection as an anomaly detection problem using AutoEncoders trained only on the non outliers data \n - We tried PCA for more features. And it didn't work\n - We tried TSNE. It didn't work  \n - We tried FM and FFM. It did not work\n - We tried isolation forest. Nope. Did not work. \n - We had a Ridge-based pairwise ranker that we intended to use for outliers detection but it didn't match with the approach we had.\n - We tried a lot of weak models in the hope of adding diversity (simple tree-based, linear, svm, etc.). And guess what? It did not work.\n\nAnyway, that was a rough wrap up of all the things we did. \n\nThank you for sparing the time to read this. \n\nAnd remember to always trust CV :D \n\n  [1]: https://www.kaggle.com/kangzhang\n  [2]: https://www.kaggle.com/nabilblk\n  [3]: https://www.kaggle.com/a763337092\n  [4]: https://towardsdatascience.com/a-feature-selection-tool-for-machine-learning-in-python-b64dd23710f0\n  [5]: https://www.kaggle.com/progression",
    "479719": "Cooperate happily",
    "479729": "Congratulation for your work !",
    "479745": "Congratulations @sc0d3r.",
    "479750": "Congratulation~~~",
    "479799": "Congrats !",
    "479823": "Congrats @sc0d3r and team. Thanks for sharing your solution overview.",
    "479827": "Congratulations on achieving such a good rank!\n\nI was curious on the part where you talked about combining correlated features with least correlated features. Could you describe more on this?",
    "479894": "Congrats!!\n\nThanks for sharing.",
    "479905": "Congrats. What a brilliant solution to learn from.",
    "479969": "Congratulation zakaria and thank you for sharing, especially `FeatureSelector`. I'll try it!",
    "479984": "Excellent work and congrats. How many fold did you choose for the single model? I found that by choose different number of folds, the CV score vary a lot (5-10cv, 3.648x-3=644x CV score).",
    "479991": "Congratulation！Thanks for sharing.",
    "480298": "Congrats. Nice works were done by you guys.",
    "480549": "I did the same post processing method - let different models vote for outliers. It really worked for public LB. But it was bad for private LB.",
    "480637": "Congrats Zakaria!",
    "480785": "Congrats ans tks for sharing",
    "482073": "That's an awesome summary and treasure trove of many things to try on future competitions. Thanks for sharing the great insights. :)",
    "482274": "Congrats! Nice Work!",
    "502682": "Congratulations and thank you for sharing! May I ask where in the pipeline the outlier classifiers were applied? Is it before the 23 models, after or in some other manner?",
    "739466": "sc0d3r, @nabilblk, @a763337092, @kangzhang, congratulations! Thanks for sharing your approaches and magic! I added your post to my collection of the best kernels and posts of winners of Kaggle Prize Competitions: [Data Science for tabular data: Advanced Techniques] (https://www.kaggle.com/vbmokin/data-science-for-tabular-data-advanced-techniques)"
  },
  "source": "meta"
}