{
  "id": 82166,
  "title": "16th Place Summary ",
  "url": "/competitions/elo-merchant-category-recommendation/writeups/nlgn-16th-place-summary",
  "author_name": "",
  "post_date": "2019-02-27T17:40:22.510Z",
  "votes": 43,
  "comment_count": 15,
  "views": 0,
  "content": "<h2>Feature Selection</h2>\n\n<p><strong>Just before training</strong></p>\n\n<p>I used the code from here: <a href=\"https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/77537\">https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/77537</a></p>\n\n<p>It is to remove features with different distribution on train and test set.</p>\n\n<p><strong>After training</strong></p>\n\n<p>I used the null selection idea (modified).\n<a href=\"https://www.kaggle.com/ogrellier/feature-selection-with-null-importances\">https://www.kaggle.com/ogrellier/feature-selection-with-null-importances</a></p>\n\n<p>It is basically a permutation test to adjust the features ranking. Trying to avoid features that work so well on random data.</p>\n\n<p><strong>After getting the features ranking</strong></p>\n\n<p>I cut the features with a loop to see what feature size works best. i.e. 90%, 80% .... 10% of features.</p>\n\n<h2>Modified 3-model Model</h2>\n\n<p><img src=\"https://i.imgur.com/6otyev8.png\" alt=\"enter image description here\"></p>\n\n<p><strong>Cross-validating the path of the outlier detection -&gt; model without outliers:</strong></p>\n\n<p>For the original public kernel (training with the manually selected non-outliers (&gt;-33), If the outlier classifier is not good enough, the CV&amp;PB error from the \"model without outliers\" will be huge.</p>\n\n<p>That's why I came up with a (relatively more stable but with less improvement) modified stacking method to simulate the error \"inheriting\" from the outlier classifier to the \"model without outliers\".</p>\n\n<p>I used the predicted non-outliers to train the model without outliers. So that the local cross validation covers the outliers prediction of the test set:</p>\n\n<p><code>\nall_train_df['outliers'] = all_train_df['prediction'] &amp;gt;= all_train_df['prediction'].quantile(0.9)\n</code></p>\n\n<p><code>\nall_test_df['outliers'] = all_test_df['prediction'] &amp;gt;= all_train_df['prediction'].quantile(0.9)\n</code></p>\n\n<h2>Blending</h2>\n\n<p><img src=\"https://i.imgur.com/xTZjWms.png\" alt=\"enter image description here\"></p>\n\n<p>The final blending consists of 16 models. They are single models, 3-model models, outlier detection models and 2 public kernels.</p>\n\n<h2>Models Summary</h2>\n\n<p><strong>Best Single Model</strong>: CV: 3.63964, Public: 3.690, Private: 3.606</p>\n\n<p><strong>Best 3-model Model</strong>: CV: 3.63469, Public: 3.685, Private: 3.604</p>",
  "messages": [
    {
      "id": "479945",
      "postDate": "02/27/2019 15:49:56",
      "content": "<h2>Feature Selection</h2>\n\n<p><strong>Just before training</strong></p>\n\n<p>I used the code from here: <a href=\"https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/77537\">https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/77537</a></p>\n\n<p>It is to remove features with different distribution on train and test set.</p>\n\n<p><strong>After training</strong></p>\n\n<p>I used the null selection idea (modified).\n<a href=\"https://www.kaggle.com/ogrellier/feature-selection-with-null-importances\">https://www.kaggle.com/ogrellier/feature-selection-with-null-importances</a></p>\n\n<p>It is basically a permutation test to adjust the features ranking. Trying to avoid features that work so well on random data.</p>\n\n<p><strong>After getting the features ranking</strong></p>\n\n<p>I cut the features with a loop to see what feature size works best. i.e. 90%, 80% .... 10% of features.</p>\n\n<h2>Modified 3-model Model</h2>\n\n<p><img src=\"https://i.imgur.com/6otyev8.png\" alt=\"enter image description here\"></p>\n\n<p><strong>Cross-validating the path of the outlier detection -&gt; model without outliers:</strong></p>\n\n<p>For the original public kernel (training with the manually selected non-outliers (&gt;-33), If the outlier classifier is not good enough, the CV&amp;PB error from the \"model without outliers\" will be huge.</p>\n\n<p>That's why I came up with a (relatively more stable but with less improvement) modified stacking method to simulate the error \"inheriting\" from the outlier classifier to the \"model without outliers\".</p>\n\n<p>I used the predicted non-outliers to train the model without outliers. So that the local cross validation covers the outliers prediction of the test set:</p>\n\n<p><code>\nall_train_df['outliers'] = all_train_df['prediction'] &amp;gt;= all_train_df['prediction'].quantile(0.9)\n</code></p>\n\n<p><code>\nall_test_df['outliers'] = all_test_df['prediction'] &amp;gt;= all_train_df['prediction'].quantile(0.9)\n</code></p>\n\n<h2>Blending</h2>\n\n<p><img src=\"https://i.imgur.com/xTZjWms.png\" alt=\"enter image description here\"></p>\n\n<p>The final blending consists of 16 models. They are single models, 3-model models, outlier detection models and 2 public kernels.</p>\n\n<h2>Models Summary</h2>\n\n<p><strong>Best Single Model</strong>: CV: 3.63964, Public: 3.690, Private: 3.606</p>\n\n<p><strong>Best 3-model Model</strong>: CV: 3.63469, Public: 3.685, Private: 3.604</p>",
      "rawMarkdown": "## Feature Selection ##\n\n**Just before training**\n\nI used the code from here: https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/77537\n\nIt is to remove features with different distribution on train and test set.\n\n**After training**\n\nI used the null selection idea (modified).\nhttps://www.kaggle.com/ogrellier/feature-selection-with-null-importances\n\nIt is basically a permutation test to adjust the features ranking. Trying to avoid features that work so well on random data.\n\n**After getting the features ranking**\n\nI cut the features with a loop to see what feature size works best. i.e. 90%, 80% .... 10% of features.\n\n\n## Modified 3-model Model ##\n\n![enter image description here][1]\n\n\n**Cross-validating the path of the outlier detection -&gt; model without outliers:**\n\nFor the original public kernel (training with the manually selected non-outliers (&gt;-33), If the outlier classifier is not good enough, the CV&amp;PB error from the \"model without outliers\" will be huge.\n\nThat's why I came up with a (relatively more stable but with less improvement) modified stacking method to simulate the error \"inheriting\" from the outlier classifier to the \"model without outliers\".\n\nI used the predicted non-outliers to train the model without outliers. So that the local cross validation covers the outliers prediction of the test set:\n\n```\nall_train_df['outliers'] = all_train_df['prediction'] &gt;= all_train_df['prediction'].quantile(0.9)\n```\n\n```\nall_test_df['outliers'] = all_test_df['prediction'] &gt;= all_train_df['prediction'].quantile(0.9)\n```\n\n\n## Blending ##\n![enter image description here][2]\n\nThe final blending consists of 16 models. They are single models, 3-model models, outlier detection models and 2 public kernels.\n\n## Models Summary ##\n\n**Best Single Model**: CV: 3.63964, Public: 3.690, Private: 3.606\n\n**Best 3-model Model**: CV: 3.63469, Public: 3.685, Private: 3.604\n\n\n  [1]: https://i.imgur.com/6otyev8.png\n  [2]: https://i.imgur.com/xTZjWms.png",
      "votes": null
    },
    {
      "id": "480013",
      "postDate": "02/27/2019 17:20:54",
      "content": "<p>Congrats for the result, and thanks for sharing this nice solution!</p>",
      "rawMarkdown": "Congrats for the result, and thanks for sharing this nice solution!",
      "votes": null
    },
    {
      "id": "480037",
      "postDate": "02/27/2019 17:54:10",
      "content": "<p>Thanks so much for your kind comment. </p>",
      "rawMarkdown": "Thanks so much for your kind comment.",
      "votes": null
    },
    {
      "id": "480057",
      "postDate": "02/27/2019 18:24:52",
      "content": "<p>great work <a href=\"/kingychiu\">@kingychiu</a>. Thanks for posting the summary of your approach.</p>",
      "rawMarkdown": "great work @kingychiu. Thanks for posting the summary of your approach.",
      "votes": null
    },
    {
      "id": "480272",
      "postDate": "02/28/2019 02:08:56",
      "content": "<p>Congrats! I have a question that in 3-model blending, length of the output of regressor(trained on non-outliers) is not equal to that of single regressor(trained on full dataset), so how to blengding them together?</p>",
      "rawMarkdown": "Congrats! I have a question that in 3-model blending, length of the output of regressor(trained on non-outliers) is not equal to that of single regressor(trained on full dataset), so how to blengding them together?",
      "votes": null
    },
    {
      "id": "480276",
      "postDate": "02/28/2019 02:16:59",
      "content": "<p>and how to decide the threhold of the outlier classifier?</p>",
      "rawMarkdown": "and how to decide the threhold of the outlier classifier?",
      "votes": null
    },
    {
      "id": "480283",
      "postDate": "02/28/2019 02:28:07",
      "content": "<p>For predicted outliers: the output will be just from the regression trained on all samples. I wasn't doing any post processing here, because my outlier detector wasn't super good.</p>\n\n<p>For predicted non-outliers: the output will be the naive average, it can be better like using a Ridge etc...</p>",
      "rawMarkdown": "For predicted outliers: the output will be just from the regression trained on all samples. I wasn't doing any post processing here, because my outlier detector wasn't super good.\n\nFor predicted non-outliers: the output will be the naive average, it can be better like using a Ridge etc...",
      "votes": null
    },
    {
      "id": "480284",
      "postDate": "02/28/2019 02:28:13",
      "content": "<p>By looking at the CV. I didn't spend much time adjusting it actually.</p>",
      "rawMarkdown": "By looking at the CV. I didn't spend much time adjusting it actually.",
      "votes": null
    },
    {
      "id": "480438",
      "postDate": "02/28/2019 07:11:26",
      "content": "<p>Fantastic, Congratulations!</p>",
      "rawMarkdown": "Fantastic, Congratulations!",
      "votes": null
    },
    {
      "id": "480475",
      "postDate": "02/28/2019 08:25:48",
      "content": "<p>Thanks for your reply, and I'm fully understand it, sorry to bother you for my misunderstand.</p>",
      "rawMarkdown": "Thanks for your reply, and I'm fully understand it, sorry to bother you for my misunderstand.",
      "votes": null
    },
    {
      "id": "480483",
      "postDate": "02/28/2019 08:31:45",
      "content": "<p>Congratulations,  <a href=\"/kingychiu\">@kingychiu</a> ! Thank you for sharing your solution. </p>",
      "rawMarkdown": "Congratulations,  @kingychiu ! Thank you for sharing your solution.",
      "votes": null
    },
    {
      "id": "480972",
      "postDate": "03/01/2019 00:09:06",
      "content": "<p>My bad, I didn't explain clearly that part.</p>",
      "rawMarkdown": "My bad, I didn't explain clearly that part.",
      "votes": null
    },
    {
      "id": "482025",
      "postDate": "03/02/2019 07:54:42",
      "content": "<p>Congrats <a href=\"/kingychiu\">@kingychiu</a> for a strong finish and thanks for sharing your solution overview.</p>",
      "rawMarkdown": "Congrats @kingychiu for a strong finish and thanks for sharing your solution overview.",
      "votes": null
    },
    {
      "id": "483691",
      "postDate": "03/05/2019 01:48:01",
      "content": "<p>Great work !! Congratulations <a href=\"/kingychiu\">@kingychiu</a> </p>",
      "rawMarkdown": "Great work !! Congratulations @kingychiu",
      "votes": null
    },
    {
      "id": "529085",
      "postDate": "05/09/2019 06:55:58",
      "content": "<p>congratulations! awesome work! </p>",
      "rawMarkdown": "congratulations! awesome work!",
      "votes": null
    },
    {
      "id": "739467",
      "postDate": "02/07/2020 21:08:24",
      "content": "<p><a href=\"/kingychiu\">@kingychiu</a>, congratulations! Thanks for sharing your approaches and magic! I added your post to my collection of the best kernels and posts of winners of Kaggle Prize Competitions: [Data Science for tabular data: Advanced Techniques] (<a href=\"https://www.kaggle.com/vbmokin/data-science-for-tabular-data-advanced-techniques\">https://www.kaggle.com/vbmokin/data-science-for-tabular-data-advanced-techniques</a>) </p>",
      "rawMarkdown": "kingychiu, congratulations! Thanks for sharing your approaches and magic! I added your post to my collection of the best kernels and posts of winners of Kaggle Prize Competitions: [Data Science for tabular data: Advanced Techniques] (https://www.kaggle.com/vbmokin/data-science-for-tabular-data-advanced-techniques)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 480013,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "02/27/2019 17:20:54",
      "content": "<p>Congrats for the result, and thanks for sharing this nice solution!</p>",
      "votes": null,
      "replies": [
        {
          "id": 480037,
          "author_name": "kingychiu",
          "author_url": "",
          "post_date": "02/27/2019 17:54:10",
          "content": "<p>Thanks so much for your kind comment. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 480057,
      "author_name": "karthik7395",
      "author_url": "",
      "post_date": "02/27/2019 18:24:52",
      "content": "<p>great work <a href=\"/kingychiu\">@kingychiu</a>. Thanks for posting the summary of your approach.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 480272,
      "author_name": "liewmanchoi",
      "author_url": "",
      "post_date": "02/28/2019 02:08:56",
      "content": "<p>Congrats! I have a question that in 3-model blending, length of the output of regressor(trained on non-outliers) is not equal to that of single regressor(trained on full dataset), so how to blengding them together?</p>",
      "votes": null,
      "replies": [
        {
          "id": 480283,
          "author_name": "kingychiu",
          "author_url": "",
          "post_date": "02/28/2019 02:28:07",
          "content": "<p>For predicted outliers: the output will be just from the regression trained on all samples. I wasn't doing any post processing here, because my outlier detector wasn't super good.</p>\n\n<p>For predicted non-outliers: the output will be the naive average, it can be better like using a Ridge etc...</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 480475,
          "author_name": "liewmanchoi",
          "author_url": "",
          "post_date": "02/28/2019 08:25:48",
          "content": "<p>Thanks for your reply, and I'm fully understand it, sorry to bother you for my misunderstand.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 480972,
          "author_name": "kingychiu",
          "author_url": "",
          "post_date": "03/01/2019 00:09:06",
          "content": "<p>My bad, I didn't explain clearly that part.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 480276,
      "author_name": "liewmanchoi",
      "author_url": "",
      "post_date": "02/28/2019 02:16:59",
      "content": "<p>and how to decide the threhold of the outlier classifier?</p>",
      "votes": null,
      "replies": [
        {
          "id": 480284,
          "author_name": "kingychiu",
          "author_url": "",
          "post_date": "02/28/2019 02:28:13",
          "content": "<p>By looking at the CV. I didn't spend much time adjusting it actually.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 480438,
      "author_name": "abhay31",
      "author_url": "",
      "post_date": "02/28/2019 07:11:26",
      "content": "<p>Fantastic, Congratulations!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 480483,
      "author_name": "gpreda",
      "author_url": "",
      "post_date": "02/28/2019 08:31:45",
      "content": "<p>Congratulations,  <a href=\"/kingychiu\">@kingychiu</a> ! Thank you for sharing your solution. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 482025,
      "author_name": "sheriytm",
      "author_url": "",
      "post_date": "03/02/2019 07:54:42",
      "content": "<p>Congrats <a href=\"/kingychiu\">@kingychiu</a> for a strong finish and thanks for sharing your solution overview.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 483691,
      "author_name": "ranjanhunt",
      "author_url": "",
      "post_date": "03/05/2019 01:48:01",
      "content": "<p>Great work !! Congratulations <a href=\"/kingychiu\">@kingychiu</a> </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 529085,
      "author_name": "lianyhai",
      "author_url": "",
      "post_date": "05/09/2019 06:55:58",
      "content": "<p>congratulations! awesome work! </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 739467,
      "author_name": "vbmokin",
      "author_url": "",
      "post_date": "02/07/2020 21:08:24",
      "content": "<p><a href=\"/kingychiu\">@kingychiu</a>, congratulations! Thanks for sharing your approaches and magic! I added your post to my collection of the best kernels and posts of winners of Kaggle Prize Competitions: [Data Science for tabular data: Advanced Techniques] (<a href=\"https://www.kaggle.com/vbmokin/data-science-for-tabular-data-advanced-techniques\">https://www.kaggle.com/vbmokin/data-science-for-tabular-data-advanced-techniques</a>) </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "479945": "## Feature Selection ##\n\n**Just before training**\n\nI used the code from here: https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/77537\n\nIt is to remove features with different distribution on train and test set.\n\n**After training**\n\nI used the null selection idea (modified).\nhttps://www.kaggle.com/ogrellier/feature-selection-with-null-importances\n\nIt is basically a permutation test to adjust the features ranking. Trying to avoid features that work so well on random data.\n\n**After getting the features ranking**\n\nI cut the features with a loop to see what feature size works best. i.e. 90%, 80% .... 10% of features.\n\n\n## Modified 3-model Model ##\n\n![enter image description here][1]\n\n\n**Cross-validating the path of the outlier detection -&gt; model without outliers:**\n\nFor the original public kernel (training with the manually selected non-outliers (&gt;-33), If the outlier classifier is not good enough, the CV&amp;PB error from the \"model without outliers\" will be huge.\n\nThat's why I came up with a (relatively more stable but with less improvement) modified stacking method to simulate the error \"inheriting\" from the outlier classifier to the \"model without outliers\".\n\nI used the predicted non-outliers to train the model without outliers. So that the local cross validation covers the outliers prediction of the test set:\n\n```\nall_train_df['outliers'] = all_train_df['prediction'] &gt;= all_train_df['prediction'].quantile(0.9)\n```\n\n```\nall_test_df['outliers'] = all_test_df['prediction'] &gt;= all_train_df['prediction'].quantile(0.9)\n```\n\n\n## Blending ##\n![enter image description here][2]\n\nThe final blending consists of 16 models. They are single models, 3-model models, outlier detection models and 2 public kernels.\n\n## Models Summary ##\n\n**Best Single Model**: CV: 3.63964, Public: 3.690, Private: 3.606\n\n**Best 3-model Model**: CV: 3.63469, Public: 3.685, Private: 3.604\n\n\n  [1]: https://i.imgur.com/6otyev8.png\n  [2]: https://i.imgur.com/xTZjWms.png",
    "480013": "Congrats for the result, and thanks for sharing this nice solution!",
    "480037": "Thanks so much for your kind comment.",
    "480057": "great work @kingychiu. Thanks for posting the summary of your approach.",
    "480272": "Congrats! I have a question that in 3-model blending, length of the output of regressor(trained on non-outliers) is not equal to that of single regressor(trained on full dataset), so how to blengding them together?",
    "480276": "and how to decide the threhold of the outlier classifier?",
    "480283": "For predicted outliers: the output will be just from the regression trained on all samples. I wasn't doing any post processing here, because my outlier detector wasn't super good.\n\nFor predicted non-outliers: the output will be the naive average, it can be better like using a Ridge etc...",
    "480284": "By looking at the CV. I didn't spend much time adjusting it actually.",
    "480438": "Fantastic, Congratulations!",
    "480475": "Thanks for your reply, and I'm fully understand it, sorry to bother you for my misunderstand.",
    "480483": "Congratulations,  @kingychiu ! Thank you for sharing your solution.",
    "480972": "My bad, I didn't explain clearly that part.",
    "482025": "Congrats @kingychiu for a strong finish and thanks for sharing your solution overview.",
    "483691": "Great work !! Congratulations @kingychiu",
    "529085": "congratulations! awesome work!",
    "739467": "kingychiu, congratulations! Thanks for sharing your approaches and magic! I added your post to my collection of the best kernels and posts of winners of Kaggle Prize Competitions: [Data Science for tabular data: Advanced Techniques] (https://www.kaggle.com/vbmokin/data-science-for-tabular-data-advanced-techniques)"
  },
  "source": "existing_discussion",
  "complete": true,
  "competition_id": 10445,
  "competition_slug": "elo-merchant-category-recommendation",
  "team_id": 2627980,
  "solution_url": "/competitions/elo-merchant-category-recommendation/writeups/nlgn-16th-place-summary"
}