{
  "id": 82314,
  "title": "#5 solution",
  "url": "/competitions/elo-merchant-category-recommendation/writeups/evgeny-patekha-5-solution",
  "author_name": "",
  "post_date": "2019-02-28T17:18:22.748460900Z",
  "votes": 68,
  "comment_count": 21,
  "views": 0,
  "content": "<p>Congrats to all winners and all who survived shakeup.</p>\n\n<p>My solution consists of a few models:</p>\n\n<ol>\n<li>Regression (full dataset)</li>\n<li>Binary classification for outliers prediction (cards without repeated transactions in test period). <br>\n This model was used to split train and test datasets by threshold 0.015 for creation 2 other regression models (low and high probabilities). \n I also used predictions from this model as a feature for high prob model (stacking with predictions from  full regression model)</li>\n<li>Low prob model - regression with small concentration of outliers based on predicts of binary classification. For rare outliers I used decreased weights 0.4.</li>\n<li>High prob model - regression with high concentration of outliers based on predictions of binary classification. The main features of this model were predictions from binary classification and full regression. Plus some other features. This model helped me to avoid post processing which was main source of shakeup. </li>\n<li>For submit I combine together predictions from low and high prob models and blend them with full regression.</li>\n</ol>\n\n<p>Features</p>\n\n<p>I created and checked a few thousands features. Finally, after selection, I used about one hundred features.</p>\n\n<p>Some of the features were created by target encoding (target as is or binary target). \nTarget encoding was two step process - first I calculated target means for features combinations and joined results with transactions and after that I aggregated to cards level. For target means and aggregation I used weights with decline for early month_lags.</p>\n\n<p>I believe that train-test split was random and it was a good strategy to trust local cross validation instead of public leaderboard. </p>",
  "messages": [
    {
      "id": "480794",
      "postDate": "02/28/2019 17:18:22",
      "content": "<p>Congrats to all winners and all who survived shakeup.</p>\n\n<p>My solution consists of a few models:</p>\n\n<ol>\n<li>Regression (full dataset)</li>\n<li>Binary classification for outliers prediction (cards without repeated transactions in test period). <br>\n This model was used to split train and test datasets by threshold 0.015 for creation 2 other regression models (low and high probabilities). \n I also used predictions from this model as a feature for high prob model (stacking with predictions from  full regression model)</li>\n<li>Low prob model - regression with small concentration of outliers based on predicts of binary classification. For rare outliers I used decreased weights 0.4.</li>\n<li>High prob model - regression with high concentration of outliers based on predictions of binary classification. The main features of this model were predictions from binary classification and full regression. Plus some other features. This model helped me to avoid post processing which was main source of shakeup. </li>\n<li>For submit I combine together predictions from low and high prob models and blend them with full regression.</li>\n</ol>\n\n<p>Features</p>\n\n<p>I created and checked a few thousands features. Finally, after selection, I used about one hundred features.</p>\n\n<p>Some of the features were created by target encoding (target as is or binary target). \nTarget encoding was two step process - first I calculated target means for features combinations and joined results with transactions and after that I aggregated to cards level. For target means and aggregation I used weights with decline for early month_lags.</p>\n\n<p>I believe that train-test split was random and it was a good strategy to trust local cross validation instead of public leaderboard. </p>",
      "rawMarkdown": "Congrats to all winners and all who survived shakeup.\n\nMy solution consists of a few models:\n\n1. Regression (full dataset)\n2. Binary classification for outliers prediction (cards without repeated transactions in test period).  \n     This model was used to split train and test datasets by threshold 0.015 for creation 2 other regression models (low and high probabilities). \n     I also used predictions from this model as a feature for high prob model (stacking with predictions from  full regression model)\n3. Low prob model - regression with small concentration of outliers based on predicts of binary classification. For rare outliers I used decreased weights 0.4.\n4. High prob model - regression with high concentration of outliers based on predictions of binary classification. The main features of this model were predictions from binary classification and full regression. Plus some other features. This model helped me to avoid post processing which was main source of shakeup. \n5. For submit I combine together predictions from low and high prob models and blend them with full regression.\n\n\nFeatures\n\nI created and checked a few thousands features. Finally, after selection, I used about one hundred features.\n\nSome of the features were created by target encoding (target as is or binary target). \nTarget encoding was two step process - first I calculated target means for features combinations and joined results with transactions and after that I aggregated to cards level. For target means and aggregation I used weights with decline for early month_lags.\n\nI believe that train-test split was random and it was a good strategy to trust local cross validation instead of public leaderboard.",
      "votes": null
    },
    {
      "id": "480798",
      "postDate": "02/28/2019 17:25:29",
      "content": "<p>Congratulations grandmaster <a href=\"/johnpateha\">@johnpateha</a>. Thanks for sharing your approach. </p>",
      "rawMarkdown": "Congratulations grandmaster @johnpateha. Thanks for sharing your approach.",
      "votes": null
    },
    {
      "id": "480912",
      "postDate": "02/28/2019 21:03:38",
      "content": "<p>Congratulations!</p>",
      "rawMarkdown": "Congratulations!",
      "votes": null
    },
    {
      "id": "480962",
      "postDate": "02/28/2019 23:50:37",
      "content": "<p>congratulation <a href=\"/johnpateha\">@johnpateha</a>! thank you for the clear and concise description. Every competition I look forward to your solution to learn new and innovative ways to structure experiment and build models  - it has always been great source of learning for me. </p>",
      "rawMarkdown": "congratulation @johnpateha! thank you for the clear and concise description. Every competition I look forward to your solution to learn new and innovative ways to structure experiment and build models  - it has always been great source of learning for me.",
      "votes": null
    },
    {
      "id": "481049",
      "postDate": "03/01/2019 02:08:47",
      "content": "<p>Congratulations about your excellent work! So, what is <code>target means</code>? Is that one <code>card_id</code> coresponding to several targets in your model?</p>",
      "rawMarkdown": "Congratulations about your excellent work! So, what is `target means`? Is that one `card_id` coresponding to several targets in your model?",
      "votes": null
    },
    {
      "id": "481138",
      "postDate": "03/01/2019 04:46:01",
      "content": "<p>we could calculate mean target for some features combination and use it as a new feature. For this competition we need to use two step process. \nFor example\nwe calculate mean target for every merchant_id (merchant_id_TE). After that we join new column merchant_id_TE to transactions data by merchant_id as a key. And now we could aggregate merchant_id_TE by card_id and use it as a feature to train.\nSimilar for features combination (for example authorized_flag + category1 + subsector_id)</p>\n\n<p>When we use target encoding we should be careful about overfitting - to calc target means in double out of fold mode plus regularization. I recommend to find more about target encoding in solution of previous competitions. </p>",
      "rawMarkdown": "we could calculate mean target for some features combination and use it as a new feature. For this competition we need to use two step process. \nFor example\nwe calculate mean target for every merchant_id (merchant_id_TE). After that we join new column merchant_id_TE to transactions data by merchant_id as a key. And now we could aggregate merchant_id_TE by card_id and use it as a feature to train.\nSimilar for features combination (for example authorized_flag + category1 + subsector_id)\n\nWhen we use target encoding we should be careful about overfitting - to calc target means in double out of fold mode plus regularization. I recommend to find more about target encoding in solution of previous competitions.",
      "votes": null
    },
    {
      "id": "481189",
      "postDate": "03/01/2019 06:10:53",
      "content": "<p>Cool!, in my experiments had trying to do same thing, but not having exp in kagle my idea was failed with strong overfitting, thank you for info! could you tell , what competition had use this technique before? sorry if my question stypid, but i am just learning ;-)</p>",
      "rawMarkdown": "Cool!, in my experiments had trying to do same thing, but not having exp in kagle my idea was failed with strong overfitting, thank you for info! could you tell , what competition had use this technique before? sorry if my question stypid, but i am just learning ;-)",
      "votes": null
    },
    {
      "id": "481252",
      "postDate": "03/01/2019 07:38:48",
      "content": "<p>As I know this method became popular after BNP Paribas competition, where Stas Semenov explain the idea. I found many sources about it, but mostly in Russian.\nI'm not sure but may be youtube auto-translate could help - this video was very helpful for me when I studied target encoding <a href=\"https://www.youtube.com/watch?v=NVKDSNM702k\">https://www.youtube.com/watch?v=NVKDSNM702k</a>\nThis article could be helpful too <a href=\"https://habr.com/ru/company/ods/blog/426227/\">https://habr.com/ru/company/ods/blog/426227/</a></p>",
      "rawMarkdown": "As I know this method became popular after BNP Paribas competition, where Stas Semenov explain the idea. I found many sources about it, but mostly in Russian.\nI'm not sure but may be youtube auto-translate could help - this video was very helpful for me when I studied target encoding https://www.youtube.com/watch?v=NVKDSNM702k\nThis article could be helpful too https://habr.com/ru/company/ods/blog/426227/",
      "votes": null
    },
    {
      "id": "481270",
      "postDate": "03/01/2019 07:58:55",
      "content": "<p>Huge thank to you! This is very usefull for me and may be for somebody else, i hope</p>",
      "rawMarkdown": "Huge thank to you! This is very usefull for me and may be for somebody else, i hope",
      "votes": null
    },
    {
      "id": "481398",
      "postDate": "03/01/2019 10:54:00",
      "content": "<p>Thank you very much. Though I'm a Chinese, so it's hard for me to understand the content.</p>",
      "rawMarkdown": "Thank you very much. Though I'm a Chinese, so it's hard for me to understand the content.",
      "votes": null
    },
    {
      "id": "481970",
      "postDate": "03/02/2019 05:51:05",
      "content": "<p>Congrats <a href=\"/johnpateha\">@johnpateha</a> for a strong solo finish and thanks for sharing.</p>",
      "rawMarkdown": "Congrats @johnpateha for a strong solo finish and thanks for sharing.",
      "votes": null
    },
    {
      "id": "482216",
      "postDate": "03/02/2019 14:25:45",
      "content": "<p>Congratulations! In my understanding of your excellent work, the train set in <strong>Low prob model</strong> is the data that target is not -33.21928095, and the test set is the test data that is predicted to be non-outlier by binary classification. </p>\n\n<p>Is my understanding correct? <br>\nAnd Does threshold 0.015 in your binary classification model means the threshold for outliers and non-outliers? \nAnd what are rare outliers in <strong>Low prob model</strong>?</p>",
      "rawMarkdown": "Congratulations! In my understanding of your excellent work, the train set in **Low prob model** is the data that target is not -33.21928095, and the test set is the test data that is predicted to be non-outlier by binary classification. \n\nIs my understanding correct?  \nAnd Does threshold 0.015 in your binary classification model means the threshold for outliers and non-outliers? \nAnd what are rare outliers in **Low prob model**?",
      "votes": null
    },
    {
      "id": "484222",
      "postDate": "03/05/2019 17:57:20",
      "content": "<p>Evgeny, congratulations sincerely! You have an unexpected \"angle of attack\". What method did you use <strong>to select the threshold (0.015)</strong>?\nCould you say a few words about the binary classifier (test period, target, metric, models, cv scores, etc...). Thank you very much!</p>",
      "rawMarkdown": "Evgeny, congratulations sincerely! You have an unexpected \"angle of attack\". What method did you use **to select the threshold (0.015)**?\nCould you say a few words about the binary classifier (test period, target, metric, models, cv scores, etc...). Thank you very much!",
      "votes": null
    },
    {
      "id": "487169",
      "postDate": "03/10/2019 09:15:29",
      "content": "<p>No special methods - I just tried different thresholds from .007 to .015 and checked them on local validation.\nFor binary classification I used LightGBM ( GOSS ). Stratified 5-folds CV.  Metric for early stop = logloss. \nBest single model with 54 features had roc auc 0.9131. Blend of 4 classifiers with slightly different parameters/features had roc auc 0.9141\nThe final combined prediction had local rmse 3.609</p>",
      "rawMarkdown": "No special methods - I just tried different thresholds from .007 to .015 and checked them on local validation.\nFor binary classification I used LightGBM ( GOSS ). Stratified 5-folds CV.  Metric for early stop = logloss. \nBest single model with 54 features had roc auc 0.9131. Blend of 4 classifiers with slightly different parameters/features had roc auc 0.9141\nThe final combined prediction had local rmse 3.609",
      "votes": null
    },
    {
      "id": "487171",
      "postDate": "03/10/2019 09:19:01",
      "content": "<p>My binary classification predicted outliers, but it was not perfect and small amount of outliers were in the low part as my classifier didn't catch them.  </p>",
      "rawMarkdown": "My binary classification predicted outliers, but it was not perfect and small amount of outliers were in the low part as my classifier didn't catch them.",
      "votes": null
    },
    {
      "id": "488323",
      "postDate": "03/12/2019 09:32:37",
      "content": "<p>Congratulations!</p>",
      "rawMarkdown": "Congratulations!",
      "votes": null
    },
    {
      "id": "494740",
      "postDate": "03/20/2019 08:01:31",
      "content": "<p>Congrats for a strong solo finish and thanks for sharing. \nWhile I can't catch the idea \"to calc target means in double out of fold mode plus regularization\".\nI always do the target encoding with smooth such as this method(<a href=\"https://www.kaggle.com/ogrellier/python-target-encoding-for-categorical-features\">https://www.kaggle.com/ogrellier/python-target-encoding-for-categorical-features</a>).\nSo would you like to  share  some details about the \"double out of fold mode plus regularization\"?\nThank you very much.</p>",
      "rawMarkdown": "Congrats for a strong solo finish and thanks for sharing. \nWhile I can't catch the idea \"to calc target means in double out of fold mode plus regularization\".\nI always do the target encoding with smooth such as this method(https://www.kaggle.com/ogrellier/python-target-encoding-for-categorical-features).\nSo would you like to  share  some details about the \"double out of fold mode plus regularization\"?\nThank you very much.",
      "votes": null
    },
    {
      "id": "495132",
      "postDate": "03/20/2019 17:23:38",
      "content": "<p>Congratulations!</p>",
      "rawMarkdown": "Congratulations!",
      "votes": null
    },
    {
      "id": "525737",
      "postDate": "05/01/2019 16:20:32",
      "content": "<p>Congratulations, thanks for sharing!</p>",
      "rawMarkdown": "Congratulations, thanks for sharing!",
      "votes": null
    },
    {
      "id": "739462",
      "postDate": "02/07/2020 21:03:30",
      "content": "<p><a href=\"/johnpateha\">@johnpateha</a>, congratulations! Thanks for sharing your approaches and magic! I added your post to my collection of the best kernels and posts of winners of Kaggle Prize Competitions: [Data Science for tabular data: Advanced Techniques] (<a href=\"https://www.kaggle.com/vbmokin/data-science-for-tabular-data-advanced-techniques\">https://www.kaggle.com/vbmokin/data-science-for-tabular-data-advanced-techniques</a>)</p>",
      "rawMarkdown": "johnpateha, congratulations! Thanks for sharing your approaches and magic! I added your post to my collection of the best kernels and posts of winners of Kaggle Prize Competitions: [Data Science for tabular data: Advanced Techniques] (https://www.kaggle.com/vbmokin/data-science-for-tabular-data-advanced-techniques)",
      "votes": null
    },
    {
      "id": "1070847",
      "postDate": "11/06/2020 08:30:12",
      "content": "<p>Thanks for sharing very helpful!!</p>",
      "rawMarkdown": "Thanks for sharing very helpful!!",
      "votes": null
    },
    {
      "id": "1185901",
      "postDate": "02/04/2021 13:23:46",
      "content": "<p>Thanks for sharing very helpful!!</p>",
      "rawMarkdown": "Thanks for sharing very helpful!!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1070847,
      "author_name": "kamlesh577",
      "author_url": "",
      "post_date": "11/06/2020 08:30:12",
      "content": "<p>Thanks for sharing very helpful!!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1185901,
      "author_name": "mahmoodabdali",
      "author_url": "",
      "post_date": "02/04/2021 13:23:46",
      "content": "<p>Thanks for sharing very helpful!!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 480798,
      "author_name": "karthik7395",
      "author_url": "",
      "post_date": "02/28/2019 17:25:29",
      "content": "<p>Congratulations grandmaster <a href=\"/johnpateha\">@johnpateha</a>. Thanks for sharing your approach. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 480912,
      "author_name": "alijs1",
      "author_url": "",
      "post_date": "02/28/2019 21:03:38",
      "content": "<p>Congratulations!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 480962,
      "author_name": "yifanxie",
      "author_url": "",
      "post_date": "02/28/2019 23:50:37",
      "content": "<p>congratulation <a href=\"/johnpateha\">@johnpateha</a>! thank you for the clear and concise description. Every competition I look forward to your solution to learn new and innovative ways to structure experiment and build models  - it has always been great source of learning for me. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 481049,
      "author_name": "liewmanchoi",
      "author_url": "",
      "post_date": "03/01/2019 02:08:47",
      "content": "<p>Congratulations about your excellent work! So, what is <code>target means</code>? Is that one <code>card_id</code> coresponding to several targets in your model?</p>",
      "votes": null,
      "replies": [
        {
          "id": 481138,
          "author_name": "johnpateha",
          "author_url": "",
          "post_date": "03/01/2019 04:46:01",
          "content": "<p>we could calculate mean target for some features combination and use it as a new feature. For this competition we need to use two step process. \nFor example\nwe calculate mean target for every merchant_id (merchant_id_TE). After that we join new column merchant_id_TE to transactions data by merchant_id as a key. And now we could aggregate merchant_id_TE by card_id and use it as a feature to train.\nSimilar for features combination (for example authorized_flag + category1 + subsector_id)</p>\n\n<p>When we use target encoding we should be careful about overfitting - to calc target means in double out of fold mode plus regularization. I recommend to find more about target encoding in solution of previous competitions. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 481189,
          "author_name": "alexeych",
          "author_url": "",
          "post_date": "03/01/2019 06:10:53",
          "content": "<p>Cool!, in my experiments had trying to do same thing, but not having exp in kagle my idea was failed with strong overfitting, thank you for info! could you tell , what competition had use this technique before? sorry if my question stypid, but i am just learning ;-)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 481252,
          "author_name": "johnpateha",
          "author_url": "",
          "post_date": "03/01/2019 07:38:48",
          "content": "<p>As I know this method became popular after BNP Paribas competition, where Stas Semenov explain the idea. I found many sources about it, but mostly in Russian.\nI'm not sure but may be youtube auto-translate could help - this video was very helpful for me when I studied target encoding <a href=\"https://www.youtube.com/watch?v=NVKDSNM702k\">https://www.youtube.com/watch?v=NVKDSNM702k</a>\nThis article could be helpful too <a href=\"https://habr.com/ru/company/ods/blog/426227/\">https://habr.com/ru/company/ods/blog/426227/</a></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 481270,
          "author_name": "alexeych",
          "author_url": "",
          "post_date": "03/01/2019 07:58:55",
          "content": "<p>Huge thank to you! This is very usefull for me and may be for somebody else, i hope</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 481398,
          "author_name": "liewmanchoi",
          "author_url": "",
          "post_date": "03/01/2019 10:54:00",
          "content": "<p>Thank you very much. Though I'm a Chinese, so it's hard for me to understand the content.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 494740,
          "author_name": "leonyang",
          "author_url": "",
          "post_date": "03/20/2019 08:01:31",
          "content": "<p>Congrats for a strong solo finish and thanks for sharing. \nWhile I can't catch the idea \"to calc target means in double out of fold mode plus regularization\".\nI always do the target encoding with smooth such as this method(<a href=\"https://www.kaggle.com/ogrellier/python-target-encoding-for-categorical-features\">https://www.kaggle.com/ogrellier/python-target-encoding-for-categorical-features</a>).\nSo would you like to  share  some details about the \"double out of fold mode plus regularization\"?\nThank you very much.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 481970,
      "author_name": "sheriytm",
      "author_url": "",
      "post_date": "03/02/2019 05:51:05",
      "content": "<p>Congrats <a href=\"/johnpateha\">@johnpateha</a> for a strong solo finish and thanks for sharing.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 482216,
      "author_name": "yanhe1",
      "author_url": "",
      "post_date": "03/02/2019 14:25:45",
      "content": "<p>Congratulations! In my understanding of your excellent work, the train set in <strong>Low prob model</strong> is the data that target is not -33.21928095, and the test set is the test data that is predicted to be non-outlier by binary classification. </p>\n\n<p>Is my understanding correct? <br>\nAnd Does threshold 0.015 in your binary classification model means the threshold for outliers and non-outliers? \nAnd what are rare outliers in <strong>Low prob model</strong>?</p>",
      "votes": null,
      "replies": [
        {
          "id": 487171,
          "author_name": "johnpateha",
          "author_url": "",
          "post_date": "03/10/2019 09:19:01",
          "content": "<p>My binary classification predicted outliers, but it was not perfect and small amount of outliers were in the low part as my classifier didn't catch them.  </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 484222,
      "author_name": "maatkara",
      "author_url": "",
      "post_date": "03/05/2019 17:57:20",
      "content": "<p>Evgeny, congratulations sincerely! You have an unexpected \"angle of attack\". What method did you use <strong>to select the threshold (0.015)</strong>?\nCould you say a few words about the binary classifier (test period, target, metric, models, cv scores, etc...). Thank you very much!</p>",
      "votes": null,
      "replies": [
        {
          "id": 487169,
          "author_name": "johnpateha",
          "author_url": "",
          "post_date": "03/10/2019 09:15:29",
          "content": "<p>No special methods - I just tried different thresholds from .007 to .015 and checked them on local validation.\nFor binary classification I used LightGBM ( GOSS ). Stratified 5-folds CV.  Metric for early stop = logloss. \nBest single model with 54 features had roc auc 0.9131. Blend of 4 classifiers with slightly different parameters/features had roc auc 0.9141\nThe final combined prediction had local rmse 3.609</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 488323,
      "author_name": "gvakhrushev",
      "author_url": "",
      "post_date": "03/12/2019 09:32:37",
      "content": "<p>Congratulations!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 495132,
      "author_name": "msgonzaga101",
      "author_url": "",
      "post_date": "03/20/2019 17:23:38",
      "content": "<p>Congratulations!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 525737,
      "author_name": "akmurray",
      "author_url": "",
      "post_date": "05/01/2019 16:20:32",
      "content": "<p>Congratulations, thanks for sharing!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 739462,
      "author_name": "vbmokin",
      "author_url": "",
      "post_date": "02/07/2020 21:03:30",
      "content": "<p><a href=\"/johnpateha\">@johnpateha</a>, congratulations! Thanks for sharing your approaches and magic! I added your post to my collection of the best kernels and posts of winners of Kaggle Prize Competitions: [Data Science for tabular data: Advanced Techniques] (<a href=\"https://www.kaggle.com/vbmokin/data-science-for-tabular-data-advanced-techniques\">https://www.kaggle.com/vbmokin/data-science-for-tabular-data-advanced-techniques</a>)</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "480794": "Congrats to all winners and all who survived shakeup.\n\nMy solution consists of a few models:\n\n1. Regression (full dataset)\n2. Binary classification for outliers prediction (cards without repeated transactions in test period).  \n     This model was used to split train and test datasets by threshold 0.015 for creation 2 other regression models (low and high probabilities). \n     I also used predictions from this model as a feature for high prob model (stacking with predictions from  full regression model)\n3. Low prob model - regression with small concentration of outliers based on predicts of binary classification. For rare outliers I used decreased weights 0.4.\n4. High prob model - regression with high concentration of outliers based on predictions of binary classification. The main features of this model were predictions from binary classification and full regression. Plus some other features. This model helped me to avoid post processing which was main source of shakeup. \n5. For submit I combine together predictions from low and high prob models and blend them with full regression.\n\n\nFeatures\n\nI created and checked a few thousands features. Finally, after selection, I used about one hundred features.\n\nSome of the features were created by target encoding (target as is or binary target). \nTarget encoding was two step process - first I calculated target means for features combinations and joined results with transactions and after that I aggregated to cards level. For target means and aggregation I used weights with decline for early month_lags.\n\nI believe that train-test split was random and it was a good strategy to trust local cross validation instead of public leaderboard.",
    "480798": "Congratulations grandmaster @johnpateha. Thanks for sharing your approach.",
    "480912": "Congratulations!",
    "480962": "congratulation @johnpateha! thank you for the clear and concise description. Every competition I look forward to your solution to learn new and innovative ways to structure experiment and build models  - it has always been great source of learning for me.",
    "481049": "Congratulations about your excellent work! So, what is `target means`? Is that one `card_id` coresponding to several targets in your model?",
    "481138": "we could calculate mean target for some features combination and use it as a new feature. For this competition we need to use two step process. \nFor example\nwe calculate mean target for every merchant_id (merchant_id_TE). After that we join new column merchant_id_TE to transactions data by merchant_id as a key. And now we could aggregate merchant_id_TE by card_id and use it as a feature to train.\nSimilar for features combination (for example authorized_flag + category1 + subsector_id)\n\nWhen we use target encoding we should be careful about overfitting - to calc target means in double out of fold mode plus regularization. I recommend to find more about target encoding in solution of previous competitions.",
    "481189": "Cool!, in my experiments had trying to do same thing, but not having exp in kagle my idea was failed with strong overfitting, thank you for info! could you tell , what competition had use this technique before? sorry if my question stypid, but i am just learning ;-)",
    "481252": "As I know this method became popular after BNP Paribas competition, where Stas Semenov explain the idea. I found many sources about it, but mostly in Russian.\nI'm not sure but may be youtube auto-translate could help - this video was very helpful for me when I studied target encoding https://www.youtube.com/watch?v=NVKDSNM702k\nThis article could be helpful too https://habr.com/ru/company/ods/blog/426227/",
    "481270": "Huge thank to you! This is very usefull for me and may be for somebody else, i hope",
    "481398": "Thank you very much. Though I'm a Chinese, so it's hard for me to understand the content.",
    "481970": "Congrats @johnpateha for a strong solo finish and thanks for sharing.",
    "482216": "Congratulations! In my understanding of your excellent work, the train set in **Low prob model** is the data that target is not -33.21928095, and the test set is the test data that is predicted to be non-outlier by binary classification. \n\nIs my understanding correct?  \nAnd Does threshold 0.015 in your binary classification model means the threshold for outliers and non-outliers? \nAnd what are rare outliers in **Low prob model**?",
    "484222": "Evgeny, congratulations sincerely! You have an unexpected \"angle of attack\". What method did you use **to select the threshold (0.015)**?\nCould you say a few words about the binary classifier (test period, target, metric, models, cv scores, etc...). Thank you very much!",
    "487169": "No special methods - I just tried different thresholds from .007 to .015 and checked them on local validation.\nFor binary classification I used LightGBM ( GOSS ). Stratified 5-folds CV.  Metric for early stop = logloss. \nBest single model with 54 features had roc auc 0.9131. Blend of 4 classifiers with slightly different parameters/features had roc auc 0.9141\nThe final combined prediction had local rmse 3.609",
    "487171": "My binary classification predicted outliers, but it was not perfect and small amount of outliers were in the low part as my classifier didn't catch them.",
    "488323": "Congratulations!",
    "494740": "Congrats for a strong solo finish and thanks for sharing. \nWhile I can't catch the idea \"to calc target means in double out of fold mode plus regularization\".\nI always do the target encoding with smooth such as this method(https://www.kaggle.com/ogrellier/python-target-encoding-for-categorical-features).\nSo would you like to  share  some details about the \"double out of fold mode plus regularization\"?\nThank you very much.",
    "495132": "Congratulations!",
    "525737": "Congratulations, thanks for sharing!",
    "739462": "johnpateha, congratulations! Thanks for sharing your approaches and magic! I added your post to my collection of the best kernels and posts of winners of Kaggle Prize Competitions: [Data Science for tabular data: Advanced Techniques] (https://www.kaggle.com/vbmokin/data-science-for-tabular-data-advanced-techniques)",
    "1070847": "Thanks for sharing very helpful!!",
    "1185901": "Thanks for sharing very helpful!!"
  },
  "source": "existing_discussion",
  "complete": true,
  "competition_id": 10445,
  "competition_slug": "elo-merchant-category-recommendation",
  "team_id": 2564551,
  "solution_url": "/competitions/elo-merchant-category-recommendation/writeups/evgeny-patekha-5-solution"
}