{
  "id": 82126,
  "title": "32nd Place - Barely shaken up - It was all about diversity",
  "url": "/competitions/elo-merchant-category-recommendation/discussion/82126",
  "author_name": "",
  "post_date": "2019-02-27T10:28:25.392805600Z",
  "votes": 11,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Hi Everyone,\nFirst let me say my thanks to Kaggle, ELO, my teammate <a href=\"/mks2192\">@mks2192</a> and this entire awesome community.</p>\n\n<p>We were 34th place in Public LB and brushed up to 32nd place in Private LB.\nI'll tell my side of the story.\nI set myself that in this competition i'll work mainly on my Feature Selection and Stacking skills and R codebase, and that's exactly what I did.</p>\n\n<hr>\n\n<p><strong>The final model consists of ridge\\lasso regression between:</strong>\n1- Regression model (Stacked 20 LGBM\\XGB models using Ridge regression, each model is a bagging of 3 seeds)</p>\n\n<p>2- Classifier model (Stacked 12   LGBM\\XGB models using OOF calculated weighted averages, each model is a bagging of 3 seeds)</p>\n\n<p>3- Regression with mean-replaced outliers (Stacked 9  LGBM\\XGB models using  Ridge regression, each model is a bagging of 3 seeds )</p>\n\n<hr>\n\n<p><strong>Diversity in models was created by:</strong>\n1- changing hyperparameters to be more\\less constricting</p>\n\n<p>2- using different loss functions such as poisson</p>\n\n<p>3- different sets of features (mostly 20-60 each model, except for one 177 features)</p>\n\n<p>4- binning target into 5 classes and running multi-class classifier and features based on it (this was not used in final model, gave more value when there were fewer models in the stack)</p>\n\n<p>5- Mean of 80 undersampled outlier classifier LGBM + SVM (this was not used in final model, gave more value when there were fewer models in the stack)</p>\n\n<hr>\n\n<p>I tried many NNs but failed.\nCV and LB scores went together most of the way, final CV score was 3.6253.</p>\n\n<p>My Feature selection consisted of Forward Selection and  Backward Selection that did not go very well. It was too costly for me to run permutation importance.\nHyperparameters were selected using random grid search.</p>\n\n<p>I will put up another discussion asking some feature selection questions that came up.\nThanks to my teammate <a href=\"/mks2192\">@mks2192</a> that put a lot of his time and effort in Feature engineering and Models for the stack.</p>\n\n<p>This competition got me sleeping an average of 3.5 hours in the last 2 weeks,\nNow back to having regular sleep (until next time).\nThanks for reading!</p>",
  "messages": [
    {
      "id": "479665",
      "postDate": "02/27/2019 10:28:25",
      "content": "<p>Hi Everyone,\nFirst let me say my thanks to Kaggle, ELO, my teammate <a href=\"/mks2192\">@mks2192</a> and this entire awesome community.</p>\n\n<p>We were 34th place in Public LB and brushed up to 32nd place in Private LB.\nI'll tell my side of the story.\nI set myself that in this competition i'll work mainly on my Feature Selection and Stacking skills and R codebase, and that's exactly what I did.</p>\n\n<hr>\n\n<p><strong>The final model consists of ridge\\lasso regression between:</strong>\n1- Regression model (Stacked 20 LGBM\\XGB models using Ridge regression, each model is a bagging of 3 seeds)</p>\n\n<p>2- Classifier model (Stacked 12   LGBM\\XGB models using OOF calculated weighted averages, each model is a bagging of 3 seeds)</p>\n\n<p>3- Regression with mean-replaced outliers (Stacked 9  LGBM\\XGB models using  Ridge regression, each model is a bagging of 3 seeds )</p>\n\n<hr>\n\n<p><strong>Diversity in models was created by:</strong>\n1- changing hyperparameters to be more\\less constricting</p>\n\n<p>2- using different loss functions such as poisson</p>\n\n<p>3- different sets of features (mostly 20-60 each model, except for one 177 features)</p>\n\n<p>4- binning target into 5 classes and running multi-class classifier and features based on it (this was not used in final model, gave more value when there were fewer models in the stack)</p>\n\n<p>5- Mean of 80 undersampled outlier classifier LGBM + SVM (this was not used in final model, gave more value when there were fewer models in the stack)</p>\n\n<hr>\n\n<p>I tried many NNs but failed.\nCV and LB scores went together most of the way, final CV score was 3.6253.</p>\n\n<p>My Feature selection consisted of Forward Selection and  Backward Selection that did not go very well. It was too costly for me to run permutation importance.\nHyperparameters were selected using random grid search.</p>\n\n<p>I will put up another discussion asking some feature selection questions that came up.\nThanks to my teammate <a href=\"/mks2192\">@mks2192</a> that put a lot of his time and effort in Feature engineering and Models for the stack.</p>\n\n<p>This competition got me sleeping an average of 3.5 hours in the last 2 weeks,\nNow back to having regular sleep (until next time).\nThanks for reading!</p>",
      "rawMarkdown": "Hi Everyone,\nFirst let me say my thanks to Kaggle, ELO, my teammate @mks2192 and this entire awesome community.\n\n\nWe were 34th place in Public LB and brushed up to 32nd place in Private LB.\nI'll tell my side of the story.\nI set myself that in this competition i'll work mainly on my Feature Selection and Stacking skills and R codebase, and that's exactly what I did.\n\n\n----------\n\n\n**The final model consists of ridge\\lasso regression between:**\n1- Regression model (Stacked 20 LGBM\\XGB models using Ridge regression, each model is a bagging of 3 seeds)\n\n2- Classifier model (Stacked 12   LGBM\\XGB models using OOF calculated weighted averages, each model is a bagging of 3 seeds)\n\n3- Regression with mean-replaced outliers (Stacked 9  LGBM\\XGB models using  Ridge regression, each model is a bagging of 3 seeds )\n\n\n----------\n\n\n**Diversity in models was created by:**\n1- changing hyperparameters to be more\\less constricting\n\n2- using different loss functions such as poisson\n\n3- different sets of features (mostly 20-60 each model, except for one 177 features)\n\n4- binning target into 5 classes and running multi-class classifier and features based on it (this was not used in final model, gave more value when there were fewer models in the stack)\n\n5- Mean of 80 undersampled outlier classifier LGBM + SVM (this was not used in final model, gave more value when there were fewer models in the stack)\n\n\n----------\n\n\nI tried many NNs but failed.\nCV and LB scores went together most of the way, final CV score was 3.6253.\n\n\nMy Feature selection consisted of Forward Selection and  Backward Selection that did not go very well. It was too costly for me to run permutation importance.\nHyperparameters were selected using random grid search.\n\n\nI will put up another discussion asking some feature selection questions that came up.\nThanks to my teammate @mks2192 that put a lot of his time and effort in Feature engineering and Models for the stack.\n\nThis competition got me sleeping an average of 3.5 hours in the last 2 weeks,\nNow back to having regular sleep (until next time).\nThanks for reading!",
      "votes": null
    },
    {
      "id": "479671",
      "postDate": "02/27/2019 10:32:38",
      "content": "<p>well done ! it's very interesting to see a so good result using regressions ! Congratulation</p>",
      "rawMarkdown": "well done ! it's very interesting to see a so good result using regressions ! Congratulation",
      "votes": null
    },
    {
      "id": "479675",
      "postDate": "02/27/2019 10:35:07",
      "content": "<p>thanks Amir (<a href=\"/tpthegreat\">@tpthegreat</a>) it was really great working with you. Happy sleeping :)</p>",
      "rawMarkdown": "thanks Amir (@tpthegreat) it was really great working with you. Happy sleeping :)",
      "votes": null
    },
    {
      "id": "479764",
      "postDate": "02/27/2019 12:04:14",
      "content": "<p>Great job!</p>",
      "rawMarkdown": "Great job!",
      "votes": null
    },
    {
      "id": "479851",
      "postDate": "02/27/2019 13:52:36",
      "content": "<p>@AmirH Great Job! Thanks for sharing!</p>",
      "rawMarkdown": "AmirH Great Job! Thanks for sharing!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 479671,
      "author_name": "francoisdubois",
      "author_url": "",
      "post_date": "02/27/2019 10:32:38",
      "content": "<p>well done ! it's very interesting to see a so good result using regressions ! Congratulation</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 479675,
      "author_name": "mks2192",
      "author_url": "",
      "post_date": "02/27/2019 10:35:07",
      "content": "<p>thanks Amir (<a href=\"/tpthegreat\">@tpthegreat</a>) it was really great working with you. Happy sleeping :)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 479764,
      "author_name": "liewmanchoi",
      "author_url": "",
      "post_date": "02/27/2019 12:04:14",
      "content": "<p>Great job!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 479851,
      "author_name": "cywwayne",
      "author_url": "",
      "post_date": "02/27/2019 13:52:36",
      "content": "<p>@AmirH Great Job! Thanks for sharing!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "479665": "Hi Everyone,\nFirst let me say my thanks to Kaggle, ELO, my teammate @mks2192 and this entire awesome community.\n\n\nWe were 34th place in Public LB and brushed up to 32nd place in Private LB.\nI'll tell my side of the story.\nI set myself that in this competition i'll work mainly on my Feature Selection and Stacking skills and R codebase, and that's exactly what I did.\n\n\n----------\n\n\n**The final model consists of ridge\\lasso regression between:**\n1- Regression model (Stacked 20 LGBM\\XGB models using Ridge regression, each model is a bagging of 3 seeds)\n\n2- Classifier model (Stacked 12   LGBM\\XGB models using OOF calculated weighted averages, each model is a bagging of 3 seeds)\n\n3- Regression with mean-replaced outliers (Stacked 9  LGBM\\XGB models using  Ridge regression, each model is a bagging of 3 seeds )\n\n\n----------\n\n\n**Diversity in models was created by:**\n1- changing hyperparameters to be more\\less constricting\n\n2- using different loss functions such as poisson\n\n3- different sets of features (mostly 20-60 each model, except for one 177 features)\n\n4- binning target into 5 classes and running multi-class classifier and features based on it (this was not used in final model, gave more value when there were fewer models in the stack)\n\n5- Mean of 80 undersampled outlier classifier LGBM + SVM (this was not used in final model, gave more value when there were fewer models in the stack)\n\n\n----------\n\n\nI tried many NNs but failed.\nCV and LB scores went together most of the way, final CV score was 3.6253.\n\n\nMy Feature selection consisted of Forward Selection and  Backward Selection that did not go very well. It was too costly for me to run permutation importance.\nHyperparameters were selected using random grid search.\n\n\nI will put up another discussion asking some feature selection questions that came up.\nThanks to my teammate @mks2192 that put a lot of his time and effort in Feature engineering and Models for the stack.\n\nThis competition got me sleeping an average of 3.5 hours in the last 2 weeks,\nNow back to having regular sleep (until next time).\nThanks for reading!",
    "479671": "well done ! it's very interesting to see a so good result using regressions ! Congratulation",
    "479675": "thanks Amir (@tpthegreat) it was really great working with you. Happy sleeping :)",
    "479764": "Great job!",
    "479851": "AmirH Great Job! Thanks for sharing!"
  },
  "source": "meta"
}