Hello everyone, thanks for the great competition and discussion. Much was was learned and put into practice.
\n\nI went into this competition marked by Yifan Xie words:
\n\n\n\n\nEach and every kaggle competition shall be treated as a work/study project in which you should setup a proper workflow to structure your work.
\n
With that in mind I hope someone can take from this just as I learnt a lot from all the discussions.
\n\nGithub for code, reproducible jupyter notebook, lgbm/xgboost for testing things, ensemble of 4 models for submitting. The train_model function here was a great find.
\n\nMuch feature engineering was done. It would not have been possible with the much used memory usage script frequently shared here. Here is everything that wen into the final models, in order of what I tried. Since most ideas are not new, I will link to other users example implementation
\n\nBasic time features were initially done, things such as this
More heavy time feature engineering, things such as here, here and here.
Negative month lag features
RFM, from both the customer_id and merchant_id, weighted and not weighted
Aggregations that only had few examples. For this, a especially consideration was used to not include information from whatever was in the currently validation set, ie, for each pass of the fold I would calculate it from the training data of that fold only.
More and more aggregation and stats
Final model was based on combining model with and without outliers as mentioned here with some extra modifications; basically, with a 3rd model to predict outliers, I found a ideal threshold amount to set as outliers and for those instances, the prediction was the average of the two models; for all others the prediction was based on the model without outliers.
\n\nEach model was in itself a ensemble of xgboost, catboost, lightgbm - stacking was tried, but the results weren't a improvement.
\n\nLate into the competition I had to take a break due to work/traveling, so some things in the pipeline could not be tried to the end.
\n\nI was playing with the idea of manually identifying states (and possibly a number of cities from correlation of spending/population) and including external economics and social data based on my findings that category 2 is identifying the 5 different regions in Brasil, which other people also found out.
\n\nNeural networks were also one of my next go-to things to try but alas, no time :/
\n\nFM and FFM - spent some time understanding the theory, but in the end all the models built were very sub par.
\n\nFeaturetools - It kind of worked and some of the features were slightly useful, but not what I expected, although I was also lacking in computing power by this point
\n\nThanks everyone for the great competition and discussion!
", "messages": [ { "id": "480206", "postDate": "02/27/2019 23:05:06", "content": "Hello everyone, thanks for the great competition and discussion. Much was was learned and put into practice.
\n\nI went into this competition marked by Yifan Xie words:
\n\n\n\n\nEach and every kaggle competition shall be treated as a work/study project in which you should setup a proper workflow to structure your work.
\n
With that in mind I hope someone can take from this just as I learnt a lot from all the discussions.
\n\nGithub for code, reproducible jupyter notebook, lgbm/xgboost for testing things, ensemble of 4 models for submitting. The train_model function here was a great find.
\n\nMuch feature engineering was done. It would not have been possible with the much used memory usage script frequently shared here. Here is everything that wen into the final models, in order of what I tried. Since most ideas are not new, I will link to other users example implementation
\n\nBasic time features were initially done, things such as this
More heavy time feature engineering, things such as here, here and here.
Negative month lag features
RFM, from both the customer_id and merchant_id, weighted and not weighted
Aggregations that only had few examples. For this, a especially consideration was used to not include information from whatever was in the currently validation set, ie, for each pass of the fold I would calculate it from the training data of that fold only.
More and more aggregation and stats
Final model was based on combining model with and without outliers as mentioned here with some extra modifications; basically, with a 3rd model to predict outliers, I found a ideal threshold amount to set as outliers and for those instances, the prediction was the average of the two models; for all others the prediction was based on the model without outliers.
\n\nEach model was in itself a ensemble of xgboost, catboost, lightgbm - stacking was tried, but the results weren't a improvement.
\n\nLate into the competition I had to take a break due to work/traveling, so some things in the pipeline could not be tried to the end.
\n\nI was playing with the idea of manually identifying states (and possibly a number of cities from correlation of spending/population) and including external economics and social data based on my findings that category 2 is identifying the 5 different regions in Brasil, which other people also found out.
\n\nNeural networks were also one of my next go-to things to try but alas, no time :/
\n\nFM and FFM - spent some time understanding the theory, but in the end all the models built were very sub par.
\n\nFeaturetools - It kind of worked and some of the features were slightly useful, but not what I expected, although I was also lacking in computing power by this point
\n\nThanks everyone for the great competition and discussion!
", "rawMarkdown": "Hello everyone, thanks for the great competition and discussion. Much was was learned and put into practice.\n\nI went into this competition marked by [Yifan Xie][1] [words][2]: \n\n> Each and every kaggle competition shall be treated as a work/study project in which you should setup a proper workflow to structure your work.\n\nWith that in mind I hope someone can take from this just as I learnt a lot from all the discussions.\n\n## Organization ##\nGithub for code, reproducible jupyter notebook, lgbm/xgboost for testing things, ensemble of 4 models for submitting. The train_model function [here][3] was a great find.\n\n\n## Feature Engineering ##\nMuch feature engineering was done. It would not have been possible with the much used [memory usage script][4] frequently shared here. Here is everything that wen into the final models, in order of what I tried. Since most ideas are not new, I will link to other users example implementation\n\n - Basic time features were initially done, things [such as this][5]\n\n - More heavy time feature engineering, things such as [here][6], [here][7] and [here][8].\n \n - Negative [month lag][9] features \n\n - [RFM][10], from both the customer_id and merchant_id, weighted and not weighted\n\n - Aggregations that only had few examples. For this, a especially consideration was used to not include information from whatever was in the currently validation set, ie, for each pass of the fold I would calculate it from the training data of that fold only.\n\n - More and more aggregation and stats \n\n## Models\n\nFinal model was based on combining model with and without outliers as mentioned [here][11] with some extra modifications; basically, with a 3rd model to predict outliers, I found a ideal threshold amount to set as outliers and for those instances, the prediction was the average of the two models; for all others the prediction was based on the model without outliers.\n\nEach model was in itself a ensemble of xgboost, catboost, lightgbm - stacking was tried, but the results weren't a improvement.\n\n\n## Things I didn't have the time to do ##\nLate into the competition I had to take a break due to work/traveling, so some things in the pipeline could not be tried to the end. \n\nI was playing with the idea of manually identifying states (and possibly a number of cities from correlation of spending/population) and including external economics and social data based on [my findings][12] that category 2 is identifying the 5 different regions in Brasil, which other people also found out.\n\nNeural networks were also one of my next go-to things to try but alas, no time :/\n\n\n## Ideas that failed ##\nFM and [FFM -][13] spent some time understanding the theory, but in the end all the models built were very sub par.\n\n[Featuretools][14] - It kind of worked and some of the features were slightly useful, but not what I expected, although I was also lacking in computing power by this point\n\nThanks everyone for the great competition and discussion!\n\n [1]: https://www.kaggle.com/yifanxie\n [2]: https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/75935\n [3]: https://www.kaggle.com/artgor/elo-eda-and-models\n [4]: https://www.kaggle.com/gemartin/load-data-reduce-memory-usage\n [5]: https://www.kaggle.com/ashishpatel26/lightgbm-goss-dart-parameter-tuning\n [6]: https://www.kaggle.com/rayidura/elo-lag-and-lgbm-scores/code\n [7]: https://www.kaggle.com/tunguz/eloda-with-feature-engineering-and-stacking\n [8]: https://www.kaggle.com/gpreda/elo-world-high-score-without-blending/codea\n [9]: https://www.kaggle.com/rayidura/elo-don-t-forget-the-past/code\n [10]: https://en.wikipedia.org/wiki/RFM_(customer_value)\n [11]: https://www.kaggle.com/waitingli/combining-your-model-with-a-model-without-outlier\n [12]: https://www.kaggle.com/tapioca/category-1-2-3-in-transactions\n [13]: https://www.youtube.com/watch?v=1cRGpDXTJC8&t=1s\n [14]: https://docs.featuretools.com/", "votes": null }, { "id": "480228", "postDate": "02/28/2019 00:15:59", "content": "Good workflow, Congrats!
", "rawMarkdown": "Good workflow, Congrats!", "votes": null }, { "id": "480396", "postDate": "02/28/2019 06:15:00", "content": "Thanks for your sharing and congrats!
", "rawMarkdown": "Thanks for your sharing and congrats!", "votes": null } ], "comments": [ { "id": 480228, "author_name": "jmourad100", "author_url": "", "post_date": "02/28/2019 00:15:59", "content": "Good workflow, Congrats!
", "votes": null, "replies": [] }, { "id": 480396, "author_name": "liewmanchoi", "author_url": "", "post_date": "02/28/2019 06:15:00", "content": "Thanks for your sharing and congrats!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "480206": "Hello everyone, thanks for the great competition and discussion. Much was was learned and put into practice.\n\nI went into this competition marked by [Yifan Xie][1] [words][2]: \n\n> Each and every kaggle competition shall be treated as a work/study project in which you should setup a proper workflow to structure your work.\n\nWith that in mind I hope someone can take from this just as I learnt a lot from all the discussions.\n\n## Organization ##\nGithub for code, reproducible jupyter notebook, lgbm/xgboost for testing things, ensemble of 4 models for submitting. The train_model function [here][3] was a great find.\n\n\n## Feature Engineering ##\nMuch feature engineering was done. It would not have been possible with the much used [memory usage script][4] frequently shared here. Here is everything that wen into the final models, in order of what I tried. Since most ideas are not new, I will link to other users example implementation\n\n - Basic time features were initially done, things [such as this][5]\n\n - More heavy time feature engineering, things such as [here][6], [here][7] and [here][8].\n \n - Negative [month lag][9] features \n\n - [RFM][10], from both the customer_id and merchant_id, weighted and not weighted\n\n - Aggregations that only had few examples. For this, a especially consideration was used to not include information from whatever was in the currently validation set, ie, for each pass of the fold I would calculate it from the training data of that fold only.\n\n - More and more aggregation and stats \n\n## Models\n\nFinal model was based on combining model with and without outliers as mentioned [here][11] with some extra modifications; basically, with a 3rd model to predict outliers, I found a ideal threshold amount to set as outliers and for those instances, the prediction was the average of the two models; for all others the prediction was based on the model without outliers.\n\nEach model was in itself a ensemble of xgboost, catboost, lightgbm - stacking was tried, but the results weren't a improvement.\n\n\n## Things I didn't have the time to do ##\nLate into the competition I had to take a break due to work/traveling, so some things in the pipeline could not be tried to the end. \n\nI was playing with the idea of manually identifying states (and possibly a number of cities from correlation of spending/population) and including external economics and social data based on [my findings][12] that category 2 is identifying the 5 different regions in Brasil, which other people also found out.\n\nNeural networks were also one of my next go-to things to try but alas, no time :/\n\n\n## Ideas that failed ##\nFM and [FFM -][13] spent some time understanding the theory, but in the end all the models built were very sub par.\n\n[Featuretools][14] - It kind of worked and some of the features were slightly useful, but not what I expected, although I was also lacking in computing power by this point\n\nThanks everyone for the great competition and discussion!\n\n [1]: https://www.kaggle.com/yifanxie\n [2]: https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/75935\n [3]: https://www.kaggle.com/artgor/elo-eda-and-models\n [4]: https://www.kaggle.com/gemartin/load-data-reduce-memory-usage\n [5]: https://www.kaggle.com/ashishpatel26/lightgbm-goss-dart-parameter-tuning\n [6]: https://www.kaggle.com/rayidura/elo-lag-and-lgbm-scores/code\n [7]: https://www.kaggle.com/tunguz/eloda-with-feature-engineering-and-stacking\n [8]: https://www.kaggle.com/gpreda/elo-world-high-score-without-blending/codea\n [9]: https://www.kaggle.com/rayidura/elo-don-t-forget-the-past/code\n [10]: https://en.wikipedia.org/wiki/RFM_(customer_value)\n [11]: https://www.kaggle.com/waitingli/combining-your-model-with-a-model-without-outlier\n [12]: https://www.kaggle.com/tapioca/category-1-2-3-in-transactions\n [13]: https://www.youtube.com/watch?v=1cRGpDXTJC8&t=1s\n [14]: https://docs.featuretools.com/", "480228": "Good workflow, Congrats!", "480396": "Thanks for your sharing and congrats!" }, "source": "meta" }