So this competition has been a long long journey for me.
\nIt felt way longer than it actually was, which is just 40 days. 40days ago, I wanted to take a break from another competition, and decided to do a \"40days challenge\".
I started off by aggregating a lot of the time-series, but it was hard to even beat the kernel. The data seemed to have so little signal, almost everything that I did didn't work.
\nAfter a week or two, I somehow managed to score 3.636-3.630 locally, which happened to be around 3.608-3.605 private. I was running out of ideas, so I decided to team merge.
After merging with my team, we merged our features and re-run our models. Our best single model improved about +0.002 at that time.\nWe searched for a good way to ensemble our models, but couldn't find a good way, especially with all the post-process that was going on.
\nJust one week before the deadline, @marcuslin finally found that a simple ridge including non-outlier model + binary-prediction model works really well. Our score jumped to the 3.66x range in public, which was 3.600 range in private.
For our final submission, we were torn between 3subs. One with heavy post-process, one with conservative post-process, and one without post-process.
\nWe were pessimistic about our final position, so we took the risk and selected the two with post-process. This was actually a mistake, but still good enough for 18th place.
\nHad we selected our no-post-process sub, we would have placed around 10th place, but that is just another story.
I really feel lucky/happy about the final result, since in the heat of the competition, I could neither trust the CV nor could I trust the LB, which means I was flying blind.
\n\nSince some people may have interest, I will describe the technical side of things, my features and my models below.
\n\nUsed features
\n1)Basic features: very good
\nAggregations like the kernel. Not much to say.
2)Time features: includes magic features
\nFeatures about When there was a purchase.\nThe kernel lack \"the last day of transaction\", which is magically good.
3)Conditional features: good
\nThe aggregation when (authorized_flag == 1, 0), (city_id == -1)
4)Recent features: good
\nThe aggregation when (month_lag >= -2, or 0)
5)Prediction of time: probably good
\nPredict the magic_feature:(last day of month_lag2 - last purchase_day) from only the historical_transaction features.\nThen do (predicted_days - actual days), and drop the predicted days.
6)Time features: maybe good?
\nhttps://www.kaggle.com/denzo123/a-closer-look-at-date-variables\nHour and DayOfWeek seems to be very important (but hard to improve the cv-score).\nI binned \"hour\" to [6-12, 12-17, 17-21, 21-05], which seems to add very small value(maybe).\nBy the way, 00:00:00 has a very high target value (probably because it is periodic purchases), but I couldn't improve the CV score with a flag, maybe because there are too few occurrence...
7)Target-encoding features: maybe good? destablized the score...
\nOOF-target encoding of categories(merchant_id, subsector_id).
8)(+, -, /, *) of two features: maybe stablizes the score? not sure...
\n\nUnusued features
\nI had tons of ideas which didn't work. Here are some of the interesting ones
LDA features: thrown away
\nthe LDA topic of each categories (like in TalkingData competition).\nI guess categories have no information!
Sequential authorized_flag=\"N\": thrown away
\nSometimes, there are many authorized_flag=\"N\", in a sequence.
Periodic buys: thrown away
\nSometimes, there are periodic purchases. Like, always once a month with the same price, at the same merchant.
Encode features: thrown away
\nEncode the categories by mean_purchase_amount.\nTake the difference (purchase_amount - mean_purchase_amount).
Models/post-process
\nIn general, stacking with simpler algorithms worked better, so Ridge/Linear regression seemed better than LGBM.\nMy teammates were very successful with stacking, so I will leave it for them to write.
Models which didn't work/weren't used
\nTime-series prediction: didn't work
\nLike in the HomeCredit competition, predict the target based on the raw time-series/monthly time-series. Neither worked.
Isotonic regression: didn't work
\nI probably did this wrong, but I couldn't make it work
Random stacking: didn't use
\nFor stacking, I stacked upon automatically-generated-random-models by randomly selecting a subset of my features. We did not select the stack, since the publicLB score was bad, but the CV and privateLB score(3.660) is actually better than our final submission.\nNeed some more experiments to confirm.
Pseudo-labeling: didn't use
\nPseudo-labeling with our best stack seems to improve our base model on the CV and privateLB. But it did bad on the publicLB, so we didn't use it.\nPL probably worked both on LB/CV earlier with NN, but we didn't have enough time to squeeze it into our final stack.
I really want to thank my teammates for doing a fantastic job in this competition.
\nWithout my team, I would not have been in this place.
\nI also want to thank and congratulate @philculliton for his first competition without a leak!\nI hope you won't ban more than 221 people so that I remain in gold :)
And.. I guess that is it.\nThank you all for participating in this competition, and good luck ;)
", "messages": [ { "id": 479529, "postDate": "2019-02-27T08:15:02.473Z", "content": "So this competition has been a long long journey for me.
\nIt felt way longer than it actually was, which is just 40 days. 40days ago, I wanted to take a break from another competition, and decided to do a \"40days challenge\".
I started off by aggregating a lot of the time-series, but it was hard to even beat the kernel. The data seemed to have so little signal, almost everything that I did didn't work.
\nAfter a week or two, I somehow managed to score 3.636-3.630 locally, which happened to be around 3.608-3.605 private. I was running out of ideas, so I decided to team merge.
After merging with my team, we merged our features and re-run our models. Our best single model improved about +0.002 at that time.\nWe searched for a good way to ensemble our models, but couldn't find a good way, especially with all the post-process that was going on.
\nJust one week before the deadline, @marcuslin finally found that a simple ridge including non-outlier model + binary-prediction model works really well. Our score jumped to the 3.66x range in public, which was 3.600 range in private.
For our final submission, we were torn between 3subs. One with heavy post-process, one with conservative post-process, and one without post-process.
\nWe were pessimistic about our final position, so we took the risk and selected the two with post-process. This was actually a mistake, but still good enough for 18th place.
\nHad we selected our no-post-process sub, we would have placed around 10th place, but that is just another story.
I really feel lucky/happy about the final result, since in the heat of the competition, I could neither trust the CV nor could I trust the LB, which means I was flying blind.
\n\nSince some people may have interest, I will describe the technical side of things, my features and my models below.
\n\nUsed features
\n1)Basic features: very good
\nAggregations like the kernel. Not much to say.
2)Time features: includes magic features
\nFeatures about When there was a purchase.\nThe kernel lack \"the last day of transaction\", which is magically good.
3)Conditional features: good
\nThe aggregation when (authorized_flag == 1, 0), (city_id == -1)
4)Recent features: good
\nThe aggregation when (month_lag >= -2, or 0)
5)Prediction of time: probably good
\nPredict the magic_feature:(last day of month_lag2 - last purchase_day) from only the historical_transaction features.\nThen do (predicted_days - actual days), and drop the predicted days.
6)Time features: maybe good?
\nhttps://www.kaggle.com/denzo123/a-closer-look-at-date-variables\nHour and DayOfWeek seems to be very important (but hard to improve the cv-score).\nI binned \"hour\" to [6-12, 12-17, 17-21, 21-05], which seems to add very small value(maybe).\nBy the way, 00:00:00 has a very high target value (probably because it is periodic purchases), but I couldn't improve the CV score with a flag, maybe because there are too few occurrence...
7)Target-encoding features: maybe good? destablized the score...
\nOOF-target encoding of categories(merchant_id, subsector_id).
8)(+, -, /, *) of two features: maybe stablizes the score? not sure...
\n\nUnusued features
\nI had tons of ideas which didn't work. Here are some of the interesting ones
LDA features: thrown away
\nthe LDA topic of each categories (like in TalkingData competition).\nI guess categories have no information!
Sequential authorized_flag=\"N\": thrown away
\nSometimes, there are many authorized_flag=\"N\", in a sequence.
Periodic buys: thrown away
\nSometimes, there are periodic purchases. Like, always once a month with the same price, at the same merchant.
Encode features: thrown away
\nEncode the categories by mean_purchase_amount.\nTake the difference (purchase_amount - mean_purchase_amount).
Models/post-process
\nIn general, stacking with simpler algorithms worked better, so Ridge/Linear regression seemed better than LGBM.\nMy teammates were very successful with stacking, so I will leave it for them to write.
Models which didn't work/weren't used
\nTime-series prediction: didn't work
\nLike in the HomeCredit competition, predict the target based on the raw time-series/monthly time-series. Neither worked.
Isotonic regression: didn't work
\nI probably did this wrong, but I couldn't make it work
Random stacking: didn't use
\nFor stacking, I stacked upon automatically-generated-random-models by randomly selecting a subset of my features. We did not select the stack, since the publicLB score was bad, but the CV and privateLB score(3.660) is actually better than our final submission.\nNeed some more experiments to confirm.
Pseudo-labeling: didn't use
\nPseudo-labeling with our best stack seems to improve our base model on the CV and privateLB. But it did bad on the publicLB, so we didn't use it.\nPL probably worked both on LB/CV earlier with NN, but we didn't have enough time to squeeze it into our final stack.
I really want to thank my teammates for doing a fantastic job in this competition.
\nWithout my team, I would not have been in this place.
\nI also want to thank and congratulate @philculliton for his first competition without a leak!\nI hope you won't ban more than 221 people so that I remain in gold :)
And.. I guess that is it.\nThank you all for participating in this competition, and good luck ;)
", "rawMarkdown": "My journey\n------------------------------------------\nSo this competition has been a long long journey for me. \nIt felt way longer than it actually was, which is just 40 days. 40days ago, I wanted to take a break from another competition, and decided to do a \"40days challenge\".\n\nI started off by aggregating a lot of the time-series, but it was hard to even beat the kernel. The data seemed to have so little signal, almost everything that I did didn't work. \nAfter a week or two, I somehow managed to score 3.636-3.630 locally, which happened to be around 3.608-3.605 private. I was running out of ideas, so I decided to team merge.\n\nAfter merging with my team, we merged our features and re-run our models. Our best single model improved about +0.002 at that time.\nWe searched for a good way to ensemble our models, but couldn't find a good way, especially with all the post-process that was going on. \nJust one week before the deadline, @marcuslin finally found that a simple ridge including non-outlier model + binary-prediction model works really well. Our score jumped to the 3.66x range in public, which was 3.600 range in private.\n\nFor our final submission, we were torn between 3subs. One with heavy post-process, one with conservative post-process, and one without post-process. \nWe were pessimistic about our final position, so we took the risk and selected the two with post-process. This was actually a mistake, but still good enough for 18th place. \nHad we selected our no-post-process sub, we would have placed around 10th place, but that is just another story.\n\nI really feel lucky/happy about the final result, since in the heat of the competition, I could neither trust the CV nor could I trust the LB, which means I was flying blind.\n\nSince some people may have interest, I will describe the technical side of things, my features and my models below.\n\n------------------------------------------\nFeatures\n------------------------------------------\n**Used features** \n1)Basic features: very good \nAggregations like the kernel. Not much to say.\n\n2)Time features: includes magic features \nFeatures about When there was a purchase.\nThe kernel lack \"the last day of transaction\", which is magically good.\n\n3)Conditional features: good \nThe aggregation when (authorized_flag == 1, 0), (city_id == -1)\n\n4)Recent features: good \nThe aggregation when (month_lag >= -2, or 0)\n\n5)Prediction of time: probably good \nPredict the magic_feature:(last day of month_lag2 - last purchase_day) from only the historical_transaction features.\nThen do (predicted_days - actual days), and drop the predicted days.\n\n6)Time features: maybe good? \nhttps://www.kaggle.com/denzo123/a-closer-look-at-date-variables\nHour and DayOfWeek seems to be very important (but hard to improve the cv-score).\nI binned \"hour\" to [6-12, 12-17, 17-21, 21-05], which seems to add very small value(maybe).\nBy the way, 00:00:00 has a very high target value (probably because it is periodic purchases), but I couldn't improve the CV score with a flag, maybe because there are too few occurrence...\n\n7)Target-encoding features: maybe good? destablized the score... \nOOF-target encoding of categories(merchant_id, subsector_id).\n\n8)(+, -, /, *) of two features: maybe stablizes the score? not sure...\n\n------------------------------------------------------\n**Unusued features** \nI had tons of ideas which didn't work. Here are some of the interesting ones\n\nLDA features: thrown away \nthe LDA topic of each categories (like in TalkingData competition).\nI guess categories have no information!\n\nSequential authorized_flag=\"N\": thrown away \nSometimes, there are many authorized_flag=\"N\", in a sequence.\n\nPeriodic buys: thrown away \nSometimes, there are periodic purchases. Like, always once a month with the same price, at the same merchant.\n\nEncode features: thrown away \nEncode the categories by mean_purchase_amount.\nTake the difference (purchase_amount - mean_purchase_amount).\n\n------------------------------------------\nModels\n------------------------------------------\n**Models/post-process** \nIn general, stacking with simpler algorithms worked better, so Ridge/Linear regression seemed better than LGBM.\nMy teammates were very successful with stacking, so I will leave it for them to write.\n\n------------------------------------------\n**Models which didn't work/weren't used** \nTime-series prediction: didn't work \nLike in the HomeCredit competition, predict the target based on the raw time-series/monthly time-series. Neither worked.\n\nIsotonic regression: didn't work \nI probably did this wrong, but I couldn't make it work\n\nRandom stacking: didn't use \nFor stacking, I stacked upon automatically-generated-random-models by randomly selecting a subset of my features. We did not select the stack, since the publicLB score was bad, but the CV and privateLB score(3.660) is actually better than our final submission.\nNeed some more experiments to confirm.\n\nPseudo-labeling: didn't use \nPseudo-labeling with our best stack seems to improve our base model on the CV and privateLB. But it did bad on the publicLB, so we didn't use it.\nPL probably worked both on LB/CV earlier with NN, but we didn't have enough time to squeeze it into our final stack.\n\n------------------------------------------\nConclusion\n------------------------------------------\nI really want to thank my teammates for doing a fantastic job in this competition. \nWithout my team, I would not have been in this place. \nI also want to thank and congratulate @philculliton for his first competition without a leak!\nI hope you won't ban more than 221 people so that I remain in gold :)\n\nAnd.. I guess that is it.\nThank you all for participating in this competition, and good luck ;)\n\n\n\n\n", "votes": 52 }, { "id": 739468, "postDate": "2020-02-07T21:09:12.997Z", "content": "@pocketsuteado, congratulations! Thanks for sharing your approaches and magic! I added your post to my collection of the best kernels and posts of winners of Kaggle Prize Competitions: [Data Science for tabular data: Advanced Techniques] (https://www.kaggle.com/vbmokin/data-science-for-tabular-data-advanced-techniques)
", "rawMarkdown": "@pocketsuteado, congratulations! Thanks for sharing your approaches and magic! I added your post to my collection of the best kernels and posts of winners of Kaggle Prize Competitions: [Data Science for tabular data: Advanced Techniques] (https://www.kaggle.com/vbmokin/data-science-for-tabular-data-advanced-techniques) ", "votes": 1 }, { "id": 479570, "postDate": "2019-02-27T08:51:29.550Z", "content": "Thank you too, nice to team up with you!
", "rawMarkdown": "Thank you too, nice to team up with you!", "votes": 3, "replies": [ { "id": 479585, "postDate": "2019-02-27T09:02:35.920Z", "content": "Thanks! Nice to hear that from you :)\nYou guys were such a beast.
", "rawMarkdown": "Thanks! Nice to hear that from you :)\nYou guys were such a beast.", "votes": 1 } ] }, { "id": 483697, "postDate": "2019-03-05T01:58:47.857Z", "content": "Could you share you code to me please?
", "rawMarkdown": "Could you share you code to me please?", "votes": 1, "replies": [ { "id": 483845, "postDate": "2019-03-05T08:40:20.423Z", "content": "I usually upload them to Github few weeks after the competition ends.
", "rawMarkdown": "I usually upload them to Github few weeks after the competition ends." } ] }, { "id": 480681, "postDate": "2019-02-28T14:14:40.920Z", "content": "Congrats pocket, good work !
", "rawMarkdown": "Congrats pocket, good work ! ", "votes": 1 }, { "id": 480575, "postDate": "2019-02-28T11:23:00.453Z", "content": "Congratulations on getting gold. I wanted to understand a couple of things.\n1) Can you explain \" last day of transaction\" feature. What was the reference date you used?\n2) What is pseudo labeling? Is it about adding noice to the label?
", "rawMarkdown": "Congratulations on getting gold. I wanted to understand a couple of things.\n1) Can you explain \" last day of transaction\" feature. What was the reference date you used?\n2) What is pseudo labeling? Is it about adding noice to the label? ", "votes": 1, "replies": [ { "id": 483843, "postDate": "2019-03-05T08:38:24.237Z", "content": "1) The day of the month of the last transaction.\n2) Google is your friend. You can also read the Toxic Comment competition winning solution for more detail.
", "rawMarkdown": "1) The day of the month of the last transaction.\n2) Google is your friend. You can also read the Toxic Comment competition winning solution for more detail.", "votes": 2 } ] }, { "id": 480297, "postDate": "2019-02-28T02:39:38.377Z", "content": "Congrats. It is amazing you did it in just 40 days.
", "rawMarkdown": "Congrats. It is amazing you did it in just 40 days.", "votes": 1 }, { "id": 479985, "postDate": "2019-02-27T16:42:50.103Z", "content": "Thanks for the detailed write-up. Very informative and congratulations to you and your team!
", "rawMarkdown": "Thanks for the detailed write-up. Very informative and congratulations to you and your team!", "votes": 1 }, { "id": 479852, "postDate": "2019-02-27T13:54:54.143Z", "content": "Congrats..!!
\n\nThanks for sharing.
", "rawMarkdown": "Congrats..!!\n\nThanks for sharing.", "votes": 1, "replies": [ { "id": 479865, "postDate": "2019-02-27T14:10:29.727Z", "content": "Thanks!!
", "rawMarkdown": "Thanks!!" } ] }, { "id": 479695, "postDate": "2019-02-27T10:49:19.340Z", "content": "Congratulations @pocketsuteado and team for surviving the shakeup still in gold medal territory. Thanks for sharing your solution.
", "rawMarkdown": "Congratulations @pocketsuteado and team for surviving the shakeup still in gold medal territory. Thanks for sharing your solution.", "votes": 1, "replies": [ { "id": 479861, "postDate": "2019-02-27T14:07:30.210Z", "content": "Thanks! I actually thought there will be about 30% chance that I will survive, based on my experience :)
", "rawMarkdown": "Thanks! I actually thought there will be about 30% chance that I will survive, based on my experience :)" }, { "id": 480208, "postDate": "2019-02-27T23:14:28.233Z", "content": "I also knew I would jump up in the shakeup but not sure by how much. Mostly because I did not go with the crowd on the outlier stuff. @pocketsuteado, you definately have a better intuition than me about this shakeup :-)
", "rawMarkdown": "I also knew I would jump up in the shakeup but not sure by how much. Mostly because I did not go with the crowd on the outlier stuff. @pocketsuteado, you definately have a better intuition than me about this shakeup :-)" } ] }, { "id": 479599, "postDate": "2019-02-27T09:16:54.417Z", "content": "Nice job!! Congrats!
", "rawMarkdown": "Nice job!! Congrats!", "votes": 1, "replies": [ { "id": 479860, "postDate": "2019-02-27T14:06:48.880Z", "content": "Thanks!
", "rawMarkdown": "Thanks!" } ] }, { "id": 479560, "postDate": "2019-02-27T08:44:34.590Z", "content": "Congratulations!!! What is the output of binary-prediction model?
Just like everyone is doing, I re-labeled the target to 0(not -33) and 1(is -33), and did a prediction.
\nSo the output is an array of length(card_id), each element being a float between 0 and 1
Congratulations! Can you give more detail about ''finally found that a simple ridge including non-outlier model + binary-prediction model works really well.\" Thank you!
", "rawMarkdown": "Congratulations! Can you give more detail about ''finally found that a simple ridge including non-outlier model + binary-prediction model works really well.\" Thank you! ", "votes": 1, "replies": [ { "id": 479582, "postDate": "2019-02-27T09:00:04.030Z", "content": "Thanks!
\nI think Marcus would be the better person to explain, but for example,\nI made 3 out-of-fold models of (binary, non-outlier, all-data) and used the 3 predictions as the sole input into a ridge regression, which made the final prediction.
Congratulations @pocketsuteado thanks for sharing your work.
", "rawMarkdown": "Congratulations @pocketsuteado thanks for sharing your work.\n", "votes": 1, "replies": [ { "id": 479578, "postDate": "2019-02-27T08:54:57.357Z", "content": "Thanks!
", "rawMarkdown": "Thanks!" } ] }, { "id": 479845, "postDate": "2019-02-27T13:44:47.973Z", "content": "Brilliant write-up, and thanks for sharing - congratulation to everyone in the team :)
", "rawMarkdown": "Brilliant write-up, and thanks for sharing - congratulation to everyone in the team :)", "votes": 2, "replies": [ { "id": 479864, "postDate": "2019-02-27T14:10:18.637Z", "content": "Thanks :)
\nActually, I expected you to shake up into gold, but maybe we were really lucky this competition.
\nWish you luck in the next one!
Congrats to the team for saying in gold zone despite the shakeup, I was rooting for my former Plasticc team mates!
", "rawMarkdown": "Congrats to the team for saying in gold zone despite the shakeup, I was rooting for my former Plasticc team mates!", "votes": 2, "replies": [ { "id": 479835, "postDate": "2019-02-27T13:35:08.927Z", "content": "haha, thank you!
", "rawMarkdown": "haha, thank you!", "votes": 1 }, { "id": 479858, "postDate": "2019-02-27T14:04:34.540Z", "content": "You know what?
\nI was like \"If our team fails to stay in gold, the blame will be on me, because they managed to get gold in Plasticc with CPMP...\" so I tried really hard. hahaha
haha, I think they would have got a gold without me in Plasticc anyway!
", "rawMarkdown": "haha, I think they would have got a gold without me in Plasticc anyway!", "votes": 1 } ] }, { "id": 479553, "postDate": "2019-02-27T08:36:42.923Z", "content": "Congrats Pocket-san
", "rawMarkdown": "Congrats Pocket-san", "votes": 2, "replies": [ { "id": 479572, "postDate": "2019-02-27T08:53:17.800Z", "content": "Thanks senkin-san!
\nNowadays, I never seem to be able to beat you ;)
no no, you got solo gold earlier than me,let's see who can be gm first.
", "rawMarkdown": "no no, you got solo gold earlier than me,let's see who can be gm first.", "votes": 1 } ] }, { "id": 479547, "postDate": "2019-02-27T08:32:37.433Z", "content": "Congrats Pocket and the rest of the team. And thank you for your detailled solution (as always :)) !
", "rawMarkdown": "Congrats Pocket and the rest of the team. And thank you for your detailled solution (as always :)) !", "votes": 2, "replies": [ { "id": 479577, "postDate": "2019-02-27T08:54:38.090Z", "content": "Thanks :)
\nI never thought that anyone could be following all my solutions :)
Is conditional features about split hist/new_trans into subset according to the value of authorized_flag or card_id?
Yes the former. I split the hist/newtrans by the value of categories, and aggregated within the subset.\nYou could make a lot of features with different categories, but those were the ones that survived my feature selection.
", "rawMarkdown": "Yes the former. I split the hist/newtrans by the value of categories, and aggregated within the subset.\nYou could make a lot of features with different categories, but those were the ones that survived my feature selection." } ] }, { "id": 705616, "postDate": "2019-12-29T06:47:01.877Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 739468, "author_name": "Vitalii Mokin", "author_url": "", "post_date": "2020-02-07T21:09:12.997000", "content": "@pocketsuteado, congratulations! Thanks for sharing your approaches and magic! I added your post to my collection of the best kernels and posts of winners of Kaggle Prize Competitions: [Data Science for tabular data: Advanced Techniques] (https://www.kaggle.com/vbmokin/data-science-for-tabular-data-advanced-techniques)
", "votes": 1, "replies": [] }, { "id": 479570, "author_name": "khyeh", "author_url": "", "post_date": "2019-02-27T08:51:29.550000", "content": "Thank you too, nice to team up with you!
", "votes": 3, "replies": [ { "id": 479585, "author_name": "pocket", "author_url": "", "post_date": "2019-02-27T09:02:35.920000", "content": "Thanks! Nice to hear that from you :)\nYou guys were such a beast.
", "votes": 1, "replies": [] } ] }, { "id": 483697, "author_name": "Mike", "author_url": "", "post_date": "2019-03-05T01:58:47.857000", "content": "Could you share you code to me please?
", "votes": 1, "replies": [ { "id": 483845, "author_name": "pocket", "author_url": "", "post_date": "2019-03-05T08:40:20.423000", "content": "I usually upload them to Github few weeks after the competition ends.
", "votes": 0, "replies": [] } ] }, { "id": 480681, "author_name": "Miljan Stojiljkovic", "author_url": "", "post_date": "2019-02-28T14:14:40.920000", "content": "Congrats pocket, good work !
", "votes": 1, "replies": [] }, { "id": 480575, "author_name": "Yuvraj", "author_url": "", "post_date": "2019-02-28T11:23:00.453000", "content": "Congratulations on getting gold. I wanted to understand a couple of things.\n1) Can you explain \" last day of transaction\" feature. What was the reference date you used?\n2) What is pseudo labeling? Is it about adding noice to the label?
", "votes": 1, "replies": [ { "id": 483843, "author_name": "pocket", "author_url": "", "post_date": "2019-03-05T08:38:24.237000", "content": "1) The day of the month of the last transaction.\n2) Google is your friend. You can also read the Toxic Comment competition winning solution for more detail.
", "votes": 2, "replies": [] } ] }, { "id": 480297, "author_name": "Anthony Chiu", "author_url": "", "post_date": "2019-02-28T02:39:38.377000", "content": "Congrats. It is amazing you did it in just 40 days.
", "votes": 1, "replies": [] }, { "id": 479985, "author_name": "RVPrakash", "author_url": "", "post_date": "2019-02-27T16:42:50.103000", "content": "Thanks for the detailed write-up. Very informative and congratulations to you and your team!
", "votes": 1, "replies": [] }, { "id": 479852, "author_name": "Sreedevi Sasidharan", "author_url": "", "post_date": "2019-02-27T13:54:54.143000", "content": "Congrats..!!
\n\nThanks for sharing.
", "votes": 1, "replies": [ { "id": 479865, "author_name": "pocket", "author_url": "", "post_date": "2019-02-27T14:10:29.727000", "content": "Thanks!!
", "votes": 0, "replies": [] } ] }, { "id": 479695, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-02-27T10:49:19.340000", "content": "Congratulations @pocketsuteado and team for surviving the shakeup still in gold medal territory. Thanks for sharing your solution.
", "votes": 1, "replies": [ { "id": 479861, "author_name": "pocket", "author_url": "", "post_date": "2019-02-27T14:07:30.210000", "content": "Thanks! I actually thought there will be about 30% chance that I will survive, based on my experience :)
", "votes": 0, "replies": [] }, { "id": 480208, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-02-27T23:14:28.233000", "content": "I also knew I would jump up in the shakeup but not sure by how much. Mostly because I did not go with the crowd on the outlier stuff. @pocketsuteado, you definately have a better intuition than me about this shakeup :-)
", "votes": 0, "replies": [] } ] }, { "id": 479599, "author_name": "Paco Diaz", "author_url": "", "post_date": "2019-02-27T09:16:54.417000", "content": "Nice job!! Congrats!
", "votes": 1, "replies": [ { "id": 479860, "author_name": "pocket", "author_url": "", "post_date": "2019-02-27T14:06:48.880000", "content": "Thanks!
", "votes": 0, "replies": [] } ] }, { "id": 479560, "author_name": "Manchoi", "author_url": "", "post_date": "2019-02-27T08:44:34.590000", "content": "Congratulations!!! What is the output of binary-prediction model?
Just like everyone is doing, I re-labeled the target to 0(not -33) and 1(is -33), and did a prediction.
\nSo the output is an array of length(card_id), each element being a float between 0 and 1
Congratulations! Can you give more detail about ''finally found that a simple ridge including non-outlier model + binary-prediction model works really well.\" Thank you!
", "votes": 1, "replies": [ { "id": 479582, "author_name": "pocket", "author_url": "", "post_date": "2019-02-27T09:00:04.030000", "content": "Thanks!
\nI think Marcus would be the better person to explain, but for example,\nI made 3 out-of-fold models of (binary, non-outlier, all-data) and used the 3 predictions as the sole input into a ridge regression, which made the final prediction.
Congratulations @pocketsuteado thanks for sharing your work.
", "votes": 1, "replies": [ { "id": 479578, "author_name": "pocket", "author_url": "", "post_date": "2019-02-27T08:54:57.357000", "content": "Thanks!
", "votes": 0, "replies": [] } ] }, { "id": 479845, "author_name": "Yifan Xie", "author_url": "", "post_date": "2019-02-27T13:44:47.973000", "content": "Brilliant write-up, and thanks for sharing - congratulation to everyone in the team :)
", "votes": 2, "replies": [ { "id": 479864, "author_name": "pocket", "author_url": "", "post_date": "2019-02-27T14:10:18.637000", "content": "Thanks :)
\nActually, I expected you to shake up into gold, but maybe we were really lucky this competition.
\nWish you luck in the next one!
Congrats to the team for saying in gold zone despite the shakeup, I was rooting for my former Plasticc team mates!
", "votes": 2, "replies": [ { "id": 479835, "author_name": "khyeh", "author_url": "", "post_date": "2019-02-27T13:35:08.927000", "content": "haha, thank you!
", "votes": 1, "replies": [] }, { "id": 479858, "author_name": "pocket", "author_url": "", "post_date": "2019-02-27T14:04:34.540000", "content": "You know what?
\nI was like \"If our team fails to stay in gold, the blame will be on me, because they managed to get gold in Plasticc with CPMP...\" so I tried really hard. hahaha
haha, I think they would have got a gold without me in Plasticc anyway!
", "votes": 1, "replies": [] } ] }, { "id": 479553, "author_name": "senkin13", "author_url": "", "post_date": "2019-02-27T08:36:42.923000", "content": "Congrats Pocket-san
", "votes": 2, "replies": [ { "id": 479572, "author_name": "pocket", "author_url": "", "post_date": "2019-02-27T08:53:17.800000", "content": "Thanks senkin-san!
\nNowadays, I never seem to be able to beat you ;)
no no, you got solo gold earlier than me,let's see who can be gm first.
", "votes": 1, "replies": [] } ] }, { "id": 479547, "author_name": "Arthur Llau", "author_url": "", "post_date": "2019-02-27T08:32:37.433000", "content": "Congrats Pocket and the rest of the team. And thank you for your detailled solution (as always :)) !
", "votes": 2, "replies": [ { "id": 479577, "author_name": "pocket", "author_url": "", "post_date": "2019-02-27T08:54:38.090000", "content": "Thanks :)
\nI never thought that anyone could be following all my solutions :)
Is conditional features about split hist/new_trans into subset according to the value of authorized_flag or card_id?
Yes the former. I split the hist/newtrans by the value of categories, and aggregated within the subset.\nYou could make a lot of features with different categories, but those were the ones that survived my feature selection.
", "votes": 0, "replies": [] } ] }, { "id": 705616, "author_name": "", "author_url": "", "post_date": "2019-12-29T06:47:01.877000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "479529": "My journey\n------------------------------------------\nSo this competition has been a long long journey for me. \nIt felt way longer than it actually was, which is just 40 days. 40days ago, I wanted to take a break from another competition, and decided to do a \"40days challenge\".\n\nI started off by aggregating a lot of the time-series, but it was hard to even beat the kernel. The data seemed to have so little signal, almost everything that I did didn't work. \nAfter a week or two, I somehow managed to score 3.636-3.630 locally, which happened to be around 3.608-3.605 private. I was running out of ideas, so I decided to team merge.\n\nAfter merging with my team, we merged our features and re-run our models. Our best single model improved about +0.002 at that time.\nWe searched for a good way to ensemble our models, but couldn't find a good way, especially with all the post-process that was going on. \nJust one week before the deadline, @marcuslin finally found that a simple ridge including non-outlier model + binary-prediction model works really well. Our score jumped to the 3.66x range in public, which was 3.600 range in private.\n\nFor our final submission, we were torn between 3subs. One with heavy post-process, one with conservative post-process, and one without post-process. \nWe were pessimistic about our final position, so we took the risk and selected the two with post-process. This was actually a mistake, but still good enough for 18th place. \nHad we selected our no-post-process sub, we would have placed around 10th place, but that is just another story.\n\nI really feel lucky/happy about the final result, since in the heat of the competition, I could neither trust the CV nor could I trust the LB, which means I was flying blind.\n\nSince some people may have interest, I will describe the technical side of things, my features and my models below.\n\n------------------------------------------\nFeatures\n------------------------------------------\n**Used features** \n1)Basic features: very good \nAggregations like the kernel. Not much to say.\n\n2)Time features: includes magic features \nFeatures about When there was a purchase.\nThe kernel lack \"the last day of transaction\", which is magically good.\n\n3)Conditional features: good \nThe aggregation when (authorized_flag == 1, 0), (city_id == -1)\n\n4)Recent features: good \nThe aggregation when (month_lag >= -2, or 0)\n\n5)Prediction of time: probably good \nPredict the magic_feature:(last day of month_lag2 - last purchase_day) from only the historical_transaction features.\nThen do (predicted_days - actual days), and drop the predicted days.\n\n6)Time features: maybe good? \nhttps://www.kaggle.com/denzo123/a-closer-look-at-date-variables\nHour and DayOfWeek seems to be very important (but hard to improve the cv-score).\nI binned \"hour\" to [6-12, 12-17, 17-21, 21-05], which seems to add very small value(maybe).\nBy the way, 00:00:00 has a very high target value (probably because it is periodic purchases), but I couldn't improve the CV score with a flag, maybe because there are too few occurrence...\n\n7)Target-encoding features: maybe good? destablized the score... \nOOF-target encoding of categories(merchant_id, subsector_id).\n\n8)(+, -, /, *) of two features: maybe stablizes the score? not sure...\n\n------------------------------------------------------\n**Unusued features** \nI had tons of ideas which didn't work. Here are some of the interesting ones\n\nLDA features: thrown away \nthe LDA topic of each categories (like in TalkingData competition).\nI guess categories have no information!\n\nSequential authorized_flag=\"N\": thrown away \nSometimes, there are many authorized_flag=\"N\", in a sequence.\n\nPeriodic buys: thrown away \nSometimes, there are periodic purchases. Like, always once a month with the same price, at the same merchant.\n\nEncode features: thrown away \nEncode the categories by mean_purchase_amount.\nTake the difference (purchase_amount - mean_purchase_amount).\n\n------------------------------------------\nModels\n------------------------------------------\n**Models/post-process** \nIn general, stacking with simpler algorithms worked better, so Ridge/Linear regression seemed better than LGBM.\nMy teammates were very successful with stacking, so I will leave it for them to write.\n\n------------------------------------------\n**Models which didn't work/weren't used** \nTime-series prediction: didn't work \nLike in the HomeCredit competition, predict the target based on the raw time-series/monthly time-series. Neither worked.\n\nIsotonic regression: didn't work \nI probably did this wrong, but I couldn't make it work\n\nRandom stacking: didn't use \nFor stacking, I stacked upon automatically-generated-random-models by randomly selecting a subset of my features. We did not select the stack, since the publicLB score was bad, but the CV and privateLB score(3.660) is actually better than our final submission.\nNeed some more experiments to confirm.\n\nPseudo-labeling: didn't use \nPseudo-labeling with our best stack seems to improve our base model on the CV and privateLB. But it did bad on the publicLB, so we didn't use it.\nPL probably worked both on LB/CV earlier with NN, but we didn't have enough time to squeeze it into our final stack.\n\n------------------------------------------\nConclusion\n------------------------------------------\nI really want to thank my teammates for doing a fantastic job in this competition. \nWithout my team, I would not have been in this place. \nI also want to thank and congratulate @philculliton for his first competition without a leak!\nI hope you won't ban more than 221 people so that I remain in gold :)\n\nAnd.. I guess that is it.\nThank you all for participating in this competition, and good luck ;)\n\n\n\n\n", "739468": "@pocketsuteado, congratulations! Thanks for sharing your approaches and magic! I added your post to my collection of the best kernels and posts of winners of Kaggle Prize Competitions: [Data Science for tabular data: Advanced Techniques] (https://www.kaggle.com/vbmokin/data-science-for-tabular-data-advanced-techniques) ", "479570": "Thank you too, nice to team up with you!", "483697": "Could you share you code to me please?", "480681": "Congrats pocket, good work ! ", "480575": "Congratulations on getting gold. I wanted to understand a couple of things.\n1) Can you explain \" last day of transaction\" feature. What was the reference date you used?\n2) What is pseudo labeling? Is it about adding noice to the label? ", "480297": "Congrats. It is amazing you did it in just 40 days.", "479985": "Thanks for the detailed write-up. Very informative and congratulations to you and your team!", "479852": "Congrats..!!\n\nThanks for sharing.", "479695": "Congratulations @pocketsuteado and team for surviving the shakeup still in gold medal territory. Thanks for sharing your solution.", "479599": "Nice job!! Congrats!", "479560": "Congratulations!!! What is the output of `binary-prediction model`? ", "479536": "Congratulations! Can you give more detail about ''finally found that a simple ridge including non-outlier model + binary-prediction model works really well.\" Thank you! ", "479532": "Congratulations @pocketsuteado thanks for sharing your work.\n", "479845": "Brilliant write-up, and thanks for sharing - congratulation to everyone in the team :)", "479686": "Congrats to the team for saying in gold zone despite the shakeup, I was rooting for my former Plasticc team mates!", "479553": "Congrats Pocket-san", "479547": "Congrats Pocket and the rest of the team. And thank you for your detailled solution (as always :)) !", "479562": "Is `conditional features` about split hist/new_trans into subset according to the value of authorized_flag or card_id?", "705616": "" } }