Three months after I posted my “believes” of this competition and with quite a decent result it’s a time for a quick recap.
\n\nFirst, the bad part: The leakage issue in this one was severe and impacted when many kagglers had already put a big effort on it. As a result many were discouraged to continue after the redesign. Another bad consequence was that public leaderboard feedbak was rendered meaningless. Last, but not least a door was left open for getting better data through late web scraping what implied to choose either to take all the aditional fuss to scrape or to assume a big disadvantage.
\n\nAll that said, it’s a pitty. Because, in my opinion all the problems have hidden the interesting parts of a quite dificult competition. I was quite uninterested in the initial format as the looking ahead possibility made it useless from a business point of view.
\n\nBut after reboot… wow. A variant of CLV business problem with almost no transactions. Almost all noise. But, in theory, signal was in there. I was interested.
\n\n1) Data “augmentation”
\n\nIt was necessary to augment data somehow. As the problem is essentially a time window to time window prediction the obvious path was to use available training data to replicate a good number of time windows. But if the data exhibited -yearly- seasonality it was possibly game over. Luckily (and strangely) some simple plots showed no meaningful yearly seasonality. This was weird, possibly due to some kind of balancing and sampling of training data. But it was also good news, there was a good possibility of enough “stationarity” of the problem to use multiple time windows.
\n\nSo, a series of overlapping time windows beginning every 15 days were constructed and agregated in a single dataset. As temporal relationship was what I intended to model the same user could be used multiple times with multiple target values depending on the window. Each window would include multiple features related on number of transactions, with or without purchases, time deltas vs. windows limits and all the usuall CLV features. That led to a nice dataset with like 10 million observations, big enough to find a needle in a haystack.
\n\n2) -Almost- no tunning
\n\nSome initial attemps to model with boosting showed huge inestability. What made sense. If you don't have data to spend the smallest problem is to assess model performance, the big problem is you can not even tune hyperparameters. Even if you stick to defaults in boosting still early stopping is an important hyperparameter that you can not afford with no data. Risky as it may seem a zero tunning model was necessary. Obvious choice was random forest. It was evidently more stable and good enough. A single RF model. (Stacking was also discarded the moment overfitting was more than a possibility and no data was available to tune nothing. I don't stack that much anyway).
\n\nSo, that’s it. Those two big decissions made me quite confident to do well without overfitting, provided there was actually signal and not many people used the available scraping option.
\n\nI hope the interesting parts of this competition will \"survive\" all the problems that ocurred. It is clear that many kagglers were affected, in spite of the efforts made by Kaggle team, that in my view were quite evident and well intended.
\n\nAt the very least, looking into the future, this will be an interesting and unusual CLV variant competition to have a look at.
", "messages": [ { "id": "476536", "postDate": "02/22/2019 10:05:06", "content": "Three months after I posted my “believes” of this competition and with quite a decent result it’s a time for a quick recap.
\n\nFirst, the bad part: The leakage issue in this one was severe and impacted when many kagglers had already put a big effort on it. As a result many were discouraged to continue after the redesign. Another bad consequence was that public leaderboard feedbak was rendered meaningless. Last, but not least a door was left open for getting better data through late web scraping what implied to choose either to take all the aditional fuss to scrape or to assume a big disadvantage.
\n\nAll that said, it’s a pitty. Because, in my opinion all the problems have hidden the interesting parts of a quite dificult competition. I was quite uninterested in the initial format as the looking ahead possibility made it useless from a business point of view.
\n\nBut after reboot… wow. A variant of CLV business problem with almost no transactions. Almost all noise. But, in theory, signal was in there. I was interested.
\n\n1) Data “augmentation”
\n\nIt was necessary to augment data somehow. As the problem is essentially a time window to time window prediction the obvious path was to use available training data to replicate a good number of time windows. But if the data exhibited -yearly- seasonality it was possibly game over. Luckily (and strangely) some simple plots showed no meaningful yearly seasonality. This was weird, possibly due to some kind of balancing and sampling of training data. But it was also good news, there was a good possibility of enough “stationarity” of the problem to use multiple time windows.
\n\nSo, a series of overlapping time windows beginning every 15 days were constructed and agregated in a single dataset. As temporal relationship was what I intended to model the same user could be used multiple times with multiple target values depending on the window. Each window would include multiple features related on number of transactions, with or without purchases, time deltas vs. windows limits and all the usuall CLV features. That led to a nice dataset with like 10 million observations, big enough to find a needle in a haystack.
\n\n2) -Almost- no tunning
\n\nSome initial attemps to model with boosting showed huge inestability. What made sense. If you don't have data to spend the smallest problem is to assess model performance, the big problem is you can not even tune hyperparameters. Even if you stick to defaults in boosting still early stopping is an important hyperparameter that you can not afford with no data. Risky as it may seem a zero tunning model was necessary. Obvious choice was random forest. It was evidently more stable and good enough. A single RF model. (Stacking was also discarded the moment overfitting was more than a possibility and no data was available to tune nothing. I don't stack that much anyway).
\n\nSo, that’s it. Those two big decissions made me quite confident to do well without overfitting, provided there was actually signal and not many people used the available scraping option.
\n\nI hope the interesting parts of this competition will \"survive\" all the problems that ocurred. It is clear that many kagglers were affected, in spite of the efforts made by Kaggle team, that in my view were quite evident and well intended.
\n\nAt the very least, looking into the future, this will be an interesting and unusual CLV variant competition to have a look at.
", "rawMarkdown": "Three months after I posted my [][1] [“believes” of this competition][1] and with quite a decent result it’s a time for a quick recap.\n\nFirst, the bad part: The leakage issue in this one was severe and impacted when many kagglers had already put a big effort on it. As a result many were discouraged to continue after the redesign. Another bad consequence was that public leaderboard feedbak was rendered meaningless. Last, but not least a door was left open for getting better data through late web scraping what implied to choose either to take all the aditional fuss to scrape or to assume a big disadvantage.\n\nAll that said, it’s a pitty. Because, in my opinion all the problems have hidden the interesting parts of a quite dificult competition. I was quite uninterested in the initial format as the looking ahead possibility made it useless from a business point of view. \n\nBut after reboot… wow. A variant of CLV business problem with almost no transactions. Almost all noise. But, in theory, signal was in there. I was interested.\n\n**1) Data “augmentation”**\n\nIt was necessary to augment data somehow. As the problem is essentially a time window to time window prediction the obvious path was to use available training data to replicate a good number of time windows. But if the data exhibited -yearly- seasonality it was possibly game over. Luckily (and strangely) some simple plots showed no meaningful yearly seasonality. This was weird, possibly due to some kind of balancing and sampling of training data. But it was also good news, there was a good possibility of enough “stationarity” of the problem to use multiple time windows.\n\nSo, a series of overlapping time windows beginning every 15 days were constructed and agregated in a single dataset. As **temporal relationship** was what I intended to model the same user could be used multiple times with multiple target values depending on the window. Each window would include multiple features related on number of transactions, with or without purchases, time deltas vs. windows limits and all the usuall CLV features. That led to a nice dataset with like 10 million observations, big enough to find a needle in a haystack.\n\n**2) -Almost- no tunning**\n\nSome initial attemps to model with boosting showed huge inestability. What made sense. If you don't have data to spend the smallest problem is to assess model performance, the big problem is you can not even tune hyperparameters. Even if you stick to defaults in boosting still early stopping is an important hyperparameter that you can not afford with no data. Risky as it may seem a zero tunning model was necessary. Obvious choice was **random forest**. It was evidently more stable and good enough. A single RF model. (Stacking was also discarded the moment overfitting was more than a possibility and no data was available to tune nothing. I don't stack that much anyway). \n\nSo, that’s it. **Those two big decissions** made me quite confident to do well without overfitting, provided there was actually signal and not many people used the available scraping option.\n\nI hope the interesting parts of this competition will \"survive\" all the problems that ocurred. It is clear that many kagglers were affected, in spite of the efforts made by Kaggle team, that in my view were quite evident and well intended.\n\nAt the very least, looking into the future, this will be an interesting and unusual CLV variant competition to have a look at.\n\n\n\n\n\n\n [1]: https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/73378", "votes": null }, { "id": "476580", "postDate": "02/22/2019 11:16:16", "content": "Congratulations @miguelpm I agree that this competition had some unusual challenges at least compared to other Kaggle competitions. You found good solutions to those where many others struggled. I agree with your assessment that tuning did not help although I had a few attempts at it! I also broke the data down into temporal segments but not with the elegance that you achieved especially with your use of augmentation. Barring any major disqualifications you should have a solo gold. Well done.
", "rawMarkdown": "Congratulations @miguelpm I agree that this competition had some unusual challenges at least compared to other Kaggle competitions. You found good solutions to those where many others struggled. I agree with your assessment that tuning did not help although I had a few attempts at it! I also broke the data down into temporal segments but not with the elegance that you achieved especially with your use of augmentation. Barring any major disqualifications you should have a solo gold. Well done.", "votes": null }, { "id": "476593", "postDate": "02/22/2019 11:28:03", "content": "Thanks for sharing and Congratulations on the Solo Gold! \nI think you would have won the R prize as well?
", "rawMarkdown": "Thanks for sharing and Congratulations on the Solo Gold! \nI think you would have won the R prize as well?", "votes": null }, { "id": "476597", "postDate": "02/22/2019 11:40:51", "content": "Thanks @MichaelP, and congrats to you for your great bronze result :-)
", "rawMarkdown": "Thanks @MichaelP, and congrats to you for your great bronze result :-)", "votes": null }, { "id": "476601", "postDate": "02/22/2019 11:43:42", "content": "Thank you @SanyamButhani, About R prize I've a full R solution but any of top solutions may have used R, so really no idea... we'll find out in some weeks!
", "rawMarkdown": "Thank you @SanyamButhani, About R prize I've a full R solution but any of top solutions may have used R, so really no idea... we'll find out in some weeks!", "votes": null }, { "id": "476658", "postDate": "02/22/2019 13:37:52", "content": "Thanks, it is good to hopefully get a bronze but nowhere near as impressive as getting a gold!
\n\nBtw, I am @datahobbit - legacy of choice I made when signing up to Kaggle. @MichaelP probably gets a lot of messages intended for me - sorry about that... In case anyone else has similar issues I normally check who someone is these days by quickly hovering over their picture to reveal their kaggle id or clicking on them.
", "rawMarkdown": "Thanks, it is good to hopefully get a bronze but nowhere near as impressive as getting a gold! \n\nBtw, I am @datahobbit - legacy of choice I made when signing up to Kaggle. @MichaelP probably gets a lot of messages intended for me - sorry about that... In case anyone else has similar issues I normally check who someone is these days by quickly hovering over their picture to reveal their kaggle id or clicking on them.", "votes": null }, { "id": "476816", "postDate": "02/23/2019 08:33:14", "content": "Congratulations @miguelpm. Thanks for sharing.
", "rawMarkdown": "Congratulations @miguelpm. Thanks for sharing.", "votes": null }, { "id": "704866", "postDate": "12/28/2019 05:33:58", "content": "Thanks for sharing
", "rawMarkdown": "Thanks for sharing", "votes": null }, { "id": "1185527", "postDate": "02/04/2021 07:43:32", "content": "\"So, a series of overlapping time windows beginning every 15 days were constructed and agregated in a single dataset\".
\nI am a little confused by this part, how would the time window be created and would the size of each window be number of entries between each 15 day period?
\nCan anyone explain this in simpler terms.
\nThanks.
\"So, a series of overlapping time windows beginning every 15 days were constructed and agregated in a single dataset\".
\nI am a little confused by this part, how would the time window be created and would the size of each window be number of entries between each 15 day period?
\nCan anyone explain this in simpler terms.
\nThanks.
Congratulations @miguelpm I agree that this competition had some unusual challenges at least compared to other Kaggle competitions. You found good solutions to those where many others struggled. I agree with your assessment that tuning did not help although I had a few attempts at it! I also broke the data down into temporal segments but not with the elegance that you achieved especially with your use of augmentation. Barring any major disqualifications you should have a solo gold. Well done.
", "votes": null, "replies": [ { "id": 476597, "author_name": "miguelpm", "author_url": "", "post_date": "02/22/2019 11:40:51", "content": "Thanks @MichaelP, and congrats to you for your great bronze result :-)
", "votes": null, "replies": [] }, { "id": 476658, "author_name": "datahobbit", "author_url": "", "post_date": "02/22/2019 13:37:52", "content": "Thanks, it is good to hopefully get a bronze but nowhere near as impressive as getting a gold!
\n\nBtw, I am @datahobbit - legacy of choice I made when signing up to Kaggle. @MichaelP probably gets a lot of messages intended for me - sorry about that... In case anyone else has similar issues I normally check who someone is these days by quickly hovering over their picture to reveal their kaggle id or clicking on them.
", "votes": null, "replies": [] } ] }, { "id": 476593, "author_name": "init27", "author_url": "", "post_date": "02/22/2019 11:28:03", "content": "Thanks for sharing and Congratulations on the Solo Gold! \nI think you would have won the R prize as well?
", "votes": null, "replies": [ { "id": 476601, "author_name": "miguelpm", "author_url": "", "post_date": "02/22/2019 11:43:42", "content": "Thank you @SanyamButhani, About R prize I've a full R solution but any of top solutions may have used R, so really no idea... we'll find out in some weeks!
", "votes": null, "replies": [] } ] }, { "id": 476816, "author_name": "karthik7395", "author_url": "", "post_date": "02/23/2019 08:33:14", "content": "Congratulations @miguelpm. Thanks for sharing.
", "votes": null, "replies": [] }, { "id": 704866, "author_name": "chitreshkr", "author_url": "", "post_date": "12/28/2019 05:33:58", "content": "Thanks for sharing
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "476536": "Three months after I posted my [][1] [“believes” of this competition][1] and with quite a decent result it’s a time for a quick recap.\n\nFirst, the bad part: The leakage issue in this one was severe and impacted when many kagglers had already put a big effort on it. As a result many were discouraged to continue after the redesign. Another bad consequence was that public leaderboard feedbak was rendered meaningless. Last, but not least a door was left open for getting better data through late web scraping what implied to choose either to take all the aditional fuss to scrape or to assume a big disadvantage.\n\nAll that said, it’s a pitty. Because, in my opinion all the problems have hidden the interesting parts of a quite dificult competition. I was quite uninterested in the initial format as the looking ahead possibility made it useless from a business point of view. \n\nBut after reboot… wow. A variant of CLV business problem with almost no transactions. Almost all noise. But, in theory, signal was in there. I was interested.\n\n**1) Data “augmentation”**\n\nIt was necessary to augment data somehow. As the problem is essentially a time window to time window prediction the obvious path was to use available training data to replicate a good number of time windows. But if the data exhibited -yearly- seasonality it was possibly game over. Luckily (and strangely) some simple plots showed no meaningful yearly seasonality. This was weird, possibly due to some kind of balancing and sampling of training data. But it was also good news, there was a good possibility of enough “stationarity” of the problem to use multiple time windows.\n\nSo, a series of overlapping time windows beginning every 15 days were constructed and agregated in a single dataset. As **temporal relationship** was what I intended to model the same user could be used multiple times with multiple target values depending on the window. Each window would include multiple features related on number of transactions, with or without purchases, time deltas vs. windows limits and all the usuall CLV features. That led to a nice dataset with like 10 million observations, big enough to find a needle in a haystack.\n\n**2) -Almost- no tunning**\n\nSome initial attemps to model with boosting showed huge inestability. What made sense. If you don't have data to spend the smallest problem is to assess model performance, the big problem is you can not even tune hyperparameters. Even if you stick to defaults in boosting still early stopping is an important hyperparameter that you can not afford with no data. Risky as it may seem a zero tunning model was necessary. Obvious choice was **random forest**. It was evidently more stable and good enough. A single RF model. (Stacking was also discarded the moment overfitting was more than a possibility and no data was available to tune nothing. I don't stack that much anyway). \n\nSo, that’s it. **Those two big decissions** made me quite confident to do well without overfitting, provided there was actually signal and not many people used the available scraping option.\n\nI hope the interesting parts of this competition will \"survive\" all the problems that ocurred. It is clear that many kagglers were affected, in spite of the efforts made by Kaggle team, that in my view were quite evident and well intended.\n\nAt the very least, looking into the future, this will be an interesting and unusual CLV variant competition to have a look at.\n\n\n\n\n\n\n [1]: https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/73378", "476580": "Congratulations @miguelpm I agree that this competition had some unusual challenges at least compared to other Kaggle competitions. You found good solutions to those where many others struggled. I agree with your assessment that tuning did not help although I had a few attempts at it! I also broke the data down into temporal segments but not with the elegance that you achieved especially with your use of augmentation. Barring any major disqualifications you should have a solo gold. Well done.", "476593": "Thanks for sharing and Congratulations on the Solo Gold! \nI think you would have won the R prize as well?", "476597": "Thanks @MichaelP, and congrats to you for your great bronze result :-)", "476601": "Thank you @SanyamButhani, About R prize I've a full R solution but any of top solutions may have used R, so really no idea... we'll find out in some weeks!", "476658": "Thanks, it is good to hopefully get a bronze but nowhere near as impressive as getting a gold! \n\nBtw, I am @datahobbit - legacy of choice I made when signing up to Kaggle. @MichaelP probably gets a lot of messages intended for me - sorry about that... In case anyone else has similar issues I normally check who someone is these days by quickly hovering over their picture to reveal their kaggle id or clicking on them.", "476816": "Congratulations @miguelpm. Thanks for sharing.", "704866": "Thanks for sharing", "1185527": "\"So, a series of overlapping time windows beginning every 15 days were constructed and agregated in a single dataset\". \nI am a little confused by this part, how would the time window be created and would the size of each window be number of entries between each 15 day period?\nCan anyone explain this in simpler terms.\nThanks." }, "source": "existing_discussion", "complete": true, "competition_id": 10038, "competition_slug": "ga-customer-revenue-prediction", "team_id": 2129670, "solution_url": "/competitions/ga-customer-revenue-prediction/writeups/miguel-perez-r-the-needle-in-a-haystack-12th-place" }