So, there went our submissions. Still more than two months before we know better but by now we can speak of some beliefs, here are mine:
\n\nAbout point 3, imo it is a flaw of the competition that this \"allowed\" scraping was left as an open option. Maybe just me, but I find it more interesting when we can focus our atention on same shared dataset.
\n\nRelated to this, and being fair, I think the solution given to the huge leakage disaster was quite good. Even with the good side-effect of making it a real business problem.
\n\nBut... one unintended consequence (this is, apart of the \"scraping\" issue): This could easily be the easiest to overfit competition held in Kaggle, considering noise to signal ratio implied by the ridiculously small number of \"positives\".
\n\nAll that said, for me it has been a fun exercise on how not to overfit an almost imposible dataset, looking forward to see how predictable all this was in the end.
", "messages": [ { "id": "431558", "postDate": "12/02/2018 13:45:44", "content": "So, there went our submissions. Still more than two months before we know better but by now we can speak of some beliefs, here are mine:
\n\nAbout point 3, imo it is a flaw of the competition that this \"allowed\" scraping was left as an open option. Maybe just me, but I find it more interesting when we can focus our atention on same shared dataset.
\n\nRelated to this, and being fair, I think the solution given to the huge leakage disaster was quite good. Even with the good side-effect of making it a real business problem.
\n\nBut... one unintended consequence (this is, apart of the \"scraping\" issue): This could easily be the easiest to overfit competition held in Kaggle, considering noise to signal ratio implied by the ridiculously small number of \"positives\".
\n\nAll that said, for me it has been a fun exercise on how not to overfit an almost imposible dataset, looking forward to see how predictable all this was in the end.
", "rawMarkdown": "So, there went our submissions. Still more than two months before we know better but by now we can speak of some beliefs, here are mine:\n\n- 1 (main belief): This competition, relaunched (v2), is not completely a lottery. Only a huge noise/signal ratio. But **there should be signal**.\n- 2 The competition is indeed **of some value in \"real world\"**, as it fits in CLV prediction without leakages in quite extreme setup. (This was not true in its initial \"future looking\" form)\n- 3 The posibility of \"scraping\" data of the gap period (that I didn't exploit ) should give a clear advantage to anyone who properly uses it, as **recency** is such important part of CLV prediction. \n\nAbout point 3, imo it is **a flaw** of the competition that this \"allowed\" scraping was left as an open option. Maybe just me, but I find it more interesting when we can focus our atention on same shared dataset.\n\nRelated to this, and being fair, I think the solution given to the huge leakage disaster was quite good. Even with the good side-effect of making it a real business problem. \n\nBut... one unintended consequence (this is, apart of the \"scraping\" issue): **This could easily be the easiest to overfit competition held in Kaggle**, considering noise to signal ratio implied by the ridiculously small number of \"positives\".\n\nAll that said, for me it has been a fun exercise on how not to overfit an almost imposible dataset, looking forward to see how predictable all this was in the end.", "votes": null }, { "id": "431726", "postDate": "12/02/2018 20:11:00", "content": "I tottally agree with the fact that dealing with overfiting was crucial in this competition. My biggest fear (and the reason I changed many times my training/validation scheme) was underestimating the validation error.
", "rawMarkdown": "I tottally agree with the fact that dealing with overfiting was crucial in this competition. My biggest fear (and the reason I changed many times my training/validation scheme) was underestimating the validation error.", "votes": null }, { "id": "432048", "postDate": "12/03/2018 10:11:03", "content": "@miguel perez You are so right about that scraping part! It's such an obvious advantage, I am slamming my head to wall now because I didn't even think of this. :)
", "rawMarkdown": "[@miguel perez](https://www.kaggle.com/miguelpm) You are so right about that scraping part! It's such an obvious advantage, I am slamming my head to wall now because I didn't even think of this. :)", "votes": null }, { "id": "432109", "postDate": "12/03/2018 12:28:59", "content": "@miguelpm I agree with your conclusions, but I think the easiest to overfit competition is the current Two Sigma one. Financial data are 99% noise, 1% signal, with lots and lots of noise outliers, that are tempting for models to overfit.
", "rawMarkdown": "miguelpm I agree with your conclusions, but I think the easiest to overfit competition is the current Two Sigma one. Financial data are 99% noise, 1% signal, with lots and lots of noise outliers, that are tempting for models to overfit.", "votes": null }, { "id": "432139", "postDate": "12/03/2018 13:04:13", "content": "That's a good point, @BurritoDan. Financial prices are -most of the time- so close to random walks that I was not even considering 2Sigma one as candidate for \"worst overfit-able\" prize :-)
", "rawMarkdown": "That's a good point, @BurritoDan. Financial prices are -most of the time- so close to random walks that I was not even considering 2Sigma one as candidate for \"worst overfit-able\" prize :-)", "votes": null }, { "id": "434730", "postDate": "12/06/2018 21:23:03", "content": "Nice summary=) About 2nd point - this competition could be even close to real world problem, for example:
\n\ngetting rid of gap between test and target;
reducing the target time interval (2 month) to lower values (2-4 weeks for ex.);
predicting not sum of revenues of individual users, but some apriori defined user groups (To my mind g-store may possibly knew some parts of their customer segmentation - marketing budgets will be spent on groups not individuals =) but of cause my suggestions could be wrong).
On the second point, I think actually a longer forecast interval would help. Since the median transaction rate for repeat purchasers is about once a month, and many customers are a lot less frequent, a forecast window of 2 months is quite short. Even a regular monthly purchaser might not transact in the window by pure chance. This pushes the predictions towards zero, and makes the model more uniformly conservative.
\n\nIn a business context, ideally we’d be forecasting the “customer lifetime value”, but of course in practice this is not measurable, but I reckon 6 months would do it. Because of the data leak, the organisers couldn’t easily hold back 6 months worth of data of course.
", "rawMarkdown": "On the second point, I think actually a longer forecast interval would help. Since the median transaction rate for repeat purchasers is about once a month, and many customers are a lot less frequent, a forecast window of 2 months is quite short. Even a regular monthly purchaser might not transact in the window by pure chance. This pushes the predictions towards zero, and makes the model more uniformly conservative.\n\nIn a business context, ideally we’d be forecasting the “customer lifetime value”, but of course in practice this is not measurable, but I reckon 6 months would do it. Because of the data leak, the organisers couldn’t easily hold back 6 months worth of data of course.", "votes": null }, { "id": "435190", "postDate": "12/07/2018 16:39:05", "content": "All those are sensible alternative time frames/flavours of the CLV, and agreed could make the comp. closer to real world. I didn't mean point 2 to sound like this comp. is a flawless CLV modelling;
\n\nTo clarify a bit my comment on \"some value in real world\", it's not that the design of the comp. is optimal, or exactly how CLV real problems present for most business, the point just expresses my view that, even if noisy, the most succesfull approach for this problem will be also good to tackle with many other flavours of CLV problem, hence \"useful in real life\".
", "rawMarkdown": "All those are sensible alternative time frames/flavours of the CLV, and agreed could make the comp. closer to real world. I didn't mean point 2 to sound like this comp. is a flawless CLV modelling;\n\nTo clarify a bit my comment on \"some value in real world\", it's not that the design of the comp. is optimal, or exactly how CLV real problems present for most business, the point just expresses my view that, even if noisy, the most succesfull approach for this problem will be also good to tackle with many other flavours of CLV problem, hence \"useful in real life\".", "votes": null }, { "id": "435352", "postDate": "12/07/2018 22:18:35", "content": "Burrito Dan, definitely data leak played its role in final problem formulation =)
\n\nBack to forecasting time interval - you marked a good point about purchasing frequency of regular customers, this value should be used to set the forecasting interval, and 2 month sounds reasonable in this context as you mentioned.
\n\nWhat do you think about setting sum of revenues of some group of users (not individuals) as the potential target? 2 month could be redundant time interval in this case.
", "rawMarkdown": "Burrito Dan, definitely data leak played its role in final problem formulation =) \n\nBack to forecasting time interval - you marked a good point about purchasing frequency of regular customers, this value should be used to set the forecasting interval, and 2 month sounds reasonable in this context as you mentioned. \n\nWhat do you think about setting sum of revenues of some group of users (not individuals) as the potential target? 2 month could be redundant time interval in this case.", "votes": null }, { "id": "435674", "postDate": "12/08/2018 14:16:23", "content": "Yes, if the purchasers are independent, their aggregate should be much more predictable, even over a short timeframe.
", "rawMarkdown": "Yes, if the purchasers are independent, their aggregate should be much more predictable, even over a short timeframe.", "votes": null }, { "id": "436344", "postDate": "12/10/2018 06:38:32", "content": "I joined this competition from day one. With leaks, data and the end decision to continue it. I dropped it off. I totally lost enthusiasm now and switched to Quoras Insincere competition. This competition was clearly a infamous compared to all other competition
", "rawMarkdown": "I joined this competition from day one. With leaks, data and the end decision to continue it. I dropped it off. I totally lost enthusiasm now and switched to Quoras Insincere competition. This competition was clearly a infamous compared to all other competition", "votes": null }, { "id": "453325", "postDate": "01/10/2019 02:35:25", "content": "There is a gap between train_v2.csv and test_v2.csv, I can find that the competition did not provide information of all users. If scrap the data of the gap period, I think it is important to focus on the fullVisitorId.
", "rawMarkdown": "There is a gap between train_v2.csv and test_v2.csv, I can find that the competition did not provide information of all users. If scrap the data of the gap period, I think it is important to focus on the fullVisitorId.", "votes": null } ], "comments": [ { "id": 431726, "author_name": "luisduque440", "author_url": "", "post_date": "12/02/2018 20:11:00", "content": "I tottally agree with the fact that dealing with overfiting was crucial in this competition. My biggest fear (and the reason I changed many times my training/validation scheme) was underestimating the validation error.
", "votes": null, "replies": [] }, { "id": 432048, "author_name": "moritzbach", "author_url": "", "post_date": "12/03/2018 10:11:03", "content": "@miguel perez You are so right about that scraping part! It's such an obvious advantage, I am slamming my head to wall now because I didn't even think of this. :)
", "votes": null, "replies": [] }, { "id": 432109, "author_name": "burritodan", "author_url": "", "post_date": "12/03/2018 12:28:59", "content": "@miguelpm I agree with your conclusions, but I think the easiest to overfit competition is the current Two Sigma one. Financial data are 99% noise, 1% signal, with lots and lots of noise outliers, that are tempting for models to overfit.
", "votes": null, "replies": [ { "id": 432139, "author_name": "miguelpm", "author_url": "", "post_date": "12/03/2018 13:04:13", "content": "That's a good point, @BurritoDan. Financial prices are -most of the time- so close to random walks that I was not even considering 2Sigma one as candidate for \"worst overfit-able\" prize :-)
", "votes": null, "replies": [] } ] }, { "id": 434730, "author_name": "chesnokov", "author_url": "", "post_date": "12/06/2018 21:23:03", "content": "Nice summary=) About 2nd point - this competition could be even close to real world problem, for example:
\n\ngetting rid of gap between test and target;
reducing the target time interval (2 month) to lower values (2-4 weeks for ex.);
predicting not sum of revenues of individual users, but some apriori defined user groups (To my mind g-store may possibly knew some parts of their customer segmentation - marketing budgets will be spent on groups not individuals =) but of cause my suggestions could be wrong).
On the second point, I think actually a longer forecast interval would help. Since the median transaction rate for repeat purchasers is about once a month, and many customers are a lot less frequent, a forecast window of 2 months is quite short. Even a regular monthly purchaser might not transact in the window by pure chance. This pushes the predictions towards zero, and makes the model more uniformly conservative.
\n\nIn a business context, ideally we’d be forecasting the “customer lifetime value”, but of course in practice this is not measurable, but I reckon 6 months would do it. Because of the data leak, the organisers couldn’t easily hold back 6 months worth of data of course.
", "votes": null, "replies": [] }, { "id": 435190, "author_name": "miguelpm", "author_url": "", "post_date": "12/07/2018 16:39:05", "content": "All those are sensible alternative time frames/flavours of the CLV, and agreed could make the comp. closer to real world. I didn't mean point 2 to sound like this comp. is a flawless CLV modelling;
\n\nTo clarify a bit my comment on \"some value in real world\", it's not that the design of the comp. is optimal, or exactly how CLV real problems present for most business, the point just expresses my view that, even if noisy, the most succesfull approach for this problem will be also good to tackle with many other flavours of CLV problem, hence \"useful in real life\".
", "votes": null, "replies": [] }, { "id": 435352, "author_name": "chesnokov", "author_url": "", "post_date": "12/07/2018 22:18:35", "content": "Burrito Dan, definitely data leak played its role in final problem formulation =)
\n\nBack to forecasting time interval - you marked a good point about purchasing frequency of regular customers, this value should be used to set the forecasting interval, and 2 month sounds reasonable in this context as you mentioned.
\n\nWhat do you think about setting sum of revenues of some group of users (not individuals) as the potential target? 2 month could be redundant time interval in this case.
", "votes": null, "replies": [] }, { "id": 435674, "author_name": "burritodan", "author_url": "", "post_date": "12/08/2018 14:16:23", "content": "Yes, if the purchasers are independent, their aggregate should be much more predictable, even over a short timeframe.
", "votes": null, "replies": [] } ] }, { "id": 436344, "author_name": "shaz13", "author_url": "", "post_date": "12/10/2018 06:38:32", "content": "I joined this competition from day one. With leaks, data and the end decision to continue it. I dropped it off. I totally lost enthusiasm now and switched to Quoras Insincere competition. This competition was clearly a infamous compared to all other competition
", "votes": null, "replies": [] }, { "id": 453325, "author_name": "simonwang95", "author_url": "", "post_date": "01/10/2019 02:35:25", "content": "There is a gap between train_v2.csv and test_v2.csv, I can find that the competition did not provide information of all users. If scrap the data of the gap period, I think it is important to focus on the fullVisitorId.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "431558": "So, there went our submissions. Still more than two months before we know better but by now we can speak of some beliefs, here are mine:\n\n- 1 (main belief): This competition, relaunched (v2), is not completely a lottery. Only a huge noise/signal ratio. But **there should be signal**.\n- 2 The competition is indeed **of some value in \"real world\"**, as it fits in CLV prediction without leakages in quite extreme setup. (This was not true in its initial \"future looking\" form)\n- 3 The posibility of \"scraping\" data of the gap period (that I didn't exploit ) should give a clear advantage to anyone who properly uses it, as **recency** is such important part of CLV prediction. \n\nAbout point 3, imo it is **a flaw** of the competition that this \"allowed\" scraping was left as an open option. Maybe just me, but I find it more interesting when we can focus our atention on same shared dataset.\n\nRelated to this, and being fair, I think the solution given to the huge leakage disaster was quite good. Even with the good side-effect of making it a real business problem. \n\nBut... one unintended consequence (this is, apart of the \"scraping\" issue): **This could easily be the easiest to overfit competition held in Kaggle**, considering noise to signal ratio implied by the ridiculously small number of \"positives\".\n\nAll that said, for me it has been a fun exercise on how not to overfit an almost imposible dataset, looking forward to see how predictable all this was in the end.", "431726": "I tottally agree with the fact that dealing with overfiting was crucial in this competition. My biggest fear (and the reason I changed many times my training/validation scheme) was underestimating the validation error.", "432048": "[@miguel perez](https://www.kaggle.com/miguelpm) You are so right about that scraping part! It's such an obvious advantage, I am slamming my head to wall now because I didn't even think of this. :)", "432109": "miguelpm I agree with your conclusions, but I think the easiest to overfit competition is the current Two Sigma one. Financial data are 99% noise, 1% signal, with lots and lots of noise outliers, that are tempting for models to overfit.", "432139": "That's a good point, @BurritoDan. Financial prices are -most of the time- so close to random walks that I was not even considering 2Sigma one as candidate for \"worst overfit-able\" prize :-)", "434730": "Nice summary=) About 2nd point - this competition could be even close to real world problem, for example: \n\n- getting rid of gap between test and target;\n\n- reducing the target time interval (2 month) to lower values (2-4 weeks for ex.);\n\n- predicting not sum of revenues of individual users, but some apriori defined user groups (To my mind g-store may possibly knew some parts of their customer segmentation - marketing budgets will be spent on groups not individuals =) but of cause my suggestions could be wrong).", "434951": "On the second point, I think actually a longer forecast interval would help. Since the median transaction rate for repeat purchasers is about once a month, and many customers are a lot less frequent, a forecast window of 2 months is quite short. Even a regular monthly purchaser might not transact in the window by pure chance. This pushes the predictions towards zero, and makes the model more uniformly conservative.\n\nIn a business context, ideally we’d be forecasting the “customer lifetime value”, but of course in practice this is not measurable, but I reckon 6 months would do it. Because of the data leak, the organisers couldn’t easily hold back 6 months worth of data of course.", "435190": "All those are sensible alternative time frames/flavours of the CLV, and agreed could make the comp. closer to real world. I didn't mean point 2 to sound like this comp. is a flawless CLV modelling;\n\nTo clarify a bit my comment on \"some value in real world\", it's not that the design of the comp. is optimal, or exactly how CLV real problems present for most business, the point just expresses my view that, even if noisy, the most succesfull approach for this problem will be also good to tackle with many other flavours of CLV problem, hence \"useful in real life\".", "435352": "Burrito Dan, definitely data leak played its role in final problem formulation =) \n\nBack to forecasting time interval - you marked a good point about purchasing frequency of regular customers, this value should be used to set the forecasting interval, and 2 month sounds reasonable in this context as you mentioned. \n\nWhat do you think about setting sum of revenues of some group of users (not individuals) as the potential target? 2 month could be redundant time interval in this case.", "435674": "Yes, if the purchasers are independent, their aggregate should be much more predictable, even over a short timeframe.", "436344": "I joined this competition from day one. With leaks, data and the end decision to continue it. I dropped it off. I totally lost enthusiasm now and switched to Quoras Insincere competition. This competition was clearly a infamous compared to all other competition", "453325": "There is a gap between train_v2.csv and test_v2.csv, I can find that the competition did not provide information of all users. If scrap the data of the gap period, I think it is important to focus on the fullVisitorId." }, "source": "meta" }