Looking at the zero transactions, i see that almost the 98.6% per user has zero transactions value. Also, when i had 1.6 CV my LB was 1.66, now with a CV of ~1.58 the LB rises to 1.74!!!. I always use the log1p of total transactions. Concerning the really high imbalance of the dataset, i think that the 30% LB is totally useless....So, what are your CV-LB combination? do you experience the same divergences?
", "messages": [ { "id": 390333, "postDate": "2018-09-20T03:39:23.560Z", "content": "I have more stable CV-LB when split by time vs. split by stratified random sampling. I actually think 30% LB is quite decent but note that it is in a very different time period.
", "rawMarkdown": "I have more stable CV-LB when split by time vs. split by stratified random sampling. I actually think 30% LB is quite decent but note that it is in a very different time period.", "votes": 5, "replies": [ { "id": 390722, "postDate": "2018-09-20T17:46:30.503Z", "content": "Hi @yilihome can you elaborate on what do you mean split by time?
", "rawMarkdown": "Hi @yilihome can you elaborate on what do you mean split by time?\n" }, { "id": 391334, "postDate": "2018-09-21T16:09:44.073Z", "content": "Train using data <= 20170413, validate using date > 20170413 in my case.
", "rawMarkdown": "Train using data <= 20170413, validate using date > 20170413 in my case.", "votes": 1 } ] }, { "id": 390186, "postDate": "2018-09-19T20:43:29.887Z", "content": "Looking at the zero transactions, i see that almost the 98.6% per user has zero transactions value. Also, when i had 1.6 CV my LB was 1.66, now with a CV of ~1.58 the LB rises to 1.74!!!. I always use the log1p of total transactions. Concerning the really high imbalance of the dataset, i think that the 30% LB is totally useless....So, what are your CV-LB combination? do you experience the same divergences?
", "rawMarkdown": "Looking at the zero transactions, i see that almost the 98.6% per user has zero transactions value. Also, when i had 1.6 CV my LB was 1.66, now with a CV of ~1.58 the LB rises to 1.74!!!. I always use the log1p of total transactions. Concerning the really high imbalance of the dataset, i think that the 30% LB is totally useless....So, what are your CV-LB combination? do you experience the same divergences?", "votes": 3 }, { "id": 390675, "postDate": "2018-09-20T16:13:22.453Z", "content": "I am using the last three months and it gives a directional match, with LB - CV ~ 0.06 consistently.
", "rawMarkdown": "I am using the last three months and it gives a directional match, with LB - CV ~ 0.06 consistently. ", "votes": 4, "replies": [ { "id": 390682, "postDate": "2018-09-20T16:29:31.597Z", "content": "Konrad, thanks for sharing. \nBut what do you mean by \"using the last three month\"? Train only on this time range?
", "rawMarkdown": "Konrad, thanks for sharing. \nBut what do you mean by \"using the last three month\"? Train only on this time range?", "votes": 1 }, { "id": 390696, "postDate": "2018-09-20T16:46:22.003Z", "content": "No - last three months of training set for validation, and then retraining on complete set for submission. What puzzles me is that i can't seem to beat the public kernels, where plenty of people roll with kfold...
", "rawMarkdown": "No - last three months of training set for validation, and then retraining on complete set for submission. What puzzles me is that i can't seem to beat the public kernels, where plenty of people roll with kfold...", "votes": 4 }, { "id": 390729, "postDate": "2018-09-20T18:02:40.870Z", "content": "Thanks for clarifying, so it similar to TalkingData validation scheme. What confuse me is that in this competition we have entire year to predict, and not few days like in TalkingData.
\n\nSuppose public kernels use sum(log), and not log(sum), which gives better public score, but there is no good explanation why. I tried to repeat the same on train, but log(sum) always works better locally.\nThere is an ongoing discussion on the forum.
", "rawMarkdown": "Thanks for clarifying, so it similar to TalkingData validation scheme. What confuse me is that in this competition we have entire year to predict, and not few days like in TalkingData.\n\nSuppose public kernels use sum(log), and not log(sum), which gives better public score, but there is no good explanation why. I tried to repeat the same on train, but log(sum) always works better locally.\nThere is an ongoing discussion on the forum.", "votes": 1 }, { "id": 390928, "postDate": "2018-09-21T03:33:02.763Z", "content": "@alexfir I agree it doesn't make sense to me neither.
", "rawMarkdown": "@alexfir I agree it doesn't make sense to me neither.", "votes": -1 }, { "id": 390971, "postDate": "2018-09-21T05:20:35.077Z", "rawMarkdown": "", "isDeleted": true }, { "id": 391007, "postDate": "2018-09-21T06:18:52.623Z", "content": "@Konrad Banachewicz\nI am using kfold with newly engineered features, it gives 1.603 CV, 1.667 LB.
\n\nIt seems valid to use validation data split by time rather than kfold.\nBut in that case, what if one wants to generate OOF files?\nAs last 3 months are separated, How to generate OOF files?\nOOF file won't be the same size of train rows right?
\n\nThanks.
", "rawMarkdown": "@Konrad Banachewicz\nI am using kfold with newly engineered features, it gives 1.603 CV, 1.667 LB.\n\nIt seems valid to use validation data split by time rather than kfold.\nBut in that case, what if one wants to generate OOF files?\nAs last 3 months are separated, How to generate OOF files?\nOOF file won't be the same size of train rows right?\n\nThanks.", "votes": 2 }, { "id": 391427, "postDate": "2018-09-21T19:47:49.770Z", "content": "@Prashant Kikani You have to retrain the model with all the train data after validation.
\n\nI was looking at Sberbank competition which was a regression problem with both options (time based split and k-fold) and the winning solution used KFold (from alijis I think)
", "rawMarkdown": "@Prashant Kikani You have to retrain the model with all the train data after validation.\n\nI was looking at Sberbank competition which was a regression problem with both options (time based split and k-fold) and the winning solution used KFold (from alijis I think)", "votes": 2 }, { "id": 391795, "postDate": "2018-09-22T12:53:57.107Z", "content": "Indeed, i tried all kinds of validations and the time based split was indeed most close to LB than any kind of Kfold. What puzzles me is that Kfold gives me my best LB score though...
", "rawMarkdown": "Indeed, i tried all kinds of validations and the time based split was indeed most close to LB than any kind of Kfold. What puzzles me is that Kfold gives me my best LB score though..." } ] }, { "id": 391179, "postDate": "2018-09-21T11:18:38.083Z", "content": "1.62 CV 1.66 LB
", "rawMarkdown": "1.62 CV 1.66 LB " } ], "comments": [ { "id": 390333, "author_name": "yilisg", "author_url": "", "post_date": "2018-09-20T03:39:23.560000", "content": "I have more stable CV-LB when split by time vs. split by stratified random sampling. I actually think 30% LB is quite decent but note that it is in a very different time period.
", "votes": 5, "replies": [ { "id": 390722, "author_name": "maverick", "author_url": "", "post_date": "2018-09-20T17:46:30.503000", "content": "Hi @yilihome can you elaborate on what do you mean split by time?
", "votes": 0, "replies": [] }, { "id": 391334, "author_name": "yilisg", "author_url": "", "post_date": "2018-09-21T16:09:44.073000", "content": "Train using data <= 20170413, validate using date > 20170413 in my case.
", "votes": 1, "replies": [] } ] }, { "id": 390675, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2018-09-20T16:13:22.453000", "content": "I am using the last three months and it gives a directional match, with LB - CV ~ 0.06 consistently.
", "votes": 4, "replies": [ { "id": 390682, "author_name": "Alexander Firsov", "author_url": "", "post_date": "2018-09-20T16:29:31.597000", "content": "Konrad, thanks for sharing. \nBut what do you mean by \"using the last three month\"? Train only on this time range?
", "votes": 1, "replies": [] }, { "id": 390696, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2018-09-20T16:46:22.003000", "content": "No - last three months of training set for validation, and then retraining on complete set for submission. What puzzles me is that i can't seem to beat the public kernels, where plenty of people roll with kfold...
", "votes": 4, "replies": [] }, { "id": 390729, "author_name": "Alexander Firsov", "author_url": "", "post_date": "2018-09-20T18:02:40.870000", "content": "Thanks for clarifying, so it similar to TalkingData validation scheme. What confuse me is that in this competition we have entire year to predict, and not few days like in TalkingData.
\n\nSuppose public kernels use sum(log), and not log(sum), which gives better public score, but there is no good explanation why. I tried to repeat the same on train, but log(sum) always works better locally.\nThere is an ongoing discussion on the forum.
", "votes": 1, "replies": [] }, { "id": 390928, "author_name": "Antoine", "author_url": "", "post_date": "2018-09-21T03:33:02.763000", "content": "@alexfir I agree it doesn't make sense to me neither.
", "votes": -1, "replies": [] }, { "id": 390971, "author_name": "", "author_url": "", "post_date": "2018-09-21T05:20:35.077000", "content": "", "votes": 0, "replies": [] }, { "id": 391007, "author_name": "Prashant Kikani", "author_url": "", "post_date": "2018-09-21T06:18:52.623000", "content": "@Konrad Banachewicz\nI am using kfold with newly engineered features, it gives 1.603 CV, 1.667 LB.
\n\nIt seems valid to use validation data split by time rather than kfold.\nBut in that case, what if one wants to generate OOF files?\nAs last 3 months are separated, How to generate OOF files?\nOOF file won't be the same size of train rows right?
\n\nThanks.
", "votes": 2, "replies": [] }, { "id": 391427, "author_name": "Aguiar", "author_url": "", "post_date": "2018-09-21T19:47:49.770000", "content": "@Prashant Kikani You have to retrain the model with all the train data after validation.
\n\nI was looking at Sberbank competition which was a regression problem with both options (time based split and k-fold) and the winning solution used KFold (from alijis I think)
", "votes": 2, "replies": [] }, { "id": 391795, "author_name": "Dimosthenis Karaflos", "author_url": "", "post_date": "2018-09-22T12:53:57.107000", "content": "Indeed, i tried all kinds of validations and the time based split was indeed most close to LB than any kind of Kfold. What puzzles me is that Kfold gives me my best LB score though...
", "votes": 0, "replies": [] } ] }, { "id": 391179, "author_name": "Arnold Kuo", "author_url": "", "post_date": "2018-09-21T11:18:38.083000", "content": "1.62 CV 1.66 LB
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "390333": "I have more stable CV-LB when split by time vs. split by stratified random sampling. I actually think 30% LB is quite decent but note that it is in a very different time period.", "390186": "Looking at the zero transactions, i see that almost the 98.6% per user has zero transactions value. Also, when i had 1.6 CV my LB was 1.66, now with a CV of ~1.58 the LB rises to 1.74!!!. I always use the log1p of total transactions. Concerning the really high imbalance of the dataset, i think that the 30% LB is totally useless....So, what are your CV-LB combination? do you experience the same divergences?", "390675": "I am using the last three months and it gives a directional match, with LB - CV ~ 0.06 consistently. ", "391179": "1.62 CV 1.66 LB " } }