The classic topic :-) I was wondering how others were handling the performance validation. I have tried different combinations of the training period and recent period for validation, but the best I am seeing is kinda directional agreement - the absolute values are waaay off. I haven't touched more advanced stuff like adversarial validation yet.
", "messages": [ { "id": "234159", "postDate": "10/22/2017 13:32:22", "content": "The classic topic :-) I was wondering how others were handling the performance validation. I have tried different combinations of the training period and recent period for validation, but the best I am seeing is kinda directional agreement - the absolute values are waaay off. I haven't touched more advanced stuff like adversarial validation yet.
", "rawMarkdown": "The classic topic :-) I was wondering how others were handling the performance validation. I have tried different combinations of the training period and recent period for validation, but the best I am seeing is kinda directional agreement - the absolute values are waaay off. I haven't touched more advanced stuff like adversarial validation yet.", "votes": null }, { "id": "234726", "postDate": "10/23/2017 22:41:38", "content": "I have tried using somewhat exact overlapping historical time window in 2016, 2015, 2014 (ensure the weekdays are matched). The validation gap seems still quite large.
", "rawMarkdown": "I have tried using somewhat exact overlapping historical time window in 2016, 2015, 2014 (ensure the weekdays are matched). The validation gap seems still quite large.", "votes": null }, { "id": "234873", "postDate": "10/24/2017 08:40:58", "content": "Out of curiosity, how big is your gap? In my early attempts (before I dropped the approach based on last two weeks) it was sth like 0.591 valid -> 0.795 LB and 0.567 -> 0.779.
", "rawMarkdown": "Out of curiosity, how big is your gap? In my early attempts (before I dropped the approach based on last two weeks) it was sth like 0.591 valid -> 0.795 LB and 0.567 -> 0.779.", "votes": null }, { "id": "234875", "postDate": "10/24/2017 08:43:09", "content": "My very simple median models show contrary results for validation and leaderboard. The recent one gave me 0.47 on validation and 0.67 on the leaderboard. The previous was 0.63 on validation and 0.59 on the leaderboard. I'm validating on 3 folds by two weeks each from 2017-07-01 till 2017-08-15 and error seems to be quite stationary for all folds. Maybe period 2017-08-16:2017-08-31 is somehow different?
", "rawMarkdown": "My very simple median models show contrary results for validation and leaderboard. The recent one gave me 0.47 on validation and 0.67 on the leaderboard. The previous was 0.63 on validation and 0.59 on the leaderboard. I'm validating on 3 folds by two weeks each from 2017-07-01 till 2017-08-15 and error seems to be quite stationary for all folds. Maybe period 2017-08-16:2017-08-31 is somehow different?", "votes": null }, { "id": "234878", "postDate": "10/24/2017 08:50:31", "content": "That is my suspicion as well - sth is different about this last period. Then again, time series prediction is notoriously tricky in terms of performance validation - stationarity is a nice concept, but pretty rare in real life...
", "rawMarkdown": "That is my suspicion as well - sth is different about this last period. Then again, time series prediction is notoriously tricky in terms of performance validation - stationarity is a nice concept, but pretty rare in real life...", "votes": null }, { "id": "234884", "postDate": "10/24/2017 09:03:52", "content": "It depends also on what part of test dataset is selected for the public leaderboard. I'm curious to test my validation on some items splits, maybe the selection is biased.
\n\nAbove you mentioned that you dropped the last two weeks approach. How are you validating your results now?
", "rawMarkdown": "It depends also on what part of test dataset is selected for the public leaderboard. I'm curious to test my validation on some items splits, maybe the selection is biased.\n\nAbove you mentioned that you dropped the last two weeks approach. How are you validating your results now?", "votes": null }, { "id": "234889", "postDate": "10/24/2017 09:07:40", "content": "I am not :-) I am working on a coherent framework, none working at the moment (discarded a few other ideas apart from the two weeks one). I think adversarial validation might be the way to go, but I to create some features for that to start making sense.
", "rawMarkdown": "I am not :-) I am working on a coherent framework, none working at the moment (discarded a few other ideas apart from the two weeks one). I think adversarial validation might be the way to go, but I to create some features for that to start making sense.", "votes": null }, { "id": "235130", "postDate": "10/24/2017 19:19:55", "content": "I guess one of the important things is:
\n\n\"NOTE: The training data does not include rows for items that had zero unit_sales for a store/date combination.\" from https://www.kaggle.com/c/favorita-grocery-sales-forecasting/data .
\n\nMaybe one can augment the dataset with observations of 0 unit_sales
", "rawMarkdown": "I guess one of the important things is:\n\n\"NOTE: The training data does not include rows for items that had zero unit_sales for a store/date combination.\" from https://www.kaggle.com/c/favorita-grocery-sales-forecasting/data .\n\nMaybe one can augment the dataset with observations of 0 unit_sales", "votes": null }, { "id": "235167", "postDate": "10/24/2017 20:47:25", "content": "I don't know about that - you think having more zeroes in the training set might actually help? I'm kinda agnostic on the topic: I have no intuition why it would tbh...
", "rawMarkdown": "I don't know about that - you think having more zeroes in the training set might actually help? I'm kinda agnostic on the topic: I have no intuition why it would tbh...", "votes": null }, { "id": "235183", "postDate": "10/24/2017 21:05:10", "content": "Well - if we have no zeroes in validation set, than scoring any row zero would give a higher error in local validation. Even though it might be a good score for some observations in the test set.\nIf we train without any zero as an output we will bias the model towards a higher positive value than it is in reality. \nOr am I wrong here?
\n\nI wonder where does this contest restriction come from - I guess Favorita can tell 'zero sales' from 'no stock', so this information is not given either to shrink the size of the dataset, or to make it an additional challenge.
", "rawMarkdown": "Well - if we have no zeroes in validation set, than scoring any row zero would give a higher error in local validation. Even though it might be a good score for some observations in the test set.\nIf we train without any zero as an output we will bias the model towards a higher positive value than it is in reality. \nOr am I wrong here?\n\nI wonder where does this contest restriction come from - I guess Favorita can tell 'zero sales' from 'no stock', so this information is not given either to shrink the size of the dataset, or to make it an additional challenge.", "votes": null }, { "id": "235186", "postDate": "10/24/2017 21:22:50", "content": "You're not wrong, I'm just wondering if the effect is strong enough to matter - proportion of zeroes in the training set is below 0.01 pct...
\n\nEdit: i actually thought about and the effect probably does matter - I missed the fact that zero sales days are absent from the dataset (actually reading the data description page helps), so those need to be corrected for.
", "rawMarkdown": "You're not wrong, I'm just wondering if the effect is strong enough to matter - proportion of zeroes in the training set is below 0.01 pct...\n\nEdit: i actually thought about and the effect probably does matter - I missed the fact that zero sales days are absent from the dataset (actually reading the data description page helps), so those need to be corrected for.", "votes": null }, { "id": "235400", "postDate": "10/25/2017 10:10:46", "content": "I think, the problem is that we have ~40k new item-store pairs in the test dataset. That can explain the gap. By the way, train dataset also contains items which do not occur in the test, approx. 8k records.
", "rawMarkdown": "I think, the problem is that we have ~40k new item-store pairs in the test dataset. That can explain the gap. By the way, train dataset also contains items which do not occur in the test, approx. 8k records.", "votes": null }, { "id": "235721", "postDate": "10/25/2017 23:41:11", "content": "Hi Konrad, execute me for a bit delayed response here. Here are my results:
\nuse 16 days data in Aug 2015 , valid 0.9151, LB 0.59
\nuse 16 days data in Aug 2016, valid 0.830, LB 0.59
maybe we can try blend the historical data with more recent weeks in 2017 to get a balanced valid dataset? Just a thought.
", "rawMarkdown": "Hi Konrad, execute me for a bit delayed response here. Here are my results: \nuse 16 days data in Aug 2015 , valid 0.9151, LB 0.59 \nuse 16 days data in Aug 2016, valid 0.830, LB 0.59 \n \nmaybe we can try blend the historical data with more recent weeks in 2017 to get a balanced valid dataset? Just a thought.", "votes": null }, { "id": "235724", "postDate": "10/25/2017 23:48:17", "content": "Also I forgot to mention, there are quite a large amount of store-item number missing from the historical sales, I guess that's the challenge of forecasting those un-seen items in the test set based on their similarity with other items (e.g. same category, etc.) Right now, I am just imputing them zero.
", "rawMarkdown": "Also I forgot to mention, there are quite a large amount of store-item number missing from the historical sales, I guess that's the challenge of forecasting those un-seen items in the test set based on their similarity with other items (e.g. same category, etc.) Right now, I am just imputing them zero.", "votes": null }, { "id": "235898", "postDate": "10/26/2017 08:57:22", "content": "I think you're right - the unseen items seem like the primary cause of the discrepancy. I will need to check performance purely on the ones already seen.
", "rawMarkdown": "I think you're right - the unseen items seem like the primary cause of the discrepancy. I will need to check performance purely on the ones already seen.", "votes": null } ], "comments": [ { "id": 234726, "author_name": "dongxu027", "author_url": "", "post_date": "10/23/2017 22:41:38", "content": "I have tried using somewhat exact overlapping historical time window in 2016, 2015, 2014 (ensure the weekdays are matched). The validation gap seems still quite large.
", "votes": null, "replies": [ { "id": 234873, "author_name": "konradb", "author_url": "", "post_date": "10/24/2017 08:40:58", "content": "Out of curiosity, how big is your gap? In my early attempts (before I dropped the approach based on last two weeks) it was sth like 0.591 valid -> 0.795 LB and 0.567 -> 0.779.
", "votes": null, "replies": [] }, { "id": 235721, "author_name": "dongxu027", "author_url": "", "post_date": "10/25/2017 23:41:11", "content": "Hi Konrad, execute me for a bit delayed response here. Here are my results:
\nuse 16 days data in Aug 2015 , valid 0.9151, LB 0.59
\nuse 16 days data in Aug 2016, valid 0.830, LB 0.59
maybe we can try blend the historical data with more recent weeks in 2017 to get a balanced valid dataset? Just a thought.
", "votes": null, "replies": [] }, { "id": 235724, "author_name": "dongxu027", "author_url": "", "post_date": "10/25/2017 23:48:17", "content": "Also I forgot to mention, there are quite a large amount of store-item number missing from the historical sales, I guess that's the challenge of forecasting those un-seen items in the test set based on their similarity with other items (e.g. same category, etc.) Right now, I am just imputing them zero.
", "votes": null, "replies": [] }, { "id": 235898, "author_name": "konradb", "author_url": "", "post_date": "10/26/2017 08:57:22", "content": "I think you're right - the unseen items seem like the primary cause of the discrepancy. I will need to check performance purely on the ones already seen.
", "votes": null, "replies": [] } ] }, { "id": 234875, "author_name": "tsdaemon", "author_url": "", "post_date": "10/24/2017 08:43:09", "content": "My very simple median models show contrary results for validation and leaderboard. The recent one gave me 0.47 on validation and 0.67 on the leaderboard. The previous was 0.63 on validation and 0.59 on the leaderboard. I'm validating on 3 folds by two weeks each from 2017-07-01 till 2017-08-15 and error seems to be quite stationary for all folds. Maybe period 2017-08-16:2017-08-31 is somehow different?
", "votes": null, "replies": [ { "id": 234878, "author_name": "konradb", "author_url": "", "post_date": "10/24/2017 08:50:31", "content": "That is my suspicion as well - sth is different about this last period. Then again, time series prediction is notoriously tricky in terms of performance validation - stationarity is a nice concept, but pretty rare in real life...
", "votes": null, "replies": [] }, { "id": 234884, "author_name": "tsdaemon", "author_url": "", "post_date": "10/24/2017 09:03:52", "content": "It depends also on what part of test dataset is selected for the public leaderboard. I'm curious to test my validation on some items splits, maybe the selection is biased.
\n\nAbove you mentioned that you dropped the last two weeks approach. How are you validating your results now?
", "votes": null, "replies": [] }, { "id": 234889, "author_name": "konradb", "author_url": "", "post_date": "10/24/2017 09:07:40", "content": "I am not :-) I am working on a coherent framework, none working at the moment (discarded a few other ideas apart from the two weeks one). I think adversarial validation might be the way to go, but I to create some features for that to start making sense.
", "votes": null, "replies": [] } ] }, { "id": 235130, "author_name": "kretes", "author_url": "", "post_date": "10/24/2017 19:19:55", "content": "I guess one of the important things is:
\n\n\"NOTE: The training data does not include rows for items that had zero unit_sales for a store/date combination.\" from https://www.kaggle.com/c/favorita-grocery-sales-forecasting/data .
\n\nMaybe one can augment the dataset with observations of 0 unit_sales
", "votes": null, "replies": [ { "id": 235167, "author_name": "konradb", "author_url": "", "post_date": "10/24/2017 20:47:25", "content": "I don't know about that - you think having more zeroes in the training set might actually help? I'm kinda agnostic on the topic: I have no intuition why it would tbh...
", "votes": null, "replies": [] }, { "id": 235183, "author_name": "kretes", "author_url": "", "post_date": "10/24/2017 21:05:10", "content": "Well - if we have no zeroes in validation set, than scoring any row zero would give a higher error in local validation. Even though it might be a good score for some observations in the test set.\nIf we train without any zero as an output we will bias the model towards a higher positive value than it is in reality. \nOr am I wrong here?
\n\nI wonder where does this contest restriction come from - I guess Favorita can tell 'zero sales' from 'no stock', so this information is not given either to shrink the size of the dataset, or to make it an additional challenge.
", "votes": null, "replies": [] }, { "id": 235186, "author_name": "konradb", "author_url": "", "post_date": "10/24/2017 21:22:50", "content": "You're not wrong, I'm just wondering if the effect is strong enough to matter - proportion of zeroes in the training set is below 0.01 pct...
\n\nEdit: i actually thought about and the effect probably does matter - I missed the fact that zero sales days are absent from the dataset (actually reading the data description page helps), so those need to be corrected for.
", "votes": null, "replies": [] } ] }, { "id": 235400, "author_name": "tsdaemon", "author_url": "", "post_date": "10/25/2017 10:10:46", "content": "I think, the problem is that we have ~40k new item-store pairs in the test dataset. That can explain the gap. By the way, train dataset also contains items which do not occur in the test, approx. 8k records.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "234159": "The classic topic :-) I was wondering how others were handling the performance validation. I have tried different combinations of the training period and recent period for validation, but the best I am seeing is kinda directional agreement - the absolute values are waaay off. I haven't touched more advanced stuff like adversarial validation yet.", "234726": "I have tried using somewhat exact overlapping historical time window in 2016, 2015, 2014 (ensure the weekdays are matched). The validation gap seems still quite large.", "234873": "Out of curiosity, how big is your gap? In my early attempts (before I dropped the approach based on last two weeks) it was sth like 0.591 valid -> 0.795 LB and 0.567 -> 0.779.", "234875": "My very simple median models show contrary results for validation and leaderboard. The recent one gave me 0.47 on validation and 0.67 on the leaderboard. The previous was 0.63 on validation and 0.59 on the leaderboard. I'm validating on 3 folds by two weeks each from 2017-07-01 till 2017-08-15 and error seems to be quite stationary for all folds. Maybe period 2017-08-16:2017-08-31 is somehow different?", "234878": "That is my suspicion as well - sth is different about this last period. Then again, time series prediction is notoriously tricky in terms of performance validation - stationarity is a nice concept, but pretty rare in real life...", "234884": "It depends also on what part of test dataset is selected for the public leaderboard. I'm curious to test my validation on some items splits, maybe the selection is biased.\n\nAbove you mentioned that you dropped the last two weeks approach. How are you validating your results now?", "234889": "I am not :-) I am working on a coherent framework, none working at the moment (discarded a few other ideas apart from the two weeks one). I think adversarial validation might be the way to go, but I to create some features for that to start making sense.", "235130": "I guess one of the important things is:\n\n\"NOTE: The training data does not include rows for items that had zero unit_sales for a store/date combination.\" from https://www.kaggle.com/c/favorita-grocery-sales-forecasting/data .\n\nMaybe one can augment the dataset with observations of 0 unit_sales", "235167": "I don't know about that - you think having more zeroes in the training set might actually help? I'm kinda agnostic on the topic: I have no intuition why it would tbh...", "235183": "Well - if we have no zeroes in validation set, than scoring any row zero would give a higher error in local validation. Even though it might be a good score for some observations in the test set.\nIf we train without any zero as an output we will bias the model towards a higher positive value than it is in reality. \nOr am I wrong here?\n\nI wonder where does this contest restriction come from - I guess Favorita can tell 'zero sales' from 'no stock', so this information is not given either to shrink the size of the dataset, or to make it an additional challenge.", "235186": "You're not wrong, I'm just wondering if the effect is strong enough to matter - proportion of zeroes in the training set is below 0.01 pct...\n\nEdit: i actually thought about and the effect probably does matter - I missed the fact that zero sales days are absent from the dataset (actually reading the data description page helps), so those need to be corrected for.", "235400": "I think, the problem is that we have ~40k new item-store pairs in the test dataset. That can explain the gap. By the way, train dataset also contains items which do not occur in the test, approx. 8k records.", "235721": "Hi Konrad, execute me for a bit delayed response here. Here are my results: \nuse 16 days data in Aug 2015 , valid 0.9151, LB 0.59 \nuse 16 days data in Aug 2016, valid 0.830, LB 0.59 \n \nmaybe we can try blend the historical data with more recent weeks in 2017 to get a balanced valid dataset? Just a thought.", "235724": "Also I forgot to mention, there are quite a large amount of store-item number missing from the historical sales, I guess that's the challenge of forecasting those un-seen items in the test set based on their similarity with other items (e.g. same category, etc.) Right now, I am just imputing them zero.", "235898": "I think you're right - the unseen items seem like the primary cause of the discrepancy. I will need to check performance purely on the ones already seen." }, "source": "meta" }