{ "id": 71745, "title": "Let's talk about validation scheme", "url": "/competitions/ga-customer-revenue-prediction/discussion/71745", "author_name": "", "post_date": "2018-11-16T05:50:51.405879200Z", "votes": 3, "comment_count": 2, "views": 0, "content": "

I've tried three validation schemes:

\n\n
    \n
  1. Training set from beginning ~ 2017.10.15, validation set from 2017.12.1 ~ 2018.1.31. (mimic the test set)
    2. \n
  2. Training set from beginning ~ 2018.6.15 validation set from 2018.8.1 ~ 2018.9.30. (most training data)
    3. \n
  3. Shift the (end date of training, start date of validation, end date of validation) in 1 every two month and average all results. (most validation data to reduce possible randomness)
    4. \n
\n\n

And different validation schemes end up choosing drastically different models, not sure which is the best way to do validation.

", "messages": [ { "id": "422363", "postDate": "11/16/2018 05:50:51", "content": "

I've tried three validation schemes:

\n\n
    \n
  1. Training set from beginning ~ 2017.10.15, validation set from 2017.12.1 ~ 2018.1.31. (mimic the test set)
    2. \n
  2. Training set from beginning ~ 2018.6.15 validation set from 2018.8.1 ~ 2018.9.30. (most training data)
    3. \n
  3. Shift the (end date of training, start date of validation, end date of validation) in 1 every two month and average all results. (most validation data to reduce possible randomness)
    4. \n
\n\n

And different validation schemes end up choosing drastically different models, not sure which is the best way to do validation.

", "rawMarkdown": "I've tried three validation schemes:\n\n1. Training set from beginning ~ 2017.10.15, validation set from 2017.12.1 ~ 2018.1.31. (mimic the test set)\n2. Training set from beginning ~ 2018.6.15 validation set from 2018.8.1 ~ 2018.9.30. (most training data)\n3. Shift the (end date of training, start date of validation, end date of validation) in 1 every two month and average all results. (most validation data to reduce possible randomness)\n\nAnd different validation schemes end up choosing drastically different models, not sure which is the best way to do validation.", "votes": null }, { "id": "422366", "postDate": "11/16/2018 05:58:39", "content": "

Or maybe the randomness in the test set (2018-12 ~ 2019-1) will have a greater influence on the final private LB than the model.

", "rawMarkdown": "Or maybe the randomness in the test set (2018-12 ~ 2019-1) will have a greater influence on the final private LB than the model.", "votes": null }, { "id": "427227", "postDate": "11/24/2018 22:10:47", "content": "

I'm using 2018-08-15 to 2018-10-15 as the test set, with everything prior to 2018-06-29 (a 48 day lag) as the training data. Within the the training data, I'm then appending different lags to create the target variable. For example lag #1 is 2018-04-29 to 2018-06-29 as the target, prior to 2018-03-13 predictors, lag #2 2018-05-30 to 2018-03-30 target, prior to 2018-02-11 predictors. I'm extracting predictors such as \"month of first day of target period\" and hoping this will enable generalization to the Dec to Jan time period.

", "rawMarkdown": "I'm using 2018-08-15 to 2018-10-15 as the test set, with everything prior to 2018-06-29 (a 48 day lag) as the training data. Within the the training data, I'm then appending different lags to create the target variable. For example lag #1 is 2018-04-29 to 2018-06-29 as the target, prior to 2018-03-13 predictors, lag #2 2018-05-30 to 2018-03-30 target, prior to 2018-02-11 predictors. I'm extracting predictors such as \"month of first day of target period\" and hoping this will enable generalization to the Dec to Jan time period.", "votes": null } ], "comments": [ { "id": 422366, "author_name": "zzhhrr001", "author_url": "", "post_date": "11/16/2018 05:58:39", "content": "

Or maybe the randomness in the test set (2018-12 ~ 2019-1) will have a greater influence on the final private LB than the model.

", "votes": null, "replies": [] }, { "id": 427227, "author_name": "johepaton", "author_url": "", "post_date": "11/24/2018 22:10:47", "content": "

I'm using 2018-08-15 to 2018-10-15 as the test set, with everything prior to 2018-06-29 (a 48 day lag) as the training data. Within the the training data, I'm then appending different lags to create the target variable. For example lag #1 is 2018-04-29 to 2018-06-29 as the target, prior to 2018-03-13 predictors, lag #2 2018-05-30 to 2018-03-30 target, prior to 2018-02-11 predictors. I'm extracting predictors such as \"month of first day of target period\" and hoping this will enable generalization to the Dec to Jan time period.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "422363": "I've tried three validation schemes:\n\n1. Training set from beginning ~ 2017.10.15, validation set from 2017.12.1 ~ 2018.1.31. (mimic the test set)\n2. Training set from beginning ~ 2018.6.15 validation set from 2018.8.1 ~ 2018.9.30. (most training data)\n3. Shift the (end date of training, start date of validation, end date of validation) in 1 every two month and average all results. (most validation data to reduce possible randomness)\n\nAnd different validation schemes end up choosing drastically different models, not sure which is the best way to do validation.", "422366": "Or maybe the randomness in the test set (2018-12 ~ 2019-1) will have a greater influence on the final private LB than the model.", "427227": "I'm using 2018-08-15 to 2018-10-15 as the test set, with everything prior to 2018-06-29 (a 48 day lag) as the training data. Within the the training data, I'm then appending different lags to create the target variable. For example lag #1 is 2018-04-29 to 2018-06-29 as the target, prior to 2018-03-13 predictors, lag #2 2018-05-30 to 2018-03-30 target, prior to 2018-02-11 predictors. I'm extracting predictors such as \"month of first day of target period\" and hoping this will enable generalization to the Dec to Jan time period." }, "source": "meta" }