Are you using a CV or just a single holdout set?
\n\nI started with a holdout, but after a few submissions saw my local scores diverge from the LB. Now, I'm switching to CV and observe that the scores vary substantially from one fold to another; my standard deviations are of the order .015.
\n\nAnybody with similar experiences?
", "messages": [ { "id": "134315", "postDate": "09/06/2016 04:39:07", "content": "Are you using a CV or just a single holdout set?
\n\nI started with a holdout, but after a few submissions saw my local scores diverge from the LB. Now, I'm switching to CV and observe that the scores vary substantially from one fold to another; my standard deviations are of the order .015.
\n\nAnybody with similar experiences?
", "rawMarkdown": "Are you using a CV or just a single holdout set?\r\n\r\nI started with a holdout, but after a few submissions saw my local scores diverge from the LB. Now, I'm switching to CV and observe that the scores vary substantially from one fold to another; my standard deviations are of the order .015.\r\n\r\nAnybody with similar experiences?", "votes": null }, { "id": "134330", "postDate": "09/06/2016 05:43:07", "content": "It very much depends on how you establish the CV in this case. If you take a simple random split, then you may get very different distributions of 0's and 1's, because apparently these are not exactly randomly split over time. A stratified K-fold split, one that looks at the y-values to do the splitting, is likely the best strategy in this case, since we are looking at anomalies and you want to maintain the distribution of 1's vs 0's for each fold that you are generating.
\n\nOne thing to consider there is shuffling the trainset prior to running that. The effect of shuffling may be that you're taking 0's out of the "neighborhood" of specific 1's, where there's a slightly higher density of 1's in time. This could either hurt or help your LB score. My intuition tells me that shuffling in this case helps to build a better generalized (and therefore useful) model.
", "rawMarkdown": "It very much depends on how you establish the CV in this case. If you take a simple random split, then you may get very different distributions of 0's and 1's, because apparently these are not exactly randomly split over time. A stratified K-fold split, one that looks at the y-values to do the splitting, is likely the best strategy in this case, since we are looking at anomalies and you want to maintain the distribution of 1's vs 0's for each fold that you are generating.\r\n\r\nOne thing to consider there is shuffling the trainset prior to running that. The effect of shuffling may be that you're taking 0's out of the \"neighborhood\" of specific 1's, where there's a slightly higher density of 1's in time. This could either hurt or help your LB score. My intuition tells me that shuffling in this case helps to build a better generalized (and therefore useful) model.", "votes": null }, { "id": "134360", "postDate": "09/06/2016 09:59:57", "content": "Data does not seem to be sorted by time hence I am not sure you need to shuffle.
", "rawMarkdown": "Data does not seem to be sorted by time hence I am not sure you need to shuffle.", "votes": null }, { "id": "134462", "postDate": "09/06/2016 18:42:07", "content": "Many thanks for your help, Gerard! Then, I will use StratifiedKFold for my CV.
", "rawMarkdown": "Many thanks for your help, Gerard! Then, I will use StratifiedKFold for my CV.", "votes": null } ], "comments": [ { "id": 134330, "author_name": "remap1", "author_url": "", "post_date": "09/06/2016 05:43:07", "content": "It very much depends on how you establish the CV in this case. If you take a simple random split, then you may get very different distributions of 0's and 1's, because apparently these are not exactly randomly split over time. A stratified K-fold split, one that looks at the y-values to do the splitting, is likely the best strategy in this case, since we are looking at anomalies and you want to maintain the distribution of 1's vs 0's for each fold that you are generating.
\n\nOne thing to consider there is shuffling the trainset prior to running that. The effect of shuffling may be that you're taking 0's out of the "neighborhood" of specific 1's, where there's a slightly higher density of 1's in time. This could either hurt or help your LB score. My intuition tells me that shuffling in this case helps to build a better generalized (and therefore useful) model.
", "votes": null, "replies": [] }, { "id": 134360, "author_name": "cpmpml", "author_url": "", "post_date": "09/06/2016 09:59:57", "content": "Data does not seem to be sorted by time hence I am not sure you need to shuffle.
", "votes": null, "replies": [] }, { "id": 134462, "author_name": "dollardollar", "author_url": "", "post_date": "09/06/2016 18:42:07", "content": "Many thanks for your help, Gerard! Then, I will use StratifiedKFold for my CV.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "134315": "Are you using a CV or just a single holdout set?\r\n\r\nI started with a holdout, but after a few submissions saw my local scores diverge from the LB. Now, I'm switching to CV and observe that the scores vary substantially from one fold to another; my standard deviations are of the order .015.\r\n\r\nAnybody with similar experiences?", "134330": "It very much depends on how you establish the CV in this case. If you take a simple random split, then you may get very different distributions of 0's and 1's, because apparently these are not exactly randomly split over time. A stratified K-fold split, one that looks at the y-values to do the splitting, is likely the best strategy in this case, since we are looking at anomalies and you want to maintain the distribution of 1's vs 0's for each fold that you are generating.\r\n\r\nOne thing to consider there is shuffling the trainset prior to running that. The effect of shuffling may be that you're taking 0's out of the \"neighborhood\" of specific 1's, where there's a slightly higher density of 1's in time. This could either hurt or help your LB score. My intuition tells me that shuffling in this case helps to build a better generalized (and therefore useful) model.", "134360": "Data does not seem to be sorted by time hence I am not sure you need to shuffle.", "134462": "Many thanks for your help, Gerard! Then, I will use StratifiedKFold for my CV." }, "source": "meta" }