{ "id": 21648, "title": "Validation framework benchmark", "url": "/competitions/facebook-v-predicting-check-ins/discussion/21648", "author_name": "", "post_date": "2016-06-13T19:39:37.197Z", "votes": 1, "comment_count": 13, "views": 1350, "content": "

I set the grid as:

\n\n

train[, x_cell := pmax(ceiling(x * 4), 1L)]

\n\n

train[, y_cell := pmax(ceiling(y * 4), 1L)]

\n\n

and split train and hold @614127 (this reproduce exactly the same proportion in tran - hold as we have in train - test):

\n\n

train[, hold := as.integer(time > 614127)]

\n\n

With this validation framework I'm playing with one of the more difficult (I think) square:

\n\n

x_cell = 21\ny_cell = 27

\n\n

This square have the max place_id diversity.

\n\n

The hold part of this square has 5357 rows.

\n\n

My cv score in this is 0.5463.

\n\n

Nice to hear your scores.

", "messages": [ { "id": "123736", "postDate": "06/13/2016 19:39:37", "content": "

I set the grid as:

\n\n

train[, x_cell := pmax(ceiling(x * 4), 1L)]

\n\n

train[, y_cell := pmax(ceiling(y * 4), 1L)]

\n\n

and split train and hold @614127 (this reproduce exactly the same proportion in tran - hold as we have in train - test):

\n\n

train[, hold := as.integer(time > 614127)]

\n\n

With this validation framework I'm playing with one of the more difficult (I think) square:

\n\n

x_cell = 21\ny_cell = 27

\n\n

This square have the max place_id diversity.

\n\n

The hold part of this square has 5357 rows.

\n\n

My cv score in this is 0.5463.

\n\n

Nice to hear your scores.

", "rawMarkdown": "I set the grid as:\r\n\r\ntrain[, x_cell := pmax(ceiling(x * 4), 1L)]\r\n\r\ntrain[, y_cell := pmax(ceiling(y * 4), 1L)]\r\n\r\nand split train and hold @614127 (this reproduce exactly the same proportion in tran - hold as we have in train - test):\r\n\r\ntrain[, hold := as.integer(time > 614127)]\r\n\r\n\r\nWith this validation framework I'm playing with one of the more difficult (I think) square:\r\n\r\nx_cell = 21\r\ny_cell = 27\r\n\r\nThis square have the max place_id diversity.\r\n\r\nThe hold part of this square has 5357 rows.\r\n\r\nMy cv score in this is 0.5463.\r\n\r\nNice to hear your scores.", "votes": null }, { "id": "123739", "postDate": "06/13/2016 20:18:11", "content": "

I found early on that my lb score was substantially higher than my cv on time > 600,000. It was something like 0.51 on cv gave 0.55 on lb. It takes so long to run on my laptop now that I haven't been running cv on the entire train set since then, so I'm not sure the precise magnitude of this difference. I do know that this original measurement I made included ids that were first seen after time > 600,000, removing these ids from the hold-out set does slightly increase the local cv.

", "rawMarkdown": "I found early on that my lb score was substantially higher than my cv on time > 600,000. It was something like 0.51 on cv gave 0.55 on lb. It takes so long to run on my laptop now that I haven't been running cv on the entire train set since then, so I'm not sure the precise magnitude of this difference. I do know that this original measurement I made included ids that were first seen after time > 600,000, removing these ids from the hold-out set does slightly increase the local cv.", "votes": null }, { "id": "124053", "postDate": "06/15/2016 07:49:54", "content": "

[quote=Jared Turkewitz;123739]

\n\n

I found early on that my lb score was substantially higher than my cv on time > 600,000. It was something like 0.51 on cv gave 0.55 on lb. It takes so long to run on my laptop now that I haven't been running cv on the entire train set since then, so I'm not sure the precise magnitude of this difference. I do know that this original measurement I made included ids that were first seen after time > 600,000, removing these ids from the hold-out set does slightly increase the local cv.

\n\n

[/quote]

\n\n

Perhaps the unseen labels could explain it. If you compute the cv over all the hold data (including the new labels respect your train period) and the test set was cleaned of the new labels respect train set.\nDo you compute cv 0.51 over all the grids or over a sample?

", "rawMarkdown": "[quote=Jared Turkewitz;123739]\r\n\r\nI found early on that my lb score was substantially higher than my cv on time > 600,000. It was something like 0.51 on cv gave 0.55 on lb. It takes so long to run on my laptop now that I haven't been running cv on the entire train set since then, so I'm not sure the precise magnitude of this difference. I do know that this original measurement I made included ids that were first seen after time > 600,000, removing these ids from the hold-out set does slightly increase the local cv.\r\n\r\n[/quote]\r\n\r\nPerhaps the unseen labels could explain it. If you compute the cv over all the hold data (including the new labels respect your train period) and the test set was cleaned of the new labels respect train set.\r\nDo you compute cv 0.51 over all the grids or over a sample?", "votes": null }, { "id": "124123", "postDate": "06/15/2016 17:42:38", "content": "

[quote=José A. Guerrero;124053]

\n\n

[quote=Jared Turkewitz;123739]

\n\n

I found early on that my lb score was substantially higher than my cv on time > 600,000. It was something like 0.51 on cv gave 0.55 on lb. It takes so long to run on my laptop now that I haven't been running cv on the entire train set since then, so I'm not sure the precise magnitude of this difference. I do know that this original measurement I made included ids that were first seen after time > 600,000, removing these ids from the hold-out set does slightly increase the local cv.

\n\n

[/quote]

\n\n

Perhaps the unseen labels could explain it. If you compute the cv over all the hold data (including the new labels respect your train period) and the test set was cleaned of the new labels respect train set.\nDo you compute cv 0.51 over all the grids or over a sample?

\n\n

[/quote]

\n\n

The 0.51 corresponded to about 0.55 on the lb, and was not removing unseen labels. Recently I haven't had the time to run cv over the entire sample because of cpu constraints. After removing unseen labels from the holdout set it still seems to me that the lb score is on average about 0.02-0.03 higher than cv, at least from the random sample of grid cells that I use for cv.

", "rawMarkdown": "[quote=José A. Guerrero;124053]\r\n\r\n[quote=Jared Turkewitz;123739]\r\n\r\nI found early on that my lb score was substantially higher than my cv on time > 600,000. It was something like 0.51 on cv gave 0.55 on lb. It takes so long to run on my laptop now that I haven't been running cv on the entire train set since then, so I'm not sure the precise magnitude of this difference. I do know that this original measurement I made included ids that were first seen after time > 600,000, removing these ids from the hold-out set does slightly increase the local cv.\r\n\r\n[/quote]\r\n\r\nPerhaps the unseen labels could explain it. If you compute the cv over all the hold data (including the new labels respect your train period) and the test set was cleaned of the new labels respect train set.\r\nDo you compute cv 0.51 over all the grids or over a sample?\r\n\r\n\r\n\r\n[/quote]\r\n\r\nThe 0.51 corresponded to about 0.55 on the lb, and was not removing unseen labels. Recently I haven't had the time to run cv over the entire sample because of cpu constraints. After removing unseen labels from the holdout set it still seems to me that the lb score is on average about 0.02-0.03 higher than cv, at least from the random sample of grid cells that I use for cv.", "votes": null }, { "id": "124131", "postDate": "06/15/2016 18:42:56", "content": "

Agree with @José's analysis about the unseen businesses.

", "rawMarkdown": "Agree with @José's analysis about the unseen businesses.", "votes": null }, { "id": "124312", "postDate": "06/17/2016 03:27:39", "content": "

For comparable grid dimensions I found that my LB scores are consistently 0.03-0.04 greater than the validation score obtained on a holdout set of events at times greater than (roughly) 600000. I performed the validation on a random sampling of 1% of the grid cells (horizontal bars indicate +/- the standard error of the mean validation score so estimated).\n\"enter\nRedoing the validation excluding the unseen classes in each holdout grid cell I found that this discrepancy vanishes, and in fact the validation score is now ~0.01-0.02 greater than the LB score. But your mileage may vary...

", "rawMarkdown": "For comparable grid dimensions I found that my LB scores are consistently 0.03-0.04 greater than the validation score obtained on a holdout set of events at times greater than (roughly) 600000. I performed the validation on a random sampling of 1% of the grid cells (horizontal bars indicate +/- the standard error of the mean validation score so estimated).\r\n![enter image description here][1]\r\nRedoing the validation excluding the unseen classes in each holdout grid cell I found that this discrepancy vanishes, and in fact the validation score is now ~0.01-0.02 greater than the LB score. But your mileage may vary...\r\n\r\n\r\n [1]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4427/val_lb_unseen.png", "votes": null }, { "id": "124342", "postDate": "06/17/2016 10:22:42", "content": "

@ Tod Wright

\n\n

Thank you for your sharing.

", "rawMarkdown": "Tod Wright\r\n\r\nThank you for your sharing.", "votes": null }, { "id": "124524", "postDate": "06/19/2016 16:14:02", "content": "

Okay, I just tested it on the area of x in [0,0.2] and y in [0,10], as well as x in [0,1.2] y in [0,10]. On the smaller subset I got 0.01475 on the lb (corresponds to 0.59, on the larger subset I got 0.08875, corresponds to 0.5916, the overall lb score for this model was 0.5921. For my local cv, on the smaller subset I got 0.625, and on the larger subset I got 0.587. This was after removing unseen ids for local cv. This run took me 4 days on my 2 core laptop, so I don't have time to run cv on the entire grid.

\n\n

So it seems like overall after removing unseen ids the lb and cv are relatively close, though it depends on the subsets you are comparing.

", "rawMarkdown": "Okay, I just tested it on the area of x in [0,0.2] and y in [0,10], as well as x in [0,1.2] y in [0,10]. On the smaller subset I got 0.01475 on the lb (corresponds to 0.59, on the larger subset I got 0.08875, corresponds to 0.5916, the overall lb score for this model was 0.5921. For my local cv, on the smaller subset I got 0.625, and on the larger subset I got 0.587. This was after removing unseen ids for local cv. This run took me 4 days on my 2 core laptop, so I don't have time to run cv on the entire grid.\r\n\r\nSo it seems like overall after removing unseen ids the lb and cv are relatively close, though it depends on the subsets you are comparing.", "votes": null }, { "id": "124876", "postDate": "06/22/2016 23:07:08", "content": "

@Tod Wright, I still find my local cv (sorted by time and the last ~25% data, filtered unseen place_id) still have lower @map3 than LB. I am using KNN, what algorithm did you use for this chart?

\n\n

It's like 0.55 in local cv to 0.58 on LB.

", "rawMarkdown": "Tod Wright, I still find my local cv (sorted by time and the last ~25% data, filtered unseen place_id) still have lower @map3 than LB. I am using KNN, what algorithm did you use for this chart?\r\n\r\nIt's like 0.55 in local cv to 0.58 on LB.", "votes": null }, { "id": "124899", "postDate": "06/23/2016 07:51:22", "content": "

@ YijunMa

\n\n

I got the same problem as you before.I used knn,too. I am not sure how you filter out your unseen place_id. My original strategy is:

\n\n
    \n
  1. Split train/val based on time
    2. \n
  2. Directly filter out those place_ids in validation set which is not present in training set
    3. \n
\n\n

The cv score is lower than lb, then I came to a new strategy. The trick I found is that you have to filter out place_ids grid by grid.

\n\n

For example, we have place_id 1,2,4,6,7,10 and 4 grid A,B,C,D. place_id 1,2,4,6 are in training set while 4,6,7,10 in validation set.

\n\n

If grid A contains only place_id 1,2. then grid A only have information of place_id 1,2 to learn. If the validation data set in grid A contains,say, 4 and 6,we must remove them since the grid A only contains information of 1,2 during the training phase.

\n\n

If we follow my original strategy to split data then train, we can't avoid the case mentioned above.\nAfter I use my new strategy, my cv score(still knn) is always higher than lb by around 0.02~0.03.

\n\n

Hope this help:)

", "rawMarkdown": "YijunMa\r\n\r\nI got the same problem as you before.I used knn,too. I am not sure how you filter out your unseen place_id. My original strategy is:\r\n\r\n1. Split train/val based on time\r\n2. Directly filter out those place_ids in validation set which is not present in training set\r\n\r\nThe cv score is lower than lb, then I came to a new strategy. The trick I found is that you have to filter out place_ids **grid by grid.** \r\n\r\nFor example, we have place_id 1,2,4,6,7,10 and 4 grid A,B,C,D. place_id 1,2,4,6 are in training set while 4,6,7,10 in validation set.\r\n\r\nIf grid A contains only place_id 1,2. then grid A only have information of place_id 1,2 to learn. If the validation data set in grid A contains,say, 4 and 6,we must remove them since the grid A only contains information of 1,2 during the training phase. \r\n\r\nIf we follow my original strategy to split data then train, we can't avoid the case mentioned above.\r\nAfter I use my new strategy, my cv score(still knn) is always higher than lb by around 0.02~0.03.\r\n\r\nHope this help:)", "votes": null }, { "id": "124916", "postDate": "06/23/2016 12:59:54", "content": "

@kuan chen, thx! awesome suggestion for grid by grid.

", "rawMarkdown": "kuan chen, thx! awesome suggestion for grid by grid.", "votes": null }, { "id": "124924", "postDate": "06/23/2016 14:07:20", "content": "

@kuan chen, I am back again... o_O

\n\n

On deeper thoughts, I highly doubt the procedure of filter out place_ids in testing set grid by grid. There is no reason to assume the original test set is generated in the same way.

\n\n

On way to explain this phenomenon would be that the original test set have an additional process of filtering place_ids and the Admins didn't point this out, say, filter place_ids that appears too fewer times.

\n\n

This assumptions make more sense because there are much fewer checkin data per day in the test set.

\n\n

Do you think it is necessary to ask Admins to clarify it?

", "rawMarkdown": "kuan chen, I am back again... o_O\r\n\r\nOn deeper thoughts, I highly doubt the procedure of filter out place_ids in testing set **grid by grid**. There is no reason to assume the original test set is generated in the same way.\r\n\r\nOn way to explain this phenomenon would be that the original test set have an additional process of filtering place_ids and the Admins didn't point this out, say, filter place_ids that appears too fewer times. \r\n\r\nThis assumptions make more sense because there are much fewer checkin data per day in the test set.\r\n\r\nDo you think it is necessary to ask Admins to clarify it?", "votes": null }, { "id": "124935", "postDate": "06/23/2016 16:08:24", "content": "

@ YijunMa

\n\n

Hi,I agree with you that the original testing set is not generated in the same way. I also discover that sometimes I have very good score in my cv(around 0.66) which turn out to be not that good in lb(0.579x). The cross validation set generated by my method is sometimes not that valid in my case

\n\n

For the second part, I am not quite sure. but do you mean that the the place_id in original test set is different from those in the training set? If so, I remembered that the Admin has claimed that both contain the same in some discussion thread

", "rawMarkdown": "YijunMa\r\n\r\nHi,I agree with you that the original testing set is not generated in the same way. I also discover that sometimes I have very good score in my cv(around 0.66) which turn out to be not that good in lb(0.579x). The cross validation set generated by my method is sometimes not that valid in my case\r\n\r\nFor the second part, I am not quite sure. but do you mean that the the place_id in original test set is different from those in the training set? If so, I remembered that the Admin has claimed that both contain the same in some discussion thread", "votes": null }, { "id": "124952", "postDate": "06/23/2016 20:00:15", "content": "

@kuan chen

\n\n

Maybe I'm wrong for the second part. I just cannot find a reason to explain the higher score in LB, and guess that there are fewer place_id in the test set than it should be. I know which thread you talk about, just that the higher score in LB confuse me.

", "rawMarkdown": "kuan chen\r\n\r\nMaybe I'm wrong for the second part. I just cannot find a reason to explain the higher score in LB, and guess that there are fewer place_id in the test set than it should be. I know which thread you talk about, just that the higher score in LB confuse me.", "votes": null } ], "comments": [ { "id": 123739, "author_name": "jturkewitz", "author_url": "", "post_date": "06/13/2016 20:18:11", "content": "

I found early on that my lb score was substantially higher than my cv on time > 600,000. It was something like 0.51 on cv gave 0.55 on lb. It takes so long to run on my laptop now that I haven't been running cv on the entire train set since then, so I'm not sure the precise magnitude of this difference. I do know that this original measurement I made included ids that were first seen after time > 600,000, removing these ids from the hold-out set does slightly increase the local cv.

", "votes": null, "replies": [] }, { "id": 124053, "author_name": "blindape", "author_url": "", "post_date": "06/15/2016 07:49:54", "content": "

[quote=Jared Turkewitz;123739]

\n\n

I found early on that my lb score was substantially higher than my cv on time > 600,000. It was something like 0.51 on cv gave 0.55 on lb. It takes so long to run on my laptop now that I haven't been running cv on the entire train set since then, so I'm not sure the precise magnitude of this difference. I do know that this original measurement I made included ids that were first seen after time > 600,000, removing these ids from the hold-out set does slightly increase the local cv.

\n\n

[/quote]

\n\n

Perhaps the unseen labels could explain it. If you compute the cv over all the hold data (including the new labels respect your train period) and the test set was cleaned of the new labels respect train set.\nDo you compute cv 0.51 over all the grids or over a sample?

", "votes": null, "replies": [] }, { "id": 124123, "author_name": "jturkewitz", "author_url": "", "post_date": "06/15/2016 17:42:38", "content": "

[quote=José A. Guerrero;124053]

\n\n

[quote=Jared Turkewitz;123739]

\n\n

I found early on that my lb score was substantially higher than my cv on time > 600,000. It was something like 0.51 on cv gave 0.55 on lb. It takes so long to run on my laptop now that I haven't been running cv on the entire train set since then, so I'm not sure the precise magnitude of this difference. I do know that this original measurement I made included ids that were first seen after time > 600,000, removing these ids from the hold-out set does slightly increase the local cv.

\n\n

[/quote]

\n\n

Perhaps the unseen labels could explain it. If you compute the cv over all the hold data (including the new labels respect your train period) and the test set was cleaned of the new labels respect train set.\nDo you compute cv 0.51 over all the grids or over a sample?

\n\n

[/quote]

\n\n

The 0.51 corresponded to about 0.55 on the lb, and was not removing unseen labels. Recently I haven't had the time to run cv over the entire sample because of cpu constraints. After removing unseen labels from the holdout set it still seems to me that the lb score is on average about 0.02-0.03 higher than cv, at least from the random sample of grid cells that I use for cv.

", "votes": null, "replies": [] }, { "id": 124131, "author_name": "wendykan", "author_url": "", "post_date": "06/15/2016 18:42:56", "content": "

Agree with @José's analysis about the unseen businesses.

", "votes": null, "replies": [] }, { "id": 124312, "author_name": "tmwright", "author_url": "", "post_date": "06/17/2016 03:27:39", "content": "

For comparable grid dimensions I found that my LB scores are consistently 0.03-0.04 greater than the validation score obtained on a holdout set of events at times greater than (roughly) 600000. I performed the validation on a random sampling of 1% of the grid cells (horizontal bars indicate +/- the standard error of the mean validation score so estimated).\n\"enter\nRedoing the validation excluding the unseen classes in each holdout grid cell I found that this discrepancy vanishes, and in fact the validation score is now ~0.01-0.02 greater than the LB score. But your mileage may vary...

", "votes": null, "replies": [] }, { "id": 124342, "author_name": "kuanchen", "author_url": "", "post_date": "06/17/2016 10:22:42", "content": "

@ Tod Wright

\n\n

Thank you for your sharing.

", "votes": null, "replies": [] }, { "id": 124524, "author_name": "jturkewitz", "author_url": "", "post_date": "06/19/2016 16:14:02", "content": "

Okay, I just tested it on the area of x in [0,0.2] and y in [0,10], as well as x in [0,1.2] y in [0,10]. On the smaller subset I got 0.01475 on the lb (corresponds to 0.59, on the larger subset I got 0.08875, corresponds to 0.5916, the overall lb score for this model was 0.5921. For my local cv, on the smaller subset I got 0.625, and on the larger subset I got 0.587. This was after removing unseen ids for local cv. This run took me 4 days on my 2 core laptop, so I don't have time to run cv on the entire grid.

\n\n

So it seems like overall after removing unseen ids the lb and cv are relatively close, though it depends on the subsets you are comparing.

", "votes": null, "replies": [] }, { "id": 124876, "author_name": "ma350365879", "author_url": "", "post_date": "06/22/2016 23:07:08", "content": "

@Tod Wright, I still find my local cv (sorted by time and the last ~25% data, filtered unseen place_id) still have lower @map3 than LB. I am using KNN, what algorithm did you use for this chart?

\n\n

It's like 0.55 in local cv to 0.58 on LB.

", "votes": null, "replies": [] }, { "id": 124899, "author_name": "kuanchen", "author_url": "", "post_date": "06/23/2016 07:51:22", "content": "

@ YijunMa

\n\n

I got the same problem as you before.I used knn,too. I am not sure how you filter out your unseen place_id. My original strategy is:

\n\n
    \n
  1. Split train/val based on time
    2. \n
  2. Directly filter out those place_ids in validation set which is not present in training set
    3. \n
\n\n

The cv score is lower than lb, then I came to a new strategy. The trick I found is that you have to filter out place_ids grid by grid.

\n\n

For example, we have place_id 1,2,4,6,7,10 and 4 grid A,B,C,D. place_id 1,2,4,6 are in training set while 4,6,7,10 in validation set.

\n\n

If grid A contains only place_id 1,2. then grid A only have information of place_id 1,2 to learn. If the validation data set in grid A contains,say, 4 and 6,we must remove them since the grid A only contains information of 1,2 during the training phase.

\n\n

If we follow my original strategy to split data then train, we can't avoid the case mentioned above.\nAfter I use my new strategy, my cv score(still knn) is always higher than lb by around 0.02~0.03.

\n\n

Hope this help:)

", "votes": null, "replies": [] }, { "id": 124916, "author_name": "ma350365879", "author_url": "", "post_date": "06/23/2016 12:59:54", "content": "

@kuan chen, thx! awesome suggestion for grid by grid.

", "votes": null, "replies": [] }, { "id": 124924, "author_name": "ma350365879", "author_url": "", "post_date": "06/23/2016 14:07:20", "content": "

@kuan chen, I am back again... o_O

\n\n

On deeper thoughts, I highly doubt the procedure of filter out place_ids in testing set grid by grid. There is no reason to assume the original test set is generated in the same way.

\n\n

On way to explain this phenomenon would be that the original test set have an additional process of filtering place_ids and the Admins didn't point this out, say, filter place_ids that appears too fewer times.

\n\n

This assumptions make more sense because there are much fewer checkin data per day in the test set.

\n\n

Do you think it is necessary to ask Admins to clarify it?

", "votes": null, "replies": [] }, { "id": 124935, "author_name": "kuanchen", "author_url": "", "post_date": "06/23/2016 16:08:24", "content": "

@ YijunMa

\n\n

Hi,I agree with you that the original testing set is not generated in the same way. I also discover that sometimes I have very good score in my cv(around 0.66) which turn out to be not that good in lb(0.579x). The cross validation set generated by my method is sometimes not that valid in my case

\n\n

For the second part, I am not quite sure. but do you mean that the the place_id in original test set is different from those in the training set? If so, I remembered that the Admin has claimed that both contain the same in some discussion thread

", "votes": null, "replies": [] }, { "id": 124952, "author_name": "ma350365879", "author_url": "", "post_date": "06/23/2016 20:00:15", "content": "

@kuan chen

\n\n

Maybe I'm wrong for the second part. I just cannot find a reason to explain the higher score in LB, and guess that there are fewer place_id in the test set than it should be. I know which thread you talk about, just that the higher score in LB confuse me.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "123736": "I set the grid as:\r\n\r\ntrain[, x_cell := pmax(ceiling(x * 4), 1L)]\r\n\r\ntrain[, y_cell := pmax(ceiling(y * 4), 1L)]\r\n\r\nand split train and hold @614127 (this reproduce exactly the same proportion in tran - hold as we have in train - test):\r\n\r\ntrain[, hold := as.integer(time > 614127)]\r\n\r\n\r\nWith this validation framework I'm playing with one of the more difficult (I think) square:\r\n\r\nx_cell = 21\r\ny_cell = 27\r\n\r\nThis square have the max place_id diversity.\r\n\r\nThe hold part of this square has 5357 rows.\r\n\r\nMy cv score in this is 0.5463.\r\n\r\nNice to hear your scores.", "123739": "I found early on that my lb score was substantially higher than my cv on time > 600,000. It was something like 0.51 on cv gave 0.55 on lb. It takes so long to run on my laptop now that I haven't been running cv on the entire train set since then, so I'm not sure the precise magnitude of this difference. I do know that this original measurement I made included ids that were first seen after time > 600,000, removing these ids from the hold-out set does slightly increase the local cv.", "124053": "[quote=Jared Turkewitz;123739]\r\n\r\nI found early on that my lb score was substantially higher than my cv on time > 600,000. It was something like 0.51 on cv gave 0.55 on lb. It takes so long to run on my laptop now that I haven't been running cv on the entire train set since then, so I'm not sure the precise magnitude of this difference. I do know that this original measurement I made included ids that were first seen after time > 600,000, removing these ids from the hold-out set does slightly increase the local cv.\r\n\r\n[/quote]\r\n\r\nPerhaps the unseen labels could explain it. If you compute the cv over all the hold data (including the new labels respect your train period) and the test set was cleaned of the new labels respect train set.\r\nDo you compute cv 0.51 over all the grids or over a sample?", "124123": "[quote=José A. Guerrero;124053]\r\n\r\n[quote=Jared Turkewitz;123739]\r\n\r\nI found early on that my lb score was substantially higher than my cv on time > 600,000. It was something like 0.51 on cv gave 0.55 on lb. It takes so long to run on my laptop now that I haven't been running cv on the entire train set since then, so I'm not sure the precise magnitude of this difference. I do know that this original measurement I made included ids that were first seen after time > 600,000, removing these ids from the hold-out set does slightly increase the local cv.\r\n\r\n[/quote]\r\n\r\nPerhaps the unseen labels could explain it. If you compute the cv over all the hold data (including the new labels respect your train period) and the test set was cleaned of the new labels respect train set.\r\nDo you compute cv 0.51 over all the grids or over a sample?\r\n\r\n\r\n\r\n[/quote]\r\n\r\nThe 0.51 corresponded to about 0.55 on the lb, and was not removing unseen labels. Recently I haven't had the time to run cv over the entire sample because of cpu constraints. After removing unseen labels from the holdout set it still seems to me that the lb score is on average about 0.02-0.03 higher than cv, at least from the random sample of grid cells that I use for cv.", "124131": "Agree with @José's analysis about the unseen businesses.", "124312": "For comparable grid dimensions I found that my LB scores are consistently 0.03-0.04 greater than the validation score obtained on a holdout set of events at times greater than (roughly) 600000. I performed the validation on a random sampling of 1% of the grid cells (horizontal bars indicate +/- the standard error of the mean validation score so estimated).\r\n![enter image description here][1]\r\nRedoing the validation excluding the unseen classes in each holdout grid cell I found that this discrepancy vanishes, and in fact the validation score is now ~0.01-0.02 greater than the LB score. But your mileage may vary...\r\n\r\n\r\n [1]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4427/val_lb_unseen.png", "124342": "Tod Wright\r\n\r\nThank you for your sharing.", "124524": "Okay, I just tested it on the area of x in [0,0.2] and y in [0,10], as well as x in [0,1.2] y in [0,10]. On the smaller subset I got 0.01475 on the lb (corresponds to 0.59, on the larger subset I got 0.08875, corresponds to 0.5916, the overall lb score for this model was 0.5921. For my local cv, on the smaller subset I got 0.625, and on the larger subset I got 0.587. This was after removing unseen ids for local cv. This run took me 4 days on my 2 core laptop, so I don't have time to run cv on the entire grid.\r\n\r\nSo it seems like overall after removing unseen ids the lb and cv are relatively close, though it depends on the subsets you are comparing.", "124876": "Tod Wright, I still find my local cv (sorted by time and the last ~25% data, filtered unseen place_id) still have lower @map3 than LB. I am using KNN, what algorithm did you use for this chart?\r\n\r\nIt's like 0.55 in local cv to 0.58 on LB.", "124899": "YijunMa\r\n\r\nI got the same problem as you before.I used knn,too. I am not sure how you filter out your unseen place_id. My original strategy is:\r\n\r\n1. Split train/val based on time\r\n2. Directly filter out those place_ids in validation set which is not present in training set\r\n\r\nThe cv score is lower than lb, then I came to a new strategy. The trick I found is that you have to filter out place_ids **grid by grid.** \r\n\r\nFor example, we have place_id 1,2,4,6,7,10 and 4 grid A,B,C,D. place_id 1,2,4,6 are in training set while 4,6,7,10 in validation set.\r\n\r\nIf grid A contains only place_id 1,2. then grid A only have information of place_id 1,2 to learn. If the validation data set in grid A contains,say, 4 and 6,we must remove them since the grid A only contains information of 1,2 during the training phase. \r\n\r\nIf we follow my original strategy to split data then train, we can't avoid the case mentioned above.\r\nAfter I use my new strategy, my cv score(still knn) is always higher than lb by around 0.02~0.03.\r\n\r\nHope this help:)", "124916": "kuan chen, thx! awesome suggestion for grid by grid.", "124924": "kuan chen, I am back again... o_O\r\n\r\nOn deeper thoughts, I highly doubt the procedure of filter out place_ids in testing set **grid by grid**. There is no reason to assume the original test set is generated in the same way.\r\n\r\nOn way to explain this phenomenon would be that the original test set have an additional process of filtering place_ids and the Admins didn't point this out, say, filter place_ids that appears too fewer times. \r\n\r\nThis assumptions make more sense because there are much fewer checkin data per day in the test set.\r\n\r\nDo you think it is necessary to ask Admins to clarify it?", "124935": "YijunMa\r\n\r\nHi,I agree with you that the original testing set is not generated in the same way. I also discover that sometimes I have very good score in my cv(around 0.66) which turn out to be not that good in lb(0.579x). The cross validation set generated by my method is sometimes not that valid in my case\r\n\r\nFor the second part, I am not quite sure. but do you mean that the the place_id in original test set is different from those in the training set? If so, I remembered that the Admin has claimed that both contain the same in some discussion thread", "124952": "kuan chen\r\n\r\nMaybe I'm wrong for the second part. I just cannot find a reason to explain the higher score in LB, and guess that there are fewer place_id in the test set than it should be. I know which thread you talk about, just that the higher score in LB confuse me." }, "source": "meta" }