Hi all,
\n\nI mentioned to Olivier ( @ogrellier ) in another thread that my local CV is way higher than than the LB score now the scoring error has been fixed, and he said that he had seen the same thing.
\n\nLet n be the total number of visits in the data set and m be the number of unique visitors. y is ln1p(sum revenue), and y hat is the estimate of that value.
\n\nThe scoring metric should be:
\n\n$$\n\\sqrt{\n\\frac{\\sum_{i=1}^{m}(y_i-\\hat y_i)^2}{m}\n}\n$$
\n\nBut I've noticed that:
\n\n$$\n\\sqrt{\n\\frac{\\sum_{i=1}^{m}(y_i-\\hat y_i)^2}{n}\n}\n$$
\n\nGives me closer agreement with the leader board.
\n\nThis could be a total coincidence, but is also a plausible mistake to make. If what I've said is right, then LB scores are out by a factor of root(m/n) - which isn't the end of the world. Of course it could easily be be noise.
\n\nIs this consistent with what others are seeing? If anyone has a good match between their LB and CV then that would be good evidence that my hunch is wrong!
", "messages": [ { "id": 393657, "postDate": "2018-09-25T18:13:36.547Z", "content": "Hi all,
\n\nI mentioned to Olivier ( @ogrellier ) in another thread that my local CV is way higher than than the LB score now the scoring error has been fixed, and he said that he had seen the same thing.
\n\nLet n be the total number of visits in the data set and m be the number of unique visitors. y is ln1p(sum revenue), and y hat is the estimate of that value.
\n\nThe scoring metric should be:
\n\n$$\n\\sqrt{\n\\frac{\\sum_{i=1}^{m}(y_i-\\hat y_i)^2}{m}\n}\n$$
\n\nBut I've noticed that:
\n\n$$\n\\sqrt{\n\\frac{\\sum_{i=1}^{m}(y_i-\\hat y_i)^2}{n}\n}\n$$
\n\nGives me closer agreement with the leader board.
\n\nThis could be a total coincidence, but is also a plausible mistake to make. If what I've said is right, then LB scores are out by a factor of root(m/n) - which isn't the end of the world. Of course it could easily be be noise.
\n\nIs this consistent with what others are seeing? If anyone has a good match between their LB and CV then that would be good evidence that my hunch is wrong!
", "rawMarkdown": "Hi all,\n\nI mentioned to Olivier ( @ogrellier ) in another thread that my local CV is way higher than than the LB score now the scoring error has been fixed, and he said that he had seen the same thing.\n\nLet n be the total number of visits in the data set and m be the number of unique visitors. y is ln1p(sum revenue), and y hat is the estimate of that value.\n\nThe scoring metric should be:\n\n$$\n\\sqrt{\n\\frac{\\sum_{i=1}^{m}(y_i-\\hat y_i)^2}{m}\n}\n$$\n\nBut I've noticed that:\n\n$$\n\\sqrt{\n\\frac{\\sum_{i=1}^{m}(y_i-\\hat y_i)^2}{n}\n}\n$$\n\nGives me closer agreement with the leader board. \n\nThis could be a total coincidence, but is also a plausible mistake to make. If what I've said is right, then LB scores are out by a factor of root(m/n) - which isn't the end of the world. Of course it could easily be be noise.\n\nIs this consistent with what others are seeing? If anyone has a good match between their LB and CV then that would be good evidence that my hunch is wrong!", "votes": 32 }, { "id": 397658, "postDate": "2018-10-02T21:50:42.287Z", "content": "Hey all -
\n\nThanks for checking into this and bringing it up!
\n\nI've looked into this and the metric code is stock - we didn't do anything custom with it (or touch it, even), so it matches every other competition that's used RMSE in the past. It should be behaving as indicated by the top equation.
\n\nI also checked into my data handling to see if there was something I'd done that would have fed into an effect like this. I didn't see anything that would do so.
\n\nIt's definitely concerning, but so far I don't see any indicators that it's a bug. I'll keep an eye out.
", "rawMarkdown": "Hey all -\n\nThanks for checking into this and bringing it up!\n\nI've looked into this and the metric code is stock - we didn't do anything custom with it (or touch it, even), so it matches every other competition that's used RMSE in the past. It should be behaving as indicated by the top equation.\n\nI also checked into my data handling to see if there was something I'd done that would have fed into an effect like this. I didn't see anything that would do so.\n\nIt's definitely concerning, but so far I don't see any indicators that it's a bug. I'll keep an eye out.", "votes": 7, "replies": [ { "id": 397663, "postDate": "2018-10-02T22:02:22.793Z", "content": "Thank you, Phil!
\n\nJust to make sure I understood correctly, your score of the validation set matches the one of the test set, right?
", "rawMarkdown": "Thank you, Phil! \n\nJust to make sure I understood correctly, your score of the validation set matches the one of the test set, right? " }, { "id": 397669, "postDate": "2018-10-02T22:11:59.160Z", "content": "You're welcome, Leonid!
\n\nBy validation set, do you mean the folds in my own CV?
", "rawMarkdown": "You're welcome, Leonid!\n\nBy validation set, do you mean the folds in my own CV?" }, { "id": 397674, "postDate": "2018-10-02T22:30:08.297Z", "content": "Yes. I meant your CV vs the test (our LB).
\n\nThank you.
", "rawMarkdown": "Yes. I meant your CV vs the test (our LB). \n\nThank you. ", "votes": 1 }, { "id": 397701, "postDate": "2018-10-03T00:23:14.110Z", "content": "Ahh, okay, thanks.
\n\nI don't want to get into any details about my modeling, but I didn't see anything from my CV that made me think there was a problem with the LB.
", "rawMarkdown": "Ahh, okay, thanks.\n\nI don't want to get into any details about my modeling, but I didn't see anything from my CV that made me think there was a problem with the LB.", "votes": 2 }, { "id": 397720, "postDate": "2018-10-03T02:02:03.707Z", "content": "Fair enough! Thank you.
", "rawMarkdown": "Fair enough! Thank you. " }, { "id": 397724, "postDate": "2018-10-03T02:13:19.197Z", "content": "I guess we are really missing something then... its getting scary.
", "rawMarkdown": "I guess we are really missing something then... its getting scary.", "votes": 3 }, { "id": 398189, "postDate": "2018-10-03T16:27:44.490Z", "content": "Dear Phil,
\n\nthank you for this comment.
\n\nJust to be clear for the RMSE-calculation, the grouping is related to unique visitors and not to the number of visits/sessions?
", "rawMarkdown": "Dear Phil,\n\nthank you for this comment.\n\nJust to be clear for the RMSE-calculation, the grouping is related to unique visitors and **not** to the number of visits/sessions?\n" }, { "id": 398193, "postDate": "2018-10-03T16:35:52.807Z", "content": "Hi Justfor,
\n\nYou're welcome!
\n\nCorrect - for the RMSE calculation, the grouping is on unique users. Number of visits / sessions is not a factor. I also checked to make sure there wasn't some confounding effect in the target that might appear to cause a similar effect, and there was nothing that I could see.
", "rawMarkdown": "Hi Justfor,\n\nYou're welcome!\n\nCorrect - for the RMSE calculation, the grouping is on unique users. Number of visits / sessions is not a factor. I also checked to make sure there wasn't some confounding effect in the target that might appear to cause a similar effect, and there was nothing that I could see.", "votes": 2 } ] }, { "id": 393731, "postDate": "2018-09-25T20:13:45.237Z", "content": "Thanks Craig for your hard work.
\n\nI guess this is just another metric issue...
\n\nPS: tagging sometimes work and sometimes doesn't I have no idea why :(
", "rawMarkdown": "Thanks Craig for your hard work. \n\nI guess this is just another metric issue...\n\nPS: tagging sometimes work and sometimes doesn't I have no idea why :(", "votes": 1 }, { "id": 393661, "postDate": "2018-09-25T18:18:51.097Z", "content": "Make total sense! \nThank you and @ogrellier!
", "rawMarkdown": "Make total sense! \nThank you and @ogrellier!", "votes": 2, "replies": [ { "id": 393665, "postDate": "2018-09-25T18:28:32.700Z", "content": "No problem - but also no guarantees I'm right.... How do you tag people like that? I seem to have botched it in my post!
", "rawMarkdown": "No problem - but also no guarantees I'm right.... How do you tag people like that? I seem to have botched it in my post!", "votes": 2 }, { "id": 393674, "postDate": "2018-09-25T18:34:48.063Z", "content": "Hmmm... It looks right in your post. I don't know why it didn't work. Maybe some kind of format issue? Try maybe copy-paste it from my reply...
", "rawMarkdown": "Hmmm... It looks right in your post. I don't know why it didn't work. Maybe some kind of format issue? Try maybe copy-paste it from my reply... ", "votes": 1 } ] }, { "id": 395498, "postDate": "2018-09-28T17:29:52.847Z", "content": "Are you running the lgb at the session level ? If so, I don't think you compare the 2 metrics, one is the RMSE of the session revenue and the LB is based at the user revenue.
\n\nBut to answer your question yes most of the kernels are having the same issue since most of us are optimizing at the session level.
", "rawMarkdown": "Are you running the lgb at the session level ? If so, I don't think you compare the 2 metrics, one is the RMSE of the session revenue and the LB is based at the user revenue.\n\nBut to answer your question yes most of the kernels are having the same issue since most of us are optimizing at the session level." }, { "id": 394162, "postDate": "2018-09-26T12:49:59.370Z", "content": "I thought it was me doing things the wrong way but if I am not the only one to see this weird discrepancy between CV and LB it might actually be a mistake from Kaggle's way of computing the metrics.
", "rawMarkdown": "I thought it was me doing things the wrong way but if I am not the only one to see this weird discrepancy between CV and LB it might actually be a mistake from Kaggle's way of computing the metrics." }, { "id": 393799, "postDate": "2018-09-25T22:40:40.070Z", "content": "I'm seeing this exact CV/LB discrepancy as well. Very confusing and contrary to what I had expected.
", "rawMarkdown": "I'm seeing this exact CV/LB discrepancy as well. Very confusing and contrary to what I had expected." }, { "id": 393739, "postDate": "2018-09-25T20:26:04.747Z", "rawMarkdown": "", "votes": 1, "isDeleted": true, "replies": [ { "id": 393774, "postDate": "2018-09-25T21:42:13.180Z", "content": "hey @taober quick question, what cv strategy are you using?
", "rawMarkdown": "hey @taober quick question, what cv strategy are you using?" }, { "id": 393781, "postDate": "2018-09-25T21:49:53.383Z", "rawMarkdown": "", "votes": 2, "isDeleted": true }, { "id": 394190, "postDate": "2018-09-26T13:29:32.020Z", "content": "what is group k fold? I have been reading the docs but I couldn't understand anything. If you can post any articles related to it. it would be helpful
For user-level models, I think we can't do the time-based splitting. So for user-level models what do you think the suitable validation scheme ?
Hi Yeygeni, What CV strategy led to that score is it GroupKFold , Stratfied or time based ?
", "rawMarkdown": "Hi Yeygeni, What CV strategy led to that score is it GroupKFold , Stratfied or time based ? " }, { "id": 394613, "postDate": "2018-09-27T06:28:52.533Z", "rawMarkdown": "", "isDeleted": true } ] } ], "comments": [ { "id": 397658, "author_name": "Phil Culliton", "author_url": "", "post_date": "2018-10-02T21:50:42.287000", "content": "Hey all -
\n\nThanks for checking into this and bringing it up!
\n\nI've looked into this and the metric code is stock - we didn't do anything custom with it (or touch it, even), so it matches every other competition that's used RMSE in the past. It should be behaving as indicated by the top equation.
\n\nI also checked into my data handling to see if there was something I'd done that would have fed into an effect like this. I didn't see anything that would do so.
\n\nIt's definitely concerning, but so far I don't see any indicators that it's a bug. I'll keep an eye out.
", "votes": 7, "replies": [ { "id": 397663, "author_name": "Leonid Gilburd", "author_url": "", "post_date": "2018-10-02T22:02:22.793000", "content": "Thank you, Phil!
\n\nJust to make sure I understood correctly, your score of the validation set matches the one of the test set, right?
", "votes": 0, "replies": [] }, { "id": 397669, "author_name": "Phil Culliton", "author_url": "", "post_date": "2018-10-02T22:11:59.160000", "content": "You're welcome, Leonid!
\n\nBy validation set, do you mean the folds in my own CV?
", "votes": 0, "replies": [] }, { "id": 397674, "author_name": "Leonid Gilburd", "author_url": "", "post_date": "2018-10-02T22:30:08.297000", "content": "Yes. I meant your CV vs the test (our LB).
\n\nThank you.
", "votes": 1, "replies": [] }, { "id": 397701, "author_name": "Phil Culliton", "author_url": "", "post_date": "2018-10-03T00:23:14.110000", "content": "Ahh, okay, thanks.
\n\nI don't want to get into any details about my modeling, but I didn't see anything from my CV that made me think there was a problem with the LB.
", "votes": 2, "replies": [] }, { "id": 397720, "author_name": "Leonid Gilburd", "author_url": "", "post_date": "2018-10-03T02:02:03.707000", "content": "Fair enough! Thank you.
", "votes": 0, "replies": [] }, { "id": 397724, "author_name": "Adri", "author_url": "", "post_date": "2018-10-03T02:13:19.197000", "content": "I guess we are really missing something then... its getting scary.
", "votes": 3, "replies": [] }, { "id": 398189, "author_name": "Justfor", "author_url": "", "post_date": "2018-10-03T16:27:44.490000", "content": "Dear Phil,
\n\nthank you for this comment.
\n\nJust to be clear for the RMSE-calculation, the grouping is related to unique visitors and not to the number of visits/sessions?
", "votes": 0, "replies": [] }, { "id": 398193, "author_name": "Phil Culliton", "author_url": "", "post_date": "2018-10-03T16:35:52.807000", "content": "Hi Justfor,
\n\nYou're welcome!
\n\nCorrect - for the RMSE calculation, the grouping is on unique users. Number of visits / sessions is not a factor. I also checked to make sure there wasn't some confounding effect in the target that might appear to cause a similar effect, and there was nothing that I could see.
", "votes": 2, "replies": [] } ] }, { "id": 393731, "author_name": "olivier", "author_url": "", "post_date": "2018-09-25T20:13:45.237000", "content": "Thanks Craig for your hard work.
\n\nI guess this is just another metric issue...
\n\nPS: tagging sometimes work and sometimes doesn't I have no idea why :(
", "votes": 1, "replies": [] }, { "id": 393661, "author_name": "Leonid Gilburd", "author_url": "", "post_date": "2018-09-25T18:18:51.097000", "content": "Make total sense! \nThank you and @ogrellier!
", "votes": 2, "replies": [ { "id": 393665, "author_name": "Craig Armstrong", "author_url": "", "post_date": "2018-09-25T18:28:32.700000", "content": "No problem - but also no guarantees I'm right.... How do you tag people like that? I seem to have botched it in my post!
", "votes": 2, "replies": [] }, { "id": 393674, "author_name": "Leonid Gilburd", "author_url": "", "post_date": "2018-09-25T18:34:48.063000", "content": "Hmmm... It looks right in your post. I don't know why it didn't work. Maybe some kind of format issue? Try maybe copy-paste it from my reply...
", "votes": 1, "replies": [] } ] }, { "id": 395498, "author_name": "Antoine", "author_url": "", "post_date": "2018-09-28T17:29:52.847000", "content": "Are you running the lgb at the session level ? If so, I don't think you compare the 2 metrics, one is the RMSE of the session revenue and the LB is based at the user revenue.
\n\nBut to answer your question yes most of the kernels are having the same issue since most of us are optimizing at the session level.
", "votes": 0, "replies": [] }, { "id": 394162, "author_name": "Arthurim", "author_url": "", "post_date": "2018-09-26T12:49:59.370000", "content": "I thought it was me doing things the wrong way but if I am not the only one to see this weird discrepancy between CV and LB it might actually be a mistake from Kaggle's way of computing the metrics.
", "votes": 0, "replies": [] }, { "id": 393799, "author_name": "Nick", "author_url": "", "post_date": "2018-09-25T22:40:40.070000", "content": "I'm seeing this exact CV/LB discrepancy as well. Very confusing and contrary to what I had expected.
", "votes": 0, "replies": [] }, { "id": 393739, "author_name": "", "author_url": "", "post_date": "2018-09-25T20:26:04.747000", "content": "", "votes": 1, "replies": [ { "id": 393774, "author_name": "maverick", "author_url": "", "post_date": "2018-09-25T21:42:13.180000", "content": "hey @taober quick question, what cv strategy are you using?
", "votes": 0, "replies": [] }, { "id": 393781, "author_name": "", "author_url": "", "post_date": "2018-09-25T21:49:53.383000", "content": "", "votes": 2, "replies": [] }, { "id": 394190, "author_name": "redshellspy", "author_url": "", "post_date": "2018-09-26T13:29:32.020000", "content": "what is group k fold? I have been reading the docs but I couldn't understand anything. If you can post any articles related to it. it would be helpful
For user-level models, I think we can't do the time-based splitting. So for user-level models what do you think the suitable validation scheme ?
Hi Yeygeni, What CV strategy led to that score is it GroupKFold , Stratfied or time based ?
", "votes": 0, "replies": [] }, { "id": 394613, "author_name": "", "author_url": "", "post_date": "2018-09-27T06:28:52.533000", "content": "", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "393657": "Hi all,\n\nI mentioned to Olivier ( @ogrellier ) in another thread that my local CV is way higher than than the LB score now the scoring error has been fixed, and he said that he had seen the same thing.\n\nLet n be the total number of visits in the data set and m be the number of unique visitors. y is ln1p(sum revenue), and y hat is the estimate of that value.\n\nThe scoring metric should be:\n\n$$\n\\sqrt{\n\\frac{\\sum_{i=1}^{m}(y_i-\\hat y_i)^2}{m}\n}\n$$\n\nBut I've noticed that:\n\n$$\n\\sqrt{\n\\frac{\\sum_{i=1}^{m}(y_i-\\hat y_i)^2}{n}\n}\n$$\n\nGives me closer agreement with the leader board. \n\nThis could be a total coincidence, but is also a plausible mistake to make. If what I've said is right, then LB scores are out by a factor of root(m/n) - which isn't the end of the world. Of course it could easily be be noise.\n\nIs this consistent with what others are seeing? If anyone has a good match between their LB and CV then that would be good evidence that my hunch is wrong!", "397658": "Hey all -\n\nThanks for checking into this and bringing it up!\n\nI've looked into this and the metric code is stock - we didn't do anything custom with it (or touch it, even), so it matches every other competition that's used RMSE in the past. It should be behaving as indicated by the top equation.\n\nI also checked into my data handling to see if there was something I'd done that would have fed into an effect like this. I didn't see anything that would do so.\n\nIt's definitely concerning, but so far I don't see any indicators that it's a bug. I'll keep an eye out.", "393731": "Thanks Craig for your hard work. \n\nI guess this is just another metric issue...\n\nPS: tagging sometimes work and sometimes doesn't I have no idea why :(", "393661": "Make total sense! \nThank you and @ogrellier!", "395498": "Are you running the lgb at the session level ? If so, I don't think you compare the 2 metrics, one is the RMSE of the session revenue and the LB is based at the user revenue.\n\nBut to answer your question yes most of the kernels are having the same issue since most of us are optimizing at the session level.", "394162": "I thought it was me doing things the wrong way but if I am not the only one to see this weird discrepancy between CV and LB it might actually be a mistake from Kaggle's way of computing the metrics.", "393799": "I'm seeing this exact CV/LB discrepancy as well. Very confusing and contrary to what I had expected.", "393739": "" } }