It is a rule in Kaggle that every competition must have a thread about the gap between local validation/cv scores and leaderboard scores. We don't seem to have one yet, so here goes...
\n\nMy LB scores are usually about 0.01-0.02 higher than local validation. I'm using a fixed random set of about 1/5 of the data for validation. There seems to be a trend in which higher performing models have a bigger gap in favor of the LB. The 0.283 score that I'm showing right now got 0.257 locally. My previous public score of 0.231 was on a model that got 0.220 locally. For my lowest performing models the gap is 0. The only model that I've had for which the gap was negative turned out to have a leak in it.
\n\nPlease add your thoughts and experiences below.
", "messages": [ { "id": "136685", "postDate": "09/23/2016 23:43:20", "content": "It is a rule in Kaggle that every competition must have a thread about the gap between local validation/cv scores and leaderboard scores. We don't seem to have one yet, so here goes...
\n\nMy LB scores are usually about 0.01-0.02 higher than local validation. I'm using a fixed random set of about 1/5 of the data for validation. There seems to be a trend in which higher performing models have a bigger gap in favor of the LB. The 0.283 score that I'm showing right now got 0.257 locally. My previous public score of 0.231 was on a model that got 0.220 locally. For my lowest performing models the gap is 0. The only model that I've had for which the gap was negative turned out to have a leak in it.
\n\nPlease add your thoughts and experiences below.
", "rawMarkdown": "It is a rule in Kaggle that every competition must have a thread about the gap between local validation/cv scores and leaderboard scores. We don't seem to have one yet, so here goes... \r\n\r\nMy LB scores are usually about 0.01-0.02 higher than local validation. I'm using a fixed random set of about 1/5 of the data for validation. There seems to be a trend in which higher performing models have a bigger gap in favor of the LB. The 0.283 score that I'm showing right now got 0.257 locally. My previous public score of 0.231 was on a model that got 0.220 locally. For my lowest performing models the gap is 0. The only model that I've had for which the gap was negative turned out to have a leak in it.\r\n\r\nPlease add your thoughts and experiences below.", "votes": null }, { "id": "136697", "postDate": "09/24/2016 01:52:25", "content": "I'm getting between 0.0048 and 0.016 between the local validation and leader board score using 10 percent of the data. Validation score is always higher.
", "rawMarkdown": "I'm getting between 0.0048 and 0.016 between the local validation and leader board score using 10 percent of the data. Validation score is always higher.", "votes": null }, { "id": "136698", "postDate": "09/24/2016 02:03:40", "content": "I am doing the same CV scheme, my scores are a bit off as well but direction holds.
\n\nHere there are some models (CV vs LB):
\n0.29788 vs 0.29815
\n0.25388 vs 0.24889
Hi everyone,
\n\nMy scores are as follows now. \n$$\n\\begin{array}{|c|c|}\n\\hline \\bf{data} & \\bf{score(mcc)} \\\\ \\hline\n\\hline \\rm{Training} & 0.634 \\\\ \\hline\n\\hline \\rm{Validation} & 0.312 \\\\ \\hline\n\\hline \\rm{Public \\space Leaderboard} & 0.301 \\\\ \\hline\n\\end{array}\n$$
\n\nI use 10% of data for validation.
For our current model:
\n\nCV 0.4675
\nLB 0.4676
AUC 0.9212 ;)
", "rawMarkdown": "For our current model:\r\n\r\nCV 0.4675 \r\nLB 0.4676\r\n\r\nAUC 0.9212 ;)", "votes": null }, { "id": "136804", "postDate": "09/25/2016 11:13:18", "content": "My LB scores are about 0.01-0.02 higher than local validation as well. I believe those who claim LB 0.01-0.02 lower than validation are likely using meta features, Bayesian encoded features, or the out of fold target based features. You can correct me if I'm mistaken.
", "rawMarkdown": "My LB scores are about 0.01-0.02 higher than local validation as well. I believe those who claim LB 0.01-0.02 lower than validation are likely using meta features, Bayesian encoded features, or the out of fold target based features. You can correct me if I'm mistaken.", "votes": null }, { "id": "136850", "postDate": "09/25/2016 20:32:46", "content": "[quote=anokas;136803]
\n\nFor our current model:
\n\nCV 0.4675
\nLB 0.4676
AUC 0.9212 ;)
\n\n[/quote]
\n\nThis is impressive. Are you using some special learning framework that is different from the usual classification setting (e.g. the popular public script)? It seems the answer is yes!
", "rawMarkdown": "[quote=anokas;136803]\r\n\r\nFor our current model:\r\n\r\nCV 0.4675 \r\nLB 0.4676\r\n\r\nAUC 0.9212 ;)\r\n\r\n[/quote]\r\n\r\n\r\nThis is impressive. Are you using some special learning framework that is different from the usual classification setting (e.g. the popular public script)? It seems the answer is yes!", "votes": null }, { "id": "136860", "postDate": "09/25/2016 21:47:03", "content": "[quote=anokas;136803]
\n\nFor our current model:
\n\nCV 0.4675
\nLB 0.4676
AUC 0.9212 ;)
\n\n[/quote]\nShould not have mentioned AUC it reveals a lot ... :P
", "rawMarkdown": "[quote=anokas;136803]\r\n\r\nFor our current model:\r\n\r\nCV 0.4675 \r\nLB 0.4676\r\n\r\nAUC 0.9212 ;)\r\n\r\n[/quote]\r\nShould not have mentioned AUC it reveals a lot ... :P", "votes": null }, { "id": "136910", "postDate": "09/26/2016 11:36:52", "content": "[quote=Mike Kim;136804]
\n\nMy LB scores are about 0.01-0.02 higher than local validation as well. I believe those who claim LB 0.01-0.02 lower than validation are likely using meta features, Bayesian encoded features, or the out of fold target based features. You can correct me if I'm mistaken.
\n\n[/quote]
\n\nI have one fold giving 0.03 more than public LB. I've seen up to +0.04 too locally (but also up to -0.03). Mostly a matter of random seed as when I had such +0.03, all the other folds were under the public LB score. No feature engineering, pure stratified splitting.
\n\nIf I change the random seed I get different results, but the "dominant/outlying" fold remains. But when changing the metric, the shake up appears (the dominant/outlying fold is not anymore).
\n\nCV is consistent with Public LB +- 0.01 after testing twice.
", "rawMarkdown": "[quote=Mike Kim;136804]\r\n\r\nMy LB scores are about 0.01-0.02 higher than local validation as well. I believe those who claim LB 0.01-0.02 lower than validation are likely using meta features, Bayesian encoded features, or the out of fold target based features. You can correct me if I'm mistaken.\r\n\r\n[/quote]\r\n\r\nI have one fold giving 0.03 more than public LB. I've seen up to +0.04 too locally (but also up to -0.03). Mostly a matter of random seed as when I had such +0.03, all the other folds were under the public LB score. No feature engineering, pure stratified splitting.\r\n\r\nIf I change the random seed I get different results, but the \"dominant/outlying\" fold remains. But when changing the metric, the shake up appears (the dominant/outlying fold is not anymore).\r\n\r\nCV is consistent with Public LB +- 0.01 after testing twice.", "votes": null }, { "id": "136913", "postDate": "09/26/2016 12:25:57", "content": "CV ~0.2411\nLB ~0.258
\n\nwith 5-fold stratCV
", "rawMarkdown": "CV ~0.2411\r\nLB ~0.258\r\n\r\nwith 5-fold stratCV", "votes": null } ], "comments": [ { "id": 136697, "author_name": "notaapple", "author_url": "", "post_date": "09/24/2016 01:52:25", "content": "I'm getting between 0.0048 and 0.016 between the local validation and leader board score using 10 percent of the data. Validation score is always higher.
", "votes": null, "replies": [] }, { "id": 136698, "author_name": "carloshuertas", "author_url": "", "post_date": "09/24/2016 02:03:40", "content": "I am doing the same CV scheme, my scores are a bit off as well but direction holds.
\n\nHere there are some models (CV vs LB):
\n0.29788 vs 0.29815
\n0.25388 vs 0.24889
Hi everyone,
\n\nMy scores are as follows now. \n$$\n\\begin{array}{|c|c|}\n\\hline \\bf{data} & \\bf{score(mcc)} \\\\ \\hline\n\\hline \\rm{Training} & 0.634 \\\\ \\hline\n\\hline \\rm{Validation} & 0.312 \\\\ \\hline\n\\hline \\rm{Public \\space Leaderboard} & 0.301 \\\\ \\hline\n\\end{array}\n$$
\n\nI use 10% of data for validation.
For our current model:
\n\nCV 0.4675
\nLB 0.4676
AUC 0.9212 ;)
", "votes": null, "replies": [] }, { "id": 136804, "author_name": "mikeskim", "author_url": "", "post_date": "09/25/2016 11:13:18", "content": "My LB scores are about 0.01-0.02 higher than local validation as well. I believe those who claim LB 0.01-0.02 lower than validation are likely using meta features, Bayesian encoded features, or the out of fold target based features. You can correct me if I'm mistaken.
", "votes": null, "replies": [] }, { "id": 136850, "author_name": "amazingbob", "author_url": "", "post_date": "09/25/2016 20:32:46", "content": "[quote=anokas;136803]
\n\nFor our current model:
\n\nCV 0.4675
\nLB 0.4676
AUC 0.9212 ;)
\n\n[/quote]
\n\nThis is impressive. Are you using some special learning framework that is different from the usual classification setting (e.g. the popular public script)? It seems the answer is yes!
", "votes": null, "replies": [] }, { "id": 136860, "author_name": "outliar", "author_url": "", "post_date": "09/25/2016 21:47:03", "content": "[quote=anokas;136803]
\n\nFor our current model:
\n\nCV 0.4675
\nLB 0.4676
AUC 0.9212 ;)
\n\n[/quote]\nShould not have mentioned AUC it reveals a lot ... :P
", "votes": null, "replies": [] }, { "id": 136910, "author_name": "laurae2", "author_url": "", "post_date": "09/26/2016 11:36:52", "content": "[quote=Mike Kim;136804]
\n\nMy LB scores are about 0.01-0.02 higher than local validation as well. I believe those who claim LB 0.01-0.02 lower than validation are likely using meta features, Bayesian encoded features, or the out of fold target based features. You can correct me if I'm mistaken.
\n\n[/quote]
\n\nI have one fold giving 0.03 more than public LB. I've seen up to +0.04 too locally (but also up to -0.03). Mostly a matter of random seed as when I had such +0.03, all the other folds were under the public LB score. No feature engineering, pure stratified splitting.
\n\nIf I change the random seed I get different results, but the "dominant/outlying" fold remains. But when changing the metric, the shake up appears (the dominant/outlying fold is not anymore).
\n\nCV is consistent with Public LB +- 0.01 after testing twice.
", "votes": null, "replies": [] }, { "id": 136913, "author_name": "stgrmks", "author_url": "", "post_date": "09/26/2016 12:25:57", "content": "CV ~0.2411\nLB ~0.258
\n\nwith 5-fold stratCV
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "136685": "It is a rule in Kaggle that every competition must have a thread about the gap between local validation/cv scores and leaderboard scores. We don't seem to have one yet, so here goes... \r\n\r\nMy LB scores are usually about 0.01-0.02 higher than local validation. I'm using a fixed random set of about 1/5 of the data for validation. There seems to be a trend in which higher performing models have a bigger gap in favor of the LB. The 0.283 score that I'm showing right now got 0.257 locally. My previous public score of 0.231 was on a model that got 0.220 locally. For my lowest performing models the gap is 0. The only model that I've had for which the gap was negative turned out to have a leak in it.\r\n\r\nPlease add your thoughts and experiences below.", "136697": "I'm getting between 0.0048 and 0.016 between the local validation and leader board score using 10 percent of the data. Validation score is always higher.", "136698": "I am doing the same CV scheme, my scores are a bit off as well but direction holds.\r\n\r\nHere there are some models (CV vs LB): \r\n0.29788 vs 0.29815 \r\n0.25388 vs 0.24889", "136802": "Hi everyone,\r\n\r\nMy scores are as follows now. \r\n$$\r\n\\begin{array}{|c|c|}\r\n\\hline \\bf{data} & \\bf{score(mcc)} \\\\ \\hline\r\n\\hline \\rm{Training} & 0.634 \\\\ \\hline\r\n\\hline \\rm{Validation} & 0.312 \\\\ \\hline\r\n\\hline \\rm{Public \\space Leaderboard} & 0.301 \\\\ \\hline\r\n\\end{array}\r\n$$\r\n\r\nI use 10% of data for validation.