My cross-validation scores are considerably lower than my public LB scores, for a range of different models, and I'm wondering why. Is anyone else experiencing this?
\n\nI'm seeing CV scores in the range of 0.990 +- 0.001 in 5-fold CV (averaged over multiple random shuffles of the data). These models consistently score 0.9855 +- 0.005 on the public LB. The difference is huge and is far outside the variation range you see in CV folds.
\n\nHaving triple-checked my CV process, I can entirely rule out any leak or error on my side. There are only two possible hypotheses to explain this oddity:
\n\nthe public leaderboard dataset is ridiculously small, on the order of 10k samples or less. This hypothesis is supported by the fact that LB scores have a much higher variance than CV scores. If this is the case, then we should only rely on CV and be careful of avoiding LB overfitting.
the training samples are not independent (eg. the training set contains samples that are quasi-duplicates of each other and that are not found in the test set), which means that any model trained on the subset of this training data will perform much higher on a different subset than it would on the test data (in other words, CV won't work accurately with the training data). If that is the case, we can't rely on CV anymore. Big potential issue.
This second hypothesis would not make sense with real world particle data, but it could happen with simulated data.
\n\nThoughts?
", "messages": [ { "id": "88066", "postDate": "08/03/2015 08:29:03", "content": "My cross-validation scores are considerably lower than my public LB scores, for a range of different models, and I'm wondering why. Is anyone else experiencing this?
\n\nI'm seeing CV scores in the range of 0.990 +- 0.001 in 5-fold CV (averaged over multiple random shuffles of the data). These models consistently score 0.9855 +- 0.005 on the public LB. The difference is huge and is far outside the variation range you see in CV folds.
\n\nHaving triple-checked my CV process, I can entirely rule out any leak or error on my side. There are only two possible hypotheses to explain this oddity:
\n\nthe public leaderboard dataset is ridiculously small, on the order of 10k samples or less. This hypothesis is supported by the fact that LB scores have a much higher variance than CV scores. If this is the case, then we should only rely on CV and be careful of avoiding LB overfitting.
the training samples are not independent (eg. the training set contains samples that are quasi-duplicates of each other and that are not found in the test set), which means that any model trained on the subset of this training data will perform much higher on a different subset than it would on the test data (in other words, CV won't work accurately with the training data). If that is the case, we can't rely on CV anymore. Big potential issue.
This second hypothesis would not make sense with real world particle data, but it could happen with simulated data.
\n\nThoughts?
", "rawMarkdown": "My cross-validation scores are considerably lower than my public LB scores, for a range of different models, and I'm wondering why. Is anyone else experiencing this? \r\n\r\nI'm seeing CV scores in the range of 0.990 +- 0.001 in 5-fold CV (averaged over multiple random shuffles of the data). These models consistently score 0.9855 +- 0.005 on the public LB. The difference is huge and is far outside the variation range you see in CV folds.\r\n\r\nHaving triple-checked my CV process, I can entirely rule out any leak or error on my side. There are only two possible hypotheses to explain this oddity:\r\n\r\n- the public leaderboard dataset is ridiculously small, on the order of 10k samples or less. This hypothesis is supported by the fact that LB scores have a much higher variance than CV scores. If this is the case, then we should only rely on CV and be careful of avoiding LB overfitting.\r\n\r\n- the training samples are not independent (eg. the training set contains samples that are quasi-duplicates of each other and that are not found in the test set), which means that any model trained on the subset of this training data will perform much higher on a different subset than it would on the test data (in other words, CV won't work accurately with the training data). If that is the case, we can't rely on CV anymore. Big potential issue.\r\n\r\nThis second hypothesis would not make sense with real world particle data, but it could happen with simulated data. \r\n\r\nThoughts?", "votes": null }, { "id": "88067", "postDate": "08/03/2015 08:34:43", "content": "When computing your CV score, do you include only those events for which min_ANNmuon > 0.4? (This is something I had missed myself)
", "rawMarkdown": "When computing your CV score, do you include only those events for which min_ANNmuon > 0.4? (This is something I had missed myself)", "votes": null }, { "id": "88068", "postDate": "08/03/2015 08:37:07", "content": "You have to remember to score only rows with min_ANNmuon > 0.4. Eval code from the starter kit:
\n\ntrain_eval = train[train['min_ANNmuon'] > 0.4]\ntrain_probs = baseline.predict_proba(train_eval[variables])[:, 1]\nAUC = evaluation.roc_auc_truncated(train_eval['signal'], train_probs)\nSo train on all rows but only evaluate rows with min_ANNmuon > .4
", "rawMarkdown": "You have to remember to score only rows with min_ANNmuon > 0.4. Eval code from the starter kit:\r\n\r\n train_eval = train[train['min_ANNmuon'] > 0.4]\r\n train_probs = baseline.predict_proba(train_eval[variables])[:, 1]\r\n AUC = evaluation.roc_auc_truncated(train_eval['signal'], train_probs)\r\n\r\n\r\nSo train on all rows but only evaluate rows with min_ANNmuon > .4", "votes": null }, { "id": "88069", "postDate": "08/03/2015 08:38:58", "content": "[quote=Gilles Louppe;88067]
\n\nWhen computing your CV score, do you include only those events for which min_ANNmuon > 0.4? (This is something I had missed myself)
\n\n[/quote]
\n\nI use the score returned by the function provided in the starter kit:
\n\nroc_auc_truncated(labels, predictions, tpr_thresholds=(0.2, 0.4, 0.6, 0.8), roc_weights=(4, 3, 2, 1, 0))\nI apply this function to all data points, no filtering. Is this incorrect?
", "rawMarkdown": "[quote=Gilles Louppe;88067]\r\n\r\nWhen computing your CV score, do you include only those events for which min_ANNmuon > 0.4? (This is something I had missed myself)\r\n\r\n[/quote]\r\n\r\nI use the score returned by the function provided in the starter kit:\r\n\r\n roc_auc_truncated(labels, predictions, tpr_thresholds=(0.2, 0.4, 0.6, 0.8), roc_weights=(4, 3, 2, 1, 0))\r\n\r\nI apply this function to all data points, no filtering. Is this incorrect?", "votes": null }, { "id": "88070", "postDate": "08/03/2015 08:45:35", "content": "Ok, in that case that explains it. Thanks for the clarification!
", "rawMarkdown": "Ok, in that case that explains it. Thanks for the clarification!", "votes": null }, { "id": "89768", "postDate": "08/19/2015 08:45:35", "content": "@fchollet I have the same issue as you maybe saw in forum. I used the same function you mentioned, and I excluded the min_ANNmuon parts we don't need, but the problem still exists. My scores are consistent +- 0.005 in 10-fold CV, averaged over multiple random shuffles of the data, as you wrote, but my LB vs. CV differences are bigger, actually it is more than 0.01! I guess you find the answer, actually I think you found more than just this "min_ANNmuon issue" in your code, probably I'm struggling too with, because you own a very nice position on LB - congrats for it :) - , so can you, please share with me any thought, idea regarding this issue?
", "rawMarkdown": "fchollet I have the same issue as you maybe saw in forum. I used the same function you mentioned, and I excluded the min_ANNmuon parts we don't need, but the problem still exists. My scores are consistent +- 0.005 in 10-fold CV, averaged over multiple random shuffles of the data, as you wrote, but my LB vs. CV differences are bigger, actually it is more than 0.01! I guess you find the answer, actually I think you found more than just this \"min_ANNmuon issue\" in your code, probably I'm struggling too with, because you own a very nice position on LB - congrats for it :) - , so can you, please share with me any thought, idea regarding this issue?", "votes": null }, { "id": "89772", "postDate": "08/19/2015 10:04:44", "content": "My CVs and LB are way out - using the min cutoff. I get 0.92 CV and 0.98 LB
\n\nEDIT: it would help if I used Weighted ROC and not just ROC
", "rawMarkdown": "My CVs and LB are way out - using the min cutoff. I get 0.92 CV and 0.98 LB\r\n\r\nEDIT: it would help if I used Weighted ROC and not just ROC", "votes": null }, { "id": "89773", "postDate": "08/19/2015 10:50:11", "content": "[quote=Bluefool;89772]
\n\nMy CVs and LB are way out - using the min cutoff. I get 0.92 CV and 0.98 LB
\n\n[/quote]\nThat is the better case. However I've just submitted the \nRF + XGBoost Example (0.982253) publicly "scripted" version, and surprisingly it was 0.979 on LB for me. Maybe I do something essentially wrong about submitting or anywhere else?\nCould you please explain the term "min cutoff" in your case?
", "rawMarkdown": "[quote=Bluefool;89772]\r\n\r\nMy CVs and LB are way out - using the min cutoff. I get 0.92 CV and 0.98 LB\r\n\r\n[/quote]\r\nThat is the better case. However I've just submitted the \r\nRF + XGBoost Example (0.982253) publicly \"scripted\" version, and surprisingly it was 0.979 on LB for me. Maybe I do something essentially wrong about submitting or anywhere else?\r\nCould you please explain the term \"min cutoff\" in your case?", "votes": null }, { "id": "89774", "postDate": "08/19/2015 10:53:35", "content": "Did you submit the original? there are loads of scripts with that name.
", "rawMarkdown": "Did you submit the original? there are loads of scripts with that name.", "votes": null }, { "id": "89775", "postDate": "08/19/2015 10:56:06", "content": "[quote=Bluefool;89774]
\n\nDid you submit the original? there are loads of scripts with that name.
\n\n[/quote]\nI think I checked everything, but I check it double, if this is the original and I did make any modification on it.
", "rawMarkdown": "[quote=Bluefool;89774]\r\n\r\nDid you submit the original? there are loads of scripts with that name. \r\n\r\n[/quote]\r\nI think I checked everything, but I check it double, if this is the original and I did make any modification on it.", "votes": null }, { "id": "89785", "postDate": "08/19/2015 12:26:46", "content": "[quote=Bluefool;89774]
\n\nDid you submit the original? there are loads of scripts with that name.
\n\n[/quote]
\n\nThank you Blue. Well you was right I used a bad clone... My AUC metric says it would be 0.9843, LB says 0.9832, so this is not that (too) much difference. So I am a little step forward, but the original problem is still the same. In the case of my model my AUC metric says it would be 0.993 -+0.0015 and I get consistently 0.979 on LB. This is annoying. I don't know. Probably I haven't enough experience yet. Maybe I have to finetune the AUC metric(!) or what?
", "rawMarkdown": "[quote=Bluefool;89774]\r\n\r\nDid you submit the original? there are loads of scripts with that name. \r\n\r\n[/quote]\r\n\r\nThank you Blue. Well you was right I used a bad clone... My AUC metric says it would be 0.9843, LB says 0.9832, so this is not that (too) much difference. So I am a little step forward, but the original problem is still the same. In the case of my model my AUC metric says it would be 0.993 -+0.0015 and I get consistently 0.979 on LB. This is annoying. I don't know. Probably I haven't enough experience yet. Maybe I have to finetune the AUC metric(!) or what?", "votes": null }, { "id": "89811", "postDate": "08/19/2015 16:13:49", "content": "George, recall that the LB score is computed using a 30% of the test set so it may be distorting your CV score.
", "rawMarkdown": "George, recall that the LB score is computed using a 30% of the test set so it may be distorting your CV score.", "votes": null }, { "id": "89817", "postDate": "08/19/2015 17:00:16", "content": "[quote=Andreu;89811]
\n\nGeorge, recall that the LB score is computed using a 30% of the test set so it may be distorting your CV score.
\n\n[/quote]\nThat's OK Andreu, but this is more than just simple distorting, don't you think? How could one create of a good model without any reliable metrics? If this is true, from this moment we should think on "kaggling" like on poker, not else, not more. I hope, that the fault is somewhere in my mechanism.
\n\nUPDATE: I've just submit another model of mine with the local score 0.99 and get on LB 0.979...
\n\nIt would be glad that someone from the organizers would say a few words too...
", "rawMarkdown": "[quote=Andreu;89811]\r\n\r\nGeorge, recall that the LB score is computed using a 30% of the test set so it may be distorting your CV score.\r\n\r\n[/quote]\r\nThat's OK Andreu, but this is more than just simple distorting, don't you think? How could one create of a good model without any reliable metrics? If this is true, from this moment we should think on \"kaggling\" like on poker, not else, not more. I hope, that the fault is somewhere in my mechanism. \r\n\r\n**UPDATE:** I've just submit another model of mine with the local score **0.99** and get on LB **0.979**...\r\n\r\n**It would be glad that someone from the organizers would say a few words too...**", "votes": null }, { "id": "91320", "postDate": "09/02/2015 11:18:54", "content": "There are many reasons of such difference between your CV and LB:
\n\nThis is the part of almost any competition. Try to investigate this problem yourself!
", "rawMarkdown": "There are many reasons of such difference between your CV and LB: \r\n\r\n1. LB is computed only on 30% of the test data.\r\n2. CV AUC could be calculated in two ways: for each fold separately, then taking an average, or for the whole sample. These two cases will probably provide different results\r\n3. Quality of the model after long tuning will be overestimated and model will be overfitted on the train data (even in case you used cross-validation). \r\n4. and so on...\r\n\r\nThis is the part of almost any competition. Try to investigate this problem yourself!", "votes": null }, { "id": "91365", "postDate": "09/02/2015 16:47:54", "content": "[quote=Tatiana Likhomanenko;91320]
\n\nThere are many reasons of such difference between your CV and LB:
\n\nThis is the part of almost any competition. Try to investigate this problem yourself!
\n\n[/quote]
\n\nThank you and yes, I am working on it myself quite a while hoping some reasonable score.\nThe things you mentioned is well known and they can cause some amount of differences, but not such a huge one and not for all type of CV consistently.
\n\nBut, can you give me an example or some reference, how can I extremely overfit on training data, if I use various CV and the training data is a reliable part of test data?
", "rawMarkdown": "[quote=Tatiana Likhomanenko;91320]\r\n\r\nThere are many reasons of such difference between your CV and LB: \r\n\r\n1. LB is computed only on 30% of the test data.\r\n2. CV AUC could be calculated in two ways: for each fold separately, then taking an average, or for the whole sample. These two cases will probably provide different results\r\n3. Quality of the model after long tuning will be overestimated and model will be overfitted on the train data (even in case you used cross-validation). \r\n4. and so on...\r\n\r\nThis is the part of almost any competition. Try to investigate this problem yourself!\r\n\r\n[/quote]\r\n\r\n\r\nThank you and yes, I am working on it myself quite a while hoping some reasonable score.\r\nThe things you mentioned is well known and they can cause some amount of differences, but not such a huge one and not for all type of CV consistently. \r\n\r\nBut, can you give me an example or some reference, how can I extremely overfit on training data, if I use various CV and the training data is a reliable part of test data?", "votes": null }, { "id": "91370", "postDate": "09/02/2015 17:14:18", "content": "Maybe this link http://blog.mrtz.org/2015/03/09/competition.html will be helpful.
\n\nTo sum up, you don't even need to look at the data to overfit.
", "rawMarkdown": "Maybe this link http://blog.mrtz.org/2015/03/09/competition.html will be helpful. \r\n\r\nTo sum up, you don't even need to look at the data to overfit.", "votes": null }, { "id": "91414", "postDate": "09/02/2015 22:01:25", "content": "[quote=Tatiana Likhomanenko;91370]
\n\nMaybe this link http://blog.mrtz.org/2015/03/09/competition.html will be helpful.
\n\nTo sum up, you don't even need to look at the data to overfit.
\n\n[/quote]
\n\nThank you Tatiana, now I'm diving into it. Note, that Gábor Melis mentioned, that the data provided in the previous CERN competition was too shallow, that's why that competition was a bit lottery. Perhaps the data amount here is also a possible inducing factor of such type of AUC differences.
", "rawMarkdown": "[quote=Tatiana Likhomanenko;91370]\r\n\r\nMaybe this link http://blog.mrtz.org/2015/03/09/competition.html will be helpful. \r\n\r\nTo sum up, you don't even need to look at the data to overfit.\r\n\r\n[/quote]\r\n\r\nThank you Tatiana, now I'm diving into it. Note, that Gábor Melis mentioned, that the data provided in the previous CERN competition was too shallow, that's why that competition was a bit lottery. Perhaps the data amount here is also a possible inducing factor of such type of AUC differences.", "votes": null }, { "id": "91474", "postDate": "09/03/2015 12:17:48", "content": "George, for deeper data wait a bit for the next CERN competition )
", "rawMarkdown": "George, for deeper data wait a bit for the next CERN competition )", "votes": null }, { "id": "91503", "postDate": "09/03/2015 18:24:45", "content": "[quote=Anaderi;91474]
\n\nGeorge, for deeper data wait a bit for the next CERN competition )
\n\n[/quote]
\n\nAnaderi, I'll do so, I have a lot of time, but till then, I try to find the solution Nr#1 at this competition. :)
", "rawMarkdown": "[quote=Anaderi;91474]\r\n\r\nGeorge, for deeper data wait a bit for the next CERN competition )\r\n\r\n[/quote]\r\n\r\nAnaderi, I'll do so, I have a lot of time, but till then, I try to find the solution Nr#1 at this competition. :)", "votes": null }, { "id": "92878", "postDate": "09/18/2015 06:49:55", "content": "[quote=Gilles Louppe;88067]
\n\nWhen computing your CV score, do you include only those events for which min_ANNmuon > 0.4? (This is something I had missed myself)
\n\n[/quote]
\n\nI'm sorry I do not quite understand why we need to exclude the events when min_ANNmuon<0.4. Can you explain a little bit more?
", "rawMarkdown": "[quote=Gilles Louppe;88067]\r\n\r\nWhen computing your CV score, do you include only those events for which min_ANNmuon > 0.4? (This is something I had missed myself)\r\n\r\n[/quote]\r\n\r\nI'm sorry I do not quite understand why we need to exclude the events when min_ANNmuon<0.4. Can you explain a little bit more?", "votes": null }, { "id": "92889", "postDate": "09/18/2015 09:28:34", "content": "[quote=Sylvia Lee;92878]
\n\n[quote=Gilles Louppe;88067]
\n\nWhen computing your CV score, do you include only those events for which min_ANNmuon > 0.4? (This is something I had missed myself)
\n\n[/quote]
\n\nI'm sorry I do not quite understand why we need to exclude the events when min_ANNmuon<0.4. Can you explain a little bit more?
\n\n[/quote]
\n\nYes, this competition is a kind of puzzle. :)\nBut you will find the required details here:\nhttps://www.kaggle.com/c/flavours-of-physics/details/evaluation
", "rawMarkdown": "[quote=Sylvia Lee;92878]\r\n\r\n[quote=Gilles Louppe;88067]\r\n\r\nWhen computing your CV score, do you include only those events for which min_ANNmuon > 0.4? (This is something I had missed myself)\r\n\r\n[/quote]\r\n\r\nI'm sorry I do not quite understand why we need to exclude the events when min_ANNmuon<0.4. Can you explain a little bit more?\r\n\r\n[/quote]\r\n\r\nYes, this competition is a kind of puzzle. :)\r\nBut you will find the required details here:\r\n[https://www.kaggle.com/c/flavours-of-physics/details/evaluation][1]\r\n\r\n\r\n [1]: https://www.kaggle.com/c/flavours-of-physics/details/evaluation", "votes": null }, { "id": "93141", "postDate": "09/22/2015 04:26:08", "content": "I got CV=0.881 while LB=0.972 using RF. I don't understand how this could happen (maybe I did something wrong). How can fix this?
", "rawMarkdown": "I got CV=0.881 while LB=0.972 using RF. I don't understand how this could happen (maybe I did something wrong). How can fix this?", "votes": null }, { "id": "93147", "postDate": "09/22/2015 06:46:55", "content": "Did you use the same evaluation method as described here https://www.kaggle.com/c/flavours-of-physics/details/evaluation? And the implementation is available on the starter kit https://github.com/yandexdataschool/flavours-of-physics-start.
", "rawMarkdown": "Did you use the same evaluation method as described here https://www.kaggle.com/c/flavours-of-physics/details/evaluation? And the implementation is available on the starter kit https://github.com/yandexdataschool/flavours-of-physics-start.", "votes": null }, { "id": "93164", "postDate": "09/22/2015 10:09:09", "content": "[quote=Sylvia Lee;93141]
\n\nI got CV=0.881 while LB=0.972 using RF. I don't understand how this could happen (maybe I did something wrong). How can fix this?
\n\n[/quote]\nAre you using weighted-roc? Your results are indicating a local normal AUC for CV and a weighted AUC for LB
", "rawMarkdown": "[quote=Sylvia Lee;93141]\r\n\r\nI got CV=0.881 while LB=0.972 using RF. I don't understand how this could happen (maybe I did something wrong). How can fix this?\r\n\r\n[/quote]\r\nAre you using weighted-roc? Your results are indicating a local normal AUC for CV and a weighted AUC for LB", "votes": null }, { "id": "93356", "postDate": "09/24/2015 07:07:38", "content": "[quote=Bluefool;93164]
\n\n[quote=Sylvia Lee;93141]
\n\nI got CV=0.881 while LB=0.972 using RF. I don't understand how this could happen (maybe I did something wrong). How can fix this?
\n\n[/quote]\nAre you using weighted-roc? Your results are indicating a local normal AUC for CV and a weighted AUC for LB
\n\n[/quote]
\n\nOh, sorry I did not use weighted-roc. But then I tried that again. ROC score becomes 0.942 which is still much lower than the LB score.
", "rawMarkdown": "[quote=Bluefool;93164]\r\n\r\n[quote=Sylvia Lee;93141]\r\n\r\nI got CV=0.881 while LB=0.972 using RF. I don't understand how this could happen (maybe I did something wrong). How can fix this?\r\n\r\n[/quote]\r\nAre you using weighted-roc? Your results are indicating a local normal AUC for CV and a weighted AUC for LB\r\n\r\n[/quote]\r\n\r\nOh, sorry I did not use weighted-roc. But then I tried that again. ROC score becomes 0.942 which is still much lower than the LB score.", "votes": null }, { "id": "93357", "postDate": "09/24/2015 07:11:22", "content": "the LB is only for cases where min_ANNmuon > 0.4.\nMaybe this is why?
", "rawMarkdown": "the LB is only for cases where min_ANNmuon > 0.4.\r\nMaybe this is why?", "votes": null }, { "id": "93365", "postDate": "09/24/2015 12:14:44", "content": "It seems everybody has a lower CV vs. LB except poor me. :) I have constantly much, much higher CV vs LB both with NN and Random Forest, though the latter has smaller difference. Unfortunately on the current level of my knowledge I was not be able to figure out why (yet:)).
", "rawMarkdown": "It seems everybody has a lower CV vs. LB except poor me. :) I have constantly much, much higher CV vs LB both with NN and Random Forest, though the latter has smaller difference. Unfortunately on the current level of my knowledge I was not be able to figure out why (yet:)).", "votes": null }, { "id": "93407", "postDate": "09/25/2015 09:08:47", "content": "Increasing my cross validation accuracy from 0.86 to ~0.88 with my models resulted always in arround 0.981 score on the panel making me unable to go up on the public LB,did somewone faced same issue?
", "rawMarkdown": "Increasing my cross validation accuracy from 0.86 to ~0.88 with my models resulted always in arround 0.981 score on the panel making me unable to go up on the public LB,did somewone faced same issue?", "votes": null }, { "id": "93410", "postDate": "09/25/2015 10:27:07", "content": "[quote=DaiaAlexandru;93407]
\n\nIncreasing my cross validation accuracy from 0.86 to ~0.88 with my models resulted always in arround 0.981 score on the panel making me unable to go up on the public LB,did somewone faced same issue?
\n\n[/quote]
\n\nAs you can see, I have the same issue. Is 0.981 your locally calculated value or you got on LB?
", "rawMarkdown": "[quote=DaiaAlexandru;93407]\r\n\r\nIncreasing my cross validation accuracy from 0.86 to ~0.88 with my models resulted always in arround 0.981 score on the panel making me unable to go up on the public LB,did somewone faced same issue?\r\n\r\n[/quote]\r\n\r\nAs you can see, I have the same issue. Is 0.981 your locally calculated value or you got on LB?", "votes": null }, { "id": "93414", "postDate": "09/25/2015 11:52:29", "content": "[quote=George Solymosi;93410]
\n\n[quote=DaiaAlexandru;93407]
\n\nIncreasing my cross validation accuracy from 0.86 to ~0.88 with my models resulted always in arround 0.981 score on the panel making me unable to go up on the public LB,did somewone faced same issue?
\n\n[/quote]
\n\nAs you can see, I have the same issue. Is 0.981 your locally calculated value or you got on LB?
\n\n[/quote]\nOn LB
", "rawMarkdown": "[quote=George Solymosi;93410]\r\n\r\n[quote=DaiaAlexandru;93407]\r\n\r\nIncreasing my cross validation accuracy from 0.86 to ~0.88 with my models resulted always in arround 0.981 score on the panel making me unable to go up on the public LB,did somewone faced same issue?\r\n\r\n[/quote]\r\n\r\nAs you can see, I have the same issue. Is 0.981 your locally calculated value or you got on LB?\r\n\r\n[/quote]\r\nOn LB", "votes": null }, { "id": "93416", "postDate": "09/25/2015 11:56:55", "content": "\n\n\nOn LB
\n
and what shows your own calculation of AUC?
", "rawMarkdown": "> On LB\r\n\r\nand what shows your own calculation of AUC?", "votes": null }, { "id": "93420", "postDate": "09/25/2015 12:51:37", "content": "Between 0.86-0.88, my cv score -every value from that range is getting lower score then the actual one on the lb
", "rawMarkdown": "Between 0.86-0.88, my cv score -every value from that range is getting lower score then the actual one on the lb", "votes": null }, { "id": "93424", "postDate": "09/25/2015 13:57:49", "content": "[quote=DaiaAlexandru;93420]
\n\nBetween 0.86-0.88, my cv score -every value from that range is getting lower score then the actual one on the lb
\n\n[/quote]\nGot it, you wrote "accuracy" but then 0.86-0.88 were the AUC scores?
", "rawMarkdown": "[quote=DaiaAlexandru;93420]\r\n\r\nBetween 0.86-0.88, my cv score -every value from that range is getting lower score then the actual one on the lb\r\n\r\n[/quote]\r\nGot it, you wrote \"accuracy\" but then 0.86-0.88 were the AUC scores?", "votes": null }, { "id": "93427", "postDate": "09/25/2015 15:01:25", "content": "Ok , \nSo tryied domething different :\nThe xgb tuning like in the nice script , gives me this ('Raw AUC score:', 0.94349133232843629) for the best params.Hoewer Ia have not found time to continue grid search on different params.\nOn the train eval it is yielding me ('AUC', 0.99277232983594832) representing the roc_auc_truncated.Since I did not made the features from train both to test, I must wait a little until I would submit!
", "rawMarkdown": "Ok , \r\nSo tryied domething different :\r\nThe xgb tuning like in the nice script , gives me this ('Raw AUC score:', 0.94349133232843629) for the best params.Hoewer Ia have not found time to continue grid search on different params.\r\nOn the train eval it is yielding me ('AUC', 0.99277232983594832) representing the roc_auc_truncated.Since I did not made the features from train both to test, I must wait a little until I would submit!", "votes": null }, { "id": "93479", "postDate": "09/26/2015 12:44:14", "content": "[quote=DaiaAlexandru;93427]
\n\nOk , \nSo tryied domething different :\nThe xgb tuning like in the nice script , gives me this ('Raw AUC score:', 0.94349133232843629) for the best params.Hoewer Ia have not found time to continue grid search on different params.\nOn the train eval it is yielding me ('AUC', 0.99277232983594832) representing the roc_auc_truncated.Since I did not made the features from train both to test, I must wait a little until I would submit!
\n\n[/quote]
\n\n@Daia What was your result on LB?
", "rawMarkdown": "[quote=DaiaAlexandru;93427]\r\n\r\nOk , \r\nSo tryied domething different :\r\nThe xgb tuning like in the nice script , gives me this ('Raw AUC score:', 0.94349133232843629) for the best params.Hoewer Ia have not found time to continue grid search on different params.\r\nOn the train eval it is yielding me ('AUC', 0.99277232983594832) representing the roc_auc_truncated.Since I did not made the features from train both to test, I must wait a little until I would submit!\r\n\r\n[/quote]\r\n\r\n@Daia What was your result on LB?", "votes": null }, { "id": "93481", "postDate": "09/26/2015 13:01:01", "content": "It failed to pass the test :) .
", "rawMarkdown": "It failed to pass the test :) .", "votes": null } ], "comments": [ { "id": 88067, "author_name": "glouppe", "author_url": "", "post_date": "08/03/2015 08:34:43", "content": "When computing your CV score, do you include only those events for which min_ANNmuon > 0.4? (This is something I had missed myself)
", "votes": null, "replies": [] }, { "id": 88068, "author_name": "guidotapia", "author_url": "", "post_date": "08/03/2015 08:37:07", "content": "You have to remember to score only rows with min_ANNmuon > 0.4. Eval code from the starter kit:
\n\ntrain_eval = train[train['min_ANNmuon'] > 0.4]\ntrain_probs = baseline.predict_proba(train_eval[variables])[:, 1]\nAUC = evaluation.roc_auc_truncated(train_eval['signal'], train_probs)\nSo train on all rows but only evaluate rows with min_ANNmuon > .4
", "votes": null, "replies": [] }, { "id": 88069, "author_name": "fchollet", "author_url": "", "post_date": "08/03/2015 08:38:58", "content": "[quote=Gilles Louppe;88067]
\n\nWhen computing your CV score, do you include only those events for which min_ANNmuon > 0.4? (This is something I had missed myself)
\n\n[/quote]
\n\nI use the score returned by the function provided in the starter kit:
\n\nroc_auc_truncated(labels, predictions, tpr_thresholds=(0.2, 0.4, 0.6, 0.8), roc_weights=(4, 3, 2, 1, 0))\nI apply this function to all data points, no filtering. Is this incorrect?
", "votes": null, "replies": [] }, { "id": 88070, "author_name": "fchollet", "author_url": "", "post_date": "08/03/2015 08:45:35", "content": "Ok, in that case that explains it. Thanks for the clarification!
", "votes": null, "replies": [] }, { "id": 89768, "author_name": "", "author_url": "", "post_date": "08/19/2015 08:45:35", "content": "@fchollet I have the same issue as you maybe saw in forum. I used the same function you mentioned, and I excluded the min_ANNmuon parts we don't need, but the problem still exists. My scores are consistent +- 0.005 in 10-fold CV, averaged over multiple random shuffles of the data, as you wrote, but my LB vs. CV differences are bigger, actually it is more than 0.01! I guess you find the answer, actually I think you found more than just this "min_ANNmuon issue" in your code, probably I'm struggling too with, because you own a very nice position on LB - congrats for it :) - , so can you, please share with me any thought, idea regarding this issue?
", "votes": null, "replies": [] }, { "id": 89772, "author_name": "domcastro", "author_url": "", "post_date": "08/19/2015 10:04:44", "content": "My CVs and LB are way out - using the min cutoff. I get 0.92 CV and 0.98 LB
\n\nEDIT: it would help if I used Weighted ROC and not just ROC
", "votes": null, "replies": [] }, { "id": 89773, "author_name": "", "author_url": "", "post_date": "08/19/2015 10:50:11", "content": "[quote=Bluefool;89772]
\n\nMy CVs and LB are way out - using the min cutoff. I get 0.92 CV and 0.98 LB
\n\n[/quote]\nThat is the better case. However I've just submitted the \nRF + XGBoost Example (0.982253) publicly "scripted" version, and surprisingly it was 0.979 on LB for me. Maybe I do something essentially wrong about submitting or anywhere else?\nCould you please explain the term "min cutoff" in your case?
", "votes": null, "replies": [] }, { "id": 89774, "author_name": "domcastro", "author_url": "", "post_date": "08/19/2015 10:53:35", "content": "Did you submit the original? there are loads of scripts with that name.
", "votes": null, "replies": [] }, { "id": 89775, "author_name": "", "author_url": "", "post_date": "08/19/2015 10:56:06", "content": "[quote=Bluefool;89774]
\n\nDid you submit the original? there are loads of scripts with that name.
\n\n[/quote]\nI think I checked everything, but I check it double, if this is the original and I did make any modification on it.
", "votes": null, "replies": [] }, { "id": 89785, "author_name": "", "author_url": "", "post_date": "08/19/2015 12:26:46", "content": "[quote=Bluefool;89774]
\n\nDid you submit the original? there are loads of scripts with that name.
\n\n[/quote]
\n\nThank you Blue. Well you was right I used a bad clone... My AUC metric says it would be 0.9843, LB says 0.9832, so this is not that (too) much difference. So I am a little step forward, but the original problem is still the same. In the case of my model my AUC metric says it would be 0.993 -+0.0015 and I get consistently 0.979 on LB. This is annoying. I don't know. Probably I haven't enough experience yet. Maybe I have to finetune the AUC metric(!) or what?
", "votes": null, "replies": [] }, { "id": 89811, "author_name": "andreusancho", "author_url": "", "post_date": "08/19/2015 16:13:49", "content": "George, recall that the LB score is computed using a 30% of the test set so it may be distorting your CV score.
", "votes": null, "replies": [] }, { "id": 89817, "author_name": "", "author_url": "", "post_date": "08/19/2015 17:00:16", "content": "[quote=Andreu;89811]
\n\nGeorge, recall that the LB score is computed using a 30% of the test set so it may be distorting your CV score.
\n\n[/quote]\nThat's OK Andreu, but this is more than just simple distorting, don't you think? How could one create of a good model without any reliable metrics? If this is true, from this moment we should think on "kaggling" like on poker, not else, not more. I hope, that the fault is somewhere in my mechanism.
\n\nUPDATE: I've just submit another model of mine with the local score 0.99 and get on LB 0.979...
\n\nIt would be glad that someone from the organizers would say a few words too...
", "votes": null, "replies": [] }, { "id": 91320, "author_name": "tataantares", "author_url": "", "post_date": "09/02/2015 11:18:54", "content": "There are many reasons of such difference between your CV and LB:
\n\nThis is the part of almost any competition. Try to investigate this problem yourself!
", "votes": null, "replies": [] }, { "id": 91365, "author_name": "", "author_url": "", "post_date": "09/02/2015 16:47:54", "content": "[quote=Tatiana Likhomanenko;91320]
\n\nThere are many reasons of such difference between your CV and LB:
\n\nThis is the part of almost any competition. Try to investigate this problem yourself!
\n\n[/quote]
\n\nThank you and yes, I am working on it myself quite a while hoping some reasonable score.\nThe things you mentioned is well known and they can cause some amount of differences, but not such a huge one and not for all type of CV consistently.
\n\nBut, can you give me an example or some reference, how can I extremely overfit on training data, if I use various CV and the training data is a reliable part of test data?
", "votes": null, "replies": [] }, { "id": 91370, "author_name": "tataantares", "author_url": "", "post_date": "09/02/2015 17:14:18", "content": "Maybe this link http://blog.mrtz.org/2015/03/09/competition.html will be helpful.
\n\nTo sum up, you don't even need to look at the data to overfit.
", "votes": null, "replies": [] }, { "id": 91414, "author_name": "", "author_url": "", "post_date": "09/02/2015 22:01:25", "content": "[quote=Tatiana Likhomanenko;91370]
\n\nMaybe this link http://blog.mrtz.org/2015/03/09/competition.html will be helpful.
\n\nTo sum up, you don't even need to look at the data to overfit.
\n\n[/quote]
\n\nThank you Tatiana, now I'm diving into it. Note, that Gábor Melis mentioned, that the data provided in the previous CERN competition was too shallow, that's why that competition was a bit lottery. Perhaps the data amount here is also a possible inducing factor of such type of AUC differences.
", "votes": null, "replies": [] }, { "id": 91474, "author_name": "anaderi", "author_url": "", "post_date": "09/03/2015 12:17:48", "content": "George, for deeper data wait a bit for the next CERN competition )
", "votes": null, "replies": [] }, { "id": 91503, "author_name": "", "author_url": "", "post_date": "09/03/2015 18:24:45", "content": "[quote=Anaderi;91474]
\n\nGeorge, for deeper data wait a bit for the next CERN competition )
\n\n[/quote]
\n\nAnaderi, I'll do so, I have a lot of time, but till then, I try to find the solution Nr#1 at this competition. :)
", "votes": null, "replies": [] }, { "id": 92878, "author_name": "sylvialee", "author_url": "", "post_date": "09/18/2015 06:49:55", "content": "[quote=Gilles Louppe;88067]
\n\nWhen computing your CV score, do you include only those events for which min_ANNmuon > 0.4? (This is something I had missed myself)
\n\n[/quote]
\n\nI'm sorry I do not quite understand why we need to exclude the events when min_ANNmuon<0.4. Can you explain a little bit more?
", "votes": null, "replies": [] }, { "id": 92889, "author_name": "", "author_url": "", "post_date": "09/18/2015 09:28:34", "content": "[quote=Sylvia Lee;92878]
\n\n[quote=Gilles Louppe;88067]
\n\nWhen computing your CV score, do you include only those events for which min_ANNmuon > 0.4? (This is something I had missed myself)
\n\n[/quote]
\n\nI'm sorry I do not quite understand why we need to exclude the events when min_ANNmuon<0.4. Can you explain a little bit more?
\n\n[/quote]
\n\nYes, this competition is a kind of puzzle. :)\nBut you will find the required details here:\nhttps://www.kaggle.com/c/flavours-of-physics/details/evaluation
", "votes": null, "replies": [] }, { "id": 93141, "author_name": "sylvialee", "author_url": "", "post_date": "09/22/2015 04:26:08", "content": "I got CV=0.881 while LB=0.972 using RF. I don't understand how this could happen (maybe I did something wrong). How can fix this?
", "votes": null, "replies": [] }, { "id": 93147, "author_name": "achm6174", "author_url": "", "post_date": "09/22/2015 06:46:55", "content": "Did you use the same evaluation method as described here https://www.kaggle.com/c/flavours-of-physics/details/evaluation? And the implementation is available on the starter kit https://github.com/yandexdataschool/flavours-of-physics-start.
", "votes": null, "replies": [] }, { "id": 93164, "author_name": "domcastro", "author_url": "", "post_date": "09/22/2015 10:09:09", "content": "[quote=Sylvia Lee;93141]
\n\nI got CV=0.881 while LB=0.972 using RF. I don't understand how this could happen (maybe I did something wrong). How can fix this?
\n\n[/quote]\nAre you using weighted-roc? Your results are indicating a local normal AUC for CV and a weighted AUC for LB
", "votes": null, "replies": [] }, { "id": 93356, "author_name": "sylvialee", "author_url": "", "post_date": "09/24/2015 07:07:38", "content": "[quote=Bluefool;93164]
\n\n[quote=Sylvia Lee;93141]
\n\nI got CV=0.881 while LB=0.972 using RF. I don't understand how this could happen (maybe I did something wrong). How can fix this?
\n\n[/quote]\nAre you using weighted-roc? Your results are indicating a local normal AUC for CV and a weighted AUC for LB
\n\n[/quote]
\n\nOh, sorry I did not use weighted-roc. But then I tried that again. ROC score becomes 0.942 which is still much lower than the LB score.
", "votes": null, "replies": [] }, { "id": 93357, "author_name": "domcastro", "author_url": "", "post_date": "09/24/2015 07:11:22", "content": "the LB is only for cases where min_ANNmuon > 0.4.\nMaybe this is why?
", "votes": null, "replies": [] }, { "id": 93365, "author_name": "", "author_url": "", "post_date": "09/24/2015 12:14:44", "content": "It seems everybody has a lower CV vs. LB except poor me. :) I have constantly much, much higher CV vs LB both with NN and Random Forest, though the latter has smaller difference. Unfortunately on the current level of my knowledge I was not be able to figure out why (yet:)).
", "votes": null, "replies": [] }, { "id": 93407, "author_name": "", "author_url": "", "post_date": "09/25/2015 09:08:47", "content": "Increasing my cross validation accuracy from 0.86 to ~0.88 with my models resulted always in arround 0.981 score on the panel making me unable to go up on the public LB,did somewone faced same issue?
", "votes": null, "replies": [] }, { "id": 93410, "author_name": "", "author_url": "", "post_date": "09/25/2015 10:27:07", "content": "[quote=DaiaAlexandru;93407]
\n\nIncreasing my cross validation accuracy from 0.86 to ~0.88 with my models resulted always in arround 0.981 score on the panel making me unable to go up on the public LB,did somewone faced same issue?
\n\n[/quote]
\n\nAs you can see, I have the same issue. Is 0.981 your locally calculated value or you got on LB?
", "votes": null, "replies": [] }, { "id": 93414, "author_name": "", "author_url": "", "post_date": "09/25/2015 11:52:29", "content": "[quote=George Solymosi;93410]
\n\n[quote=DaiaAlexandru;93407]
\n\nIncreasing my cross validation accuracy from 0.86 to ~0.88 with my models resulted always in arround 0.981 score on the panel making me unable to go up on the public LB,did somewone faced same issue?
\n\n[/quote]
\n\nAs you can see, I have the same issue. Is 0.981 your locally calculated value or you got on LB?
\n\n[/quote]\nOn LB
", "votes": null, "replies": [] }, { "id": 93416, "author_name": "", "author_url": "", "post_date": "09/25/2015 11:56:55", "content": "\n\n\nOn LB
\n
and what shows your own calculation of AUC?
", "votes": null, "replies": [] }, { "id": 93420, "author_name": "", "author_url": "", "post_date": "09/25/2015 12:51:37", "content": "Between 0.86-0.88, my cv score -every value from that range is getting lower score then the actual one on the lb
", "votes": null, "replies": [] }, { "id": 93424, "author_name": "", "author_url": "", "post_date": "09/25/2015 13:57:49", "content": "[quote=DaiaAlexandru;93420]
\n\nBetween 0.86-0.88, my cv score -every value from that range is getting lower score then the actual one on the lb
\n\n[/quote]\nGot it, you wrote "accuracy" but then 0.86-0.88 were the AUC scores?
", "votes": null, "replies": [] }, { "id": 93427, "author_name": "", "author_url": "", "post_date": "09/25/2015 15:01:25", "content": "Ok , \nSo tryied domething different :\nThe xgb tuning like in the nice script , gives me this ('Raw AUC score:', 0.94349133232843629) for the best params.Hoewer Ia have not found time to continue grid search on different params.\nOn the train eval it is yielding me ('AUC', 0.99277232983594832) representing the roc_auc_truncated.Since I did not made the features from train both to test, I must wait a little until I would submit!
", "votes": null, "replies": [] }, { "id": 93479, "author_name": "", "author_url": "", "post_date": "09/26/2015 12:44:14", "content": "[quote=DaiaAlexandru;93427]
\n\nOk , \nSo tryied domething different :\nThe xgb tuning like in the nice script , gives me this ('Raw AUC score:', 0.94349133232843629) for the best params.Hoewer Ia have not found time to continue grid search on different params.\nOn the train eval it is yielding me ('AUC', 0.99277232983594832) representing the roc_auc_truncated.Since I did not made the features from train both to test, I must wait a little until I would submit!
\n\n[/quote]
\n\n@Daia What was your result on LB?
", "votes": null, "replies": [] }, { "id": 93481, "author_name": "", "author_url": "", "post_date": "09/26/2015 13:01:01", "content": "It failed to pass the test :) .
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "88066": "My cross-validation scores are considerably lower than my public LB scores, for a range of different models, and I'm wondering why. Is anyone else experiencing this? \r\n\r\nI'm seeing CV scores in the range of 0.990 +- 0.001 in 5-fold CV (averaged over multiple random shuffles of the data). These models consistently score 0.9855 +- 0.005 on the public LB. The difference is huge and is far outside the variation range you see in CV folds.\r\n\r\nHaving triple-checked my CV process, I can entirely rule out any leak or error on my side. There are only two possible hypotheses to explain this oddity:\r\n\r\n- the public leaderboard dataset is ridiculously small, on the order of 10k samples or less. This hypothesis is supported by the fact that LB scores have a much higher variance than CV scores. If this is the case, then we should only rely on CV and be careful of avoiding LB overfitting.\r\n\r\n- the training samples are not independent (eg. the training set contains samples that are quasi-duplicates of each other and that are not found in the test set), which means that any model trained on the subset of this training data will perform much higher on a different subset than it would on the test data (in other words, CV won't work accurately with the training data). If that is the case, we can't rely on CV anymore. Big potential issue.\r\n\r\nThis second hypothesis would not make sense with real world particle data, but it could happen with simulated data. \r\n\r\nThoughts?", "88067": "When computing your CV score, do you include only those events for which min_ANNmuon > 0.4? (This is something I had missed myself)", "88068": "You have to remember to score only rows with min_ANNmuon > 0.4. Eval code from the starter kit:\r\n\r\n train_eval = train[train['min_ANNmuon'] > 0.4]\r\n train_probs = baseline.predict_proba(train_eval[variables])[:, 1]\r\n AUC = evaluation.roc_auc_truncated(train_eval['signal'], train_probs)\r\n\r\n\r\nSo train on all rows but only evaluate rows with min_ANNmuon > .4", "88069": "[quote=Gilles Louppe;88067]\r\n\r\nWhen computing your CV score, do you include only those events for which min_ANNmuon > 0.4? (This is something I had missed myself)\r\n\r\n[/quote]\r\n\r\nI use the score returned by the function provided in the starter kit:\r\n\r\n roc_auc_truncated(labels, predictions, tpr_thresholds=(0.2, 0.4, 0.6, 0.8), roc_weights=(4, 3, 2, 1, 0))\r\n\r\nI apply this function to all data points, no filtering. Is this incorrect?", "88070": "Ok, in that case that explains it. Thanks for the clarification!", "89768": "fchollet I have the same issue as you maybe saw in forum. I used the same function you mentioned, and I excluded the min_ANNmuon parts we don't need, but the problem still exists. My scores are consistent +- 0.005 in 10-fold CV, averaged over multiple random shuffles of the data, as you wrote, but my LB vs. CV differences are bigger, actually it is more than 0.01! I guess you find the answer, actually I think you found more than just this \"min_ANNmuon issue\" in your code, probably I'm struggling too with, because you own a very nice position on LB - congrats for it :) - , so can you, please share with me any thought, idea regarding this issue?", "89772": "My CVs and LB are way out - using the min cutoff. I get 0.92 CV and 0.98 LB\r\n\r\nEDIT: it would help if I used Weighted ROC and not just ROC", "89773": "[quote=Bluefool;89772]\r\n\r\nMy CVs and LB are way out - using the min cutoff. I get 0.92 CV and 0.98 LB\r\n\r\n[/quote]\r\nThat is the better case. However I've just submitted the \r\nRF + XGBoost Example (0.982253) publicly \"scripted\" version, and surprisingly it was 0.979 on LB for me. Maybe I do something essentially wrong about submitting or anywhere else?\r\nCould you please explain the term \"min cutoff\" in your case?", "89774": "Did you submit the original? there are loads of scripts with that name.", "89775": "[quote=Bluefool;89774]\r\n\r\nDid you submit the original? there are loads of scripts with that name. \r\n\r\n[/quote]\r\nI think I checked everything, but I check it double, if this is the original and I did make any modification on it.", "89785": "[quote=Bluefool;89774]\r\n\r\nDid you submit the original? there are loads of scripts with that name. \r\n\r\n[/quote]\r\n\r\nThank you Blue. Well you was right I used a bad clone... My AUC metric says it would be 0.9843, LB says 0.9832, so this is not that (too) much difference. So I am a little step forward, but the original problem is still the same. In the case of my model my AUC metric says it would be 0.993 -+0.0015 and I get consistently 0.979 on LB. This is annoying. I don't know. Probably I haven't enough experience yet. Maybe I have to finetune the AUC metric(!) or what?", "89811": "George, recall that the LB score is computed using a 30% of the test set so it may be distorting your CV score.", "89817": "[quote=Andreu;89811]\r\n\r\nGeorge, recall that the LB score is computed using a 30% of the test set so it may be distorting your CV score.\r\n\r\n[/quote]\r\nThat's OK Andreu, but this is more than just simple distorting, don't you think? How could one create of a good model without any reliable metrics? If this is true, from this moment we should think on \"kaggling\" like on poker, not else, not more. I hope, that the fault is somewhere in my mechanism. \r\n\r\n**UPDATE:** I've just submit another model of mine with the local score **0.99** and get on LB **0.979**...\r\n\r\n**It would be glad that someone from the organizers would say a few words too...**", "91320": "There are many reasons of such difference between your CV and LB: \r\n\r\n1. LB is computed only on 30% of the test data.\r\n2. CV AUC could be calculated in two ways: for each fold separately, then taking an average, or for the whole sample. These two cases will probably provide different results\r\n3. Quality of the model after long tuning will be overestimated and model will be overfitted on the train data (even in case you used cross-validation). \r\n4. and so on...\r\n\r\nThis is the part of almost any competition. Try to investigate this problem yourself!", "91365": "[quote=Tatiana Likhomanenko;91320]\r\n\r\nThere are many reasons of such difference between your CV and LB: \r\n\r\n1. LB is computed only on 30% of the test data.\r\n2. CV AUC could be calculated in two ways: for each fold separately, then taking an average, or for the whole sample. These two cases will probably provide different results\r\n3. Quality of the model after long tuning will be overestimated and model will be overfitted on the train data (even in case you used cross-validation). \r\n4. and so on...\r\n\r\nThis is the part of almost any competition. Try to investigate this problem yourself!\r\n\r\n[/quote]\r\n\r\n\r\nThank you and yes, I am working on it myself quite a while hoping some reasonable score.\r\nThe things you mentioned is well known and they can cause some amount of differences, but not such a huge one and not for all type of CV consistently. \r\n\r\nBut, can you give me an example or some reference, how can I extremely overfit on training data, if I use various CV and the training data is a reliable part of test data?", "91370": "Maybe this link http://blog.mrtz.org/2015/03/09/competition.html will be helpful. \r\n\r\nTo sum up, you don't even need to look at the data to overfit.", "91414": "[quote=Tatiana Likhomanenko;91370]\r\n\r\nMaybe this link http://blog.mrtz.org/2015/03/09/competition.html will be helpful. \r\n\r\nTo sum up, you don't even need to look at the data to overfit.\r\n\r\n[/quote]\r\n\r\nThank you Tatiana, now I'm diving into it. Note, that Gábor Melis mentioned, that the data provided in the previous CERN competition was too shallow, that's why that competition was a bit lottery. Perhaps the data amount here is also a possible inducing factor of such type of AUC differences.", "91474": "George, for deeper data wait a bit for the next CERN competition )", "91503": "[quote=Anaderi;91474]\r\n\r\nGeorge, for deeper data wait a bit for the next CERN competition )\r\n\r\n[/quote]\r\n\r\nAnaderi, I'll do so, I have a lot of time, but till then, I try to find the solution Nr#1 at this competition. :)", "92878": "[quote=Gilles Louppe;88067]\r\n\r\nWhen computing your CV score, do you include only those events for which min_ANNmuon > 0.4? (This is something I had missed myself)\r\n\r\n[/quote]\r\n\r\nI'm sorry I do not quite understand why we need to exclude the events when min_ANNmuon<0.4. Can you explain a little bit more?", "92889": "[quote=Sylvia Lee;92878]\r\n\r\n[quote=Gilles Louppe;88067]\r\n\r\nWhen computing your CV score, do you include only those events for which min_ANNmuon > 0.4? (This is something I had missed myself)\r\n\r\n[/quote]\r\n\r\nI'm sorry I do not quite understand why we need to exclude the events when min_ANNmuon<0.4. Can you explain a little bit more?\r\n\r\n[/quote]\r\n\r\nYes, this competition is a kind of puzzle. :)\r\nBut you will find the required details here:\r\n[https://www.kaggle.com/c/flavours-of-physics/details/evaluation][1]\r\n\r\n\r\n [1]: https://www.kaggle.com/c/flavours-of-physics/details/evaluation", "93141": "I got CV=0.881 while LB=0.972 using RF. I don't understand how this could happen (maybe I did something wrong). How can fix this?", "93147": "Did you use the same evaluation method as described here https://www.kaggle.com/c/flavours-of-physics/details/evaluation? And the implementation is available on the starter kit https://github.com/yandexdataschool/flavours-of-physics-start.", "93164": "[quote=Sylvia Lee;93141]\r\n\r\nI got CV=0.881 while LB=0.972 using RF. I don't understand how this could happen (maybe I did something wrong). How can fix this?\r\n\r\n[/quote]\r\nAre you using weighted-roc? Your results are indicating a local normal AUC for CV and a weighted AUC for LB", "93356": "[quote=Bluefool;93164]\r\n\r\n[quote=Sylvia Lee;93141]\r\n\r\nI got CV=0.881 while LB=0.972 using RF. I don't understand how this could happen (maybe I did something wrong). How can fix this?\r\n\r\n[/quote]\r\nAre you using weighted-roc? Your results are indicating a local normal AUC for CV and a weighted AUC for LB\r\n\r\n[/quote]\r\n\r\nOh, sorry I did not use weighted-roc. But then I tried that again. ROC score becomes 0.942 which is still much lower than the LB score.", "93357": "the LB is only for cases where min_ANNmuon > 0.4.\r\nMaybe this is why?", "93365": "It seems everybody has a lower CV vs. LB except poor me. :) I have constantly much, much higher CV vs LB both with NN and Random Forest, though the latter has smaller difference. Unfortunately on the current level of my knowledge I was not be able to figure out why (yet:)).", "93407": "Increasing my cross validation accuracy from 0.86 to ~0.88 with my models resulted always in arround 0.981 score on the panel making me unable to go up on the public LB,did somewone faced same issue?", "93410": "[quote=DaiaAlexandru;93407]\r\n\r\nIncreasing my cross validation accuracy from 0.86 to ~0.88 with my models resulted always in arround 0.981 score on the panel making me unable to go up on the public LB,did somewone faced same issue?\r\n\r\n[/quote]\r\n\r\nAs you can see, I have the same issue. Is 0.981 your locally calculated value or you got on LB?", "93414": "[quote=George Solymosi;93410]\r\n\r\n[quote=DaiaAlexandru;93407]\r\n\r\nIncreasing my cross validation accuracy from 0.86 to ~0.88 with my models resulted always in arround 0.981 score on the panel making me unable to go up on the public LB,did somewone faced same issue?\r\n\r\n[/quote]\r\n\r\nAs you can see, I have the same issue. Is 0.981 your locally calculated value or you got on LB?\r\n\r\n[/quote]\r\nOn LB", "93416": "> On LB\r\n\r\nand what shows your own calculation of AUC?", "93420": "Between 0.86-0.88, my cv score -every value from that range is getting lower score then the actual one on the lb", "93424": "[quote=DaiaAlexandru;93420]\r\n\r\nBetween 0.86-0.88, my cv score -every value from that range is getting lower score then the actual one on the lb\r\n\r\n[/quote]\r\nGot it, you wrote \"accuracy\" but then 0.86-0.88 were the AUC scores?", "93427": "Ok , \r\nSo tryied domething different :\r\nThe xgb tuning like in the nice script , gives me this ('Raw AUC score:', 0.94349133232843629) for the best params.Hoewer Ia have not found time to continue grid search on different params.\r\nOn the train eval it is yielding me ('AUC', 0.99277232983594832) representing the roc_auc_truncated.Since I did not made the features from train both to test, I must wait a little until I would submit!", "93479": "[quote=DaiaAlexandru;93427]\r\n\r\nOk , \r\nSo tryied domething different :\r\nThe xgb tuning like in the nice script , gives me this ('Raw AUC score:', 0.94349133232843629) for the best params.Hoewer Ia have not found time to continue grid search on different params.\r\nOn the train eval it is yielding me ('AUC', 0.99277232983594832) representing the roc_auc_truncated.Since I did not made the features from train both to test, I must wait a little until I would submit!\r\n\r\n[/quote]\r\n\r\n@Daia What was your result on LB?", "93481": "It failed to pass the test :) ." }, "source": "meta" }