I tested the "scripted" Rf + Xgboost.\nThen I made my own solution.\nI proceeded according all the required metrics:\nCvM done.\nKS done.\nAUC done.\nI've got a quite good AUC score, so then I submitted it, but surprisingly I've got 0.979 on the public LB!\nI checked again and again: \nMade the RF + XGB according the metrics I used before: \n0.983-0.984 depending on the validation set. \nChecked again my model, and yes with the same higher AUC score. \nThen back to submission again:
\n\n\n\n\nAUC: 0.979 "no improvement etc...."
\n
I checked twice (rather much more): I applied the \n min_ANNmuon > 0.4 filter, \nI used several different validation set.
\n\nWhere could be the problem yet?\nWhere did I make a mistake?\nThanks in advance.
\n\nUPDATE: When I made two other submission with other two different model configuration, I got the same approx. 0.979 AUC value.
", "messages": [ { "id": "89182", "postDate": "08/12/2015 13:10:43", "content": "I tested the "scripted" Rf + Xgboost.\nThen I made my own solution.\nI proceeded according all the required metrics:\nCvM done.\nKS done.\nAUC done.\nI've got a quite good AUC score, so then I submitted it, but surprisingly I've got 0.979 on the public LB!\nI checked again and again: \nMade the RF + XGB according the metrics I used before: \n0.983-0.984 depending on the validation set. \nChecked again my model, and yes with the same higher AUC score. \nThen back to submission again:
\n\n\n\n\nAUC: 0.979 "no improvement etc...."
\n
I checked twice (rather much more): I applied the \n min_ANNmuon > 0.4 filter, \nI used several different validation set.
\n\nWhere could be the problem yet?\nWhere did I make a mistake?\nThanks in advance.
\n\nUPDATE: When I made two other submission with other two different model configuration, I got the same approx. 0.979 AUC value.
", "rawMarkdown": "I tested the \"scripted\" Rf + Xgboost.\r\nThen I made my own solution.\r\nI proceeded according all the required metrics:\r\nCvM done.\r\nKS done.\r\nAUC done.\r\nI've got a quite good AUC score, so then I submitted it, but surprisingly I've got **0.979** on the public LB!\r\nI checked again and again: \r\nMade the RF + XGB according the metrics I used before: \r\n**0.983-0.984** depending on the validation set. \r\nChecked again my model, and yes with the same higher AUC score. \r\nThen back to submission again: \r\n\r\n> **AUC: 0.979 \"no improvement etc....\"**\r\n\r\nI checked twice (rather much more): I applied the \r\n **min_ANNmuon > 0.4 filter**, \r\nI used several different validation set.\r\n\r\nWhere could be the problem yet?\r\nWhere did I make a mistake?\r\nThanks in advance.\r\n\r\n**UPDATE:** When I made two other submission with other two different model configuration, I got the same approx. **0.979** AUC value.", "votes": null }, { "id": "89185", "postDate": "08/12/2015 14:04:13", "content": "Consider performing another type of test validation (i.e., Cross-validation), like a Stratified K-Fold Cross-validation (see scikit page on that) when testing locally. Typically, Stratified 10-Fold CV works well for most of the cases, although depending on the data other types may be better (4-Fold CV, 5x2 CV, and so on).
", "rawMarkdown": "Consider performing another type of test validation (i.e., Cross-validation), like a Stratified K-Fold Cross-validation ([see scikit page on that][1]) when testing locally. Typically, Stratified 10-Fold CV works well for most of the cases, although depending on the data other types may be better (4-Fold CV, 5x2 CV, and so on).\r\n\r\n\r\n [1]: http://scikit-learn.org/stable/modules/cross_validation.html", "votes": null }, { "id": "89200", "postDate": "08/12/2015 17:34:13", "content": "@Andreu Thank you for your kind answer, I gave you a vote up for it, however do you really think that such an amount of difference (almost 0.01 AUC (!) could be occured by a not appropriate validation test selection? Furthermore, maybe my bad, I didn't mentioned, that I used stratified training/validation sets using Sklearn methods.
", "rawMarkdown": "Andreu Thank you for your kind answer, I gave you a vote up for it, however do you really think that such an amount of difference (almost 0.01 AUC (!) could be occured by a not appropriate validation test selection? Furthermore, maybe my bad, I didn't mentioned, that I used stratified training/validation sets using Sklearn methods.", "votes": null }, { "id": "89280", "postDate": "08/13/2015 07:55:13", "content": "Well, it may happen that the partitions used in your train/test sets are not the best ones for the learning algorithm used; Cross-validation is not perfect. Try other types, such as 6 CV, 4 CV, or 5x2 CV until you find the most similar to the public leader board. Another common approach is to repeat the Stratified K-Fold CV several times (10, 20, 30 times...) with a different random seed and average the results to avoid bias (i.e., "luck"). Also, relatively large differences may imply two possible cases: (1) that your algorithm is falling into overfitting and CV is not able of identifying that, and/or (2) that the public leader board score is not realistic (that happened in the previous LHC challenge) and we will see large differences in the private one.
\n\nI think that the second option is the most probable.
", "rawMarkdown": "Well, it may happen that the partitions used in your train/test sets are not the best ones for the learning algorithm used; Cross-validation is not perfect. Try other types, such as 6 CV, 4 CV, or 5x2 CV until you find the most similar to the public leader board. Another common approach is to repeat the Stratified K-Fold CV several times (10, 20, 30 times...) with a different random seed and average the results to avoid bias (i.e., \"luck\"). Also, relatively large differences may imply two possible cases: (1) that your algorithm is falling into overfitting and CV is not able of identifying that, and/or (2) that the public leader board score is not *realistic* (that happened in the previous LHC challenge) and we will see large differences in the private one.\r\n\r\nI think that the second option is the most probable.", "votes": null }, { "id": "89535", "postDate": "08/16/2015 22:40:15", "content": "@Andreu Thank you again! Well I will take your advice and I will test more CV approaches. Anyway you will see, if I will succeed it or will not soon. :)
\n\n\n\n\n(1) that your algorithm is falling into overfitting and CV is not\n able of identifying that:
\n
This is a bit unclear for me which cases can generate such a situation, but I am working on it.
\n\n\n\n\n(2) that the public leader board score is not realistic (that happened\n in the previous LHC challenge) and we will see large differences in\n the private one. I think that the second option is the most probable.
\n
It's a bit worrying to know about such a possibility.
", "rawMarkdown": "Andreu Thank you again! Well I will take your advice and I will test more CV approaches. Anyway you will see, if I will succeed it or will not soon. :)\r\n\r\n> (1) that your algorithm is falling into overfitting and CV is not\r\n> able of identifying that:\r\n\r\nThis is a bit unclear for me which cases can generate such a situation, but I am working on it.\r\n\r\n> (2) that the public leader board score is not realistic (that happened\r\n> in the previous LHC challenge) and we will see large differences in\r\n> the private one. I think that the second option is the most probable.\r\n\r\nIt's a bit worrying to know about such a possibility.", "votes": null }, { "id": "89738", "postDate": "08/19/2015 00:32:54", "content": "Update:
\n\nI tried 10 fold, 4 fold CV and more and the average AUC value resulted by any method is roughly the same within -+ 0.15 %. \nAt the same time the difference compared to public leader board is about -3%, which means a difference like 0.979 vs. 0.989, which is huge.
\n\nI am trying to figure out what is exactly happening, but it is quite surrealistic for me at the moment.\nAny help, thought, idea, anything would be highly appreciated, thank you.
", "rawMarkdown": "**Update:** \r\n\r\nI tried 10 fold, 4 fold CV and more and the average AUC value resulted by any method is roughly the same within **-+ 0.15 %**. \r\nAt the same time the difference compared to public leader board is about **-3%**, which means a difference like 0.979 vs. 0.989, which is huge.\r\n\r\nI am trying to figure out what is exactly happening, but it is quite surrealistic for me at the moment.\r\nAny help, thought, idea, anything would be highly appreciated, thank you.", "votes": null } ], "comments": [ { "id": 89185, "author_name": "andreusancho", "author_url": "", "post_date": "08/12/2015 14:04:13", "content": "Consider performing another type of test validation (i.e., Cross-validation), like a Stratified K-Fold Cross-validation (see scikit page on that) when testing locally. Typically, Stratified 10-Fold CV works well for most of the cases, although depending on the data other types may be better (4-Fold CV, 5x2 CV, and so on).
", "votes": null, "replies": [] }, { "id": 89200, "author_name": "", "author_url": "", "post_date": "08/12/2015 17:34:13", "content": "@Andreu Thank you for your kind answer, I gave you a vote up for it, however do you really think that such an amount of difference (almost 0.01 AUC (!) could be occured by a not appropriate validation test selection? Furthermore, maybe my bad, I didn't mentioned, that I used stratified training/validation sets using Sklearn methods.
", "votes": null, "replies": [] }, { "id": 89280, "author_name": "andreusancho", "author_url": "", "post_date": "08/13/2015 07:55:13", "content": "Well, it may happen that the partitions used in your train/test sets are not the best ones for the learning algorithm used; Cross-validation is not perfect. Try other types, such as 6 CV, 4 CV, or 5x2 CV until you find the most similar to the public leader board. Another common approach is to repeat the Stratified K-Fold CV several times (10, 20, 30 times...) with a different random seed and average the results to avoid bias (i.e., "luck"). Also, relatively large differences may imply two possible cases: (1) that your algorithm is falling into overfitting and CV is not able of identifying that, and/or (2) that the public leader board score is not realistic (that happened in the previous LHC challenge) and we will see large differences in the private one.
\n\nI think that the second option is the most probable.
", "votes": null, "replies": [] }, { "id": 89535, "author_name": "", "author_url": "", "post_date": "08/16/2015 22:40:15", "content": "@Andreu Thank you again! Well I will take your advice and I will test more CV approaches. Anyway you will see, if I will succeed it or will not soon. :)
\n\n\n\n\n(1) that your algorithm is falling into overfitting and CV is not\n able of identifying that:
\n
This is a bit unclear for me which cases can generate such a situation, but I am working on it.
\n\n\n\n\n(2) that the public leader board score is not realistic (that happened\n in the previous LHC challenge) and we will see large differences in\n the private one. I think that the second option is the most probable.
\n
It's a bit worrying to know about such a possibility.
", "votes": null, "replies": [] }, { "id": 89738, "author_name": "", "author_url": "", "post_date": "08/19/2015 00:32:54", "content": "Update:
\n\nI tried 10 fold, 4 fold CV and more and the average AUC value resulted by any method is roughly the same within -+ 0.15 %. \nAt the same time the difference compared to public leader board is about -3%, which means a difference like 0.979 vs. 0.989, which is huge.
\n\nI am trying to figure out what is exactly happening, but it is quite surrealistic for me at the moment.\nAny help, thought, idea, anything would be highly appreciated, thank you.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "89182": "I tested the \"scripted\" Rf + Xgboost.\r\nThen I made my own solution.\r\nI proceeded according all the required metrics:\r\nCvM done.\r\nKS done.\r\nAUC done.\r\nI've got a quite good AUC score, so then I submitted it, but surprisingly I've got **0.979** on the public LB!\r\nI checked again and again: \r\nMade the RF + XGB according the metrics I used before: \r\n**0.983-0.984** depending on the validation set. \r\nChecked again my model, and yes with the same higher AUC score. \r\nThen back to submission again: \r\n\r\n> **AUC: 0.979 \"no improvement etc....\"**\r\n\r\nI checked twice (rather much more): I applied the \r\n **min_ANNmuon > 0.4 filter**, \r\nI used several different validation set.\r\n\r\nWhere could be the problem yet?\r\nWhere did I make a mistake?\r\nThanks in advance.\r\n\r\n**UPDATE:** When I made two other submission with other two different model configuration, I got the same approx. **0.979** AUC value.", "89185": "Consider performing another type of test validation (i.e., Cross-validation), like a Stratified K-Fold Cross-validation ([see scikit page on that][1]) when testing locally. Typically, Stratified 10-Fold CV works well for most of the cases, although depending on the data other types may be better (4-Fold CV, 5x2 CV, and so on).\r\n\r\n\r\n [1]: http://scikit-learn.org/stable/modules/cross_validation.html", "89200": "Andreu Thank you for your kind answer, I gave you a vote up for it, however do you really think that such an amount of difference (almost 0.01 AUC (!) could be occured by a not appropriate validation test selection? Furthermore, maybe my bad, I didn't mentioned, that I used stratified training/validation sets using Sklearn methods.", "89280": "Well, it may happen that the partitions used in your train/test sets are not the best ones for the learning algorithm used; Cross-validation is not perfect. Try other types, such as 6 CV, 4 CV, or 5x2 CV until you find the most similar to the public leader board. Another common approach is to repeat the Stratified K-Fold CV several times (10, 20, 30 times...) with a different random seed and average the results to avoid bias (i.e., \"luck\"). Also, relatively large differences may imply two possible cases: (1) that your algorithm is falling into overfitting and CV is not able of identifying that, and/or (2) that the public leader board score is not *realistic* (that happened in the previous LHC challenge) and we will see large differences in the private one.\r\n\r\nI think that the second option is the most probable.", "89535": "Andreu Thank you again! Well I will take your advice and I will test more CV approaches. Anyway you will see, if I will succeed it or will not soon. :)\r\n\r\n> (1) that your algorithm is falling into overfitting and CV is not\r\n> able of identifying that:\r\n\r\nThis is a bit unclear for me which cases can generate such a situation, but I am working on it.\r\n\r\n> (2) that the public leader board score is not realistic (that happened\r\n> in the previous LHC challenge) and we will see large differences in\r\n> the private one. I think that the second option is the most probable.\r\n\r\nIt's a bit worrying to know about such a possibility.", "89738": "**Update:** \r\n\r\nI tried 10 fold, 4 fold CV and more and the average AUC value resulted by any method is roughly the same within **-+ 0.15 %**. \r\nAt the same time the difference compared to public leader board is about **-3%**, which means a difference like 0.979 vs. 0.989, which is huge.\r\n\r\nI am trying to figure out what is exactly happening, but it is quite surrealistic for me at the moment.\r\nAny help, thought, idea, anything would be highly appreciated, thank you." }, "source": "meta" }