Are all/some of the 0.99+ are physically sound?
", "messages": [ { "id": "95801", "postDate": "10/12/2015 05:57:27", "content": "Are all/some of the 0.99+ are physically sound?
", "rawMarkdown": "Are all/some of the 0.99+ are physically sound?", "votes": null }, { "id": "95827", "postDate": "10/12/2015 14:14:40", "content": "IMHO, in terms of rare decay phenomenon all of 0.98+ are also dubious results.
", "rawMarkdown": "IMHO, in terms of rare decay phenomenon all of 0.98+ are also dubious results.", "votes": null }, { "id": "95828", "postDate": "10/12/2015 14:20:54", "content": "Unfortunately, following the rules strictly, "physically sound" should have in my opinion no other meaning that "the classifier passes the agreement and correlation tests", since those were the checks chosen to verify physical soundness.
\n\nThe fact that these checks can easily be bypassed / are not actually verifying anything is another story :)
", "rawMarkdown": "Unfortunately, following the rules strictly, \"physically sound\" should have in my opinion no other meaning that \"the classifier passes the agreement and correlation tests\", since those were the checks chosen to verify physical soundness. \r\n\r\nThe fact that these checks can easily be bypassed / are not actually verifying anything is another story :)", "votes": null }, { "id": "95830", "postDate": "10/12/2015 14:33:22", "content": "A local score of .98976 gave LB .98281 -- overfit by .007.\nA two-fold CV gave .98338 -- overfit by .0006.\nUsing 1 329 d.o.f.\nWhy would it be dubious?
", "rawMarkdown": "A local score of .98976 gave LB .98281 -- overfit by .007.\r\nA two-fold CV gave .98338 -- overfit by .0006.\r\nUsing 1 329 d.o.f.\r\nWhy would it be dubious?", "votes": null }, { "id": "95833", "postDate": "10/12/2015 15:28:18", "content": ""physically sound" in my opinion is stricter than 'passing the KS/CvM test' that I always put physical meaningful features in my feature engineering, and I hope some of my features will be helpful in real life, as well as my ensemble method of balancing 'aggressive' classifiers like xgboost and 'peaceful' classifiers like UGrad and RandomForest. Passing agreement/correlation test is trivial for this task: we only have a single calibration channel, aka the control channel, so playing check_agreement.csv tricks don't make sense at all for real life. A more real life calibration examples is: do you know how many QCD photon bins for calibration in the Higgs -> diphoton search?
\n\nAs to the public LB score, here it is the old question on Kaggle: should we trust the CV or the LB score?
", "rawMarkdown": "\"physically sound\" in my opinion is stricter than 'passing the KS/CvM test' that I always put physical meaningful features in my feature engineering, and I hope some of my features will be helpful in real life, as well as my ensemble method of balancing 'aggressive' classifiers like xgboost and 'peaceful' classifiers like UGrad and RandomForest. Passing agreement/correlation test is trivial for this task: we only have a single calibration channel, aka the control channel, so playing check_agreement.csv tricks don't make sense at all for real life. A more real life calibration examples is: do you know how many QCD photon bins for calibration in the Higgs -> diphoton search?\r\n\r\nAs to the public LB score, here it is the old question on Kaggle: should we trust the CV or the LB score?", "votes": null }, { "id": "95835", "postDate": "10/12/2015 15:38:04", "content": "[quote=Glimmung;95830]\nWhy would it be dubious?\n[/quote]\nIt's my opinion. I could be mistaken.
", "rawMarkdown": "[quote=Glimmung;95830]\r\nWhy would it be dubious?\r\n[/quote]\r\nIt's my opinion. I could be mistaken.", "votes": null }, { "id": "95837", "postDate": "10/12/2015 15:48:24", "content": "@phunter: I totally agree with you. My point was that "physical soundness" is a somehow subjective concept and that, within the scope of the challenge, it was chosen to be defined in a too simple and imperfect way (for practical reasons). As a consequence, the actual soundness of the top submissions should not be expected to be higher than what is required of them. Too bad...
", "rawMarkdown": "phunter: I totally agree with you. My point was that \"physical soundness\" is a somehow subjective concept and that, within the scope of the challenge, it was chosen to be defined in a too simple and imperfect way (for practical reasons). As a consequence, the actual soundness of the top submissions should not be expected to be higher than what is required of them. Too bad...", "votes": null }, { "id": "95838", "postDate": "10/12/2015 15:51:41", "content": "@Gilles Louppe that is very true, the current top 1 is almost 1.0 AUC that I may not convince myself that the dataset has no systematic errors no noise (what a perfect simulation!)
", "rawMarkdown": "Gilles Louppe that is very true, the current top 1 is almost 1.0 AUC that I may not convince myself that the dataset has no systematic errors no noise (what a perfect simulation!)", "votes": null }, { "id": "95840", "postDate": "10/12/2015 15:54:47", "content": "[quote=phunter;95838]\n@Gilles Louppe that is very true, the current top 1 is almost 1.0 AUC that I may not convince myself that the dataset has no systematic errors no noise (what a perfect simulation!)\n[/quote]
\n\n+100000!!!!!!!!!!!!
", "rawMarkdown": "[quote=phunter;95838]\r\n@Gilles Louppe that is very true, the current top 1 is almost 1.0 AUC that I may not convince myself that the dataset has no systematic errors no noise (what a perfect simulation!)\r\n[/quote]\r\n\r\n+100000!!!!!!!!!!!!", "votes": null }, { "id": "95843", "postDate": "10/12/2015 16:02:08", "content": "Passing two tests is enough for this competition. But this is just the first step. For a physically sound model, I think the classifier should at least be checked on the full data calibration. If some useful features are found, physicists may need to try to understand those features.
", "rawMarkdown": "Passing two tests is enough for this competition. But this is just the first step. For a physically sound model, I think the classifier should at least be checked on the full data calibration. If some useful features are found, physicists may need to try to understand those features.", "votes": null } ], "comments": [ { "id": 95827, "author_name": "derkanat", "author_url": "", "post_date": "10/12/2015 14:14:40", "content": "IMHO, in terms of rare decay phenomenon all of 0.98+ are also dubious results.
", "votes": null, "replies": [] }, { "id": 95828, "author_name": "glouppe", "author_url": "", "post_date": "10/12/2015 14:20:54", "content": "Unfortunately, following the rules strictly, "physically sound" should have in my opinion no other meaning that "the classifier passes the agreement and correlation tests", since those were the checks chosen to verify physical soundness.
\n\nThe fact that these checks can easily be bypassed / are not actually verifying anything is another story :)
", "votes": null, "replies": [] }, { "id": 95830, "author_name": "glimmung", "author_url": "", "post_date": "10/12/2015 14:33:22", "content": "A local score of .98976 gave LB .98281 -- overfit by .007.\nA two-fold CV gave .98338 -- overfit by .0006.\nUsing 1 329 d.o.f.\nWhy would it be dubious?
", "votes": null, "replies": [] }, { "id": 95833, "author_name": "phunter", "author_url": "", "post_date": "10/12/2015 15:28:18", "content": ""physically sound" in my opinion is stricter than 'passing the KS/CvM test' that I always put physical meaningful features in my feature engineering, and I hope some of my features will be helpful in real life, as well as my ensemble method of balancing 'aggressive' classifiers like xgboost and 'peaceful' classifiers like UGrad and RandomForest. Passing agreement/correlation test is trivial for this task: we only have a single calibration channel, aka the control channel, so playing check_agreement.csv tricks don't make sense at all for real life. A more real life calibration examples is: do you know how many QCD photon bins for calibration in the Higgs -> diphoton search?
\n\nAs to the public LB score, here it is the old question on Kaggle: should we trust the CV or the LB score?
", "votes": null, "replies": [] }, { "id": 95835, "author_name": "derkanat", "author_url": "", "post_date": "10/12/2015 15:38:04", "content": "[quote=Glimmung;95830]\nWhy would it be dubious?\n[/quote]\nIt's my opinion. I could be mistaken.
", "votes": null, "replies": [] }, { "id": 95837, "author_name": "glouppe", "author_url": "", "post_date": "10/12/2015 15:48:24", "content": "@phunter: I totally agree with you. My point was that "physical soundness" is a somehow subjective concept and that, within the scope of the challenge, it was chosen to be defined in a too simple and imperfect way (for practical reasons). As a consequence, the actual soundness of the top submissions should not be expected to be higher than what is required of them. Too bad...
", "votes": null, "replies": [] }, { "id": 95838, "author_name": "phunter", "author_url": "", "post_date": "10/12/2015 15:51:41", "content": "@Gilles Louppe that is very true, the current top 1 is almost 1.0 AUC that I may not convince myself that the dataset has no systematic errors no noise (what a perfect simulation!)
", "votes": null, "replies": [] }, { "id": 95840, "author_name": "derkanat", "author_url": "", "post_date": "10/12/2015 15:54:47", "content": "[quote=phunter;95838]\n@Gilles Louppe that is very true, the current top 1 is almost 1.0 AUC that I may not convince myself that the dataset has no systematic errors no noise (what a perfect simulation!)\n[/quote]
\n\n+100000!!!!!!!!!!!!
", "votes": null, "replies": [] }, { "id": 95843, "author_name": "liuchengwang", "author_url": "", "post_date": "10/12/2015 16:02:08", "content": "Passing two tests is enough for this competition. But this is just the first step. For a physically sound model, I think the classifier should at least be checked on the full data calibration. If some useful features are found, physicists may need to try to understand those features.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "95801": "Are all/some of the 0.99+ are physically sound?", "95827": "IMHO, in terms of rare decay phenomenon all of 0.98+ are also dubious results.", "95828": "Unfortunately, following the rules strictly, \"physically sound\" should have in my opinion no other meaning that \"the classifier passes the agreement and correlation tests\", since those were the checks chosen to verify physical soundness. \r\n\r\nThe fact that these checks can easily be bypassed / are not actually verifying anything is another story :)", "95830": "A local score of .98976 gave LB .98281 -- overfit by .007.\r\nA two-fold CV gave .98338 -- overfit by .0006.\r\nUsing 1 329 d.o.f.\r\nWhy would it be dubious?", "95833": "\"physically sound\" in my opinion is stricter than 'passing the KS/CvM test' that I always put physical meaningful features in my feature engineering, and I hope some of my features will be helpful in real life, as well as my ensemble method of balancing 'aggressive' classifiers like xgboost and 'peaceful' classifiers like UGrad and RandomForest. Passing agreement/correlation test is trivial for this task: we only have a single calibration channel, aka the control channel, so playing check_agreement.csv tricks don't make sense at all for real life. A more real life calibration examples is: do you know how many QCD photon bins for calibration in the Higgs -> diphoton search?\r\n\r\nAs to the public LB score, here it is the old question on Kaggle: should we trust the CV or the LB score?", "95835": "[quote=Glimmung;95830]\r\nWhy would it be dubious?\r\n[/quote]\r\nIt's my opinion. I could be mistaken.", "95837": "phunter: I totally agree with you. My point was that \"physical soundness\" is a somehow subjective concept and that, within the scope of the challenge, it was chosen to be defined in a too simple and imperfect way (for practical reasons). As a consequence, the actual soundness of the top submissions should not be expected to be higher than what is required of them. Too bad...", "95838": "Gilles Louppe that is very true, the current top 1 is almost 1.0 AUC that I may not convince myself that the dataset has no systematic errors no noise (what a perfect simulation!)", "95840": "[quote=phunter;95838]\r\n@Gilles Louppe that is very true, the current top 1 is almost 1.0 AUC that I may not convince myself that the dataset has no systematic errors no noise (what a perfect simulation!)\r\n[/quote]\r\n\r\n+100000!!!!!!!!!!!!", "95843": "Passing two tests is enough for this competition. But this is just the first step. For a physically sound model, I think the classifier should at least be checked on the full data calibration. If some useful features are found, physicists may need to try to understand those features." }, "source": "meta" }