As of last week, we identified, @betatim and I, a loophole in the challenge\nsetting making it possible to train a classifier exploiting simulation\nimperfections (i.e., features discriminative between simulation and real data,\nsuch as SPDhits, rather than between signal and background) while still passing\nthe agreement test. The issue was reported to the organizers, who decided to\nmake the rules of the challenge stricter (see https://www.kaggle.com/c/flavours-of-physics/forums/t/15735/important-training-dataset-reminder).
\n\nSo that everyone can understand and think about the problem, we prepared an\nIPython notebook explaining the issue, showing how and why checks on control\nchannels to compare a classifier's response on real and simulated\nevents/samples can be bypassed (accidentally or on purpose). Overall, we argue\nthat models obtained through the challenge (at least some) might not be as\nreliable as estimated, even if tests on control channels pass. In particular, a\npassing control channel test does not guarantee that a classifier does not\nleverage discriminative simulation imperfections.
\n\n\n\nThe notebook includes in broad terms the solution making it possible to\ncurrently reach the top of the leaderboard. Since such a solution is however\ninvalid with respect to the new challenge rules, we wont post the corresponding\nsource code, in order to avoid chaos in the LB with dozens of such solutions.
\n\nFinally, since we want this to beneficial in the end for the physics, we are\ncurrently thinking about better agreement tests. We would be glad to share and discuss\nideas if you have any :)
", "messages": [ { "id": "88131", "postDate": "08/03/2015 18:54:28", "content": "As of last week, we identified, @betatim and I, a loophole in the challenge\nsetting making it possible to train a classifier exploiting simulation\nimperfections (i.e., features discriminative between simulation and real data,\nsuch as SPDhits, rather than between signal and background) while still passing\nthe agreement test. The issue was reported to the organizers, who decided to\nmake the rules of the challenge stricter (see https://www.kaggle.com/c/flavours-of-physics/forums/t/15735/important-training-dataset-reminder).
\n\nSo that everyone can understand and think about the problem, we prepared an\nIPython notebook explaining the issue, showing how and why checks on control\nchannels to compare a classifier's response on real and simulated\nevents/samples can be bypassed (accidentally or on purpose). Overall, we argue\nthat models obtained through the challenge (at least some) might not be as\nreliable as estimated, even if tests on control channels pass. In particular, a\npassing control channel test does not guarantee that a classifier does not\nleverage discriminative simulation imperfections.
\n\n\n\nThe notebook includes in broad terms the solution making it possible to\ncurrently reach the top of the leaderboard. Since such a solution is however\ninvalid with respect to the new challenge rules, we wont post the corresponding\nsource code, in order to avoid chaos in the LB with dozens of such solutions.
\n\nFinally, since we want this to beneficial in the end for the physics, we are\ncurrently thinking about better agreement tests. We would be glad to share and discuss\nideas if you have any :)
", "rawMarkdown": "As of last week, we identified, @betatim and I, a loophole in the challenge\r\nsetting making it possible to train a classifier exploiting simulation\r\nimperfections (i.e., features discriminative between simulation and real data,\r\nsuch as SPDhits, rather than between signal and background) while still passing\r\nthe agreement test. The issue was reported to the organizers, who decided to\r\nmake the rules of the challenge stricter (see https://www.kaggle.com/c/flavours-of-physics/forums/t/15735/important-training-dataset-reminder).\r\n\r\nSo that everyone can understand and think about the problem, we prepared an\r\nIPython notebook explaining the issue, showing how and why checks on control\r\nchannels to compare a classifier's response on real and simulated\r\nevents/samples can be bypassed (accidentally or on purpose). Overall, we argue\r\nthat models obtained through the challenge (at least some) might not be as\r\nreliable as estimated, even if tests on control channels pass. In particular, a\r\npassing control channel test does not guarantee that a classifier does not\r\nleverage discriminative simulation imperfections.\r\n\r\nhttp://t.co/kF40NpHHIz\r\n\r\nThe notebook includes in broad terms the solution making it possible to\r\ncurrently reach the top of the leaderboard. Since such a solution is however\r\ninvalid with respect to the new challenge rules, we wont post the corresponding\r\nsource code, in order to avoid chaos in the LB with dozens of such solutions.\r\n\r\nFinally, since we want this to beneficial in the end for the physics, we are\r\ncurrently thinking about better agreement tests. We would be glad to share and discuss\r\nideas if you have any :)", "votes": null }, { "id": "88134", "postDate": "08/03/2015 19:23:54", "content": "Thank you for reporting!
\n\nSo if algorithms that depend heavily on randomness can find this leak "accidentally" without failing the tests, is usage of these highly risky if you want to avoid cheating?
\n\nIs creating averaged out-of-fold predictions for the agreement data to see if an ensemble passes the test forbidden under: "To be eligible for the prize your final solution should rely solely on the data provided in ‘training.csv’ without any usage of ‘check_correlation.csv' or ‘check_agreement.csv' datasets."?
\n\nThough working on real data is neat, is it possible to use simulated data for both classes, or does this worsen the effect of "fitting to the algorithm"?
\n\nI used "SPDhits" in one submission and beat the highest benchmark with it (thank you for sklearn.ensemble! :) ). That is an example of a "bad submission"?
", "rawMarkdown": "Thank you for reporting!\r\n\r\nSo if algorithms that depend heavily on randomness can find this leak \"accidentally\" without failing the tests, is usage of these highly risky if you want to avoid cheating?\r\n\r\nIs creating averaged out-of-fold predictions for the agreement data to see if an ensemble passes the test forbidden under: \"To be eligible for the prize your final solution should rely solely on the data provided in ‘training.csv’ without any usage of ‘check_correlation.csv' or ‘check_agreement.csv' datasets.\"?\r\n\r\nThough working on real data is neat, is it possible to use simulated data for both classes, or does this worsen the effect of \"fitting to the algorithm\"?\r\n\r\nI used \"SPDhits\" in one submission and beat the highest benchmark with it (thank you for sklearn.ensemble! :) ). That is an example of a \"bad submission\"?", "votes": null }, { "id": "88143", "postDate": "08/03/2015 20:06:35", "content": "Very clever!
\n\nIn my opinion , the statement
\n\n\n\n\nyour final solution should rely solely on the data provided in ‘training.csv’ without any usage of ‘check_correlation.csv' or ‘check_agreement.csv' datasets
\n
must be clarified.
\n\nAt least we must be able to check locally the model output before submission ( the submission itself is using in some sense those datasets in order to verify the checks!, so after several submission some information is also leaked)
\n\nMoreover, physicist compares real data and Monte Carlo to estimate the systematic error introduced by the simulation and tries to minimize it.
\n\nFor instance, SPDhits shows an important bias in simulated data compared to real data, but the ratio of SPDhits to some variables, is distributed very similarly in real data and MC. This variables can be found ‘ by hand’ or modifying the loss function in a learning schema to programmatically found variable combinations equally distributed in real data and MC, and simultaneously able to discriminate between signal and background.
\n\nThis procedure needs to use the ‘check_agreement.csv' events in an “honest” way. For sure we have to be very careful in having an independent set to check the agreement different from the set used for finding the ‘golden’ variables (‘golden’ because are discriminative and well simulated )
\n\nMakes it sense to you guys?
", "rawMarkdown": "Very clever!\r\n\r\nIn my opinion , the statement \r\n\r\n> your final solution should rely solely on the data provided in ‘training.csv’ without **any** usage of ‘check_correlation.csv' or ‘check_agreement.csv' datasets\r\n\r\nmust be clarified.\r\n\r\n At least we must be able to check locally the model output before submission ( the submission itself is using in some sense those datasets in order to verify the checks!, so after several submission some information is also leaked)\r\n\r\n\r\nMoreover, physicist compares real data and Monte Carlo to estimate the systematic error introduced by the simulation and tries to minimize it.\r\n\r\nFor instance, SPDhits shows an important bias in simulated data compared to real data, but the ratio of SPDhits to some variables, is distributed very similarly in real data and MC. This variables can be found ‘ by hand’ or modifying the loss function in a learning schema to programmatically found variable combinations equally distributed in real data and MC, and simultaneously able to discriminate between signal and background. \r\n\r\nThis procedure needs to use the ‘check_agreement.csv' events in an “honest” way. For sure we have to be very careful in having an independent set to check the agreement different from the set used for finding the ‘golden’ variables (‘golden’ because are discriminative and well simulated )\r\n\r\nMakes it sense to you guys?", "votes": null }, { "id": "88158", "postDate": "08/03/2015 21:03:30", "content": "I dont think this is "cheating". See https://www.kaggle.com/forums/messages/quote?forumMessageId=88157
\n\nof course the admins will have the last word in this competition. They can also decide to add a new rule at the very end that blocks anyone whos name starts with "u"...
", "rawMarkdown": "I dont think this is \"cheating\". See https://www.kaggle.com/forums/messages/quote?forumMessageId=88157\r\n\r\nof course the admins will have the last word in this competition. They can also decide to add a new rule at the very end that blocks anyone whos name starts with \"u\"...", "votes": null }, { "id": "88178", "postDate": "08/04/2015 00:17:42", "content": "Nice work, Gilles Louppe! I support your suggestion. Here it is why, from an ex-physicist point of view.
\n\nIt is not very hard to distinguish Ds → φπ (control) from τ → 3μ (signal), since the control samples are manually inserted in the training dataset, and (!@#$%^&*#$%^&* sorry I don't want to tell the names but simple physics truth) can probably be useful for distinguishing them. However, the primary goal for this competition is finding a useful machine learning model which can help discovering τ → 3μ, so this model must be useful in the real life, aka a generalized model. Real life has no mercy and it doesn't just give a single control sample set: do you know how complicated the QCD photon background is in the Higgs diphoton search?
\n\nAgreement test was annoying to me, honestly, and I was very unhappy when my results got rejected because of 0.0900001 score. But it was not a reason for using some simulation features, which would make the model useless in the real life. Let me paste Richard Feynman's quote here as a reminder: "Physics is like sex: sure, it may give some practical results, but that's not why we do it." It is the same with machine learning.
", "rawMarkdown": "Nice work, Gilles Louppe! I support your suggestion. Here it is why, from an ex-physicist point of view.\r\n\r\nIt is not very hard to distinguish Ds → φπ (control) from τ → 3μ (signal), since the control samples are manually inserted in the training dataset, and (!@#$%^&*#$%^&* sorry I don't want to tell the names but simple physics truth) can probably be useful for distinguishing them. However, the primary goal for this competition is finding a useful machine learning model which can help discovering τ → 3μ, so this model must be useful in the real life, aka a generalized model. Real life has no mercy and it doesn't just give a single control sample set: do you know how complicated the QCD photon background is in the Higgs diphoton search?\r\n\r\nAgreement test was annoying to me, honestly, and I was very unhappy when my results got rejected because of 0.0900001 score. But it was not a reason for using some simulation features, which would make the model useless in the real life. Let me paste Richard Feynman's quote here as a reminder: \"Physics is like sex: sure, it may give some practical results, but that's not why we do it.\" It is the same with machine learning.", "votes": null }, { "id": "88408", "postDate": "08/04/2015 18:00:09", "content": "[quote=Gilles Louppe;88131]Finally, since we want this to beneficial in the end for the physics, we are\ncurrently thinking about better agreement tests. We would be glad to share and discuss\nideas if you have any :)[/quote]
\n\nWRT being the most "beneficial in the end for the physics", I feel contestants should be given access to real data with a proper explanation of what that data is.
\n\nUsing real data for training, the scripts would be run against a new real data set (that no one has seen) for leaderboard ranking and we start by seeing who is best at detecting known events, like the known background signal D+ → K−π+π+. Follow that up with a test on who is best at getting the second strongest background signal Ds → η(→ µ+µ−γ)µ−νµ.
\n\nOnce contestants like myself, had these scripts in hand, we would be in a far better position to launch a search for flavour violation.
", "rawMarkdown": "[quote=Gilles Louppe;88131]Finally, since we want this to beneficial in the end for the physics, we are\r\ncurrently thinking about better agreement tests. We would be glad to share and discuss\r\nideas if you have any :)[/quote]\r\n\r\nWRT being the most \"beneficial in the end for the physics\", I feel contestants should be given access to real data with a proper explanation of what that data is. \r\n\r\nUsing real data for training, the scripts would be run against a new real data set (that no one has seen) for leaderboard ranking and we start by seeing who is best at detecting known events, like the known background signal D+ → K−π+π+. Follow that up with a test on who is best at getting the second strongest background signal Ds → η(→ µ+µ−γ)µ−νµ.\r\n\r\nOnce contestants like myself, had these scripts in hand, we would be in a far better position to launch a search for flavour violation.", "votes": null }, { "id": "88552", "postDate": "08/05/2015 17:18:54", "content": "[quote=Gilles Louppe;88131]\nSo that everyone can understand and think about the problem, we prepared an\nIPython notebook explaining the issue..\n[/quote]\nThe example in your paper contains "signal data", but in this competition there are no signal data. All signal points were simulated.
\n\n[quote]\nSince such a solution is however invalid with respect to the new challenge rules..\n[/quote]\nDoes it explicitely use some data except "train.csv"?
", "rawMarkdown": "[quote=Gilles Louppe;88131]\r\nSo that everyone can understand and think about the problem, we prepared an\r\nIPython notebook explaining the issue..\r\n[/quote]\r\nThe example in your paper contains \"signal data\", but in this competition there are no signal data. All signal points were simulated.\r\n\r\n[quote]\r\nSince such a solution is however invalid with respect to the new challenge rules..\r\n[/quote]\r\nDoes it explicitely use some data except \"train.csv\"?", "votes": null }, { "id": "88553", "postDate": "08/05/2015 17:30:12", "content": "@Victor: In the toy example, the real data is only used for evaluation purposes, to highlight what would be the true performance (for the physics) if X1 were used. And yes, this trial-and-error approach is not valid since it uses the control data for model selection.
", "rawMarkdown": "Victor: In the toy example, the real data is only used for evaluation purposes, to highlight what would be the true performance (for the physics) if X1 were used. And yes, this trial-and-error approach is not valid since it uses the control data for model selection.", "votes": null }, { "id": "88665", "postDate": "08/06/2015 12:59:26", "content": "[quote=Gilles Louppe;88553]\n@Victor: In the toy example, the real data is only used for evaluation purposes, to highlight what would be the true performance (for the physics) if X1 were used. And yes, this trial-and-error approach is not valid since it uses the control data for model selection.\n[/quote]\nSurely any model that consistently passes the control tests has have been the result of a some selection process based on those control tests?
\n\nAlso, I'm super-interested in the code for how you did this, that you used to obtain 1st place. I want to make sure my understanding of this issue is 100% (not wanting to use it here). Will you please make it available after the competition has ended?
", "rawMarkdown": "[quote=Gilles Louppe;88553]\r\n@Victor: In the toy example, the real data is only used for evaluation purposes, to highlight what would be the true performance (for the physics) if X1 were used. And yes, this trial-and-error approach is not valid since it uses the control data for model selection.\r\n[/quote]\r\nSurely any model that consistently passes the control tests has have been the result of a some selection process based on those control tests?\r\n\r\nAlso, I'm super-interested in the code for how you did [this][1], that you used to obtain 1st place. I want to make sure my understanding of this issue is 100% (not wanting to use it here). Will you please make it available after the competition has ended?\r\n\r\n\r\n [1]: http://nbviewer.ipython.org/github/glouppe/notebooks/blob/master/Classification%20with%20a%20control%20channel.ipynb", "votes": null }, { "id": "88694", "postDate": "08/06/2015 17:09:27", "content": "As far as I understand, the goal of the competition is to develop a method that distinguishes between real and simulated data for channel 1. But this method must not distinguish between real and simulated data for channel 2.\nActually, we have two criteria.\nTo achieve the first criterion we have a training data, but for the second criterion we have no training data. Moreover, the only way allowed to achieve the second criterion is a blind guessing.
\n\nThe same in simplified formulation. We have a dataset 1 and a dataset 2. We need to construct a classifier that has good performance on dataset 1, but has no discriminative power on dataset 2. Also we may not even look at dataset 2.
", "rawMarkdown": "As far as I understand, the goal of the competition is to develop a method that distinguishes between real and simulated data for channel 1. But this method must not distinguish between real and simulated data for channel 2.\r\nActually, we have two criteria.\r\nTo achieve the first criterion we have a training data, but for the second criterion we have no training data. Moreover, the only way allowed to achieve the second criterion is a blind guessing.\r\n\r\nThe same in simplified formulation. We have a dataset 1 and a dataset 2. We need to construct a classifier that has good performance on dataset 1, but has no discriminative power on dataset 2. Also we may not even look at dataset 2.", "votes": null }, { "id": "90168", "postDate": "08/23/2015 18:14:42", "content": "Would this simple agreement test work?
\n\nIf it performs significantly worse on (b) it uses simulation information.
", "rawMarkdown": "Would this simple agreement test work?\r\n\r\n - Create simulated background data\r\n - (a) Run the classifier on real background + simulated signal data\r\n - (b) Run the classifier on simulated background + simulated signal data\r\n\r\nIf it performs significantly worse on (b) it uses simulation information.", "votes": null }, { "id": "90213", "postDate": "08/24/2015 09:13:38", "content": "Am I allowed to use the antics of "A simple multidimensional generator for studying selection techniques" (K Hultqvist) 1 to create a "toy MC" for the two detector data samples? And then include both in my analyses?
", "rawMarkdown": "Am I allowed to use the antics of \"A simple multidimensional generator for studying selection techniques\" (K Hultqvist) [1] to create a \"toy MC\" for the two detector data samples? And then include both in my analyses?\r\n\r\n\r\n [1]: http://delphiwww.cern.ch/pubxx.delwww/delnote/private/98_52_phys_778_prog_232.ps.gz", "votes": null } ], "comments": [ { "id": 88134, "author_name": "triskelion", "author_url": "", "post_date": "08/03/2015 19:23:54", "content": "Thank you for reporting!
\n\nSo if algorithms that depend heavily on randomness can find this leak "accidentally" without failing the tests, is usage of these highly risky if you want to avoid cheating?
\n\nIs creating averaged out-of-fold predictions for the agreement data to see if an ensemble passes the test forbidden under: "To be eligible for the prize your final solution should rely solely on the data provided in ‘training.csv’ without any usage of ‘check_correlation.csv' or ‘check_agreement.csv' datasets."?
\n\nThough working on real data is neat, is it possible to use simulated data for both classes, or does this worsen the effect of "fitting to the algorithm"?
\n\nI used "SPDhits" in one submission and beat the highest benchmark with it (thank you for sklearn.ensemble! :) ). That is an example of a "bad submission"?
", "votes": null, "replies": [] }, { "id": 88143, "author_name": "vicensgaitan", "author_url": "", "post_date": "08/03/2015 20:06:35", "content": "Very clever!
\n\nIn my opinion , the statement
\n\n\n\n\nyour final solution should rely solely on the data provided in ‘training.csv’ without any usage of ‘check_correlation.csv' or ‘check_agreement.csv' datasets
\n
must be clarified.
\n\nAt least we must be able to check locally the model output before submission ( the submission itself is using in some sense those datasets in order to verify the checks!, so after several submission some information is also leaked)
\n\nMoreover, physicist compares real data and Monte Carlo to estimate the systematic error introduced by the simulation and tries to minimize it.
\n\nFor instance, SPDhits shows an important bias in simulated data compared to real data, but the ratio of SPDhits to some variables, is distributed very similarly in real data and MC. This variables can be found ‘ by hand’ or modifying the loss function in a learning schema to programmatically found variable combinations equally distributed in real data and MC, and simultaneously able to discriminate between signal and background.
\n\nThis procedure needs to use the ‘check_agreement.csv' events in an “honest” way. For sure we have to be very careful in having an independent set to check the agreement different from the set used for finding the ‘golden’ variables (‘golden’ because are discriminative and well simulated )
\n\nMakes it sense to you guys?
", "votes": null, "replies": [] }, { "id": 88158, "author_name": "udibr1", "author_url": "", "post_date": "08/03/2015 21:03:30", "content": "I dont think this is "cheating". See https://www.kaggle.com/forums/messages/quote?forumMessageId=88157
\n\nof course the admins will have the last word in this competition. They can also decide to add a new rule at the very end that blocks anyone whos name starts with "u"...
", "votes": null, "replies": [] }, { "id": 88178, "author_name": "phunter", "author_url": "", "post_date": "08/04/2015 00:17:42", "content": "Nice work, Gilles Louppe! I support your suggestion. Here it is why, from an ex-physicist point of view.
\n\nIt is not very hard to distinguish Ds → φπ (control) from τ → 3μ (signal), since the control samples are manually inserted in the training dataset, and (!@#$%^&*#$%^&* sorry I don't want to tell the names but simple physics truth) can probably be useful for distinguishing them. However, the primary goal for this competition is finding a useful machine learning model which can help discovering τ → 3μ, so this model must be useful in the real life, aka a generalized model. Real life has no mercy and it doesn't just give a single control sample set: do you know how complicated the QCD photon background is in the Higgs diphoton search?
\n\nAgreement test was annoying to me, honestly, and I was very unhappy when my results got rejected because of 0.0900001 score. But it was not a reason for using some simulation features, which would make the model useless in the real life. Let me paste Richard Feynman's quote here as a reminder: "Physics is like sex: sure, it may give some practical results, but that's not why we do it." It is the same with machine learning.
", "votes": null, "replies": [] }, { "id": 88408, "author_name": "edguy99", "author_url": "", "post_date": "08/04/2015 18:00:09", "content": "[quote=Gilles Louppe;88131]Finally, since we want this to beneficial in the end for the physics, we are\ncurrently thinking about better agreement tests. We would be glad to share and discuss\nideas if you have any :)[/quote]
\n\nWRT being the most "beneficial in the end for the physics", I feel contestants should be given access to real data with a proper explanation of what that data is.
\n\nUsing real data for training, the scripts would be run against a new real data set (that no one has seen) for leaderboard ranking and we start by seeing who is best at detecting known events, like the known background signal D+ → K−π+π+. Follow that up with a test on who is best at getting the second strongest background signal Ds → η(→ µ+µ−γ)µ−νµ.
\n\nOnce contestants like myself, had these scripts in hand, we would be in a far better position to launch a search for flavour violation.
", "votes": null, "replies": [] }, { "id": 88552, "author_name": "nedelko", "author_url": "", "post_date": "08/05/2015 17:18:54", "content": "[quote=Gilles Louppe;88131]\nSo that everyone can understand and think about the problem, we prepared an\nIPython notebook explaining the issue..\n[/quote]\nThe example in your paper contains "signal data", but in this competition there are no signal data. All signal points were simulated.
\n\n[quote]\nSince such a solution is however invalid with respect to the new challenge rules..\n[/quote]\nDoes it explicitely use some data except "train.csv"?
", "votes": null, "replies": [] }, { "id": 88553, "author_name": "glouppe", "author_url": "", "post_date": "08/05/2015 17:30:12", "content": "@Victor: In the toy example, the real data is only used for evaluation purposes, to highlight what would be the true performance (for the physics) if X1 were used. And yes, this trial-and-error approach is not valid since it uses the control data for model selection.
", "votes": null, "replies": [] }, { "id": 88665, "author_name": "erlend", "author_url": "", "post_date": "08/06/2015 12:59:26", "content": "[quote=Gilles Louppe;88553]\n@Victor: In the toy example, the real data is only used for evaluation purposes, to highlight what would be the true performance (for the physics) if X1 were used. And yes, this trial-and-error approach is not valid since it uses the control data for model selection.\n[/quote]\nSurely any model that consistently passes the control tests has have been the result of a some selection process based on those control tests?
\n\nAlso, I'm super-interested in the code for how you did this, that you used to obtain 1st place. I want to make sure my understanding of this issue is 100% (not wanting to use it here). Will you please make it available after the competition has ended?
", "votes": null, "replies": [] }, { "id": 88694, "author_name": "nedelko", "author_url": "", "post_date": "08/06/2015 17:09:27", "content": "As far as I understand, the goal of the competition is to develop a method that distinguishes between real and simulated data for channel 1. But this method must not distinguish between real and simulated data for channel 2.\nActually, we have two criteria.\nTo achieve the first criterion we have a training data, but for the second criterion we have no training data. Moreover, the only way allowed to achieve the second criterion is a blind guessing.
\n\nThe same in simplified formulation. We have a dataset 1 and a dataset 2. We need to construct a classifier that has good performance on dataset 1, but has no discriminative power on dataset 2. Also we may not even look at dataset 2.
", "votes": null, "replies": [] }, { "id": 90168, "author_name": "sgerber", "author_url": "", "post_date": "08/23/2015 18:14:42", "content": "Would this simple agreement test work?
\n\nIf it performs significantly worse on (b) it uses simulation information.
", "votes": null, "replies": [] }, { "id": 90213, "author_name": "glimmung", "author_url": "", "post_date": "08/24/2015 09:13:38", "content": "Am I allowed to use the antics of "A simple multidimensional generator for studying selection techniques" (K Hultqvist) 1 to create a "toy MC" for the two detector data samples? And then include both in my analyses?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "88131": "As of last week, we identified, @betatim and I, a loophole in the challenge\r\nsetting making it possible to train a classifier exploiting simulation\r\nimperfections (i.e., features discriminative between simulation and real data,\r\nsuch as SPDhits, rather than between signal and background) while still passing\r\nthe agreement test. The issue was reported to the organizers, who decided to\r\nmake the rules of the challenge stricter (see https://www.kaggle.com/c/flavours-of-physics/forums/t/15735/important-training-dataset-reminder).\r\n\r\nSo that everyone can understand and think about the problem, we prepared an\r\nIPython notebook explaining the issue, showing how and why checks on control\r\nchannels to compare a classifier's response on real and simulated\r\nevents/samples can be bypassed (accidentally or on purpose). Overall, we argue\r\nthat models obtained through the challenge (at least some) might not be as\r\nreliable as estimated, even if tests on control channels pass. In particular, a\r\npassing control channel test does not guarantee that a classifier does not\r\nleverage discriminative simulation imperfections.\r\n\r\nhttp://t.co/kF40NpHHIz\r\n\r\nThe notebook includes in broad terms the solution making it possible to\r\ncurrently reach the top of the leaderboard. Since such a solution is however\r\ninvalid with respect to the new challenge rules, we wont post the corresponding\r\nsource code, in order to avoid chaos in the LB with dozens of such solutions.\r\n\r\nFinally, since we want this to beneficial in the end for the physics, we are\r\ncurrently thinking about better agreement tests. We would be glad to share and discuss\r\nideas if you have any :)", "88134": "Thank you for reporting!\r\n\r\nSo if algorithms that depend heavily on randomness can find this leak \"accidentally\" without failing the tests, is usage of these highly risky if you want to avoid cheating?\r\n\r\nIs creating averaged out-of-fold predictions for the agreement data to see if an ensemble passes the test forbidden under: \"To be eligible for the prize your final solution should rely solely on the data provided in ‘training.csv’ without any usage of ‘check_correlation.csv' or ‘check_agreement.csv' datasets.\"?\r\n\r\nThough working on real data is neat, is it possible to use simulated data for both classes, or does this worsen the effect of \"fitting to the algorithm\"?\r\n\r\nI used \"SPDhits\" in one submission and beat the highest benchmark with it (thank you for sklearn.ensemble! :) ). That is an example of a \"bad submission\"?", "88143": "Very clever!\r\n\r\nIn my opinion , the statement \r\n\r\n> your final solution should rely solely on the data provided in ‘training.csv’ without **any** usage of ‘check_correlation.csv' or ‘check_agreement.csv' datasets\r\n\r\nmust be clarified.\r\n\r\n At least we must be able to check locally the model output before submission ( the submission itself is using in some sense those datasets in order to verify the checks!, so after several submission some information is also leaked)\r\n\r\n\r\nMoreover, physicist compares real data and Monte Carlo to estimate the systematic error introduced by the simulation and tries to minimize it.\r\n\r\nFor instance, SPDhits shows an important bias in simulated data compared to real data, but the ratio of SPDhits to some variables, is distributed very similarly in real data and MC. This variables can be found ‘ by hand’ or modifying the loss function in a learning schema to programmatically found variable combinations equally distributed in real data and MC, and simultaneously able to discriminate between signal and background. \r\n\r\nThis procedure needs to use the ‘check_agreement.csv' events in an “honest” way. For sure we have to be very careful in having an independent set to check the agreement different from the set used for finding the ‘golden’ variables (‘golden’ because are discriminative and well simulated )\r\n\r\nMakes it sense to you guys?", "88158": "I dont think this is \"cheating\". See https://www.kaggle.com/forums/messages/quote?forumMessageId=88157\r\n\r\nof course the admins will have the last word in this competition. They can also decide to add a new rule at the very end that blocks anyone whos name starts with \"u\"...", "88178": "Nice work, Gilles Louppe! I support your suggestion. Here it is why, from an ex-physicist point of view.\r\n\r\nIt is not very hard to distinguish Ds → φπ (control) from τ → 3μ (signal), since the control samples are manually inserted in the training dataset, and (!@#$%^&*#$%^&* sorry I don't want to tell the names but simple physics truth) can probably be useful for distinguishing them. However, the primary goal for this competition is finding a useful machine learning model which can help discovering τ → 3μ, so this model must be useful in the real life, aka a generalized model. Real life has no mercy and it doesn't just give a single control sample set: do you know how complicated the QCD photon background is in the Higgs diphoton search?\r\n\r\nAgreement test was annoying to me, honestly, and I was very unhappy when my results got rejected because of 0.0900001 score. But it was not a reason for using some simulation features, which would make the model useless in the real life. Let me paste Richard Feynman's quote here as a reminder: \"Physics is like sex: sure, it may give some practical results, but that's not why we do it.\" It is the same with machine learning.", "88408": "[quote=Gilles Louppe;88131]Finally, since we want this to beneficial in the end for the physics, we are\r\ncurrently thinking about better agreement tests. We would be glad to share and discuss\r\nideas if you have any :)[/quote]\r\n\r\nWRT being the most \"beneficial in the end for the physics\", I feel contestants should be given access to real data with a proper explanation of what that data is. \r\n\r\nUsing real data for training, the scripts would be run against a new real data set (that no one has seen) for leaderboard ranking and we start by seeing who is best at detecting known events, like the known background signal D+ → K−π+π+. Follow that up with a test on who is best at getting the second strongest background signal Ds → η(→ µ+µ−γ)µ−νµ.\r\n\r\nOnce contestants like myself, had these scripts in hand, we would be in a far better position to launch a search for flavour violation.", "88552": "[quote=Gilles Louppe;88131]\r\nSo that everyone can understand and think about the problem, we prepared an\r\nIPython notebook explaining the issue..\r\n[/quote]\r\nThe example in your paper contains \"signal data\", but in this competition there are no signal data. All signal points were simulated.\r\n\r\n[quote]\r\nSince such a solution is however invalid with respect to the new challenge rules..\r\n[/quote]\r\nDoes it explicitely use some data except \"train.csv\"?", "88553": "Victor: In the toy example, the real data is only used for evaluation purposes, to highlight what would be the true performance (for the physics) if X1 were used. And yes, this trial-and-error approach is not valid since it uses the control data for model selection.", "88665": "[quote=Gilles Louppe;88553]\r\n@Victor: In the toy example, the real data is only used for evaluation purposes, to highlight what would be the true performance (for the physics) if X1 were used. And yes, this trial-and-error approach is not valid since it uses the control data for model selection.\r\n[/quote]\r\nSurely any model that consistently passes the control tests has have been the result of a some selection process based on those control tests?\r\n\r\nAlso, I'm super-interested in the code for how you did [this][1], that you used to obtain 1st place. I want to make sure my understanding of this issue is 100% (not wanting to use it here). Will you please make it available after the competition has ended?\r\n\r\n\r\n [1]: http://nbviewer.ipython.org/github/glouppe/notebooks/blob/master/Classification%20with%20a%20control%20channel.ipynb", "88694": "As far as I understand, the goal of the competition is to develop a method that distinguishes between real and simulated data for channel 1. But this method must not distinguish between real and simulated data for channel 2.\r\nActually, we have two criteria.\r\nTo achieve the first criterion we have a training data, but for the second criterion we have no training data. Moreover, the only way allowed to achieve the second criterion is a blind guessing.\r\n\r\nThe same in simplified formulation. We have a dataset 1 and a dataset 2. We need to construct a classifier that has good performance on dataset 1, but has no discriminative power on dataset 2. Also we may not even look at dataset 2.", "90168": "Would this simple agreement test work?\r\n\r\n - Create simulated background data\r\n - (a) Run the classifier on real background + simulated signal data\r\n - (b) Run the classifier on simulated background + simulated signal data\r\n\r\nIf it performs significantly worse on (b) it uses simulation information.", "90213": "Am I allowed to use the antics of \"A simple multidimensional generator for studying selection techniques\" (K Hultqvist) [1] to create a \"toy MC\" for the two detector data samples? And then include both in my analyses?\r\n\r\n\r\n [1]: http://delphiwww.cern.ch/pubxx.delwww/delnote/private/98_52_phys_778_prog_232.ps.gz" }, "source": "meta" }