To admins:
\n\nI understand that it is not allowed to make any use of the following data in any capacity:
\n\nAlthough the use of external data is disallowed in this competition, is it admissible to make use of a self-contained source of data in a model, such as the data produced by a random number generator?
\n\nThe reason I am asking this is that I have a theoretical solution that would allow to train a model that could entirely by-pass the agreement test, without ever looking at check_agreement.csv. This would allow to leverage the simulation defects (that are leaking the labels) as much as possible while having a low KS score. This model would score very high (although it would be useless from a physics perspective), while respecting the basic rules such as:
\n\nThis solution relies only on the use of a random number generator and a handful of hard-coded hyperparameters.
\n\nSo, are random number generators (and hard-coded hyperparameters) allowed in this competition?
\n\nOf course, it is only half a real question, since RNGs are common in many machine learning algorithms and disallowing them would be... strange, to say the least. So you could rephrase my question as: are you willing to address the core issues with this competition?
\n\nAdding new rules will NOT be sufficient to prevent entrants from finding ways to exploit simulation leaks while still passing the KS test. Exploiting these flaws is by design the only way to perform competitively in this challenge. I can promise you that the top-performing models (including mine if I don't quit right now) will not be useful to physics, while still managing to comply with your rules. You are not getting what you are looking for, and you will need to revise your basics assumptions if you want this competition to align with physics usefulness.
\n\nThank you.
", "messages": [ { "id": "88212", "postDate": "08/04/2015 02:29:54", "content": "To admins:
\n\nI understand that it is not allowed to make any use of the following data in any capacity:
\n\nAlthough the use of external data is disallowed in this competition, is it admissible to make use of a self-contained source of data in a model, such as the data produced by a random number generator?
\n\nThe reason I am asking this is that I have a theoretical solution that would allow to train a model that could entirely by-pass the agreement test, without ever looking at check_agreement.csv. This would allow to leverage the simulation defects (that are leaking the labels) as much as possible while having a low KS score. This model would score very high (although it would be useless from a physics perspective), while respecting the basic rules such as:
\n\nThis solution relies only on the use of a random number generator and a handful of hard-coded hyperparameters.
\n\nSo, are random number generators (and hard-coded hyperparameters) allowed in this competition?
\n\nOf course, it is only half a real question, since RNGs are common in many machine learning algorithms and disallowing them would be... strange, to say the least. So you could rephrase my question as: are you willing to address the core issues with this competition?
\n\nAdding new rules will NOT be sufficient to prevent entrants from finding ways to exploit simulation leaks while still passing the KS test. Exploiting these flaws is by design the only way to perform competitively in this challenge. I can promise you that the top-performing models (including mine if I don't quit right now) will not be useful to physics, while still managing to comply with your rules. You are not getting what you are looking for, and you will need to revise your basics assumptions if you want this competition to align with physics usefulness.
\n\nThank you.
", "rawMarkdown": "To admins:\r\n\r\nI understand that it is not allowed to make any use of the following data in any capacity:\r\n\r\n- check_correlation.csv\r\n- check_agreement.csv\r\n- any \"external\" data\r\n\r\nAlthough the use of external data is disallowed in this competition, is it admissible to make use of a self-contained source of data in a model, such as the data produced by a random number generator?\r\n\r\nThe reason I am asking this is that I have a theoretical solution that would allow to train a model that could entirely by-pass the agreement test, without ever looking at check_agreement.csv. This would allow to leverage the simulation defects (that are leaking the labels) as much as possible while having a low KS score. This model would score very high (although it would be useless from a physics perspective), while respecting the basic rules such as:\r\n\r\n- no use of check_agreement.csv whatsoever\r\n- all events are treated in the same way as if they are from the same channel\r\n\r\nThis solution relies only on the use of a random number generator and a handful of hard-coded hyperparameters.\r\n\r\nSo, are random number generators (and hard-coded hyperparameters) allowed in this competition?\r\n\r\nOf course, it is only half a real question, since RNGs are common in many machine learning algorithms and disallowing them would be... strange, to say the least. So you could rephrase my question as: are you willing to address the core issues with this competition? \r\n\r\nAdding new rules will NOT be sufficient to prevent entrants from finding ways to exploit simulation leaks while still passing the KS test. Exploiting these flaws is *by design* the only way to perform competitively in this challenge. I can promise you that the top-performing models (including mine if I don't quit right now) will *not* be useful to physics, while still managing to comply with your rules. You are not getting what you are looking for, and you will need to revise your basics assumptions if you want this competition to align with physics usefulness.\r\n\r\nThank you.", "votes": null }, { "id": "89279", "postDate": "08/13/2015 07:46:45", "content": "Unfortunately, this post (as well as many others) has been completely ignored by the competition admins. If you think I'm wrong, please tell me so, but ignoring these questions is a demonstration of total contempt for the people who voluntarily spend many hours of their time working to solve your problems.
\n\nIn any case, I just implemented the technique, and got 0.990680 on the LB on my first try (2nd place, could easily be tweaked to reach 1st place). As far as I can tell, the approach is within the (current) rules since it only makes use of the data from training.csv.
\n\nThe approach is also uninteresting from a physics standpoint, since it is merely exploiting simulation leaks while passing the KS test. Then again, any approach that performs highly in this competition will necessarily be based on the exploitation of simulation leaks. That's just how the competition is set up. No matter how much one might want to develop a physics-based solution, it is not possible, because the data we have does not allow to tell apart solutions that are physics based from solutions based on exploiting simulation leaks. Both are identical from an information-theoretical standpoint, given the data available.
\n\nIf the competition admins want to make my approach invalid, I recommend the following rule change: disallow random number generators. Until the next exploit, and the next rule change.
\n\nAlternatively, you could consider admitting your mistakes, and restarting the competition with new data and saner rules. In doing so, you should consult with entrants in the current competition to avoid repeating the same mistakes. There are people here who are willing to help you. For free. Just out of desire to make things right. Then again, given how utterly unresponsive to concerns and questions the competition admins have been, there is not a chance that it will happen. But one can dream.
\n\nI've seen a few failed competitions on Kaggle in the past (launching data science competitions is hard work), but this one takes the cake. Not because the mistakes made were particularly obvious (they certainly weren't, and it took weeks to fully identify them) but because of the complete mismanagement of the competition after the issues were pointed out.
\n\nToo bad. I'll shut up now, since I am obviously just wasting my time.
", "rawMarkdown": "Unfortunately, this post (as well as many others) has been completely ignored by the competition admins. If you think I'm wrong, please tell me so, but ignoring these questions is a demonstration of total contempt for the people who voluntarily spend many hours of their time working to solve your problems.\r\n\r\nIn any case, I just implemented the technique, and got 0.990680 on the LB on my first try (2nd place, could easily be tweaked to reach 1st place). As far as I can tell, the approach is within the (current) rules since it only makes use of the data from training.csv.\r\n\r\nThe approach is also uninteresting from a physics standpoint, since it is merely exploiting simulation leaks while passing the KS test. Then again, any approach that performs highly in this competition will necessarily be based on the exploitation of simulation leaks. That's just how the competition is set up. No matter how much one might want to develop a physics-based solution, it is not possible, because the data we have does not allow to tell apart solutions that are physics based from solutions based on exploiting simulation leaks. Both are identical from an information-theoretical standpoint, given the data available. \r\n\r\nIf the competition admins want to make my approach invalid, I recommend the following rule change: disallow random number generators. Until the next exploit, and the next rule change. \r\n\r\nAlternatively, you could consider admitting your mistakes, and restarting the competition with new data and saner rules. In doing so, you should consult with entrants in the current competition to avoid repeating the same mistakes. There are people here who are willing to help you. For free. Just out of desire to make things right. Then again, given how utterly unresponsive to concerns and questions the competition admins have been, there is not a chance that it will happen. But one can dream. \r\n\r\nI've seen a few failed competitions on Kaggle in the past (launching data science competitions is hard work), but this one takes the cake. Not because the mistakes made were particularly obvious (they certainly weren't, and it took weeks to fully identify them) but because of the complete mismanagement of the competition after the issues were pointed out. \r\n\r\nToo bad. I'll shut up now, since I am obviously just wasting my time.", "votes": null }, { "id": "89296", "postDate": "08/13/2015 13:10:01", "content": "Sorry, I missed your precious post. So you can of coz use random numbers. We will ask you in the solution to give the motivation, how the numbers were obtained and how we can add them to future data. We will check if the numbers are causing some differences between the calibration channel and signal channel.
\n\nLet me comment on:\nExploiting these flaws is by design the only way to perform competitively in this challenge.
\n\nThe point of the challenge is not to get an classifier that is completely useless because it treats signal channel differently than control channel. If we wanted to that kind of things obviously we wouldn't waste time on control channel at all. There are people in the top10 that have a valid classifier trained without using the control channel so that can be applied in physics application.
", "rawMarkdown": "Sorry, I missed your precious post. So you can of coz use random numbers. We will ask you in the solution to give the motivation, how the numbers were obtained and how we can add them to future data. We will check if the numbers are causing some differences between the calibration channel and signal channel. \r\n\r\nLet me comment on:\r\nExploiting these flaws is by design the only way to perform competitively in this challenge.\r\n\r\nThe point of the challenge is not to get an classifier that is completely useless because it treats signal channel differently than control channel. If we wanted to that kind of things obviously we wouldn't waste time on control channel at all. There are people in the top10 that have a valid classifier trained without using the control channel so that can be applied in physics application.", "votes": null }, { "id": "89297", "postDate": "08/13/2015 13:33:09", "content": "[quote=Albercik1905;89296]
\n\nSorry, I missed your precious post. So you can of coz use random numbers. We will ask you in the solution to give the motivation, how the numbers were obtained and how we can add them to future data. We will check if the numbers are causing some differences between the calibration channel and signal channel.
\n\nLet me comment on:\nExploiting these flaws is by design the only way to perform competitively in this challenge.
\n\nThe point of the challenge is not to get an classifier that is completely useless because it treats signal channel differently than control channel. If we wanted to that kind of things obviously we wouldn't waste time on control channel at all. There are people in the top10 that have a valid classifier trained without using the control channel so that can be applied in physics application.
\n\n[/quote]
\n\nWhen reviewing somebody's solution, you will not be able to tell whether it is grounded in physics or whether it is exploiting simulation artifacts, because the two are completely identical in this challenge.
\n\nThe winners in this challenge will be entrants who find features that successfully separate signal from background while yielding a low KS value. Two facts about these features:
\n\n[quote]\nThe point of the challenge is not to get an classifier that is completely useless because it treats signal channel differently than control channel. If we wanted to that kind of things obviously we wouldn't waste time on control channel at all. [/quote]
\n\nMy classifier treats all data points in the same way. Yet it is useless. The control channel does not do what you seem to think it does.
\n\nMy point about exploiting the simulation flaws in this challenge is this: there is no way not to do it, and those doing it will not realize their are doing it. As far as they know their solutions might well be grounded in actual physics. Physics-based solutions and solutions that exploit simulation artifacts look strictly identical in this challenge, for entrants, and for you. They are simply the solutions that are picking up the "right" features: features that discriminate signal from background while passing the KS test.
\n\nCan you define in concrete terms what a valid solution will look like? I 100% believe that my current solution fits the bill.
", "rawMarkdown": "[quote=Albercik1905;89296]\r\n\r\nSorry, I missed your precious post. So you can of coz use random numbers. We will ask you in the solution to give the motivation, how the numbers were obtained and how we can add them to future data. We will check if the numbers are causing some differences between the calibration channel and signal channel. \r\n\r\nLet me comment on:\r\nExploiting these flaws is by design the only way to perform competitively in this challenge.\r\n\r\nThe point of the challenge is not to get an classifier that is completely useless because it treats signal channel differently than control channel. If we wanted to that kind of things obviously we wouldn't waste time on control channel at all. There are people in the top10 that have a valid classifier trained without using the control channel so that can be applied in physics application. \r\n\r\n\r\n\r\n[/quote]\r\n\r\nWhen reviewing somebody's solution, you will *not* be able to tell whether it is grounded in physics or whether it is exploiting simulation artifacts, because the two are completely identical in this challenge. \r\n\r\nThe winners in this challenge will be entrants who find features that successfully separate signal from background while yielding a low KS value. Two facts about these features:\r\n\r\n- the only way to find these features is by using the data from check_agreement.csv. However this can be done implicitly, without every looking at that data during the training process. If you are not using check_agreement.csv to make sure your process passes the KS test, then what, random chance? \r\n- it is impossible to tell apart features that are simply good at picking up simulation artifacts while discriminating control events (which have a different distribution), from features that do not leverage simulation artifacts. The two are strictly identical given the data we have. The tell them part you would need to know what simulation artifacts are present, and that cannot be seen in the data we have access to. The KS test *does not* control for the use of simulation artifacts, so passing the KS test tells you nothing.\r\n\r\n[quote]\r\nThe point of the challenge is not to get an classifier that is completely useless because it treats signal channel differently than control channel. If we wanted to that kind of things obviously we wouldn't waste time on control channel at all. [/quote] \r\n\r\nMy classifier treats all data points in the same way. Yet it is useless. The control channel does not do what you seem to think it does. \r\n\r\nMy point about exploiting the simulation flaws in this challenge is this: there is no way *not* to do it, and those doing it will not realize their are doing it. As far as they know their solutions might well be grounded in actual physics. Physics-based solutions and solutions that exploit simulation artifacts look strictly identical in this challenge, for entrants, and for you. They are simply the solutions that are picking up the \"right\" features: features that discriminate signal from background while passing the KS test.\r\n\r\nCan you define in concrete terms what a valid solution will look like? I 100% believe that my current solution fits the bill.", "votes": null }, { "id": "89346", "postDate": "08/14/2015 08:39:43", "content": "So let me start by defining what we do normally. We check which variables are are in agreement and which are not. This is done even by looking at the distributions, not need to KS test. Then we pick the ones that look ok or reweigh them if we really really need to use them. One has to say for us the robustness of the classifier comes before the performance. When we designed the contests we left thous variables for you to discover on your own that they should not be used.
\n\nOf coz the tests we ask you to do here are tip of the iceberg. If we asked you guys to do all of them then the contest would have 10 participants. Some tests are dependent of what variables you use, etc. We decided to settle down for thous two simplest cross checks which already caused problem in understanding why we do them at all.
\n\nSo to answer your question in a simplest way I can I would say that the classifier you trained should super similar performance on Ds->phi pi as in tau->3mu. If this the case in your training?
\n\nSo when we did our classifer for the analysis. we first trained several classifiers picked the best one and looked at the performance and in the first goal we had a super DATA/MC agreement as we shown in the documentation. Furthermore, we had super similar performance on Ds->phi pi and tau->3mu(there are more tests but not relevant probably for this discussion). Does this a bit explain what are we aiming for?
\n\nSo if you really didn't use the Ds2phipi data for training your solution is ok with the rules. I had a look at your solution and run my private checks. Your solution will increase one of the solution systematic which is not desired in physics but it's out side the scope of this competition.
\n\nCheers,\nMarcin
", "rawMarkdown": "So let me start by defining what we do normally. We check which variables are are in agreement and which are not. This is done even by looking at the distributions, not need to KS test. Then we pick the ones that look ok or reweigh them if we really really need to use them. One has to say for us the robustness of the classifier comes before the performance. When we designed the contests we left thous variables for you to discover on your own that they should not be used. \r\n\r\nOf coz the tests we ask you to do here are tip of the iceberg. If we asked you guys to do all of them then the contest would have 10 participants. Some tests are dependent of what variables you use, etc. We decided to settle down for thous two simplest cross checks which already caused problem in understanding why we do them at all. \r\n\r\nSo to answer your question in a simplest way I can I would say that the classifier you trained should super similar performance on Ds->phi pi as in tau->3mu. If this the case in your training?\r\n\r\nSo when we did our classifer for the analysis. we first trained several classifiers picked the best one and looked at the performance and in the first goal we had a super DATA/MC agreement as we shown in the documentation. Furthermore, we had super similar performance on Ds->phi pi and tau->3mu(there are more tests but not relevant probably for this discussion). Does this a bit explain what are we aiming for?\r\n\r\nSo if you really didn't use the Ds2phipi data for training your solution is ok with the rules. I had a look at your solution and run my private checks. Your solution will increase one of the solution systematic which is not desired in physics but it's out side the scope of this competition.\r\n\r\nCheers,\r\nMarcin", "votes": null }, { "id": "89514", "postDate": "08/16/2015 16:08:10", "content": "Thanks, that was informative.
\n\nHere's my takeaway from what you are saying:
\n\nYou do have in fact much more information than the entrants, possibly to an extent that would allow you to recognize entries that exploit simulation leaks. However I doubt that is really the case, since if you had the means to do it you could simply erase simulation regularities from the simulated data, hence removing the need for the KS test and making the competition much more straightforward.
There are "shadow rules" in this competition ("Of coz the tests we ask you to do here are tip of the iceberg") that entries must comply with in order to be eligible to win. However we have no way to know what they are or if we are or not in compliance.
[quote]So if you really didn't use the Ds2phipi data for training your solution is ok with the rules. [/quote]
\n\nMy point is that the same results that can be achieved by training on the control data can also be achieved without explicitly training on the control data (hence the meaninglessness of the rule "only use training.csv"), via a number of possible tricks. So no, I did no train on the data from check_agreement.csv, but I might as well have since there is plenty of information about the control data that is leaking in my entry.
\n\nI'm pretty sure that my current entry is all about exploiting simulation leaks. If your own checks on my entry do not reveal that, then that confirms that you do not have the means to tell apart solutions that are grounded in physics from those that aren't.
", "rawMarkdown": "Thanks, that was informative. \r\n\r\nHere's my takeaway from what you are saying:\r\n\r\n- You do have in fact much more information than the entrants, possibly to an extent that would allow you to recognize entries that exploit simulation leaks. However I doubt that is really the case, since if you had the means to do it you could simply erase simulation regularities from the simulated data, hence removing the need for the KS test and making the competition much more straightforward.\r\n\r\n- There are \"shadow rules\" in this competition (\"*Of coz the tests we ask you to do here are tip of the iceberg*\") that entries must comply with in order to be eligible to win. However we have no way to know what they are or if we are or not in compliance.\r\n\r\n[quote]So if you really didn't use the Ds2phipi data for training your solution is ok with the rules. [/quote]\r\n\r\nMy point is that the same results that can be achieved by training on the control data can also be achieved without explicitly training on the control data (hence the meaninglessness of the rule \"only use training.csv\"), via a number of possible tricks. So no, I did no train on the data from check_agreement.csv, but I might as well have since there is plenty of information about the control data that is leaking in my entry. \r\n\r\nI'm pretty sure that my current entry is all about exploiting simulation leaks. If your own checks on my entry do not reveal that, then that confirms that you do not have the means to tell apart solutions that are grounded in physics from those that aren't.", "votes": null }, { "id": "89525", "postDate": "08/16/2015 20:34:05", "content": "So it's next to impossible get ride of the "simulation regularities" as you called them. Once you fix them in one place they pop up in other places.
\n\nLet me comment about:\n"However we have no way to know what they are or if we are or not in compliance." \nWe do not ask you to do. This competition would be far to complicated to you guys to check everything (this takes months in some cases). Some check are only applicable only if certain variables are used in the training etc. That's why we decided for the 2 simplest ones.
\n\n"I'm pretty sure that my current entry is all about exploiting simulation leaks. If your own checks on my entry do not reveal that, then that confirms that you do not have the means to tell apart solutions that are grounded in physics from those that aren't."
\n\nProbably I wasn't clear. Our different checks reveled that. This is why I said: " Your solution will increase one of the solution systematic". The larger this systematic is the less discriminating power your classifier has after you do the full data calibration. Doing the calibration is CPU consuming and a bit more complicated so we didn't want to put this in the metric.
\n\nLet me stress once again that your solution if you really didn't use test in the training will be ok for this contests. In real world we wouldn't choose it but in kaggle competition we did not put the calibration on purpose and we knew that there might be this kind of solutions (that's why there is a physics prize).
\n\nbtw. I am super interested how did you manged to got that solution :)
", "rawMarkdown": "So it's next to impossible get ride of the \"simulation regularities\" as you called them. Once you fix them in one place they pop up in other places.\r\n\r\nLet me comment about:\r\n\"However we have no way to know what they are or if we are or not in compliance.\" \r\nWe do not ask you to do. This competition would be far to complicated to you guys to check everything (this takes months in some cases). Some check are only applicable only if certain variables are used in the training etc. That's why we decided for the 2 simplest ones.\r\n\r\n\"I'm pretty sure that my current entry is all about exploiting simulation leaks. If your own checks on my entry do not reveal that, then that confirms that you do not have the means to tell apart solutions that are grounded in physics from those that aren't.\"\r\n\r\nProbably I wasn't clear. Our different checks reveled that. This is why I said: \" Your solution will increase one of the solution systematic\". The larger this systematic is the less discriminating power your classifier has after you do the full data calibration. Doing the calibration is CPU consuming and a bit more complicated so we didn't want to put this in the metric.\r\n\r\nLet me stress once again that your solution if you really didn't use test in the training will be ok for this contests. In real world we wouldn't choose it but in kaggle competition we did not put the calibration on purpose and we knew that there might be this kind of solutions (that's why there is a physics prize). \r\n\r\nbtw. I am super interested how did you manged to got that solution :)", "votes": null }, { "id": "89544", "postDate": "08/17/2015 06:45:33", "content": "\n\n\nLet me stress once again that your solution if you really didn't use test in the training will be ok for this contests.
\n
So let us assume that one can build a one-class classifier for training data, using training data and random samples only (for some definition of "random"). Then let us also assume one can build an overfit classifier for tau -> 3mu vs background. Is it allowed to combine these two classifiers at prediction time, either explicitly or implicitly?
\n\n(As far as the rules of the challenge are concerned, this appears to be strictly valid, even though it is useless.)
", "rawMarkdown": "> Let me stress once again that your solution if you really didn't use test in the training will be ok for this contests. \r\n\r\nSo let us assume that one can build a one-class classifier for training data, using training data and random samples only (for some definition of \"random\"). Then let us also assume one can build an overfit classifier for tau -> 3mu vs background. Is it allowed to combine these two classifiers at prediction time, either explicitly or implicitly? \r\n\r\n(As far as the rules of the challenge are concerned, this appears to be strictly valid, even though it is useless.)", "votes": null }, { "id": "91650", "postDate": "09/05/2015 14:47:42", "content": "You don't even really need to explicitly "use an RNG". If you use lots of stochastic models, with different (fixed) seeds, then perform selection using the banned data - you'll do well and no one will be able to tell.
", "rawMarkdown": "You don't even really need to explicitly \"use an RNG\". If you use lots of stochastic models, with different (fixed) seeds, then perform selection using the banned data - you'll do well and no one will be able to tell.", "votes": null } ], "comments": [ { "id": 89279, "author_name": "fchollet", "author_url": "", "post_date": "08/13/2015 07:46:45", "content": "Unfortunately, this post (as well as many others) has been completely ignored by the competition admins. If you think I'm wrong, please tell me so, but ignoring these questions is a demonstration of total contempt for the people who voluntarily spend many hours of their time working to solve your problems.
\n\nIn any case, I just implemented the technique, and got 0.990680 on the LB on my first try (2nd place, could easily be tweaked to reach 1st place). As far as I can tell, the approach is within the (current) rules since it only makes use of the data from training.csv.
\n\nThe approach is also uninteresting from a physics standpoint, since it is merely exploiting simulation leaks while passing the KS test. Then again, any approach that performs highly in this competition will necessarily be based on the exploitation of simulation leaks. That's just how the competition is set up. No matter how much one might want to develop a physics-based solution, it is not possible, because the data we have does not allow to tell apart solutions that are physics based from solutions based on exploiting simulation leaks. Both are identical from an information-theoretical standpoint, given the data available.
\n\nIf the competition admins want to make my approach invalid, I recommend the following rule change: disallow random number generators. Until the next exploit, and the next rule change.
\n\nAlternatively, you could consider admitting your mistakes, and restarting the competition with new data and saner rules. In doing so, you should consult with entrants in the current competition to avoid repeating the same mistakes. There are people here who are willing to help you. For free. Just out of desire to make things right. Then again, given how utterly unresponsive to concerns and questions the competition admins have been, there is not a chance that it will happen. But one can dream.
\n\nI've seen a few failed competitions on Kaggle in the past (launching data science competitions is hard work), but this one takes the cake. Not because the mistakes made were particularly obvious (they certainly weren't, and it took weeks to fully identify them) but because of the complete mismanagement of the competition after the issues were pointed out.
\n\nToo bad. I'll shut up now, since I am obviously just wasting my time.
", "votes": null, "replies": [] }, { "id": 89296, "author_name": "albercik1905", "author_url": "", "post_date": "08/13/2015 13:10:01", "content": "Sorry, I missed your precious post. So you can of coz use random numbers. We will ask you in the solution to give the motivation, how the numbers were obtained and how we can add them to future data. We will check if the numbers are causing some differences between the calibration channel and signal channel.
\n\nLet me comment on:\nExploiting these flaws is by design the only way to perform competitively in this challenge.
\n\nThe point of the challenge is not to get an classifier that is completely useless because it treats signal channel differently than control channel. If we wanted to that kind of things obviously we wouldn't waste time on control channel at all. There are people in the top10 that have a valid classifier trained without using the control channel so that can be applied in physics application.
", "votes": null, "replies": [] }, { "id": 89297, "author_name": "fchollet", "author_url": "", "post_date": "08/13/2015 13:33:09", "content": "[quote=Albercik1905;89296]
\n\nSorry, I missed your precious post. So you can of coz use random numbers. We will ask you in the solution to give the motivation, how the numbers were obtained and how we can add them to future data. We will check if the numbers are causing some differences between the calibration channel and signal channel.
\n\nLet me comment on:\nExploiting these flaws is by design the only way to perform competitively in this challenge.
\n\nThe point of the challenge is not to get an classifier that is completely useless because it treats signal channel differently than control channel. If we wanted to that kind of things obviously we wouldn't waste time on control channel at all. There are people in the top10 that have a valid classifier trained without using the control channel so that can be applied in physics application.
\n\n[/quote]
\n\nWhen reviewing somebody's solution, you will not be able to tell whether it is grounded in physics or whether it is exploiting simulation artifacts, because the two are completely identical in this challenge.
\n\nThe winners in this challenge will be entrants who find features that successfully separate signal from background while yielding a low KS value. Two facts about these features:
\n\n[quote]\nThe point of the challenge is not to get an classifier that is completely useless because it treats signal channel differently than control channel. If we wanted to that kind of things obviously we wouldn't waste time on control channel at all. [/quote]
\n\nMy classifier treats all data points in the same way. Yet it is useless. The control channel does not do what you seem to think it does.
\n\nMy point about exploiting the simulation flaws in this challenge is this: there is no way not to do it, and those doing it will not realize their are doing it. As far as they know their solutions might well be grounded in actual physics. Physics-based solutions and solutions that exploit simulation artifacts look strictly identical in this challenge, for entrants, and for you. They are simply the solutions that are picking up the "right" features: features that discriminate signal from background while passing the KS test.
\n\nCan you define in concrete terms what a valid solution will look like? I 100% believe that my current solution fits the bill.
", "votes": null, "replies": [] }, { "id": 89346, "author_name": "albercik1905", "author_url": "", "post_date": "08/14/2015 08:39:43", "content": "So let me start by defining what we do normally. We check which variables are are in agreement and which are not. This is done even by looking at the distributions, not need to KS test. Then we pick the ones that look ok or reweigh them if we really really need to use them. One has to say for us the robustness of the classifier comes before the performance. When we designed the contests we left thous variables for you to discover on your own that they should not be used.
\n\nOf coz the tests we ask you to do here are tip of the iceberg. If we asked you guys to do all of them then the contest would have 10 participants. Some tests are dependent of what variables you use, etc. We decided to settle down for thous two simplest cross checks which already caused problem in understanding why we do them at all.
\n\nSo to answer your question in a simplest way I can I would say that the classifier you trained should super similar performance on Ds->phi pi as in tau->3mu. If this the case in your training?
\n\nSo when we did our classifer for the analysis. we first trained several classifiers picked the best one and looked at the performance and in the first goal we had a super DATA/MC agreement as we shown in the documentation. Furthermore, we had super similar performance on Ds->phi pi and tau->3mu(there are more tests but not relevant probably for this discussion). Does this a bit explain what are we aiming for?
\n\nSo if you really didn't use the Ds2phipi data for training your solution is ok with the rules. I had a look at your solution and run my private checks. Your solution will increase one of the solution systematic which is not desired in physics but it's out side the scope of this competition.
\n\nCheers,\nMarcin
", "votes": null, "replies": [] }, { "id": 89514, "author_name": "fchollet", "author_url": "", "post_date": "08/16/2015 16:08:10", "content": "Thanks, that was informative.
\n\nHere's my takeaway from what you are saying:
\n\nYou do have in fact much more information than the entrants, possibly to an extent that would allow you to recognize entries that exploit simulation leaks. However I doubt that is really the case, since if you had the means to do it you could simply erase simulation regularities from the simulated data, hence removing the need for the KS test and making the competition much more straightforward.
There are "shadow rules" in this competition ("Of coz the tests we ask you to do here are tip of the iceberg") that entries must comply with in order to be eligible to win. However we have no way to know what they are or if we are or not in compliance.
[quote]So if you really didn't use the Ds2phipi data for training your solution is ok with the rules. [/quote]
\n\nMy point is that the same results that can be achieved by training on the control data can also be achieved without explicitly training on the control data (hence the meaninglessness of the rule "only use training.csv"), via a number of possible tricks. So no, I did no train on the data from check_agreement.csv, but I might as well have since there is plenty of information about the control data that is leaking in my entry.
\n\nI'm pretty sure that my current entry is all about exploiting simulation leaks. If your own checks on my entry do not reveal that, then that confirms that you do not have the means to tell apart solutions that are grounded in physics from those that aren't.
", "votes": null, "replies": [] }, { "id": 89525, "author_name": "albercik1905", "author_url": "", "post_date": "08/16/2015 20:34:05", "content": "So it's next to impossible get ride of the "simulation regularities" as you called them. Once you fix them in one place they pop up in other places.
\n\nLet me comment about:\n"However we have no way to know what they are or if we are or not in compliance." \nWe do not ask you to do. This competition would be far to complicated to you guys to check everything (this takes months in some cases). Some check are only applicable only if certain variables are used in the training etc. That's why we decided for the 2 simplest ones.
\n\n"I'm pretty sure that my current entry is all about exploiting simulation leaks. If your own checks on my entry do not reveal that, then that confirms that you do not have the means to tell apart solutions that are grounded in physics from those that aren't."
\n\nProbably I wasn't clear. Our different checks reveled that. This is why I said: " Your solution will increase one of the solution systematic". The larger this systematic is the less discriminating power your classifier has after you do the full data calibration. Doing the calibration is CPU consuming and a bit more complicated so we didn't want to put this in the metric.
\n\nLet me stress once again that your solution if you really didn't use test in the training will be ok for this contests. In real world we wouldn't choose it but in kaggle competition we did not put the calibration on purpose and we knew that there might be this kind of solutions (that's why there is a physics prize).
\n\nbtw. I am super interested how did you manged to got that solution :)
", "votes": null, "replies": [] }, { "id": 89544, "author_name": "glouppe", "author_url": "", "post_date": "08/17/2015 06:45:33", "content": "\n\n\nLet me stress once again that your solution if you really didn't use test in the training will be ok for this contests.
\n
So let us assume that one can build a one-class classifier for training data, using training data and random samples only (for some definition of "random"). Then let us also assume one can build an overfit classifier for tau -> 3mu vs background. Is it allowed to combine these two classifiers at prediction time, either explicitly or implicitly?
\n\n(As far as the rules of the challenge are concerned, this appears to be strictly valid, even though it is useless.)
", "votes": null, "replies": [] }, { "id": 91650, "author_name": "vwoodnz", "author_url": "", "post_date": "09/05/2015 14:47:42", "content": "You don't even really need to explicitly "use an RNG". If you use lots of stochastic models, with different (fixed) seeds, then perform selection using the banned data - you'll do well and no one will be able to tell.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "88212": "To admins:\r\n\r\nI understand that it is not allowed to make any use of the following data in any capacity:\r\n\r\n- check_correlation.csv\r\n- check_agreement.csv\r\n- any \"external\" data\r\n\r\nAlthough the use of external data is disallowed in this competition, is it admissible to make use of a self-contained source of data in a model, such as the data produced by a random number generator?\r\n\r\nThe reason I am asking this is that I have a theoretical solution that would allow to train a model that could entirely by-pass the agreement test, without ever looking at check_agreement.csv. This would allow to leverage the simulation defects (that are leaking the labels) as much as possible while having a low KS score. This model would score very high (although it would be useless from a physics perspective), while respecting the basic rules such as:\r\n\r\n- no use of check_agreement.csv whatsoever\r\n- all events are treated in the same way as if they are from the same channel\r\n\r\nThis solution relies only on the use of a random number generator and a handful of hard-coded hyperparameters.\r\n\r\nSo, are random number generators (and hard-coded hyperparameters) allowed in this competition?\r\n\r\nOf course, it is only half a real question, since RNGs are common in many machine learning algorithms and disallowing them would be... strange, to say the least. So you could rephrase my question as: are you willing to address the core issues with this competition? \r\n\r\nAdding new rules will NOT be sufficient to prevent entrants from finding ways to exploit simulation leaks while still passing the KS test. Exploiting these flaws is *by design* the only way to perform competitively in this challenge. I can promise you that the top-performing models (including mine if I don't quit right now) will *not* be useful to physics, while still managing to comply with your rules. You are not getting what you are looking for, and you will need to revise your basics assumptions if you want this competition to align with physics usefulness.\r\n\r\nThank you.", "89279": "Unfortunately, this post (as well as many others) has been completely ignored by the competition admins. If you think I'm wrong, please tell me so, but ignoring these questions is a demonstration of total contempt for the people who voluntarily spend many hours of their time working to solve your problems.\r\n\r\nIn any case, I just implemented the technique, and got 0.990680 on the LB on my first try (2nd place, could easily be tweaked to reach 1st place). As far as I can tell, the approach is within the (current) rules since it only makes use of the data from training.csv.\r\n\r\nThe approach is also uninteresting from a physics standpoint, since it is merely exploiting simulation leaks while passing the KS test. Then again, any approach that performs highly in this competition will necessarily be based on the exploitation of simulation leaks. That's just how the competition is set up. No matter how much one might want to develop a physics-based solution, it is not possible, because the data we have does not allow to tell apart solutions that are physics based from solutions based on exploiting simulation leaks. Both are identical from an information-theoretical standpoint, given the data available. \r\n\r\nIf the competition admins want to make my approach invalid, I recommend the following rule change: disallow random number generators. Until the next exploit, and the next rule change. \r\n\r\nAlternatively, you could consider admitting your mistakes, and restarting the competition with new data and saner rules. In doing so, you should consult with entrants in the current competition to avoid repeating the same mistakes. There are people here who are willing to help you. For free. Just out of desire to make things right. Then again, given how utterly unresponsive to concerns and questions the competition admins have been, there is not a chance that it will happen. But one can dream. \r\n\r\nI've seen a few failed competitions on Kaggle in the past (launching data science competitions is hard work), but this one takes the cake. Not because the mistakes made were particularly obvious (they certainly weren't, and it took weeks to fully identify them) but because of the complete mismanagement of the competition after the issues were pointed out. \r\n\r\nToo bad. I'll shut up now, since I am obviously just wasting my time.", "89296": "Sorry, I missed your precious post. So you can of coz use random numbers. We will ask you in the solution to give the motivation, how the numbers were obtained and how we can add them to future data. We will check if the numbers are causing some differences between the calibration channel and signal channel. \r\n\r\nLet me comment on:\r\nExploiting these flaws is by design the only way to perform competitively in this challenge.\r\n\r\nThe point of the challenge is not to get an classifier that is completely useless because it treats signal channel differently than control channel. If we wanted to that kind of things obviously we wouldn't waste time on control channel at all. There are people in the top10 that have a valid classifier trained without using the control channel so that can be applied in physics application.", "89297": "[quote=Albercik1905;89296]\r\n\r\nSorry, I missed your precious post. So you can of coz use random numbers. We will ask you in the solution to give the motivation, how the numbers were obtained and how we can add them to future data. We will check if the numbers are causing some differences between the calibration channel and signal channel. \r\n\r\nLet me comment on:\r\nExploiting these flaws is by design the only way to perform competitively in this challenge.\r\n\r\nThe point of the challenge is not to get an classifier that is completely useless because it treats signal channel differently than control channel. If we wanted to that kind of things obviously we wouldn't waste time on control channel at all. There are people in the top10 that have a valid classifier trained without using the control channel so that can be applied in physics application. \r\n\r\n\r\n\r\n[/quote]\r\n\r\nWhen reviewing somebody's solution, you will *not* be able to tell whether it is grounded in physics or whether it is exploiting simulation artifacts, because the two are completely identical in this challenge. \r\n\r\nThe winners in this challenge will be entrants who find features that successfully separate signal from background while yielding a low KS value. Two facts about these features:\r\n\r\n- the only way to find these features is by using the data from check_agreement.csv. However this can be done implicitly, without every looking at that data during the training process. If you are not using check_agreement.csv to make sure your process passes the KS test, then what, random chance? \r\n- it is impossible to tell apart features that are simply good at picking up simulation artifacts while discriminating control events (which have a different distribution), from features that do not leverage simulation artifacts. The two are strictly identical given the data we have. The tell them part you would need to know what simulation artifacts are present, and that cannot be seen in the data we have access to. The KS test *does not* control for the use of simulation artifacts, so passing the KS test tells you nothing.\r\n\r\n[quote]\r\nThe point of the challenge is not to get an classifier that is completely useless because it treats signal channel differently than control channel. If we wanted to that kind of things obviously we wouldn't waste time on control channel at all. [/quote] \r\n\r\nMy classifier treats all data points in the same way. Yet it is useless. The control channel does not do what you seem to think it does. \r\n\r\nMy point about exploiting the simulation flaws in this challenge is this: there is no way *not* to do it, and those doing it will not realize their are doing it. As far as they know their solutions might well be grounded in actual physics. Physics-based solutions and solutions that exploit simulation artifacts look strictly identical in this challenge, for entrants, and for you. They are simply the solutions that are picking up the \"right\" features: features that discriminate signal from background while passing the KS test.\r\n\r\nCan you define in concrete terms what a valid solution will look like? I 100% believe that my current solution fits the bill.", "89346": "So let me start by defining what we do normally. We check which variables are are in agreement and which are not. This is done even by looking at the distributions, not need to KS test. Then we pick the ones that look ok or reweigh them if we really really need to use them. One has to say for us the robustness of the classifier comes before the performance. When we designed the contests we left thous variables for you to discover on your own that they should not be used. \r\n\r\nOf coz the tests we ask you to do here are tip of the iceberg. If we asked you guys to do all of them then the contest would have 10 participants. Some tests are dependent of what variables you use, etc. We decided to settle down for thous two simplest cross checks which already caused problem in understanding why we do them at all. \r\n\r\nSo to answer your question in a simplest way I can I would say that the classifier you trained should super similar performance on Ds->phi pi as in tau->3mu. If this the case in your training?\r\n\r\nSo when we did our classifer for the analysis. we first trained several classifiers picked the best one and looked at the performance and in the first goal we had a super DATA/MC agreement as we shown in the documentation. Furthermore, we had super similar performance on Ds->phi pi and tau->3mu(there are more tests but not relevant probably for this discussion). Does this a bit explain what are we aiming for?\r\n\r\nSo if you really didn't use the Ds2phipi data for training your solution is ok with the rules. I had a look at your solution and run my private checks. Your solution will increase one of the solution systematic which is not desired in physics but it's out side the scope of this competition.\r\n\r\nCheers,\r\nMarcin", "89514": "Thanks, that was informative. \r\n\r\nHere's my takeaway from what you are saying:\r\n\r\n- You do have in fact much more information than the entrants, possibly to an extent that would allow you to recognize entries that exploit simulation leaks. However I doubt that is really the case, since if you had the means to do it you could simply erase simulation regularities from the simulated data, hence removing the need for the KS test and making the competition much more straightforward.\r\n\r\n- There are \"shadow rules\" in this competition (\"*Of coz the tests we ask you to do here are tip of the iceberg*\") that entries must comply with in order to be eligible to win. However we have no way to know what they are or if we are or not in compliance.\r\n\r\n[quote]So if you really didn't use the Ds2phipi data for training your solution is ok with the rules. [/quote]\r\n\r\nMy point is that the same results that can be achieved by training on the control data can also be achieved without explicitly training on the control data (hence the meaninglessness of the rule \"only use training.csv\"), via a number of possible tricks. So no, I did no train on the data from check_agreement.csv, but I might as well have since there is plenty of information about the control data that is leaking in my entry. \r\n\r\nI'm pretty sure that my current entry is all about exploiting simulation leaks. If your own checks on my entry do not reveal that, then that confirms that you do not have the means to tell apart solutions that are grounded in physics from those that aren't.", "89525": "So it's next to impossible get ride of the \"simulation regularities\" as you called them. Once you fix them in one place they pop up in other places.\r\n\r\nLet me comment about:\r\n\"However we have no way to know what they are or if we are or not in compliance.\" \r\nWe do not ask you to do. This competition would be far to complicated to you guys to check everything (this takes months in some cases). Some check are only applicable only if certain variables are used in the training etc. That's why we decided for the 2 simplest ones.\r\n\r\n\"I'm pretty sure that my current entry is all about exploiting simulation leaks. If your own checks on my entry do not reveal that, then that confirms that you do not have the means to tell apart solutions that are grounded in physics from those that aren't.\"\r\n\r\nProbably I wasn't clear. Our different checks reveled that. This is why I said: \" Your solution will increase one of the solution systematic\". The larger this systematic is the less discriminating power your classifier has after you do the full data calibration. Doing the calibration is CPU consuming and a bit more complicated so we didn't want to put this in the metric.\r\n\r\nLet me stress once again that your solution if you really didn't use test in the training will be ok for this contests. In real world we wouldn't choose it but in kaggle competition we did not put the calibration on purpose and we knew that there might be this kind of solutions (that's why there is a physics prize). \r\n\r\nbtw. I am super interested how did you manged to got that solution :)", "89544": "> Let me stress once again that your solution if you really didn't use test in the training will be ok for this contests. \r\n\r\nSo let us assume that one can build a one-class classifier for training data, using training data and random samples only (for some definition of \"random\"). Then let us also assume one can build an overfit classifier for tau -> 3mu vs background. Is it allowed to combine these two classifiers at prediction time, either explicitly or implicitly? \r\n\r\n(As far as the rules of the challenge are concerned, this appears to be strictly valid, even though it is useless.)", "91650": "You don't even really need to explicitly \"use an RNG\". If you use lots of stochastic models, with different (fixed) seeds, then perform selection using the banned data - you'll do well and no one will be able to tell." }, "source": "meta" }