Dear participants,
\n\nAs the competition progresses rapidly, we’d like to draw your attention to the data to be used for the purpose of training your model. It is mentioned in the data page, but we’d like it to be a bit more explicit. To be eligible for the prize your final solution should rely solely on the data provided in ‘training.csv’ without any usage of ‘check_correlation.csv' or ‘check_agreement.csv' datasets.
\n\nThe main motivation behind this restriction is that we want to make this competition as useful as possible from physics application, and having models that fit to ‘agreement_check’ data just to pass agreement test and at the same time behave quite differently on the signal channel is in general meaningless from physics point of view, but rather easy to achieve ‘accidentally' by adding ‘check_correlation.csv' or ‘check_agreement.csv' into the training process.
\n\nThe idea behind the agreement test is to check if the model takes advantage of the fact that there are differences between the samples obtained from simulation and the real data. These are due to imperfections in the simulation and a classifier useful for a physics analysis should not take advantage of the differences between samples from the same class that arise from these imperfections.
\n\nPS. We would like to encourage everyone to submit descriptions of their solutions upon the challenge finish, since even the lower-ranked solution might be useful for physicists in other analyses. In particular, approaches that are not main stream will certainly receive attention from CERN researchers and their authors could be invited to one of the post-competition ML-HEP workshops.
", "messages": [ { "id": "88129", "postDate": "08/03/2015 18:32:40", "content": "Dear participants,
\n\nAs the competition progresses rapidly, we’d like to draw your attention to the data to be used for the purpose of training your model. It is mentioned in the data page, but we’d like it to be a bit more explicit. To be eligible for the prize your final solution should rely solely on the data provided in ‘training.csv’ without any usage of ‘check_correlation.csv' or ‘check_agreement.csv' datasets.
\n\nThe main motivation behind this restriction is that we want to make this competition as useful as possible from physics application, and having models that fit to ‘agreement_check’ data just to pass agreement test and at the same time behave quite differently on the signal channel is in general meaningless from physics point of view, but rather easy to achieve ‘accidentally' by adding ‘check_correlation.csv' or ‘check_agreement.csv' into the training process.
\n\nThe idea behind the agreement test is to check if the model takes advantage of the fact that there are differences between the samples obtained from simulation and the real data. These are due to imperfections in the simulation and a classifier useful for a physics analysis should not take advantage of the differences between samples from the same class that arise from these imperfections.
\n\nPS. We would like to encourage everyone to submit descriptions of their solutions upon the challenge finish, since even the lower-ranked solution might be useful for physicists in other analyses. In particular, approaches that are not main stream will certainly receive attention from CERN researchers and their authors could be invited to one of the post-competition ML-HEP workshops.
", "rawMarkdown": "Dear participants,\r\n\r\nAs the competition progresses rapidly, we’d like to draw your attention to the data to be used for the purpose of training your model. It is mentioned in the [data page][1], but we’d like it to be a bit more explicit. **To be eligible for the prize your final solution should rely solely on the data provided in ‘training.csv’ without any usage of ‘check_correlation.csv' or ‘check_agreement.csv' datasets.**\r\n\r\nThe main motivation behind this restriction is that we want to make this competition as useful as possible from physics application, and having models that fit to ‘agreement_check’ data just to pass agreement test and at the same time behave quite differently on the signal channel is in general meaningless from physics point of view, but rather easy to achieve ‘accidentally' by adding ‘check_correlation.csv' or ‘check_agreement.csv' into the training process.\r\n\r\nThe idea behind the agreement test is to check if the model takes advantage of the fact that there are differences between the samples obtained from simulation and the real data. These are due to imperfections in the simulation and a classifier useful for a physics analysis should not take advantage of the differences between samples from the same class that arise from these imperfections.\r\n\r\nPS. We would like to encourage everyone to submit descriptions of their solutions upon the challenge finish, since even the lower-ranked solution might be useful for physicists in other analyses. In particular, approaches that are not main stream will certainly receive attention from CERN researchers and their authors could be invited to one of the [post-competition ML-HEP workshops][2].\r\n\r\n\r\n [1]: https://www.kaggle.com/c/flavours-of-physics/data\r\n [2]: https://www.kaggle.com/c/flavours-of-physics/details/prizes", "votes": null }, { "id": "88150", "postDate": "08/03/2015 20:21:34", "content": "Thank you for the clarification but I could not see any reference to this explicit limitation anywhere in your original description of the problem.\nCan you please give an explicit quote from the original description of the problem.\nChanging the rules while the competition takes place does not add much fun and undermine the entire value of a site like kaggle.
\n\nAs an off-topic, I think there is real value in taking other existing channels into account when modeling. In a real situation they do add to the noise and modeling all the possible noise is one way of improving the signal.
", "rawMarkdown": "Thank you for the clarification but I could not see any reference to this explicit limitation anywhere in your original description of the problem.\r\nCan you please give an explicit quote from the original description of the problem.\r\nChanging the rules while the competition takes place does not add much fun and undermine the entire value of a site like kaggle.\r\n\r\nAs an off-topic, I think there is real value in taking other existing channels into account when modeling. In a real situation they do add to the noise and modeling all the possible noise is one way of improving the signal.", "votes": null }, { "id": "88151", "postDate": "08/03/2015 20:27:56", "content": "Under our Data page:\n"You need to submit predictions for ALL the test entries. You will need to treat them all the same and predict as if they are all the same channel's collision events. "
", "rawMarkdown": "Under our [Data page][1]:\r\n\"You need to submit predictions for ALL the test entries. You will need to treat them all the same and predict as if they are all the same channel's collision events. \"\r\n\r\n\r\n [1]: https://www.kaggle.com/c/flavours-of-physics/data", "votes": null }, { "id": "88155", "postDate": "08/03/2015 20:59:12", "content": "udibr, you are right, there might be a real value in taking other channels into account and we do not deny people will find how find that value, that's why we have Special Physics prize in this challenge and a couple of workshops (at NIPS (if confirmed) and at Zurich) to discuss these approaches.
", "rawMarkdown": "udibr, you are right, there might be a real value in taking other channels into account and we do not deny people will find how find that value, that's why we have Special Physics prize in this challenge and a couple of workshops (at NIPS (if confirmed) and at Zurich) to discuss these approaches.", "votes": null }, { "id": "88157", "postDate": "08/03/2015 20:59:59", "content": "I am very sorry, but, perhaps I am wrong but I think that what you quoted is not relevant.\nYour quote refers to how predictions are made. I have a single prediction algorithm that runs on all examples in exactly the same way, so it fits your description of "same channel".\nThe only question is how this algorithm was trained in the first place.
\n\nAs you can see for yourself the quote you have taken is from the part talking about the test process and not about the training process.
\n\nIn any case you should refer to this page https://www.kaggle.com/c/flavours-of-physics/rules\nand not to any other page of the competition when specifying rules.\nThis is because the rules page (which I had to accept when downloading the data and no other page) begins with "These are the complete, official rules for the Competition..."
\n\nThe rules page states that "Use of external data is not permitted" and that "'Data' means the Data or Datasets linked from the Competition Website for the purpose of use by Participants in the Competition."\nthis covers all the csv files.
\n\nI would also like to add that developing methods that uses the other csv files will help the process of discovering new physics.\nThis is because the main challenge in this competition is not finding new physics now but building tools for doing it in the future and perhaps the most important tool will be an accurate simulator and test sets that take any adversary action into account. If the simulator is not accurate enough to describe the "real" physics then it needs to be improved and if the gaps it has were find in this competition then all the better.
", "rawMarkdown": "I am very sorry, but, perhaps I am wrong but I think that what you quoted is not relevant.\r\nYour quote refers to how predictions are made. I have a single prediction algorithm that runs on all examples in exactly the same way, so it fits your description of \"same channel\".\r\nThe only question is how this algorithm was trained in the first place.\r\n\r\nAs you can see for yourself the quote you have taken is from the part talking about the test process and not about the training process.\r\n\r\nIn any case you should refer to this page https://www.kaggle.com/c/flavours-of-physics/rules\r\nand not to any other page of the competition when specifying rules.\r\nThis is because the rules page (which I had to accept when downloading the data and no other page) begins with \"These are the complete, official rules for the Competition...\"\r\n\r\nThe rules page states that \"Use of external data is not permitted\" and that \"'Data' means the Data or Datasets linked from the Competition Website for the purpose of use by Participants in the Competition.\"\r\nthis covers all the csv files.\r\n\r\nI would also like to add that developing methods that uses the other csv files will help the process of discovering new physics.\r\nThis is because the main challenge in this competition is not finding new physics now but building tools for doing it in the future and perhaps the most important tool will be an accurate simulator and test sets that take any adversary action into account. If the simulator is not accurate enough to describe the \"real\" physics then it needs to be improved and if the gaps it has were find in this competition then all the better.", "votes": null }, { "id": "88163", "postDate": "08/03/2015 21:39:09", "content": "[quote]\nTo be eligible for the prize your final solution should rely solely on the data provided in ‘training.csv’ without any usage of ‘check_correlation.csv' or ‘check_agreement.csv' datasets.\n[/quote]
\n\nDoes this means we cannot run the KS and correlation tests locally? If so, then why are these datasets provided in the first place?
\n\nIt seems that what you are trying to say is, "don't train your model using these datasets". But of course, selecting a model based on whether it passes the KS test is the exact same as using the dataset ‘check_correlation.csv' to train your model. This much should be obvious to anyone.
\n\nJust in case it isn't obvious for you, here's a thought experiment: the no-training model. Your set up a logistic regression model to classify signal from background, but instead of learning the parameters through gradient descent, you randomly generate them. Then you check whether your random model passes the KS test and correlation test, and you check what the ROC AUC is. You do this with 1,000,000 models (very much realistic and computationally tractable) and you pick the model with the highest ROC AUC. This is in fact a solid strategy if you don't feel like computing the gradients of the ROC AUC and KS scores.
\n\nThis model never ever saw the data in ‘check_correlation.csv' (or any training data, for that matter), but it was still effectively trained on it... through post-selection. ANY USE of ‘check_correlation.csv' means that you are training your model (i.e. selecting its parameters) based on the data in ‘check_correlation.csv'. If you are running the KS test locally, you are breaking the rules. But if you are not, you are insane (enjoy submitting your .csv 100 times in a row).
\n\nBy the way, the same is also true with the test happening on the Kaggle servers. As long as you are selecting models based on whether they pass the KS test, then these models are in effect trained on the KS data (i.e. the parameters of your final model have been chosen based on the KS data). There is no way around it whatsoever.
\n\n[quote=udibr;88157]\nI have a single prediction algorithm that runs on all examples in\nexactly the same way, so it fits your description of "same channel".\n[/quote]
\n\nSo do I (no special case for processing the control data), and as you do I also leverage the KS data for training. Also, I can't help but remark that what we are doing inevitably results in models that pick up simulation defects while still managing to work around the KS test. It's what we optimize for.
\n\nThis competition is broken in a fundamental way. The organizers will have to backpedal, and possibly cancel the competition altogether, if their goal is a model that is useful from a physics perspective. No such model is going to emerge in the current setup.
", "rawMarkdown": "[quote]\r\nTo be eligible for the prize your final solution should rely solely on the data provided in ‘training.csv’ without any usage of ‘check_correlation.csv' or ‘check_agreement.csv' datasets.\r\n[/quote]\r\n\r\nDoes this means we cannot run the KS and correlation tests locally? If so, then why are these datasets provided in the first place?\r\n\r\nIt seems that what you are trying to say is, \"don't train your model using these datasets\". But of course, selecting a model based on whether it passes the KS test is the exact same as using the dataset ‘check_correlation.csv' to train your model. This much should be obvious to anyone.\r\n\r\nJust in case it isn't obvious for you, here's a thought experiment: the no-training model. Your set up a logistic regression model to classify signal from background, but instead of learning the parameters through gradient descent, you randomly generate them. Then you check whether your random model passes the KS test and correlation test, and you check what the ROC AUC is. You do this with 1,000,000 models (very much realistic and computationally tractable) and you pick the model with the highest ROC AUC. This is in fact a solid strategy if you don't feel like computing the gradients of the ROC AUC and KS scores.\r\n\r\nThis model never ever saw the data in ‘check_correlation.csv' (or any training data, for that matter), but it was still effectively trained on it... through post-selection. ANY USE of ‘check_correlation.csv' means that you are training your model (i.e. selecting its parameters) based on the data in ‘check_correlation.csv'. If you are running the KS test locally, you are breaking the rules. But if you are not, you are insane (enjoy submitting your .csv 100 times in a row).\r\n\r\nBy the way, the same is also true with the test happening on the Kaggle servers. As long as you are selecting models based on whether they pass the KS test, then these models are in effect trained on the KS data (i.e. the parameters of your final model have been chosen based on the KS data). There is no way around it whatsoever. \r\n\r\n[quote=udibr;88157]\r\nI have a single prediction algorithm that runs on all examples in\r\nexactly the same way, so it fits your description of \"same channel\".\r\n[/quote]\r\n\r\nSo do I (no special case for processing the control data), and as you do I also leverage the KS data for training. Also, I can't help but remark that what we are doing inevitably results in models that pick up simulation defects while still managing to work around the KS test. It's what we optimize for. \r\n\r\nThis competition is broken in a fundamental way. The organizers will have to backpedal, and possibly cancel the competition altogether, if their goal is a model that is useful from a physics perspective. No such model is going to emerge in the current setup.", "votes": null }, { "id": "88165", "postDate": "08/03/2015 21:48:36", "content": "I find no ambiguity in the statement
\n\n\n\n\nTo be eligible for the prize your final solution should rely solely on\n the data provided in ‘training.csv’ without any usage of\n ‘check_correlation.csv' or ‘check_agreement.csv' datasets.
\n
I think you are just taking the statement to its literary absurd extreme. Do not train models on these datasets. You can use them to evaluate that your models meets the competition goals and that is why they were provided.
\n\nI myself experimented using these datasets in my training (as in use them to train models) but I figured that was not in the spirit of the comp so I did not submit any of those experiments. This was before clarification above was made so I think this statement was implied (softly whispered) even before the clarification in this topic.
", "rawMarkdown": "I find no ambiguity in the statement\r\n\r\n> To be eligible for the prize your final solution should rely solely on\r\n> the data provided in ‘training.csv’ without any usage of\r\n> ‘check_correlation.csv' or ‘check_agreement.csv' datasets.\r\n\r\nI think you are just taking the statement to its literary absurd extreme. Do not train models on these datasets. You can use them to evaluate that your models meets the competition goals and that is why they were provided.\r\n\r\nI myself experimented using these datasets in my training (as in use them to train models) but I figured that was not in the spirit of the comp so I did not submit any of those experiments. This was before clarification above was made so I think this statement was implied (softly whispered) even before the clarification in this topic.", "votes": null }, { "id": "88166", "postDate": "08/03/2015 21:56:38", "content": "Also, just FYI: the control data and test data have significantly different distributions, so it is possible to by-pass the KS test entirely without ever looking at the KS data, simply by setting up a 1-class classifier (output: "belongs to the class"/"doesn't belong"). Only data points that have the same distribution as the data in training.csv would be classified, others would get a constant prediction (for instance). KS test solved. Such a classifier can be obtained by using randomly generated data as negative examples (no need for the KS data).
\n\nThink this process would violate the rules? Well, it is also possible to set up a signal/BG model that implicitly do the job of this classifier, without any explicit special case for KS control data.
", "rawMarkdown": "Also, just FYI: the control data and test data have significantly different distributions, so it is possible to by-pass the KS test entirely without ever looking at the KS data, simply by setting up a 1-class classifier (output: \"belongs to the class\"/\"doesn't belong\"). Only data points that have the same distribution as the data in training.csv would be classified, others would get a constant prediction (for instance). KS test solved. Such a classifier can be obtained by using randomly generated data as negative examples (no need for the KS data).\r\n\r\nThink this process would violate the rules? Well, it is also possible to set up a signal/BG model that implicitly do the job of this classifier, without any explicit special case for KS control data.", "votes": null }, { "id": "88167", "postDate": "08/03/2015 21:57:44", "content": "[quote=Guido Tapia;88165]\ntaking the statement to its literary absurd extreme.\n[/quote]\nI think fchollet is right. If we take "training a model" to mean "using information from" then any locally performed check adds information to your model selection pipeline. No matter how indirectly or directly, purposeful or accidental, this information will ultimately leak and end up into your submissions.
", "rawMarkdown": "[quote=Guido Tapia;88165]\r\ntaking the statement to its literary absurd extreme.\r\n[/quote]\r\nI think fchollet is right. If we take \"training a model\" to mean \"using information from\" then any locally performed check adds information to your model selection pipeline. No matter how indirectly or directly, purposeful or accidental, this information will ultimately leak and end up into your submissions.", "votes": null }, { "id": "88168", "postDate": "08/03/2015 22:01:29", "content": "[quote=Guido Tapia;88165]
\n\nI think you are just taking the statement to its literary absurd extreme. Do not train models on these datasets. You can use them to evaluate that your models meets the competition goals and that is why they were provided.
\n\n[/quote]
\n\nYou seem to misunderstand what "training a model" means. It does not mean calling the "fit" method on a sklearn classifier. It means the process of selecting the parameters of your model. Any usage of the KS data to select parameters, hyperparameters or even the type of classifier to use means that you are training your solution on the KS data.
\n\nYou know how everybody tends to overfit to the leaderboard even though we don't have access to the labels of the leaderboard? This is the same. When you are selecting any aspect of your model based on some score, it means that data used to generate this score is part of your training data.
", "rawMarkdown": "[quote=Guido Tapia;88165]\r\n\r\nI think you are just taking the statement to its literary absurd extreme. Do not train models on these datasets. You can use them to evaluate that your models meets the competition goals and that is why they were provided.\r\n\r\n[/quote]\r\n\r\nYou seem to misunderstand what \"training a model\" means. It does not mean calling the \"fit\" method on a sklearn classifier. It means the process of selecting the parameters of your model. Any usage of the KS data to select parameters, hyperparameters or even the type of classifier to use means that you are training your solution on the KS data.\r\n\r\nYou know how everybody tends to overfit to the leaderboard even though we don't have access to the labels of the leaderboard? This is the same. When you are selecting any aspect of your model based on some score, it means that data used to generate this score is part of your training data.", "votes": null }, { "id": "88169", "postDate": "08/03/2015 22:14:11", "content": "I agree people will over fit the validation datasets as they will the public leader board. But I think its a compromise that was understood and accepted. And hopefully, even with these imperfect constraints some valuable insights can be learnt.
\n\nI think this is a very complex problem and we only have the current set of tools and platforms to solve the problem, lets make the best of it. I just think suggesting the competition be scrapped because its not perfect is the wrong approach.
\n\nI'll keep going, if KS metric gets too frustrating I'll go back to caterpillar which I left because I found the craziness in the data set structure too frustrating. Again no competition is perfect and if it is then you get 3000 competitors submitting each others scripts and filling the forums with noise.
", "rawMarkdown": "I agree people will over fit the validation datasets as they will the public leader board. But I think its a compromise that was understood and accepted. And hopefully, even with these imperfect constraints some valuable insights can be learnt. \r\n\r\nI think this is a very complex problem and we only have the current set of tools and platforms to solve the problem, lets make the best of it. I just think suggesting the competition be scrapped because its not perfect is the wrong approach. \r\n\r\nI'll keep going, if KS metric gets too frustrating I'll go back to caterpillar which I left because I found the craziness in the data set structure too frustrating. Again no competition is perfect and if it is then you get 3000 competitors submitting each others scripts and filling the forums with noise.", "votes": null }, { "id": "88263", "postDate": "08/04/2015 08:11:12", "content": "Isn't the "Agreement test" meant as an estimate of the systematic error?\nThus, if you use it to train, it messes up just that?
\n\nEdit: removed rant about MC
", "rawMarkdown": "Isn't the \"Agreement test\" meant as an estimate of the systematic error?\r\nThus, if you use it to train, it messes up just that?\r\n\r\nEdit: removed rant about MC", "votes": null }, { "id": "88326", "postDate": "08/04/2015 11:21:39", "content": "As Gilles hasn't been removed from the leaderboard, I take it that he is still eligible for the Kaggle ranking points?
", "rawMarkdown": "As Gilles hasn't been removed from the leaderboard, I take it that he is still eligible for the Kaggle ranking points?", "votes": null }, { "id": "88328", "postDate": "08/04/2015 11:26:33", "content": "[quote=Bluefool;88326]
\n\nAs Gilles hasn't been removed from the leaderboard, I take it that he is still eligible for the Kaggle ranking points?
\n\n[/quote]
\n\nThere will be people in the end using the agreement test dataset just to gain Kaggle points. How do you plan to remove them? ;)
", "rawMarkdown": "[quote=Bluefool;88326]\r\n\r\nAs Gilles hasn't been removed from the leaderboard, I take it that he is still eligible for the Kaggle ranking points?\r\n\r\n[/quote]\r\n\r\nThere will be people in the end using the agreement test dataset just to gain Kaggle points. How do you plan to remove them? ;)", "votes": null }, { "id": "88329", "postDate": "08/04/2015 11:26:58", "content": "\n\n\nAs Gilles hasn't been removed from the leaderboard, I take it that he is still eligible for the Kaggle ranking points?
\n
Honestly, I couldnt care less about the Kaggle points, but I understand this may be important for some. I dont know how this works, but upon inspection of the submissions, only those deemed as valid get points, right?
", "rawMarkdown": "> As Gilles hasn't been removed from the leaderboard, I take it that he is still eligible for the Kaggle ranking points?\r\n\r\nHonestly, I couldnt care less about the Kaggle points, but I understand this may be important for some. I dont know how this works, but upon inspection of the submissions, only those deemed as valid get points, right?", "votes": null }, { "id": "88330", "postDate": "08/04/2015 11:28:05", "content": "Nope, you'll get the points
", "rawMarkdown": "Nope, you'll get the points", "votes": null }, { "id": "88332", "postDate": "08/04/2015 11:30:04", "content": "[quote=Abhishek;88328]
\n\n[quote=Bluefool;88326]
\n\nAs Gilles hasn't been removed from the leaderboard, I take it that he is still eligible for the Kaggle ranking points?
\n\n[/quote]
\n\nThere will be people in the end using the agreement test dataset just to gain Kaggle points. How do you plan to remove them? ;)
\n\n[/quote]
\n\nThat's my point - we might as well all just use the agreement test.
", "rawMarkdown": "[quote=Abhishek;88328]\r\n\r\n[quote=Bluefool;88326]\r\n\r\nAs Gilles hasn't been removed from the leaderboard, I take it that he is still eligible for the Kaggle ranking points?\r\n\r\n[/quote]\r\n\r\nThere will be people in the end using the agreement test dataset just to gain Kaggle points. How do you plan to remove them? ;)\r\n\r\n[/quote]\r\n\r\nThat's my point - we might as well all just use the agreement test.", "votes": null }, { "id": "88362", "postDate": "08/04/2015 13:48:48", "content": "Gilles,\nYou can also take the position saying that your final submission was made before the rules were changed. Therefore it remains valid.
", "rawMarkdown": "Gilles,\r\nYou can also take the position saying that your final submission was made before the rules were changed. Therefore it remains valid.", "votes": null }, { "id": "88421", "postDate": "08/04/2015 20:06:01", "content": "Sorry, but I have another question about this rule. In physics it is fairly common to actually correct for discrepancies in a particular variable between data and simulation by re-weighting events. So, for example, if you saw that in a particular bin of pT in the control region, data was 20% higher than MC, you might decide to re-weight the training events in that bin by a factor of 1.2. (The idea being that you are re-weighting events so that the distribution in data and MC for that variable matches). This is not training directly on the data in the control region, but it is using that data to improve the description of the input variables in simulation. Would an approach like this be allowed?
\n\nI certainly think it could be physically useful. If the simulation is getting the description of a variable wrong, and you know that from the data, you should be able to correct for it. You aren't using this mis-modeling for discrimination in that case. You're just improving your training sample to be more like the real world.
", "rawMarkdown": "Sorry, but I have another question about this rule. In physics it is fairly common to actually correct for discrepancies in a particular variable between data and simulation by re-weighting events. So, for example, if you saw that in a particular bin of pT in the control region, data was 20% higher than MC, you might decide to re-weight the training events in that bin by a factor of 1.2. (The idea being that you are re-weighting events so that the distribution in data and MC for that variable matches). This is not training directly on the data in the control region, but it is using that data to improve the description of the input variables in simulation. Would an approach like this be allowed? \r\n\r\nI certainly think it could be physically useful. If the simulation is getting the description of a variable wrong, and you know that from the data, you should be able to correct for it. You aren't using this mis-modeling for discrimination in that case. You're just improving your training sample to be more like the real world.", "votes": null }, { "id": "88519", "postDate": "08/05/2015 13:14:23", "content": "Yep, we often make the reweighting procedure. But in this case you should correctly apply reweighting to necessary datasets and then compute the ks-metric, the cvm-metric and the weighted AUC with obtained weights. This we cannot support in the competition because of an inability to control right modifications of the datasets by participants. If you will reweight correctly and your solution will be interesting from physics point of view, you will be able to win special physics prize (of course your quality on LB won't be correct and can be higher, that is why you should present in your solution weights which will be used to obtain the correct quality).
\n\nCheers,\nMarcin
", "rawMarkdown": "Yep, we often make the reweighting procedure. But in this case you should correctly apply reweighting to necessary datasets and then compute the ks-metric, the cvm-metric and the weighted AUC with obtained weights. This we cannot support in the competition because of an inability to control right modifications of the datasets by participants. If you will reweight correctly and your solution will be interesting from physics point of view, you will be able to win special physics prize (of course your quality on LB won't be correct and can be higher, that is why you should present in your solution weights which will be used to obtain the correct quality). \r\n\r\nCheers,\r\nMarcin", "votes": null }, { "id": "88575", "postDate": "08/05/2015 19:51:16", "content": "Am I correct in the following assumptions?
\n\n1) In the baseline notebook that the admin supplied. It is said:
\n\n\n\n\nHere we use subset of the all features to pass the agreement checking
\n
Am I allowed to perform feature selection based on running local control test?
\n\n2) In the noise notebook it is said:
\n\n\n\n\nAdd noise for the agreement checking
\n
So am I correct I am allowed to "Manually" add steps to my algorithm until I manage to pass the control test?
\n\nIf the answer to both questions is yes then what am I not allowed to perform?
", "rawMarkdown": "Am I correct in the following assumptions?\r\n\r\n1) In the [baseline notebook][1] that the admin supplied. It is said:\r\n\r\n> Here we use subset of the all features to pass the agreement checking\r\n\r\nAm I allowed to perform feature selection based on running local control test?\r\n\r\n2) In the [noise notebook][2] it is said:\r\n\r\n> Add noise for the agreement checking\r\n\r\nSo am I correct I am allowed to \"Manually\" add steps to my algorithm until I manage to pass the control test?\r\n\r\nIf the answer to both questions is yes then what am I not allowed to perform?\r\n\r\n [1]: https://github.com/yandexdataschool/flavours-of-physics-start/blob/master/baseline.ipynb\r\n [2]: https://github.com/yandexdataschool/flavours-of-physics-start/blob/master/noise.ipynb", "votes": null }, { "id": "88577", "postDate": "08/05/2015 20:11:26", "content": "It's all a bit confusing. I had some feature selection running with the checks built in but I stopped it. Now I'm not sure. At the moment, I'm only using checks before I submit, but even then I'll be selecting my models based on if they pass or not!
", "rawMarkdown": "It's all a bit confusing. I had some feature selection running with the checks built in but I stopped it. Now I'm not sure. At the moment, I'm only using checks before I submit, but even then I'll be selecting my models based on if they pass or not!", "votes": null }, { "id": "88606", "postDate": "08/06/2015 00:26:51", "content": "Hi,\n Are we allowed to use 'test.csv' for unsupervised learning and/or PCA given the new rules?\nThanks.
", "rawMarkdown": "Hi,\r\n Are we allowed to use 'test.csv' for unsupervised learning and/or PCA given the new rules?\r\nThanks.", "votes": null }, { "id": "88614", "postDate": "08/06/2015 01:35:21", "content": "[quote=udibr;88575]
\n\nAm I correct in the following assumptions?
\n\n1) In the baseline notebook that the admin supplied. It is said:
\n\n\n\n\nHere we use subset of the all features to pass the agreement checking
\n
Am I allowed to perform feature selection based on running local control test?
\n\n2) In the noise notebook it is said:
\n\n\n\n\nAdd noise for the agreement checking
\n
So am I correct I am allowed to "Manually" add steps to my algorithm until I manage to pass the control test?
\n\nIf the answer to both questions is yes then what am I not allowed to perform?
\n\n[/quote]
\n\nTo admins: if we are allowed to do feature engineering or feature selection, then we are allowed to train on check_agreeement.csv too (because the two are strictly equivalent). I.e. we are also allowed to exploit simulation flaws as much as possible, and submit models that are physically useless but highly performant.
\n\nJust in case you don't see the equivalence: first we train a shallow NN or a tree-based model on both the training data and the check_agreement.csv data, via a loss function that ensures that the model learns to discriminate between the two types of events. Then we record the features developed by such a model (e.g. the first-layer representations in the NN), and reinject these features in a model that will be trained only on training.csv. Job done.
\n\nThere would be no trace of the initial training on check_agreement.csv, just a bunch of hard-coded magical features.
\n\nYou cannot escape the fact that, by design, the only way to perform well in this competition is by exploiting the leaks in the data. Which is easy to do, because the KS test is easy to pass while exploiting the leaks, no matter how much you change the rules. You will not gain any physical insight from this competition.
\n\nIgnore it and face a bigger debacle later on.
", "rawMarkdown": "[quote=udibr;88575]\r\n\r\nAm I correct in the following assumptions?\r\n\r\n1) In the [baseline notebook][1] that the admin supplied. It is said:\r\n\r\n> Here we use subset of the all features to pass the agreement checking\r\n\r\nAm I allowed to perform feature selection based on running local control test?\r\n\r\n2) In the [noise notebook][2] it is said:\r\n\r\n> Add noise for the agreement checking\r\n\r\nSo am I correct I am allowed to \"Manually\" add steps to my algorithm until I manage to pass the control test?\r\n\r\nIf the answer to both questions is yes then what am I not allowed to perform?\r\n\r\n [1]: https://github.com/yandexdataschool/flavours-of-physics-start/blob/master/baseline.ipynb\r\n [2]: https://github.com/yandexdataschool/flavours-of-physics-start/blob/master/noise.ipynb\r\n\r\n[/quote]\r\n\r\nTo admins: if we are allowed to do feature engineering or feature selection, then we are allowed to train on check_agreeement.csv too (because the two are strictly equivalent). I.e. we are also allowed to exploit simulation flaws as much as possible, and submit models that are physically useless but highly performant.\r\n\r\nJust in case you don't see the equivalence: first we train a shallow NN or a tree-based model on both the training data and the check_agreement.csv data, via a loss function that ensures that the model learns to discriminate between the two types of events. Then we record the features developed by such a model (e.g. the first-layer representations in the NN), and reinject these features in a model that will be trained only on training.csv. Job done. \r\n\r\nThere would be no trace of the initial training on check_agreement.csv, just a bunch of hard-coded magical features.\r\n\r\nYou cannot escape the fact that, *by design*, the only way to perform well in this competition is by exploiting the leaks in the data. Which is easy to do, because the KS test is easy to pass while exploiting the leaks, no matter how much you change the rules. You will *not* gain any physical insight from this competition. \r\n\r\nIgnore it and face a bigger debacle later on.", "votes": null }, { "id": "88617", "postDate": "08/06/2015 01:43:49", "content": "[quote=Mark Vicuna;88606]
\n\nHi,\n Are we allowed to use 'test.csv' for unsupervised learning and/or PCA given the new rules?\nThanks.
\n\n[/quote]
\n\nIf we were allowed to use the data in test.csv, it would be easy to exploit it to learn to discriminate between test events and control events (since they have quite different distributions).
\n\nShort list of ways to exploit simulation leaks while passing the KS test:
\n\nClearly, you have to add the following rules to the competition to make it even funnier:
\n\n[quote=fchollet;88617]
\n\n[quote=Mark Vicuna;88606]
\n\nHi,\n Are we allowed to use 'test.csv' for unsupervised learning and/or PCA given the new rules?\nThanks.
\n\n[/quote]
\n\nIf we were allowed to use the data in test.csv, it would be easy to exploit it to learn to discriminate between test events and control events (since they have quite different distributions).
\n\nShort list of ways to exploit simulation leaks while passing the KS test:
\n\nClearly, you have to add the following rules to the competition to make it even funnier:
\n\n[/quote]
\n\nOk. Thanks. I hadn't put two and two together and realized they added specific data to the test.csv that isn't present at all in the training.csv data to confuse models.
\n\nI think you are absolutely right there is no good answer to the RNG problem. The seed and algorythim are now meta-parameters that you can use to find the model that finds the leaky data. You can then completely remove that step from the submitted model and whose to say '47562' isn't your favorite number?
\n\nThey would have to provide an RNG and enforce a seed to work around that loophole IMO.
", "rawMarkdown": "[quote=fchollet;88617]\r\n\r\n[quote=Mark Vicuna;88606]\r\n\r\nHi,\r\n Are we allowed to use 'test.csv' for unsupervised learning and/or PCA given the new rules?\r\nThanks.\r\n\r\n[/quote]\r\n\r\nIf we were allowed to use the data in test.csv, it would be easy to exploit it to learn to discriminate between test events and control events (since they have quite different distributions). \r\n\r\n**Short list of ways to exploit simulation leaks while passing the KS test:**\r\n\r\n- post-classification filtering of control events via a control classifier trained on training.csv and check_agreement.csv **[newly disallowed]**\r\n- direct training with a custom loss function on both training.csv and check_agreement.csv **[newly disallowed]**\r\n- either of the methods above with control data that you generate yourself via a RNG **[likely allowed]**\r\n- using the test.csv data to learn to discriminate between the test events and control events **[likely disallowed]**\r\n- feature engineering step before training **[likely allowed]**\r\n\r\n\r\nClearly, you have to add the following rules to the competition to make it even funnier:\r\n\r\n- disallow random number generators\r\n- no feature engineering allowed\r\n\r\n[/quote]\r\n\r\nOk. Thanks. I hadn't put two and two together and realized they added specific data to the test.csv that isn't present at all in the training.csv data to confuse models.\r\n\r\nI think you are absolutely right there is no good answer to the RNG problem. The seed and algorythim are now meta-parameters that you can use to find the model that finds the leaky data. You can then completely remove that step from the submitted model and whose to say '47562' isn't your favorite number?\r\n\r\nThey would have to provide an RNG and enforce a seed to work around that loophole IMO.", "votes": null }, { "id": "88664", "postDate": "08/06/2015 12:52:17", "content": "Hi Mark, I'd be very interested to see how one would implement your second point (direct training with a custom loss function on both training.csv and check_agreement.csv). I'm not interested in using to in this competition, but I'd like to know how to do this in general since I think it's complicated by the fact that the agreement check is really checking a global systemic property.
\n\nWould you, or someone, be willing to demonstrate code for that (either during or after the competition)?
\n\nThanks,\nErlend
", "rawMarkdown": "Hi Mark, I'd be very interested to see how one would implement your second point (direct training with a custom loss function on both training.csv and check_agreement.csv). I'm not interested in using to in this competition, but I'd like to know how to do this in general since I think it's complicated by the fact that the agreement check is really checking a global systemic property. \r\n\r\nWould you, or someone, be willing to demonstrate code for that (either during or after the competition)?\r\n\r\nThanks,\r\nErlend", "votes": null }, { "id": "88679", "postDate": "08/06/2015 15:02:30", "content": "the issues raised in this forum can sink the entire kaggle web site if not addressed correctly.\nsave kaggle
", "rawMarkdown": "the issues raised in this forum can sink the entire kaggle web site if not addressed correctly.\r\n[save kaggle][1]\r\n\r\n\r\n [1]: https://www.kaggle.com/forums/f/15/kaggle-forum/t/15805/save-kaggle/88678#post88678", "votes": null }, { "id": "88705", "postDate": "08/06/2015 19:01:49", "content": "Another issue to consider...from the initial post the motivation for disallowing use of check_agreement and check_correlation is that you don't want to get models that exploit the differences between real data and simulation. In that case, wouldn't it make sense to use the control data to do feature selection? If I were to make plots of different variables to see how well they discriminate between signal and background, I would also want to look in the control region to see if that variable is actually discriminative or if it is simply mis-modeled. Is using the check_agreement dataset to eliminate features that ruin the agreement allowed? That seems to me to be one of the whole points of including that data.
", "rawMarkdown": "Another issue to consider...from the initial post the motivation for disallowing use of check_agreement and check_correlation is that you don't want to get models that exploit the differences between real data and simulation. In that case, wouldn't it make sense to use the control data to do feature selection? If I were to make plots of different variables to see how well they discriminate between signal and background, I would also want to look in the control region to see if that variable is actually discriminative or if it is simply mis-modeled. Is using the check_agreement dataset to eliminate features that ruin the agreement allowed? That seems to me to be one of the whole points of including that data.", "votes": null }, { "id": "88711", "postDate": "08/06/2015 20:13:52", "content": "[quote=fizzicks;88421]
\n\nSorry, but I have another question about this rule. In physics it is fairly common to actually correct for discrepancies in a particular variable between data and simulation by re-weighting events.
\n\n[/quote]
\n\nin fact, we usually check the agreement / reweight the one-dimensional projections between background and data but then go ahead and exploit the differences between signal and background in the full N-dimensional space. I have only seen one example yet (ATLAS Higgs to bb search) where also two-dimensional projections were checked.
", "rawMarkdown": "[quote=fizzicks;88421]\r\n\r\nSorry, but I have another question about this rule. In physics it is fairly common to actually correct for discrepancies in a particular variable between data and simulation by re-weighting events. \r\n\r\n[/quote]\r\n\r\nin fact, we usually check the agreement / reweight the one-dimensional projections between background and data but then go ahead and exploit the differences between signal and background in the full N-dimensional space. I have only seen one example yet (ATLAS Higgs to bb search) where also two-dimensional projections were checked.", "votes": null }, { "id": "88751", "postDate": "08/07/2015 00:56:21", "content": "[quote=udibr;88679]
\n\nthe issues raised in this forum can sink the entire kaggle web site if not addressed correctly.\nsave kaggle
\n\n[/quote]
\n\nNo, I don't think what happened in this competition will have any impact on future Kaggle competitions.
\n\nBut there are no doubts that the current competition is over at this point. I don't see what could possibly save it, short of starting from scratch with different data and different rules (and no KS test).
\n\nSure, it is in bad taste to change the rules in the middle of the competition, especially in a way that invalidates the work of many entrants (including many (all?) top entrants). Sure, it's in bad taste not to reply to questions and concerns addressed to admins, too. Sure, it was unfortunate to submit a competition that was flawed by design, but then again, it's something that has happened a few times on Kaggle before. Data science is hard. Organizing a competition requires strong data science competence. Teams with such strong data science competence are less likely to turn to Kaggle competitions to solve their problems...
\n\nClearly, the only viable answer here is just to move away from this competition. The first dozen hours you spent on this challenge have been thoroughly wasted, so just don't spend any more time.
\n\nLet's move on.
", "rawMarkdown": "[quote=udibr;88679]\r\n\r\nthe issues raised in this forum can sink the entire kaggle web site if not addressed correctly.\r\n[save kaggle][1]\r\n\r\n\r\n [1]: https://www.kaggle.com/forums/f/15/kaggle-forum/t/15805/save-kaggle/88678#post88678\r\n\r\n[/quote]\r\n\r\nNo, I don't think what happened in this competition will have any impact on future Kaggle competitions. \r\n\r\nBut there are no doubts that the current competition is over at this point. I don't see what could possibly save it, short of starting from scratch with different data and different rules (and no KS test). \r\n\r\nSure, it is in bad taste to change the rules in the middle of the competition, especially in a way that invalidates the work of many entrants (including many (all?) top entrants). Sure, it's in bad taste not to reply to questions and concerns addressed to admins, too. Sure, it was unfortunate to submit a competition that was flawed by design, but then again, it's something that has happened a few times on Kaggle before. Data science is hard. Organizing a competition requires strong data science competence. Teams with such strong data science competence are less likely to turn to Kaggle competitions to solve their problems...\r\n\r\nClearly, the only viable answer here is just to move away from this competition. The first dozen hours you spent on this challenge have been thoroughly wasted, so just don't spend any more time.\r\n\r\nLet's move on.", "votes": null }, { "id": "88752", "postDate": "08/07/2015 01:04:37", "content": "[quote=Erlend Davidson;88664]
\n\nHi Mark, I'd be very interested to see how one would implement your second point (direct training with a custom loss function on both training.csv and check_agreement.csv). I'm not interested in using to in this competition, but I'd like to know how to do this in general since I think it's complicated by the fact that the agreement check is really checking a global systemic property.
\n\nWould you, or someone, be willing to demonstrate code for that (either during or after the competition)?
\n\nThanks,\nErlend
\n\n[/quote]
\n\nI believe this comment was actually meant as a reply to my post.
\n\nMy first entries in this competition were based on this scheme (a custom loss function to train a classifier that, at the same time, classifies signal from background in the test data, and outputs similar probabilities for signal and background of the control data).
\n\nSure, ROC AUC is a global property. It also has a statistical / information-theoretical meaning, and therefore you can optimize for it via per-sample proxy metrics from statistics or information theory...
\n\nThe most basic thing you can do is use binary crossentropy. Minimize crossentropy of signal/background on the test data while maintaining the crossentropy of signal/background on the control data at some carefully selected value (typically the maximum entropy value). The latter is done via minimization of the difference between the crossentropy and the target value.
\n\nThis can be treated as a global optimization problem by using as objective a weighted sum of these two objectives.
\n\nThe result, in practice, are highly performant models (e.g. up to 0.996 for a single NN, no ensembles, and after very little search/optimization of the setup) with very low KS scores (generally around 0.05-0.06).
", "rawMarkdown": "[quote=Erlend Davidson;88664]\r\n\r\nHi Mark, I'd be very interested to see how one would implement your second point (direct training with a custom loss function on both training.csv and check_agreement.csv). I'm not interested in using to in this competition, but I'd like to know how to do this in general since I think it's complicated by the fact that the agreement check is really checking a global systemic property. \r\n\r\nWould you, or someone, be willing to demonstrate code for that (either during or after the competition)?\r\n\r\nThanks,\r\nErlend\r\n\r\n[/quote]\r\n\r\nI believe this comment was actually meant as a reply to my post. \r\n\r\nMy first entries in this competition were based on this scheme (a custom loss function to train a classifier that, at the same time, classifies signal from background in the test data, and outputs similar probabilities for signal and background of the control data). \r\n\r\nSure, ROC AUC is a global property. It also has a statistical / information-theoretical meaning, and therefore you can optimize for it via per-sample proxy metrics from statistics or information theory...\r\n\r\nThe most basic thing you can do is use binary crossentropy. Minimize crossentropy of signal/background on the test data while maintaining the crossentropy of signal/background on the control data at some carefully selected value (typically the maximum entropy value). The latter is done via minimization of the difference between the crossentropy and the target value. \r\n\r\nThis can be treated as a global optimization problem by using as objective a weighted sum of these two objectives. \r\n\r\nThe result, in practice, are highly performant models (e.g. up to 0.996 for a single NN, no ensembles, and after very little search/optimization of the setup) with very low KS scores (generally around 0.05-0.06).", "votes": null }, { "id": "88755", "postDate": "08/07/2015 01:45:45", "content": "I used a simpler method that worked nicely. Build a predictor to predict if a sample is from the training set or the check_agreement set and use the output of this predictor to reduce the features that are known to increase the KS score (for example SPDhits) whenever a sample looks like it is coming from check_agreement and not from training.
\n\nThe reduction was done as follows: convert the feature (SPDhits) to log(Psignal/Pbackground) and then multiply by the probability to be from the training set and not from the check_agreement set
", "rawMarkdown": "I used a simpler method that worked nicely. Build a predictor to predict if a sample is from the training set or the check_agreement set and use the output of this predictor to reduce the features that are known to increase the KS score (for example SPDhits) whenever a sample looks like it is coming from check_agreement and not from training.\r\n\r\nThe reduction was done as follows: convert the feature (SPDhits) to log(Psignal/Pbackground) and then multiply by the probability to be from the training set and not from the check_agreement set", "votes": null }, { "id": "88756", "postDate": "08/07/2015 01:49:05", "content": "[quote=fchollet;88751]
\n\nLet's move on.
\n\n[/quote]
\n\nThanks, will do
", "rawMarkdown": "[quote=fchollet;88751]\r\n\r\nLet's move on.\r\n\r\n[/quote]\r\n\r\nThanks, will do", "votes": null }, { "id": "88760", "postDate": "08/07/2015 02:29:06", "content": "To admin, one final thing. I am not sure about this but it looks to me that a classifier that avoids the KS score ( for example in the method I just described ) gives better prediction results than a simpler classifier that ignores the KS test.
\n\nIf this is true than it can be useful for real future research in this area. Perhaps the background contains D decay events and the reduction method helps in detecting these events and correctly placing them in the background bin.
", "rawMarkdown": "To admin, one final thing. I am not sure about this but it looks to me that a classifier that avoids the KS score ( for example in the method I just described ) gives better prediction results than a simpler classifier that ignores the KS test. \r\n\r\nIf this is true than it can be useful for real future research in this area. Perhaps the background contains D decay events and the reduction method helps in detecting these events and correctly placing them in the background bin.", "votes": null }, { "id": "88768", "postDate": "08/07/2015 04:07:47", "content": "[quote=udibr;88760]
\n\nTo admin, one final thing. I am not sure about this but it looks to me that a classifier that avoids the KS score ( for example in the method I just described ) gives better prediction results than a simpler classifier that ignores the KS test.
\n\nIf this is true than it can be useful for real future research in this area. Perhaps the background contains D decay events and the reduction method helps in detecting these events and correctly placing them in the background bin.
\n\n[/quote]
\n\nThe idea behind the custom loss function I was using was the following:
\n\nIn theory it would work perfectly, except in practice it doesn't. Because the assumption that the KS score is meaningful is just plain false. So what happens in practice is that a classifier trained in this way learns to discriminate control data while exploiting simulation artifacts.
\n\nIt is also worth pointing that the problem of finding which features are "valid" (physics-based) and which are invalid, isn't solvable with the data provided. It cannot be found from the training data (because events are either real and background, or simulated and signal), and it cannot be found from the control data (because it has a completely different distribution from the test data, and therefore can be discriminated against to cheat the test).
\n\nI.e. the problem that the organizers wanted to solve cannot be solved with the data that the organizers provided. Which is why nothing interesting will come out of this competition.
", "rawMarkdown": "[quote=udibr;88760]\r\n\r\nTo admin, one final thing. I am not sure about this but it looks to me that a classifier that avoids the KS score ( for example in the method I just described ) gives better prediction results than a simpler classifier that ignores the KS test. \r\n\r\nIf this is true than it can be useful for real future research in this area. Perhaps the background contains D decay events and the reduction method helps in detecting these events and correctly placing them in the background bin. \r\n\r\n[/quote]\r\n\r\nThe idea behind the custom loss function I was using was the following:\r\n\r\n- the data contains both \"valid\" (physics-based) signal indicators, and \"invalid\" indicators (simulation artifacts)\r\n- therefore we need a way to constrain the learning to only use the valid features\r\n- ***assuming that the KS score reliably tests for usage of such invalid features***, then the solution is to constrain training to produce a low KS score, through a custom loss function trained on both types of events.\r\n\r\nIn theory it would work perfectly, except in practice it doesn't. ***Because the assumption that the KS score is meaningful is just plain false.*** So what happens in practice is that a classifier trained in this way learns to discriminate control data while exploiting simulation artifacts.\r\n\r\nIt is also worth pointing that the problem of finding which features are \"valid\" (physics-based) and which are invalid, **isn't solvable with the data provided.** It cannot be found from the training data (because events are either real *and* background, or simulated *and* signal), and it cannot be found from the control data (because it has a completely different distribution from the test data, and therefore can be discriminated against to cheat the test). \r\n\r\nI.e. the problem that the organizers wanted to solve cannot be solved with the data that the organizers provided. Which is why nothing interesting will come out of this competition.", "votes": null }, { "id": "89345", "postDate": "08/14/2015 08:13:32", "content": "", "rawMarkdown": "", "votes": null }, { "id": "89447", "postDate": "08/15/2015 16:37:13", "content": "Several cases just to clarify this topic.
\n\nCan we
\n\nSo for sure you can do 1. The 2 and 3. depends on what exactly you mean by: "research correlations" and "make assumptions":
\n\nIf you want to plot distributions and choose only the ones that show good agreement then yes please :) If you want to somehow to create variable that is somehow distinguishing between tau->3mu MC and Ds->Phi Pi MC then no.
\n\nCheers,\nMarcin
", "rawMarkdown": "So for sure you can do 1. The 2 and 3. depends on what exactly you mean by: \"research correlations\" and \"make assumptions\":\r\n\r\nIf you want to plot distributions and choose only the ones that show good agreement then yes please :) If you want to somehow to create variable that is somehow distinguishing between tau->3mu MC and Ds->Phi Pi MC then no. \r\n\r\nCheers,\r\nMarcin", "votes": null }, { "id": "89554", "postDate": "08/17/2015 10:25:44", "content": "[quote]
\n\nCan we
\n\n[/quote]
\n\n[quote] So for sure you can do 1. [/quote]
\n\nExcellent. Just to be clear that you fully understand what you are stating here, here's a code skeleton that is a straightforward implementation of 1. Is this code skeleton valid by your rules?
\n\nX_agreement, y_agreement, w_agreement = get_agreement_data()\nX, y = get_training_data()\ninitial_params = initialize() # just random {x1..xn}\nmodel = model_from_params(initial_params, X, y)\ncurrent_score = compute_roc_auc(model, X, y)\ncurrent_ks = compute_ks(model, X_agreement, y_agreement, w_agreement)\n\nwhile not stopping_criterion.check():\n # just increment / decrement initial {x1..xn}\n initial_params = update_parameters(initial_params, current_score, current_ks)\n\n # model_from_params could even make no use of X, y\n # i.e. this step isn't even necessary\n model = model_from_params(initial_params, X, y)\n\n current_score = compute_roc_auc(model, X, y)\n current_ks = compute_ks(model, X_agreement, y_agreement, w_agreement)\n\n stopping_criterion.update(current_score, current_ks)\nOk. I'll try to generalize method.\nAny "black box" (imagine, that we have to submit our test set instead of running local tests) analysis is allowed (in assumption, that we haven't got correlation and agreement datasets) ?
", "rawMarkdown": "Ok. I'll try to generalize method.\r\nAny \"black box\" (imagine, that we have to submit our test set instead of running local tests) analysis is allowed (in assumption, that we haven't got correlation and agreement datasets) ?", "votes": null }, { "id": "89595", "postDate": "08/17/2015 19:38:01", "content": "Mikhail, I'm not sure I got your question, but in the end winning solution should be accompanied with compilable & runnable code (that produces the same results) and explanation of how this solution operates on given data.
", "rawMarkdown": "Mikhail, I'm not sure I got your question, but in the end winning solution should be accompanied with compilable & runnable code (that produces the same results) and explanation of how this solution operates on given data.", "votes": null }, { "id": "89600", "postDate": "08/17/2015 19:48:24", "content": "Francois, your solution skeleton would qualify as "using check_agreement for model testing". If you follow the logic from the post https://www.kaggle.com/c/flavours-of-physics/forums/t/15974/machine-learning-approach-in-high-energy-physics-explanation, most likely it should be fine. To be sure that solution is OK we demand participants to provide the compilable&runnable code and verbal description of their approaches in the end.
\n\nNB we realize that things happening inside loop while not stopping_criterion.check() can also be treated as model training effectively overfitting to model restrictions. It is not much worse than overfitting to public leaderboard (for example http://blog.mrtz.org/2015/03/09/competition.html and http://jmlr.org/proceedings/papers/v37/blum15.pdf) and this is something all challenges have to cope with for the given competition framework.
Ok, so, but what if a solution strictly follows this code skeleton, but the explanation of how it works is, as you state, "(over)fitting to model restrictions" (actually just fitting, i.e. training)? Is the solution no longer eligible then, even though it is following a training pattern that has been claimed as acceptable?
\n\nBasically, why "most likely it should be fine"? Is this training pattern valid, or not? If it is valid, it should stay valid no matter what goes into the functions model_from_params(init_params, X, y) and update_parameters(init_params, score, ks)? (of course these functions would not be be accessing new data, or anything sketchy like that. Just straightforward logic applied to their arguments).
Francois, unless anything "sketchy" happens inside the functions you've mentioned (that's the reason for 'most likely') the pattern is fine.
", "rawMarkdown": "Francois, unless anything \"sketchy\" happens inside the functions you've mentioned (that's the reason for '**most likely**') the pattern is fine.", "votes": null }, { "id": "89607", "postDate": "08/17/2015 20:19:23", "content": "By sketchy I just mean external data access. Of course these functions would only be processing the arguments that we pass them. There is no trick behind my code skeleton, it would do exactly what the function names imply.
\n\nAlright, if this pattern is allowed, that's good news for us, but you do realize that this pattern is all about training on the control data, right? Via update_parameters(init_params, score, ks), which injects KS score information into the model. By repeating the loop as many times as needed, you can end up leaking as much KS score information as you want.
Just like "overfitting the leaderboard" would eventually result in perfect scores if we didn't have daily submission limits.
", "rawMarkdown": "By sketchy I just mean external data access. Of course these functions would only be processing the arguments that we pass them. There is no trick behind my code skeleton, it would do exactly what the function names imply.\r\n\r\nAlright, if this pattern is allowed, that's good news for us, but you do realize that this pattern is all about training on the control data, right? Via `update_parameters(init_params, score, ks)`, which injects KS score information into the model. By repeating the loop as many times as needed, you can end up leaking as much KS score information as you want. \r\n\r\nJust like \"overfitting the leaderboard\" would eventually result in perfect scores if we didn't have daily submission limits.", "votes": null }, { "id": "89612", "postDate": "08/17/2015 22:01:20", "content": "Actually, the shown code is exactly what i mean, when saying "black box" analisis.\nSounds good, thank you.
", "rawMarkdown": "Actually, the shown code is exactly what i mean, when saying \"black box\" analisis.\r\nSounds good, thank you.", "votes": null }, { "id": "89802", "postDate": "08/19/2015 14:49:14", "content": "Just to be explicit, that also means that a custom loss function where KS is included as a penalty is strictly allowed? I thought I read in the forum somewhere that doing so was not allowed. But this is essentially doing the same thing as fchollet's code snippet.
", "rawMarkdown": "Just to be explicit, that also means that a custom loss function where KS is included as a penalty is strictly allowed? I thought I read in the forum somewhere that doing so was not allowed. But this is essentially doing the same thing as fchollet's code snippet.", "votes": null }, { "id": "89818", "postDate": "08/19/2015 17:06:28", "content": "\n\n\nBut this is essentially doing the same thing as fchollet's code snippet.
\n
It is indeed. It's just semantics. You are allowed to "train" on the control data, but training on the control data is forbidden. My personal take on this is that you should figure out ways to train on the control data as much as possible while camouflaging the process to keep some facade of respectability in the eyes of the organizers (which my snippet does). At least that's how I interpret these recent developments.
", "rawMarkdown": "> But this is essentially doing the same thing as fchollet's code snippet.\r\n\r\nIt is indeed. It's just semantics. You are allowed to \"train\" on the control data, but training on the control data is forbidden. My personal take on this is that you should figure out ways to train on the control data as much as possible while camouflaging the process to keep some facade of respectability in the eyes of the organizers (which my snippet does). At least that's how I interpret these recent developments.", "votes": null }, { "id": "89880", "postDate": "08/20/2015 09:01:53", "content": "[quote=fchollet;89818]
\n\nIt is indeed. It's just semantics. You are allowed to "train" on the control data, but training on the control data is forbidden.
\n\n[/quote]
\n\nI guess the original intent of the competition is that one uses compute_ks(..) as an 'oracle' without looking at the entries in check_agreement.csv (although I suspect that given a huge amount of CPU time one could even infer the contents of check_agreement.csv this way....)
", "rawMarkdown": "[quote=fchollet;89818]\r\n\r\nIt is indeed. It's just semantics. You are allowed to \"train\" on the control data, but training on the control data is forbidden. \r\n\r\n[/quote]\r\n\r\nI guess the original intent of the competition is that one uses compute_ks(..) as an 'oracle' without looking at the entries in check_agreement.csv (although I suspect that given a huge amount of CPU time one could even infer the contents of check_agreement.csv this way....)", "votes": null }, { "id": "89887", "postDate": "08/20/2015 10:46:46", "content": "[quote=Andre Holzner;89880]
\n\n[quote=fchollet;89818]
\n\nIt is indeed. It's just semantics. You are allowed to "train" on the control data, but training on the control data is forbidden.
\n\n[/quote]
\n\nI guess the original intent of the competition is that one uses compute_ks(..) as an 'oracle' without looking at the entries in check_agreement.csv (although I suspect that given a huge amount of CPU time one could even infer the contents of check_agreement.csv this way....)
\n\n[/quote]
\n\nIt doesn't matter if you "look" (i.e. open the file and read the contents of it), if you accept/reject your models based on some test (i.e. the KS-metric) derived from the data in check_agreement.csv then some of check_agreement.csv will leak into your data. The same sort of problem can happen if you score your models using just a cross-validation on a static test-set.
\n\nThis is described in the first two paragraphs of http://k-d-w.org/node/95, and also in §7.10.2 of the book here: http://statweb.stanford.edu/~tibs/ElemStatLearn/.
", "rawMarkdown": "[quote=Andre Holzner;89880]\r\n\r\n[quote=fchollet;89818]\r\n\r\nIt is indeed. It's just semantics. You are allowed to \"train\" on the control data, but training on the control data is forbidden. \r\n\r\n[/quote]\r\n\r\nI guess the original intent of the competition is that one uses compute_ks(..) as an 'oracle' without looking at the entries in check_agreement.csv (although I suspect that given a huge amount of CPU time one could even infer the contents of check_agreement.csv this way....)\r\n\r\n[/quote]\r\n\r\nIt doesn't matter if you \"look\" (i.e. open the file and read the contents of it), if you accept/reject your models based on some test (i.e. the KS-metric) derived from the data in check_agreement.csv then some of check_agreement.csv will leak into your data. The same sort of problem can happen if you score your models using just a cross-validation on a static test-set.\r\n\r\nThis is described in the first two paragraphs of http://k-d-w.org/node/95, and also in §7.10.2 of the book here: http://statweb.stanford.edu/~tibs/ElemStatLearn/.", "votes": null }, { "id": "90877", "postDate": "08/29/2015 19:41:06", "content": "Wow, I find the discussion on this thread to be discouraging. The point of the competition is to help move forward an understanding of physics and yet a number of contributors seem more intent on arguing about the rules changes as if it were a casual game. Yes you may have to go back and do things over, or come up with a different approach. The rule changes are there to provide for usable data for physicists. \nThat is the ultimate point of the challenge, not whether you can successively argue for rules in your favor or use clever interpretations to get an upper hand.
", "rawMarkdown": "Wow, I find the discussion on this thread to be discouraging. The point of the competition is to help move forward an understanding of physics and yet a number of contributors seem more intent on arguing about the rules changes as if it were a casual game. Yes you may have to go back and do things over, or come up with a different approach. The rule changes are there to provide for usable data for physicists. \r\nThat is the ultimate point of the challenge, not whether you can successively argue for rules in your favor or use clever interpretations to get an upper hand.", "votes": null }, { "id": "91685", "postDate": "09/06/2015 04:55:59", "content": "[quote=Chrisfs;90877]\n The point of the competition is to help move forward an understanding of physics and yet a number of contributors seem more intent on arguing about the rules changes as if it were a casual game. Yes you may have to go back and do things over, or come up with a different approach. The rule changes are there to provide for usable data for physicists. \n[/quote]
\n\nNot really. If you set up a system that scores entries, people will optimise for that measure. If you want to complain later that the optimal solution for that measure didn't solve your initial problem, well then it's your fault for posing the problem incorrectly.
", "rawMarkdown": "[quote=Chrisfs;90877]\r\n The point of the competition is to help move forward an understanding of physics and yet a number of contributors seem more intent on arguing about the rules changes as if it were a casual game. Yes you may have to go back and do things over, or come up with a different approach. The rule changes are there to provide for usable data for physicists. \r\n[/quote]\r\n\r\n\r\nNot really. If you set up a system that scores entries, people will optimise for that measure. If you want to complain later that the optimal solution for that measure didn't solve your initial problem, well then it's your fault for posing the problem incorrectly.", "votes": null }, { "id": "91900", "postDate": "09/09/2015 03:51:49", "content": "[quote=fchollet;88751]
\n\nNo, I don't think what happened in this competition will have any impact on future Kaggle competitions.
\n\n[/quote]
\n\nWhy on earth should we learn from mistakes? If it's not in a file called "train.csv" we don't care!
", "rawMarkdown": "[quote=fchollet;88751]\r\n\r\nNo, I don't think what happened in this competition will have any impact on future Kaggle competitions. \r\n\r\n[/quote]\r\n\r\nWhy on earth should we learn from mistakes? If it's not in a file called \"train.csv\" we don't care!", "votes": null } ], "comments": [ { "id": 88150, "author_name": "udibr1", "author_url": "", "post_date": "08/03/2015 20:21:34", "content": "Thank you for the clarification but I could not see any reference to this explicit limitation anywhere in your original description of the problem.\nCan you please give an explicit quote from the original description of the problem.\nChanging the rules while the competition takes place does not add much fun and undermine the entire value of a site like kaggle.
\n\nAs an off-topic, I think there is real value in taking other existing channels into account when modeling. In a real situation they do add to the noise and modeling all the possible noise is one way of improving the signal.
", "votes": null, "replies": [] }, { "id": 88151, "author_name": "wendykan", "author_url": "", "post_date": "08/03/2015 20:27:56", "content": "Under our Data page:\n"You need to submit predictions for ALL the test entries. You will need to treat them all the same and predict as if they are all the same channel's collision events. "
", "votes": null, "replies": [] }, { "id": 88155, "author_name": "anaderi", "author_url": "", "post_date": "08/03/2015 20:59:12", "content": "udibr, you are right, there might be a real value in taking other channels into account and we do not deny people will find how find that value, that's why we have Special Physics prize in this challenge and a couple of workshops (at NIPS (if confirmed) and at Zurich) to discuss these approaches.
", "votes": null, "replies": [] }, { "id": 88157, "author_name": "udibr1", "author_url": "", "post_date": "08/03/2015 20:59:59", "content": "I am very sorry, but, perhaps I am wrong but I think that what you quoted is not relevant.\nYour quote refers to how predictions are made. I have a single prediction algorithm that runs on all examples in exactly the same way, so it fits your description of "same channel".\nThe only question is how this algorithm was trained in the first place.
\n\nAs you can see for yourself the quote you have taken is from the part talking about the test process and not about the training process.
\n\nIn any case you should refer to this page https://www.kaggle.com/c/flavours-of-physics/rules\nand not to any other page of the competition when specifying rules.\nThis is because the rules page (which I had to accept when downloading the data and no other page) begins with "These are the complete, official rules for the Competition..."
\n\nThe rules page states that "Use of external data is not permitted" and that "'Data' means the Data or Datasets linked from the Competition Website for the purpose of use by Participants in the Competition."\nthis covers all the csv files.
\n\nI would also like to add that developing methods that uses the other csv files will help the process of discovering new physics.\nThis is because the main challenge in this competition is not finding new physics now but building tools for doing it in the future and perhaps the most important tool will be an accurate simulator and test sets that take any adversary action into account. If the simulator is not accurate enough to describe the "real" physics then it needs to be improved and if the gaps it has were find in this competition then all the better.
", "votes": null, "replies": [] }, { "id": 88163, "author_name": "fchollet", "author_url": "", "post_date": "08/03/2015 21:39:09", "content": "[quote]\nTo be eligible for the prize your final solution should rely solely on the data provided in ‘training.csv’ without any usage of ‘check_correlation.csv' or ‘check_agreement.csv' datasets.\n[/quote]
\n\nDoes this means we cannot run the KS and correlation tests locally? If so, then why are these datasets provided in the first place?
\n\nIt seems that what you are trying to say is, "don't train your model using these datasets". But of course, selecting a model based on whether it passes the KS test is the exact same as using the dataset ‘check_correlation.csv' to train your model. This much should be obvious to anyone.
\n\nJust in case it isn't obvious for you, here's a thought experiment: the no-training model. Your set up a logistic regression model to classify signal from background, but instead of learning the parameters through gradient descent, you randomly generate them. Then you check whether your random model passes the KS test and correlation test, and you check what the ROC AUC is. You do this with 1,000,000 models (very much realistic and computationally tractable) and you pick the model with the highest ROC AUC. This is in fact a solid strategy if you don't feel like computing the gradients of the ROC AUC and KS scores.
\n\nThis model never ever saw the data in ‘check_correlation.csv' (or any training data, for that matter), but it was still effectively trained on it... through post-selection. ANY USE of ‘check_correlation.csv' means that you are training your model (i.e. selecting its parameters) based on the data in ‘check_correlation.csv'. If you are running the KS test locally, you are breaking the rules. But if you are not, you are insane (enjoy submitting your .csv 100 times in a row).
\n\nBy the way, the same is also true with the test happening on the Kaggle servers. As long as you are selecting models based on whether they pass the KS test, then these models are in effect trained on the KS data (i.e. the parameters of your final model have been chosen based on the KS data). There is no way around it whatsoever.
\n\n[quote=udibr;88157]\nI have a single prediction algorithm that runs on all examples in\nexactly the same way, so it fits your description of "same channel".\n[/quote]
\n\nSo do I (no special case for processing the control data), and as you do I also leverage the KS data for training. Also, I can't help but remark that what we are doing inevitably results in models that pick up simulation defects while still managing to work around the KS test. It's what we optimize for.
\n\nThis competition is broken in a fundamental way. The organizers will have to backpedal, and possibly cancel the competition altogether, if their goal is a model that is useful from a physics perspective. No such model is going to emerge in the current setup.
", "votes": null, "replies": [] }, { "id": 88165, "author_name": "guidotapia", "author_url": "", "post_date": "08/03/2015 21:48:36", "content": "I find no ambiguity in the statement
\n\n\n\n\nTo be eligible for the prize your final solution should rely solely on\n the data provided in ‘training.csv’ without any usage of\n ‘check_correlation.csv' or ‘check_agreement.csv' datasets.
\n
I think you are just taking the statement to its literary absurd extreme. Do not train models on these datasets. You can use them to evaluate that your models meets the competition goals and that is why they were provided.
\n\nI myself experimented using these datasets in my training (as in use them to train models) but I figured that was not in the spirit of the comp so I did not submit any of those experiments. This was before clarification above was made so I think this statement was implied (softly whispered) even before the clarification in this topic.
", "votes": null, "replies": [] }, { "id": 88166, "author_name": "fchollet", "author_url": "", "post_date": "08/03/2015 21:56:38", "content": "Also, just FYI: the control data and test data have significantly different distributions, so it is possible to by-pass the KS test entirely without ever looking at the KS data, simply by setting up a 1-class classifier (output: "belongs to the class"/"doesn't belong"). Only data points that have the same distribution as the data in training.csv would be classified, others would get a constant prediction (for instance). KS test solved. Such a classifier can be obtained by using randomly generated data as negative examples (no need for the KS data).
\n\nThink this process would violate the rules? Well, it is also possible to set up a signal/BG model that implicitly do the job of this classifier, without any explicit special case for KS control data.
", "votes": null, "replies": [] }, { "id": 88167, "author_name": "triskelion", "author_url": "", "post_date": "08/03/2015 21:57:44", "content": "[quote=Guido Tapia;88165]\ntaking the statement to its literary absurd extreme.\n[/quote]\nI think fchollet is right. If we take "training a model" to mean "using information from" then any locally performed check adds information to your model selection pipeline. No matter how indirectly or directly, purposeful or accidental, this information will ultimately leak and end up into your submissions.
", "votes": null, "replies": [] }, { "id": 88168, "author_name": "fchollet", "author_url": "", "post_date": "08/03/2015 22:01:29", "content": "[quote=Guido Tapia;88165]
\n\nI think you are just taking the statement to its literary absurd extreme. Do not train models on these datasets. You can use them to evaluate that your models meets the competition goals and that is why they were provided.
\n\n[/quote]
\n\nYou seem to misunderstand what "training a model" means. It does not mean calling the "fit" method on a sklearn classifier. It means the process of selecting the parameters of your model. Any usage of the KS data to select parameters, hyperparameters or even the type of classifier to use means that you are training your solution on the KS data.
\n\nYou know how everybody tends to overfit to the leaderboard even though we don't have access to the labels of the leaderboard? This is the same. When you are selecting any aspect of your model based on some score, it means that data used to generate this score is part of your training data.
", "votes": null, "replies": [] }, { "id": 88169, "author_name": "guidotapia", "author_url": "", "post_date": "08/03/2015 22:14:11", "content": "I agree people will over fit the validation datasets as they will the public leader board. But I think its a compromise that was understood and accepted. And hopefully, even with these imperfect constraints some valuable insights can be learnt.
\n\nI think this is a very complex problem and we only have the current set of tools and platforms to solve the problem, lets make the best of it. I just think suggesting the competition be scrapped because its not perfect is the wrong approach.
\n\nI'll keep going, if KS metric gets too frustrating I'll go back to caterpillar which I left because I found the craziness in the data set structure too frustrating. Again no competition is perfect and if it is then you get 3000 competitors submitting each others scripts and filling the forums with noise.
", "votes": null, "replies": [] }, { "id": 88263, "author_name": "glimmung", "author_url": "", "post_date": "08/04/2015 08:11:12", "content": "Isn't the "Agreement test" meant as an estimate of the systematic error?\nThus, if you use it to train, it messes up just that?
\n\nEdit: removed rant about MC
", "votes": null, "replies": [] }, { "id": 88326, "author_name": "domcastro", "author_url": "", "post_date": "08/04/2015 11:21:39", "content": "As Gilles hasn't been removed from the leaderboard, I take it that he is still eligible for the Kaggle ranking points?
", "votes": null, "replies": [] }, { "id": 88328, "author_name": "abhishek", "author_url": "", "post_date": "08/04/2015 11:26:33", "content": "[quote=Bluefool;88326]
\n\nAs Gilles hasn't been removed from the leaderboard, I take it that he is still eligible for the Kaggle ranking points?
\n\n[/quote]
\n\nThere will be people in the end using the agreement test dataset just to gain Kaggle points. How do you plan to remove them? ;)
", "votes": null, "replies": [] }, { "id": 88329, "author_name": "glouppe", "author_url": "", "post_date": "08/04/2015 11:26:58", "content": "\n\n\nAs Gilles hasn't been removed from the leaderboard, I take it that he is still eligible for the Kaggle ranking points?
\n
Honestly, I couldnt care less about the Kaggle points, but I understand this may be important for some. I dont know how this works, but upon inspection of the submissions, only those deemed as valid get points, right?
", "votes": null, "replies": [] }, { "id": 88330, "author_name": "domcastro", "author_url": "", "post_date": "08/04/2015 11:28:05", "content": "Nope, you'll get the points
", "votes": null, "replies": [] }, { "id": 88332, "author_name": "domcastro", "author_url": "", "post_date": "08/04/2015 11:30:04", "content": "[quote=Abhishek;88328]
\n\n[quote=Bluefool;88326]
\n\nAs Gilles hasn't been removed from the leaderboard, I take it that he is still eligible for the Kaggle ranking points?
\n\n[/quote]
\n\nThere will be people in the end using the agreement test dataset just to gain Kaggle points. How do you plan to remove them? ;)
\n\n[/quote]
\n\nThat's my point - we might as well all just use the agreement test.
", "votes": null, "replies": [] }, { "id": 88362, "author_name": "udibr1", "author_url": "", "post_date": "08/04/2015 13:48:48", "content": "Gilles,\nYou can also take the position saying that your final submission was made before the rules were changed. Therefore it remains valid.
", "votes": null, "replies": [] }, { "id": 88421, "author_name": "lazovich", "author_url": "", "post_date": "08/04/2015 20:06:01", "content": "Sorry, but I have another question about this rule. In physics it is fairly common to actually correct for discrepancies in a particular variable between data and simulation by re-weighting events. So, for example, if you saw that in a particular bin of pT in the control region, data was 20% higher than MC, you might decide to re-weight the training events in that bin by a factor of 1.2. (The idea being that you are re-weighting events so that the distribution in data and MC for that variable matches). This is not training directly on the data in the control region, but it is using that data to improve the description of the input variables in simulation. Would an approach like this be allowed?
\n\nI certainly think it could be physically useful. If the simulation is getting the description of a variable wrong, and you know that from the data, you should be able to correct for it. You aren't using this mis-modeling for discrimination in that case. You're just improving your training sample to be more like the real world.
", "votes": null, "replies": [] }, { "id": 88519, "author_name": "albercik1905", "author_url": "", "post_date": "08/05/2015 13:14:23", "content": "Yep, we often make the reweighting procedure. But in this case you should correctly apply reweighting to necessary datasets and then compute the ks-metric, the cvm-metric and the weighted AUC with obtained weights. This we cannot support in the competition because of an inability to control right modifications of the datasets by participants. If you will reweight correctly and your solution will be interesting from physics point of view, you will be able to win special physics prize (of course your quality on LB won't be correct and can be higher, that is why you should present in your solution weights which will be used to obtain the correct quality).
\n\nCheers,\nMarcin
", "votes": null, "replies": [] }, { "id": 88575, "author_name": "udibr1", "author_url": "", "post_date": "08/05/2015 19:51:16", "content": "Am I correct in the following assumptions?
\n\n1) In the baseline notebook that the admin supplied. It is said:
\n\n\n\n\nHere we use subset of the all features to pass the agreement checking
\n
Am I allowed to perform feature selection based on running local control test?
\n\n2) In the noise notebook it is said:
\n\n\n\n\nAdd noise for the agreement checking
\n
So am I correct I am allowed to "Manually" add steps to my algorithm until I manage to pass the control test?
\n\nIf the answer to both questions is yes then what am I not allowed to perform?
", "votes": null, "replies": [] }, { "id": 88577, "author_name": "domcastro", "author_url": "", "post_date": "08/05/2015 20:11:26", "content": "It's all a bit confusing. I had some feature selection running with the checks built in but I stopped it. Now I'm not sure. At the moment, I'm only using checks before I submit, but even then I'll be selecting my models based on if they pass or not!
", "votes": null, "replies": [] }, { "id": 88606, "author_name": "mcvicuna", "author_url": "", "post_date": "08/06/2015 00:26:51", "content": "Hi,\n Are we allowed to use 'test.csv' for unsupervised learning and/or PCA given the new rules?\nThanks.
", "votes": null, "replies": [] }, { "id": 88614, "author_name": "fchollet", "author_url": "", "post_date": "08/06/2015 01:35:21", "content": "[quote=udibr;88575]
\n\nAm I correct in the following assumptions?
\n\n1) In the baseline notebook that the admin supplied. It is said:
\n\n\n\n\nHere we use subset of the all features to pass the agreement checking
\n
Am I allowed to perform feature selection based on running local control test?
\n\n2) In the noise notebook it is said:
\n\n\n\n\nAdd noise for the agreement checking
\n
So am I correct I am allowed to "Manually" add steps to my algorithm until I manage to pass the control test?
\n\nIf the answer to both questions is yes then what am I not allowed to perform?
\n\n[/quote]
\n\nTo admins: if we are allowed to do feature engineering or feature selection, then we are allowed to train on check_agreeement.csv too (because the two are strictly equivalent). I.e. we are also allowed to exploit simulation flaws as much as possible, and submit models that are physically useless but highly performant.
\n\nJust in case you don't see the equivalence: first we train a shallow NN or a tree-based model on both the training data and the check_agreement.csv data, via a loss function that ensures that the model learns to discriminate between the two types of events. Then we record the features developed by such a model (e.g. the first-layer representations in the NN), and reinject these features in a model that will be trained only on training.csv. Job done.
\n\nThere would be no trace of the initial training on check_agreement.csv, just a bunch of hard-coded magical features.
\n\nYou cannot escape the fact that, by design, the only way to perform well in this competition is by exploiting the leaks in the data. Which is easy to do, because the KS test is easy to pass while exploiting the leaks, no matter how much you change the rules. You will not gain any physical insight from this competition.
\n\nIgnore it and face a bigger debacle later on.
", "votes": null, "replies": [] }, { "id": 88617, "author_name": "fchollet", "author_url": "", "post_date": "08/06/2015 01:43:49", "content": "[quote=Mark Vicuna;88606]
\n\nHi,\n Are we allowed to use 'test.csv' for unsupervised learning and/or PCA given the new rules?\nThanks.
\n\n[/quote]
\n\nIf we were allowed to use the data in test.csv, it would be easy to exploit it to learn to discriminate between test events and control events (since they have quite different distributions).
\n\nShort list of ways to exploit simulation leaks while passing the KS test:
\n\nClearly, you have to add the following rules to the competition to make it even funnier:
\n\n[quote=fchollet;88617]
\n\n[quote=Mark Vicuna;88606]
\n\nHi,\n Are we allowed to use 'test.csv' for unsupervised learning and/or PCA given the new rules?\nThanks.
\n\n[/quote]
\n\nIf we were allowed to use the data in test.csv, it would be easy to exploit it to learn to discriminate between test events and control events (since they have quite different distributions).
\n\nShort list of ways to exploit simulation leaks while passing the KS test:
\n\nClearly, you have to add the following rules to the competition to make it even funnier:
\n\n[/quote]
\n\nOk. Thanks. I hadn't put two and two together and realized they added specific data to the test.csv that isn't present at all in the training.csv data to confuse models.
\n\nI think you are absolutely right there is no good answer to the RNG problem. The seed and algorythim are now meta-parameters that you can use to find the model that finds the leaky data. You can then completely remove that step from the submitted model and whose to say '47562' isn't your favorite number?
\n\nThey would have to provide an RNG and enforce a seed to work around that loophole IMO.
", "votes": null, "replies": [] }, { "id": 88664, "author_name": "erlend", "author_url": "", "post_date": "08/06/2015 12:52:17", "content": "Hi Mark, I'd be very interested to see how one would implement your second point (direct training with a custom loss function on both training.csv and check_agreement.csv). I'm not interested in using to in this competition, but I'd like to know how to do this in general since I think it's complicated by the fact that the agreement check is really checking a global systemic property.
\n\nWould you, or someone, be willing to demonstrate code for that (either during or after the competition)?
\n\nThanks,\nErlend
", "votes": null, "replies": [] }, { "id": 88679, "author_name": "udibr1", "author_url": "", "post_date": "08/06/2015 15:02:30", "content": "the issues raised in this forum can sink the entire kaggle web site if not addressed correctly.\nsave kaggle
", "votes": null, "replies": [] }, { "id": 88705, "author_name": "lazovich", "author_url": "", "post_date": "08/06/2015 19:01:49", "content": "Another issue to consider...from the initial post the motivation for disallowing use of check_agreement and check_correlation is that you don't want to get models that exploit the differences between real data and simulation. In that case, wouldn't it make sense to use the control data to do feature selection? If I were to make plots of different variables to see how well they discriminate between signal and background, I would also want to look in the control region to see if that variable is actually discriminative or if it is simply mis-modeled. Is using the check_agreement dataset to eliminate features that ruin the agreement allowed? That seems to me to be one of the whole points of including that data.
", "votes": null, "replies": [] }, { "id": 88711, "author_name": "holzner", "author_url": "", "post_date": "08/06/2015 20:13:52", "content": "[quote=fizzicks;88421]
\n\nSorry, but I have another question about this rule. In physics it is fairly common to actually correct for discrepancies in a particular variable between data and simulation by re-weighting events.
\n\n[/quote]
\n\nin fact, we usually check the agreement / reweight the one-dimensional projections between background and data but then go ahead and exploit the differences between signal and background in the full N-dimensional space. I have only seen one example yet (ATLAS Higgs to bb search) where also two-dimensional projections were checked.
", "votes": null, "replies": [] }, { "id": 88751, "author_name": "fchollet", "author_url": "", "post_date": "08/07/2015 00:56:21", "content": "[quote=udibr;88679]
\n\nthe issues raised in this forum can sink the entire kaggle web site if not addressed correctly.\nsave kaggle
\n\n[/quote]
\n\nNo, I don't think what happened in this competition will have any impact on future Kaggle competitions.
\n\nBut there are no doubts that the current competition is over at this point. I don't see what could possibly save it, short of starting from scratch with different data and different rules (and no KS test).
\n\nSure, it is in bad taste to change the rules in the middle of the competition, especially in a way that invalidates the work of many entrants (including many (all?) top entrants). Sure, it's in bad taste not to reply to questions and concerns addressed to admins, too. Sure, it was unfortunate to submit a competition that was flawed by design, but then again, it's something that has happened a few times on Kaggle before. Data science is hard. Organizing a competition requires strong data science competence. Teams with such strong data science competence are less likely to turn to Kaggle competitions to solve their problems...
\n\nClearly, the only viable answer here is just to move away from this competition. The first dozen hours you spent on this challenge have been thoroughly wasted, so just don't spend any more time.
\n\nLet's move on.
", "votes": null, "replies": [] }, { "id": 88752, "author_name": "fchollet", "author_url": "", "post_date": "08/07/2015 01:04:37", "content": "[quote=Erlend Davidson;88664]
\n\nHi Mark, I'd be very interested to see how one would implement your second point (direct training with a custom loss function on both training.csv and check_agreement.csv). I'm not interested in using to in this competition, but I'd like to know how to do this in general since I think it's complicated by the fact that the agreement check is really checking a global systemic property.
\n\nWould you, or someone, be willing to demonstrate code for that (either during or after the competition)?
\n\nThanks,\nErlend
\n\n[/quote]
\n\nI believe this comment was actually meant as a reply to my post.
\n\nMy first entries in this competition were based on this scheme (a custom loss function to train a classifier that, at the same time, classifies signal from background in the test data, and outputs similar probabilities for signal and background of the control data).
\n\nSure, ROC AUC is a global property. It also has a statistical / information-theoretical meaning, and therefore you can optimize for it via per-sample proxy metrics from statistics or information theory...
\n\nThe most basic thing you can do is use binary crossentropy. Minimize crossentropy of signal/background on the test data while maintaining the crossentropy of signal/background on the control data at some carefully selected value (typically the maximum entropy value). The latter is done via minimization of the difference between the crossentropy and the target value.
\n\nThis can be treated as a global optimization problem by using as objective a weighted sum of these two objectives.
\n\nThe result, in practice, are highly performant models (e.g. up to 0.996 for a single NN, no ensembles, and after very little search/optimization of the setup) with very low KS scores (generally around 0.05-0.06).
", "votes": null, "replies": [] }, { "id": 88755, "author_name": "udibr1", "author_url": "", "post_date": "08/07/2015 01:45:45", "content": "I used a simpler method that worked nicely. Build a predictor to predict if a sample is from the training set or the check_agreement set and use the output of this predictor to reduce the features that are known to increase the KS score (for example SPDhits) whenever a sample looks like it is coming from check_agreement and not from training.
\n\nThe reduction was done as follows: convert the feature (SPDhits) to log(Psignal/Pbackground) and then multiply by the probability to be from the training set and not from the check_agreement set
", "votes": null, "replies": [] }, { "id": 88756, "author_name": "udibr1", "author_url": "", "post_date": "08/07/2015 01:49:05", "content": "[quote=fchollet;88751]
\n\nLet's move on.
\n\n[/quote]
\n\nThanks, will do
", "votes": null, "replies": [] }, { "id": 88760, "author_name": "udibr1", "author_url": "", "post_date": "08/07/2015 02:29:06", "content": "To admin, one final thing. I am not sure about this but it looks to me that a classifier that avoids the KS score ( for example in the method I just described ) gives better prediction results than a simpler classifier that ignores the KS test.
\n\nIf this is true than it can be useful for real future research in this area. Perhaps the background contains D decay events and the reduction method helps in detecting these events and correctly placing them in the background bin.
", "votes": null, "replies": [] }, { "id": 88768, "author_name": "fchollet", "author_url": "", "post_date": "08/07/2015 04:07:47", "content": "[quote=udibr;88760]
\n\nTo admin, one final thing. I am not sure about this but it looks to me that a classifier that avoids the KS score ( for example in the method I just described ) gives better prediction results than a simpler classifier that ignores the KS test.
\n\nIf this is true than it can be useful for real future research in this area. Perhaps the background contains D decay events and the reduction method helps in detecting these events and correctly placing them in the background bin.
\n\n[/quote]
\n\nThe idea behind the custom loss function I was using was the following:
\n\nIn theory it would work perfectly, except in practice it doesn't. Because the assumption that the KS score is meaningful is just plain false. So what happens in practice is that a classifier trained in this way learns to discriminate control data while exploiting simulation artifacts.
\n\nIt is also worth pointing that the problem of finding which features are "valid" (physics-based) and which are invalid, isn't solvable with the data provided. It cannot be found from the training data (because events are either real and background, or simulated and signal), and it cannot be found from the control data (because it has a completely different distribution from the test data, and therefore can be discriminated against to cheat the test).
\n\nI.e. the problem that the organizers wanted to solve cannot be solved with the data that the organizers provided. Which is why nothing interesting will come out of this competition.
", "votes": null, "replies": [] }, { "id": 89345, "author_name": "albercik1905", "author_url": "", "post_date": "08/14/2015 08:13:32", "content": "", "votes": null, "replies": [] }, { "id": 89447, "author_name": "mihailr812", "author_url": "", "post_date": "08/15/2015 16:37:13", "content": "Several cases just to clarify this topic.
\n\nCan we
\n\nSo for sure you can do 1. The 2 and 3. depends on what exactly you mean by: "research correlations" and "make assumptions":
\n\nIf you want to plot distributions and choose only the ones that show good agreement then yes please :) If you want to somehow to create variable that is somehow distinguishing between tau->3mu MC and Ds->Phi Pi MC then no.
\n\nCheers,\nMarcin
", "votes": null, "replies": [] }, { "id": 89554, "author_name": "fchollet", "author_url": "", "post_date": "08/17/2015 10:25:44", "content": "[quote]
\n\nCan we
\n\n[/quote]
\n\n[quote] So for sure you can do 1. [/quote]
\n\nExcellent. Just to be clear that you fully understand what you are stating here, here's a code skeleton that is a straightforward implementation of 1. Is this code skeleton valid by your rules?
\n\nX_agreement, y_agreement, w_agreement = get_agreement_data()\nX, y = get_training_data()\ninitial_params = initialize() # just random {x1..xn}\nmodel = model_from_params(initial_params, X, y)\ncurrent_score = compute_roc_auc(model, X, y)\ncurrent_ks = compute_ks(model, X_agreement, y_agreement, w_agreement)\n\nwhile not stopping_criterion.check():\n # just increment / decrement initial {x1..xn}\n initial_params = update_parameters(initial_params, current_score, current_ks)\n\n # model_from_params could even make no use of X, y\n # i.e. this step isn't even necessary\n model = model_from_params(initial_params, X, y)\n\n current_score = compute_roc_auc(model, X, y)\n current_ks = compute_ks(model, X_agreement, y_agreement, w_agreement)\n\n stopping_criterion.update(current_score, current_ks)\nOk. I'll try to generalize method.\nAny "black box" (imagine, that we have to submit our test set instead of running local tests) analysis is allowed (in assumption, that we haven't got correlation and agreement datasets) ?
", "votes": null, "replies": [] }, { "id": 89595, "author_name": "anaderi", "author_url": "", "post_date": "08/17/2015 19:38:01", "content": "Mikhail, I'm not sure I got your question, but in the end winning solution should be accompanied with compilable & runnable code (that produces the same results) and explanation of how this solution operates on given data.
", "votes": null, "replies": [] }, { "id": 89600, "author_name": "anaderi", "author_url": "", "post_date": "08/17/2015 19:48:24", "content": "Francois, your solution skeleton would qualify as "using check_agreement for model testing". If you follow the logic from the post https://www.kaggle.com/c/flavours-of-physics/forums/t/15974/machine-learning-approach-in-high-energy-physics-explanation, most likely it should be fine. To be sure that solution is OK we demand participants to provide the compilable&runnable code and verbal description of their approaches in the end.
\n\nNB we realize that things happening inside loop while not stopping_criterion.check() can also be treated as model training effectively overfitting to model restrictions. It is not much worse than overfitting to public leaderboard (for example http://blog.mrtz.org/2015/03/09/competition.html and http://jmlr.org/proceedings/papers/v37/blum15.pdf) and this is something all challenges have to cope with for the given competition framework.
Ok, so, but what if a solution strictly follows this code skeleton, but the explanation of how it works is, as you state, "(over)fitting to model restrictions" (actually just fitting, i.e. training)? Is the solution no longer eligible then, even though it is following a training pattern that has been claimed as acceptable?
\n\nBasically, why "most likely it should be fine"? Is this training pattern valid, or not? If it is valid, it should stay valid no matter what goes into the functions model_from_params(init_params, X, y) and update_parameters(init_params, score, ks)? (of course these functions would not be be accessing new data, or anything sketchy like that. Just straightforward logic applied to their arguments).
Francois, unless anything "sketchy" happens inside the functions you've mentioned (that's the reason for 'most likely') the pattern is fine.
", "votes": null, "replies": [] }, { "id": 89607, "author_name": "fchollet", "author_url": "", "post_date": "08/17/2015 20:19:23", "content": "By sketchy I just mean external data access. Of course these functions would only be processing the arguments that we pass them. There is no trick behind my code skeleton, it would do exactly what the function names imply.
\n\nAlright, if this pattern is allowed, that's good news for us, but you do realize that this pattern is all about training on the control data, right? Via update_parameters(init_params, score, ks), which injects KS score information into the model. By repeating the loop as many times as needed, you can end up leaking as much KS score information as you want.
Just like "overfitting the leaderboard" would eventually result in perfect scores if we didn't have daily submission limits.
", "votes": null, "replies": [] }, { "id": 89612, "author_name": "mihailr812", "author_url": "", "post_date": "08/17/2015 22:01:20", "content": "Actually, the shown code is exactly what i mean, when saying "black box" analisis.\nSounds good, thank you.
", "votes": null, "replies": [] }, { "id": 89802, "author_name": "jstaker7", "author_url": "", "post_date": "08/19/2015 14:49:14", "content": "Just to be explicit, that also means that a custom loss function where KS is included as a penalty is strictly allowed? I thought I read in the forum somewhere that doing so was not allowed. But this is essentially doing the same thing as fchollet's code snippet.
", "votes": null, "replies": [] }, { "id": 89818, "author_name": "fchollet", "author_url": "", "post_date": "08/19/2015 17:06:28", "content": "\n\n\nBut this is essentially doing the same thing as fchollet's code snippet.
\n
It is indeed. It's just semantics. You are allowed to "train" on the control data, but training on the control data is forbidden. My personal take on this is that you should figure out ways to train on the control data as much as possible while camouflaging the process to keep some facade of respectability in the eyes of the organizers (which my snippet does). At least that's how I interpret these recent developments.
", "votes": null, "replies": [] }, { "id": 89880, "author_name": "holzner", "author_url": "", "post_date": "08/20/2015 09:01:53", "content": "[quote=fchollet;89818]
\n\nIt is indeed. It's just semantics. You are allowed to "train" on the control data, but training on the control data is forbidden.
\n\n[/quote]
\n\nI guess the original intent of the competition is that one uses compute_ks(..) as an 'oracle' without looking at the entries in check_agreement.csv (although I suspect that given a huge amount of CPU time one could even infer the contents of check_agreement.csv this way....)
", "votes": null, "replies": [] }, { "id": 89887, "author_name": "erlend", "author_url": "", "post_date": "08/20/2015 10:46:46", "content": "[quote=Andre Holzner;89880]
\n\n[quote=fchollet;89818]
\n\nIt is indeed. It's just semantics. You are allowed to "train" on the control data, but training on the control data is forbidden.
\n\n[/quote]
\n\nI guess the original intent of the competition is that one uses compute_ks(..) as an 'oracle' without looking at the entries in check_agreement.csv (although I suspect that given a huge amount of CPU time one could even infer the contents of check_agreement.csv this way....)
\n\n[/quote]
\n\nIt doesn't matter if you "look" (i.e. open the file and read the contents of it), if you accept/reject your models based on some test (i.e. the KS-metric) derived from the data in check_agreement.csv then some of check_agreement.csv will leak into your data. The same sort of problem can happen if you score your models using just a cross-validation on a static test-set.
\n\nThis is described in the first two paragraphs of http://k-d-w.org/node/95, and also in §7.10.2 of the book here: http://statweb.stanford.edu/~tibs/ElemStatLearn/.
", "votes": null, "replies": [] }, { "id": 90877, "author_name": "chrisfs", "author_url": "", "post_date": "08/29/2015 19:41:06", "content": "Wow, I find the discussion on this thread to be discouraging. The point of the competition is to help move forward an understanding of physics and yet a number of contributors seem more intent on arguing about the rules changes as if it were a casual game. Yes you may have to go back and do things over, or come up with a different approach. The rule changes are there to provide for usable data for physicists. \nThat is the ultimate point of the challenge, not whether you can successively argue for rules in your favor or use clever interpretations to get an upper hand.
", "votes": null, "replies": [] }, { "id": 91685, "author_name": "vwoodnz", "author_url": "", "post_date": "09/06/2015 04:55:59", "content": "[quote=Chrisfs;90877]\n The point of the competition is to help move forward an understanding of physics and yet a number of contributors seem more intent on arguing about the rules changes as if it were a casual game. Yes you may have to go back and do things over, or come up with a different approach. The rule changes are there to provide for usable data for physicists. \n[/quote]
\n\nNot really. If you set up a system that scores entries, people will optimise for that measure. If you want to complain later that the optimal solution for that measure didn't solve your initial problem, well then it's your fault for posing the problem incorrectly.
", "votes": null, "replies": [] }, { "id": 91900, "author_name": "g1685786", "author_url": "", "post_date": "09/09/2015 03:51:49", "content": "[quote=fchollet;88751]
\n\nNo, I don't think what happened in this competition will have any impact on future Kaggle competitions.
\n\n[/quote]
\n\nWhy on earth should we learn from mistakes? If it's not in a file called "train.csv" we don't care!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "88129": "Dear participants,\r\n\r\nAs the competition progresses rapidly, we’d like to draw your attention to the data to be used for the purpose of training your model. It is mentioned in the [data page][1], but we’d like it to be a bit more explicit. **To be eligible for the prize your final solution should rely solely on the data provided in ‘training.csv’ without any usage of ‘check_correlation.csv' or ‘check_agreement.csv' datasets.**\r\n\r\nThe main motivation behind this restriction is that we want to make this competition as useful as possible from physics application, and having models that fit to ‘agreement_check’ data just to pass agreement test and at the same time behave quite differently on the signal channel is in general meaningless from physics point of view, but rather easy to achieve ‘accidentally' by adding ‘check_correlation.csv' or ‘check_agreement.csv' into the training process.\r\n\r\nThe idea behind the agreement test is to check if the model takes advantage of the fact that there are differences between the samples obtained from simulation and the real data. These are due to imperfections in the simulation and a classifier useful for a physics analysis should not take advantage of the differences between samples from the same class that arise from these imperfections.\r\n\r\nPS. We would like to encourage everyone to submit descriptions of their solutions upon the challenge finish, since even the lower-ranked solution might be useful for physicists in other analyses. In particular, approaches that are not main stream will certainly receive attention from CERN researchers and their authors could be invited to one of the [post-competition ML-HEP workshops][2].\r\n\r\n\r\n [1]: https://www.kaggle.com/c/flavours-of-physics/data\r\n [2]: https://www.kaggle.com/c/flavours-of-physics/details/prizes", "88150": "Thank you for the clarification but I could not see any reference to this explicit limitation anywhere in your original description of the problem.\r\nCan you please give an explicit quote from the original description of the problem.\r\nChanging the rules while the competition takes place does not add much fun and undermine the entire value of a site like kaggle.\r\n\r\nAs an off-topic, I think there is real value in taking other existing channels into account when modeling. In a real situation they do add to the noise and modeling all the possible noise is one way of improving the signal.", "88151": "Under our [Data page][1]:\r\n\"You need to submit predictions for ALL the test entries. You will need to treat them all the same and predict as if they are all the same channel's collision events. \"\r\n\r\n\r\n [1]: https://www.kaggle.com/c/flavours-of-physics/data", "88155": "udibr, you are right, there might be a real value in taking other channels into account and we do not deny people will find how find that value, that's why we have Special Physics prize in this challenge and a couple of workshops (at NIPS (if confirmed) and at Zurich) to discuss these approaches.", "88157": "I am very sorry, but, perhaps I am wrong but I think that what you quoted is not relevant.\r\nYour quote refers to how predictions are made. I have a single prediction algorithm that runs on all examples in exactly the same way, so it fits your description of \"same channel\".\r\nThe only question is how this algorithm was trained in the first place.\r\n\r\nAs you can see for yourself the quote you have taken is from the part talking about the test process and not about the training process.\r\n\r\nIn any case you should refer to this page https://www.kaggle.com/c/flavours-of-physics/rules\r\nand not to any other page of the competition when specifying rules.\r\nThis is because the rules page (which I had to accept when downloading the data and no other page) begins with \"These are the complete, official rules for the Competition...\"\r\n\r\nThe rules page states that \"Use of external data is not permitted\" and that \"'Data' means the Data or Datasets linked from the Competition Website for the purpose of use by Participants in the Competition.\"\r\nthis covers all the csv files.\r\n\r\nI would also like to add that developing methods that uses the other csv files will help the process of discovering new physics.\r\nThis is because the main challenge in this competition is not finding new physics now but building tools for doing it in the future and perhaps the most important tool will be an accurate simulator and test sets that take any adversary action into account. If the simulator is not accurate enough to describe the \"real\" physics then it needs to be improved and if the gaps it has were find in this competition then all the better.", "88163": "[quote]\r\nTo be eligible for the prize your final solution should rely solely on the data provided in ‘training.csv’ without any usage of ‘check_correlation.csv' or ‘check_agreement.csv' datasets.\r\n[/quote]\r\n\r\nDoes this means we cannot run the KS and correlation tests locally? If so, then why are these datasets provided in the first place?\r\n\r\nIt seems that what you are trying to say is, \"don't train your model using these datasets\". But of course, selecting a model based on whether it passes the KS test is the exact same as using the dataset ‘check_correlation.csv' to train your model. This much should be obvious to anyone.\r\n\r\nJust in case it isn't obvious for you, here's a thought experiment: the no-training model. Your set up a logistic regression model to classify signal from background, but instead of learning the parameters through gradient descent, you randomly generate them. Then you check whether your random model passes the KS test and correlation test, and you check what the ROC AUC is. You do this with 1,000,000 models (very much realistic and computationally tractable) and you pick the model with the highest ROC AUC. This is in fact a solid strategy if you don't feel like computing the gradients of the ROC AUC and KS scores.\r\n\r\nThis model never ever saw the data in ‘check_correlation.csv' (or any training data, for that matter), but it was still effectively trained on it... through post-selection. ANY USE of ‘check_correlation.csv' means that you are training your model (i.e. selecting its parameters) based on the data in ‘check_correlation.csv'. If you are running the KS test locally, you are breaking the rules. But if you are not, you are insane (enjoy submitting your .csv 100 times in a row).\r\n\r\nBy the way, the same is also true with the test happening on the Kaggle servers. As long as you are selecting models based on whether they pass the KS test, then these models are in effect trained on the KS data (i.e. the parameters of your final model have been chosen based on the KS data). There is no way around it whatsoever. \r\n\r\n[quote=udibr;88157]\r\nI have a single prediction algorithm that runs on all examples in\r\nexactly the same way, so it fits your description of \"same channel\".\r\n[/quote]\r\n\r\nSo do I (no special case for processing the control data), and as you do I also leverage the KS data for training. Also, I can't help but remark that what we are doing inevitably results in models that pick up simulation defects while still managing to work around the KS test. It's what we optimize for. \r\n\r\nThis competition is broken in a fundamental way. The organizers will have to backpedal, and possibly cancel the competition altogether, if their goal is a model that is useful from a physics perspective. No such model is going to emerge in the current setup.", "88165": "I find no ambiguity in the statement\r\n\r\n> To be eligible for the prize your final solution should rely solely on\r\n> the data provided in ‘training.csv’ without any usage of\r\n> ‘check_correlation.csv' or ‘check_agreement.csv' datasets.\r\n\r\nI think you are just taking the statement to its literary absurd extreme. Do not train models on these datasets. You can use them to evaluate that your models meets the competition goals and that is why they were provided.\r\n\r\nI myself experimented using these datasets in my training (as in use them to train models) but I figured that was not in the spirit of the comp so I did not submit any of those experiments. This was before clarification above was made so I think this statement was implied (softly whispered) even before the clarification in this topic.", "88166": "Also, just FYI: the control data and test data have significantly different distributions, so it is possible to by-pass the KS test entirely without ever looking at the KS data, simply by setting up a 1-class classifier (output: \"belongs to the class\"/\"doesn't belong\"). Only data points that have the same distribution as the data in training.csv would be classified, others would get a constant prediction (for instance). KS test solved. Such a classifier can be obtained by using randomly generated data as negative examples (no need for the KS data).\r\n\r\nThink this process would violate the rules? Well, it is also possible to set up a signal/BG model that implicitly do the job of this classifier, without any explicit special case for KS control data.", "88167": "[quote=Guido Tapia;88165]\r\ntaking the statement to its literary absurd extreme.\r\n[/quote]\r\nI think fchollet is right. If we take \"training a model\" to mean \"using information from\" then any locally performed check adds information to your model selection pipeline. No matter how indirectly or directly, purposeful or accidental, this information will ultimately leak and end up into your submissions.", "88168": "[quote=Guido Tapia;88165]\r\n\r\nI think you are just taking the statement to its literary absurd extreme. Do not train models on these datasets. You can use them to evaluate that your models meets the competition goals and that is why they were provided.\r\n\r\n[/quote]\r\n\r\nYou seem to misunderstand what \"training a model\" means. It does not mean calling the \"fit\" method on a sklearn classifier. It means the process of selecting the parameters of your model. Any usage of the KS data to select parameters, hyperparameters or even the type of classifier to use means that you are training your solution on the KS data.\r\n\r\nYou know how everybody tends to overfit to the leaderboard even though we don't have access to the labels of the leaderboard? This is the same. When you are selecting any aspect of your model based on some score, it means that data used to generate this score is part of your training data.", "88169": "I agree people will over fit the validation datasets as they will the public leader board. But I think its a compromise that was understood and accepted. And hopefully, even with these imperfect constraints some valuable insights can be learnt. \r\n\r\nI think this is a very complex problem and we only have the current set of tools and platforms to solve the problem, lets make the best of it. I just think suggesting the competition be scrapped because its not perfect is the wrong approach. \r\n\r\nI'll keep going, if KS metric gets too frustrating I'll go back to caterpillar which I left because I found the craziness in the data set structure too frustrating. Again no competition is perfect and if it is then you get 3000 competitors submitting each others scripts and filling the forums with noise.", "88263": "Isn't the \"Agreement test\" meant as an estimate of the systematic error?\r\nThus, if you use it to train, it messes up just that?\r\n\r\nEdit: removed rant about MC", "88326": "As Gilles hasn't been removed from the leaderboard, I take it that he is still eligible for the Kaggle ranking points?", "88328": "[quote=Bluefool;88326]\r\n\r\nAs Gilles hasn't been removed from the leaderboard, I take it that he is still eligible for the Kaggle ranking points?\r\n\r\n[/quote]\r\n\r\nThere will be people in the end using the agreement test dataset just to gain Kaggle points. How do you plan to remove them? ;)", "88329": "> As Gilles hasn't been removed from the leaderboard, I take it that he is still eligible for the Kaggle ranking points?\r\n\r\nHonestly, I couldnt care less about the Kaggle points, but I understand this may be important for some. I dont know how this works, but upon inspection of the submissions, only those deemed as valid get points, right?", "88330": "Nope, you'll get the points", "88332": "[quote=Abhishek;88328]\r\n\r\n[quote=Bluefool;88326]\r\n\r\nAs Gilles hasn't been removed from the leaderboard, I take it that he is still eligible for the Kaggle ranking points?\r\n\r\n[/quote]\r\n\r\nThere will be people in the end using the agreement test dataset just to gain Kaggle points. How do you plan to remove them? ;)\r\n\r\n[/quote]\r\n\r\nThat's my point - we might as well all just use the agreement test.", "88362": "Gilles,\r\nYou can also take the position saying that your final submission was made before the rules were changed. Therefore it remains valid.", "88421": "Sorry, but I have another question about this rule. In physics it is fairly common to actually correct for discrepancies in a particular variable between data and simulation by re-weighting events. So, for example, if you saw that in a particular bin of pT in the control region, data was 20% higher than MC, you might decide to re-weight the training events in that bin by a factor of 1.2. (The idea being that you are re-weighting events so that the distribution in data and MC for that variable matches). This is not training directly on the data in the control region, but it is using that data to improve the description of the input variables in simulation. Would an approach like this be allowed? \r\n\r\nI certainly think it could be physically useful. If the simulation is getting the description of a variable wrong, and you know that from the data, you should be able to correct for it. You aren't using this mis-modeling for discrimination in that case. You're just improving your training sample to be more like the real world.", "88519": "Yep, we often make the reweighting procedure. But in this case you should correctly apply reweighting to necessary datasets and then compute the ks-metric, the cvm-metric and the weighted AUC with obtained weights. This we cannot support in the competition because of an inability to control right modifications of the datasets by participants. If you will reweight correctly and your solution will be interesting from physics point of view, you will be able to win special physics prize (of course your quality on LB won't be correct and can be higher, that is why you should present in your solution weights which will be used to obtain the correct quality). \r\n\r\nCheers,\r\nMarcin", "88575": "Am I correct in the following assumptions?\r\n\r\n1) In the [baseline notebook][1] that the admin supplied. It is said:\r\n\r\n> Here we use subset of the all features to pass the agreement checking\r\n\r\nAm I allowed to perform feature selection based on running local control test?\r\n\r\n2) In the [noise notebook][2] it is said:\r\n\r\n> Add noise for the agreement checking\r\n\r\nSo am I correct I am allowed to \"Manually\" add steps to my algorithm until I manage to pass the control test?\r\n\r\nIf the answer to both questions is yes then what am I not allowed to perform?\r\n\r\n [1]: https://github.com/yandexdataschool/flavours-of-physics-start/blob/master/baseline.ipynb\r\n [2]: https://github.com/yandexdataschool/flavours-of-physics-start/blob/master/noise.ipynb", "88577": "It's all a bit confusing. I had some feature selection running with the checks built in but I stopped it. Now I'm not sure. At the moment, I'm only using checks before I submit, but even then I'll be selecting my models based on if they pass or not!", "88606": "Hi,\r\n Are we allowed to use 'test.csv' for unsupervised learning and/or PCA given the new rules?\r\nThanks.", "88614": "[quote=udibr;88575]\r\n\r\nAm I correct in the following assumptions?\r\n\r\n1) In the [baseline notebook][1] that the admin supplied. It is said:\r\n\r\n> Here we use subset of the all features to pass the agreement checking\r\n\r\nAm I allowed to perform feature selection based on running local control test?\r\n\r\n2) In the [noise notebook][2] it is said:\r\n\r\n> Add noise for the agreement checking\r\n\r\nSo am I correct I am allowed to \"Manually\" add steps to my algorithm until I manage to pass the control test?\r\n\r\nIf the answer to both questions is yes then what am I not allowed to perform?\r\n\r\n [1]: https://github.com/yandexdataschool/flavours-of-physics-start/blob/master/baseline.ipynb\r\n [2]: https://github.com/yandexdataschool/flavours-of-physics-start/blob/master/noise.ipynb\r\n\r\n[/quote]\r\n\r\nTo admins: if we are allowed to do feature engineering or feature selection, then we are allowed to train on check_agreeement.csv too (because the two are strictly equivalent). I.e. we are also allowed to exploit simulation flaws as much as possible, and submit models that are physically useless but highly performant.\r\n\r\nJust in case you don't see the equivalence: first we train a shallow NN or a tree-based model on both the training data and the check_agreement.csv data, via a loss function that ensures that the model learns to discriminate between the two types of events. Then we record the features developed by such a model (e.g. the first-layer representations in the NN), and reinject these features in a model that will be trained only on training.csv. Job done. \r\n\r\nThere would be no trace of the initial training on check_agreement.csv, just a bunch of hard-coded magical features.\r\n\r\nYou cannot escape the fact that, *by design*, the only way to perform well in this competition is by exploiting the leaks in the data. Which is easy to do, because the KS test is easy to pass while exploiting the leaks, no matter how much you change the rules. You will *not* gain any physical insight from this competition. \r\n\r\nIgnore it and face a bigger debacle later on.", "88617": "[quote=Mark Vicuna;88606]\r\n\r\nHi,\r\n Are we allowed to use 'test.csv' for unsupervised learning and/or PCA given the new rules?\r\nThanks.\r\n\r\n[/quote]\r\n\r\nIf we were allowed to use the data in test.csv, it would be easy to exploit it to learn to discriminate between test events and control events (since they have quite different distributions). \r\n\r\n**Short list of ways to exploit simulation leaks while passing the KS test:**\r\n\r\n- post-classification filtering of control events via a control classifier trained on training.csv and check_agreement.csv **[newly disallowed]**\r\n- direct training with a custom loss function on both training.csv and check_agreement.csv **[newly disallowed]**\r\n- either of the methods above with control data that you generate yourself via a RNG **[likely allowed]**\r\n- using the test.csv data to learn to discriminate between the test events and control events **[likely disallowed]**\r\n- feature engineering step before training **[likely allowed]**\r\n\r\n\r\nClearly, you have to add the following rules to the competition to make it even funnier:\r\n\r\n- disallow random number generators\r\n- no feature engineering allowed", "88620": "[quote=fchollet;88617]\r\n\r\n[quote=Mark Vicuna;88606]\r\n\r\nHi,\r\n Are we allowed to use 'test.csv' for unsupervised learning and/or PCA given the new rules?\r\nThanks.\r\n\r\n[/quote]\r\n\r\nIf we were allowed to use the data in test.csv, it would be easy to exploit it to learn to discriminate between test events and control events (since they have quite different distributions). \r\n\r\n**Short list of ways to exploit simulation leaks while passing the KS test:**\r\n\r\n- post-classification filtering of control events via a control classifier trained on training.csv and check_agreement.csv **[newly disallowed]**\r\n- direct training with a custom loss function on both training.csv and check_agreement.csv **[newly disallowed]**\r\n- either of the methods above with control data that you generate yourself via a RNG **[likely allowed]**\r\n- using the test.csv data to learn to discriminate between the test events and control events **[likely disallowed]**\r\n- feature engineering step before training **[likely allowed]**\r\n\r\n\r\nClearly, you have to add the following rules to the competition to make it even funnier:\r\n\r\n- disallow random number generators\r\n- no feature engineering allowed\r\n\r\n[/quote]\r\n\r\nOk. Thanks. I hadn't put two and two together and realized they added specific data to the test.csv that isn't present at all in the training.csv data to confuse models.\r\n\r\nI think you are absolutely right there is no good answer to the RNG problem. The seed and algorythim are now meta-parameters that you can use to find the model that finds the leaky data. You can then completely remove that step from the submitted model and whose to say '47562' isn't your favorite number?\r\n\r\nThey would have to provide an RNG and enforce a seed to work around that loophole IMO.", "88664": "Hi Mark, I'd be very interested to see how one would implement your second point (direct training with a custom loss function on both training.csv and check_agreement.csv). I'm not interested in using to in this competition, but I'd like to know how to do this in general since I think it's complicated by the fact that the agreement check is really checking a global systemic property. \r\n\r\nWould you, or someone, be willing to demonstrate code for that (either during or after the competition)?\r\n\r\nThanks,\r\nErlend", "88679": "the issues raised in this forum can sink the entire kaggle web site if not addressed correctly.\r\n[save kaggle][1]\r\n\r\n\r\n [1]: https://www.kaggle.com/forums/f/15/kaggle-forum/t/15805/save-kaggle/88678#post88678", "88705": "Another issue to consider...from the initial post the motivation for disallowing use of check_agreement and check_correlation is that you don't want to get models that exploit the differences between real data and simulation. In that case, wouldn't it make sense to use the control data to do feature selection? If I were to make plots of different variables to see how well they discriminate between signal and background, I would also want to look in the control region to see if that variable is actually discriminative or if it is simply mis-modeled. Is using the check_agreement dataset to eliminate features that ruin the agreement allowed? That seems to me to be one of the whole points of including that data.", "88711": "[quote=fizzicks;88421]\r\n\r\nSorry, but I have another question about this rule. In physics it is fairly common to actually correct for discrepancies in a particular variable between data and simulation by re-weighting events. \r\n\r\n[/quote]\r\n\r\nin fact, we usually check the agreement / reweight the one-dimensional projections between background and data but then go ahead and exploit the differences between signal and background in the full N-dimensional space. I have only seen one example yet (ATLAS Higgs to bb search) where also two-dimensional projections were checked.", "88751": "[quote=udibr;88679]\r\n\r\nthe issues raised in this forum can sink the entire kaggle web site if not addressed correctly.\r\n[save kaggle][1]\r\n\r\n\r\n [1]: https://www.kaggle.com/forums/f/15/kaggle-forum/t/15805/save-kaggle/88678#post88678\r\n\r\n[/quote]\r\n\r\nNo, I don't think what happened in this competition will have any impact on future Kaggle competitions. \r\n\r\nBut there are no doubts that the current competition is over at this point. I don't see what could possibly save it, short of starting from scratch with different data and different rules (and no KS test). \r\n\r\nSure, it is in bad taste to change the rules in the middle of the competition, especially in a way that invalidates the work of many entrants (including many (all?) top entrants). Sure, it's in bad taste not to reply to questions and concerns addressed to admins, too. Sure, it was unfortunate to submit a competition that was flawed by design, but then again, it's something that has happened a few times on Kaggle before. Data science is hard. Organizing a competition requires strong data science competence. Teams with such strong data science competence are less likely to turn to Kaggle competitions to solve their problems...\r\n\r\nClearly, the only viable answer here is just to move away from this competition. The first dozen hours you spent on this challenge have been thoroughly wasted, so just don't spend any more time.\r\n\r\nLet's move on.", "88752": "[quote=Erlend Davidson;88664]\r\n\r\nHi Mark, I'd be very interested to see how one would implement your second point (direct training with a custom loss function on both training.csv and check_agreement.csv). I'm not interested in using to in this competition, but I'd like to know how to do this in general since I think it's complicated by the fact that the agreement check is really checking a global systemic property. \r\n\r\nWould you, or someone, be willing to demonstrate code for that (either during or after the competition)?\r\n\r\nThanks,\r\nErlend\r\n\r\n[/quote]\r\n\r\nI believe this comment was actually meant as a reply to my post. \r\n\r\nMy first entries in this competition were based on this scheme (a custom loss function to train a classifier that, at the same time, classifies signal from background in the test data, and outputs similar probabilities for signal and background of the control data). \r\n\r\nSure, ROC AUC is a global property. It also has a statistical / information-theoretical meaning, and therefore you can optimize for it via per-sample proxy metrics from statistics or information theory...\r\n\r\nThe most basic thing you can do is use binary crossentropy. Minimize crossentropy of signal/background on the test data while maintaining the crossentropy of signal/background on the control data at some carefully selected value (typically the maximum entropy value). The latter is done via minimization of the difference between the crossentropy and the target value. \r\n\r\nThis can be treated as a global optimization problem by using as objective a weighted sum of these two objectives. \r\n\r\nThe result, in practice, are highly performant models (e.g. up to 0.996 for a single NN, no ensembles, and after very little search/optimization of the setup) with very low KS scores (generally around 0.05-0.06).", "88755": "I used a simpler method that worked nicely. Build a predictor to predict if a sample is from the training set or the check_agreement set and use the output of this predictor to reduce the features that are known to increase the KS score (for example SPDhits) whenever a sample looks like it is coming from check_agreement and not from training.\r\n\r\nThe reduction was done as follows: convert the feature (SPDhits) to log(Psignal/Pbackground) and then multiply by the probability to be from the training set and not from the check_agreement set", "88756": "[quote=fchollet;88751]\r\n\r\nLet's move on.\r\n\r\n[/quote]\r\n\r\nThanks, will do", "88760": "To admin, one final thing. I am not sure about this but it looks to me that a classifier that avoids the KS score ( for example in the method I just described ) gives better prediction results than a simpler classifier that ignores the KS test. \r\n\r\nIf this is true than it can be useful for real future research in this area. Perhaps the background contains D decay events and the reduction method helps in detecting these events and correctly placing them in the background bin.", "88768": "[quote=udibr;88760]\r\n\r\nTo admin, one final thing. I am not sure about this but it looks to me that a classifier that avoids the KS score ( for example in the method I just described ) gives better prediction results than a simpler classifier that ignores the KS test. \r\n\r\nIf this is true than it can be useful for real future research in this area. Perhaps the background contains D decay events and the reduction method helps in detecting these events and correctly placing them in the background bin. \r\n\r\n[/quote]\r\n\r\nThe idea behind the custom loss function I was using was the following:\r\n\r\n- the data contains both \"valid\" (physics-based) signal indicators, and \"invalid\" indicators (simulation artifacts)\r\n- therefore we need a way to constrain the learning to only use the valid features\r\n- ***assuming that the KS score reliably tests for usage of such invalid features***, then the solution is to constrain training to produce a low KS score, through a custom loss function trained on both types of events.\r\n\r\nIn theory it would work perfectly, except in practice it doesn't. ***Because the assumption that the KS score is meaningful is just plain false.*** So what happens in practice is that a classifier trained in this way learns to discriminate control data while exploiting simulation artifacts.\r\n\r\nIt is also worth pointing that the problem of finding which features are \"valid\" (physics-based) and which are invalid, **isn't solvable with the data provided.** It cannot be found from the training data (because events are either real *and* background, or simulated *and* signal), and it cannot be found from the control data (because it has a completely different distribution from the test data, and therefore can be discriminated against to cheat the test). \r\n\r\nI.e. the problem that the organizers wanted to solve cannot be solved with the data that the organizers provided. Which is why nothing interesting will come out of this competition.", "89345": "", "89447": "Several cases just to clarify this topic.\r\n\r\nCan we \r\n\r\n1. \r\n- Train model using **train dataset only** and initial **parametrs {x1..xn}**\r\n- Run correlation and agreement tests\r\n- increment or decrement x1..xh based on tests result\r\n- repeat several times\r\n\r\n2. \r\n- Research correlation and agreement tests code and make assumptions\r\n- Train model using **train dataset**, **knowledge from correlation and agreement tests code** and initial **parametrs {x1..xn}** \r\n- Run tests check_correlation and check_agreement tests\r\n- increment or decrement x1..xn based on tests result\r\n- repeat several times\r\n\r\n3. \r\n- Research correlation and agreement tests code and make assumptions\r\n- Train model using **train dataset**, **knowledge from correlation and agreement tests code**\r\n- Run correlation and agreement tests", "89528": "So for sure you can do 1. The 2 and 3. depends on what exactly you mean by: \"research correlations\" and \"make assumptions\":\r\n\r\nIf you want to plot distributions and choose only the ones that show good agreement then yes please :) If you want to somehow to create variable that is somehow distinguishing between tau->3mu MC and Ds->Phi Pi MC then no. \r\n\r\nCheers,\r\nMarcin", "89554": "[quote]\r\n\r\nCan we\r\n\r\n1. \r\n\r\n- Train model using train dataset only and initial parametrs {x1..xn}\r\n- Run correlation and agreement tests\r\n- increment or decrement x1..xh based on tests result\r\n- repeat several times\r\n\r\n[/quote]\r\n\r\n[quote] So for sure you can do 1. [/quote]\r\n\r\nExcellent. Just to be clear that you fully understand what you are stating here, here's a code skeleton that is a straightforward implementation of 1. Is this code skeleton valid by your rules?\r\n\r\n X_agreement, y_agreement, w_agreement = get_agreement_data()\r\n X, y = get_training_data()\r\n initial_params = initialize() # just random {x1..xn}\r\n model = model_from_params(initial_params, X, y)\r\n current_score = compute_roc_auc(model, X, y)\r\n current_ks = compute_ks(model, X_agreement, y_agreement, w_agreement)\r\n\r\n while not stopping_criterion.check():\r\n # just increment / decrement initial {x1..xn}\r\n initial_params = update_parameters(initial_params, current_score, current_ks)\r\n\r\n # model_from_params could even make no use of X, y\r\n # i.e. this step isn't even necessary\r\n model = model_from_params(initial_params, X, y)\r\n\r\n current_score = compute_roc_auc(model, X, y)\r\n current_ks = compute_ks(model, X_agreement, y_agreement, w_agreement)\r\n\r\n stopping_criterion.update(current_score, current_ks)", "89582": "Ok. I'll try to generalize method.\r\nAny \"black box\" (imagine, that we have to submit our test set instead of running local tests) analysis is allowed (in assumption, that we haven't got correlation and agreement datasets) ?", "89595": "Mikhail, I'm not sure I got your question, but in the end winning solution should be accompanied with compilable & runnable code (that produces the same results) and explanation of how this solution operates on given data.", "89600": "Francois, your solution skeleton would qualify as \"using check_agreement for model testing\". If you follow the logic from the post https://www.kaggle.com/c/flavours-of-physics/forums/t/15974/machine-learning-approach-in-high-energy-physics-explanation, most likely it should be fine. To be sure that solution is OK we demand participants to provide the compilable&runnable code and verbal description of their approaches in the end.\r\n\r\nNB we realize that things happening inside loop `while not stopping_criterion.check()` can also be treated as model training effectively overfitting to model restrictions. It is not much worse than overfitting to public leaderboard (for example http://blog.mrtz.org/2015/03/09/competition.html and http://jmlr.org/proceedings/papers/v37/blum15.pdf) and this is something all challenges have to cope with for the given competition framework.", "89601": "Ok, so, but what if a solution strictly follows this code skeleton, but the explanation of how it works is, as you state, \"(over)fitting to model restrictions\" (actually just fitting, i.e. training)? Is the solution no longer eligible then, even though it is following a training pattern that has been claimed as acceptable?\r\n\r\nBasically, why \"***most likely*** it should be fine\"? Is this training pattern valid, or not? If it is valid, it should stay valid no matter what goes into the functions `model_from_params(init_params, X, y)` and `update_parameters(init_params, score, ks)`? (of course these functions would not be be accessing new data, or anything sketchy like that. Just straightforward logic applied to their arguments).", "89605": "Francois, unless anything \"sketchy\" happens inside the functions you've mentioned (that's the reason for '**most likely**') the pattern is fine.", "89607": "By sketchy I just mean external data access. Of course these functions would only be processing the arguments that we pass them. There is no trick behind my code skeleton, it would do exactly what the function names imply.\r\n\r\nAlright, if this pattern is allowed, that's good news for us, but you do realize that this pattern is all about training on the control data, right? Via `update_parameters(init_params, score, ks)`, which injects KS score information into the model. By repeating the loop as many times as needed, you can end up leaking as much KS score information as you want. \r\n\r\nJust like \"overfitting the leaderboard\" would eventually result in perfect scores if we didn't have daily submission limits.", "89612": "Actually, the shown code is exactly what i mean, when saying \"black box\" analisis.\r\nSounds good, thank you.", "89802": "Just to be explicit, that also means that a custom loss function where KS is included as a penalty is strictly allowed? I thought I read in the forum somewhere that doing so was not allowed. But this is essentially doing the same thing as fchollet's code snippet.", "89818": "> But this is essentially doing the same thing as fchollet's code snippet.\r\n\r\nIt is indeed. It's just semantics. You are allowed to \"train\" on the control data, but training on the control data is forbidden. My personal take on this is that you should figure out ways to train on the control data as much as possible while camouflaging the process to keep some facade of respectability in the eyes of the organizers (which my snippet does). At least that's how I interpret these recent developments.", "89880": "[quote=fchollet;89818]\r\n\r\nIt is indeed. It's just semantics. You are allowed to \"train\" on the control data, but training on the control data is forbidden. \r\n\r\n[/quote]\r\n\r\nI guess the original intent of the competition is that one uses compute_ks(..) as an 'oracle' without looking at the entries in check_agreement.csv (although I suspect that given a huge amount of CPU time one could even infer the contents of check_agreement.csv this way....)", "89887": "[quote=Andre Holzner;89880]\r\n\r\n[quote=fchollet;89818]\r\n\r\nIt is indeed. It's just semantics. You are allowed to \"train\" on the control data, but training on the control data is forbidden. \r\n\r\n[/quote]\r\n\r\nI guess the original intent of the competition is that one uses compute_ks(..) as an 'oracle' without looking at the entries in check_agreement.csv (although I suspect that given a huge amount of CPU time one could even infer the contents of check_agreement.csv this way....)\r\n\r\n[/quote]\r\n\r\nIt doesn't matter if you \"look\" (i.e. open the file and read the contents of it), if you accept/reject your models based on some test (i.e. the KS-metric) derived from the data in check_agreement.csv then some of check_agreement.csv will leak into your data. The same sort of problem can happen if you score your models using just a cross-validation on a static test-set.\r\n\r\nThis is described in the first two paragraphs of http://k-d-w.org/node/95, and also in §7.10.2 of the book here: http://statweb.stanford.edu/~tibs/ElemStatLearn/.", "90877": "Wow, I find the discussion on this thread to be discouraging. The point of the competition is to help move forward an understanding of physics and yet a number of contributors seem more intent on arguing about the rules changes as if it were a casual game. Yes you may have to go back and do things over, or come up with a different approach. The rule changes are there to provide for usable data for physicists. \r\nThat is the ultimate point of the challenge, not whether you can successively argue for rules in your favor or use clever interpretations to get an upper hand.", "91685": "[quote=Chrisfs;90877]\r\n The point of the competition is to help move forward an understanding of physics and yet a number of contributors seem more intent on arguing about the rules changes as if it were a casual game. Yes you may have to go back and do things over, or come up with a different approach. The rule changes are there to provide for usable data for physicists. \r\n[/quote]\r\n\r\n\r\nNot really. If you set up a system that scores entries, people will optimise for that measure. If you want to complain later that the optimal solution for that measure didn't solve your initial problem, well then it's your fault for posing the problem incorrectly.", "91900": "[quote=fchollet;88751]\r\n\r\nNo, I don't think what happened in this competition will have any impact on future Kaggle competitions. \r\n\r\n[/quote]\r\n\r\nWhy on earth should we learn from mistakes? If it's not in a file called \"train.csv\" we don't care!" }, "source": "meta" }