As a suggestion from Tim Head, just tested my model using the synthetic data from the ipython notebook http://t.co/kF40NpHHIz with interesting results
\n\nFor the minimal value of KS the model obtains
\n\nAUC =0.9984177, AUC_R=0.9652983, KS=0.14805
\n\nHalfway to the optimal values obtained by completely ignoring the X1 variable
\n\nAUC=0.98223417, AUC_R=0.98136828, KS=0.04544
\n\nSo there is still room for improvement.
\n\nThis methodology doesn’t try to identify control data, but introduce confusion in the training samples when their variables match control data easily identified as Real data or MonteCarlo.
\n\nIn this sense, probably is not cheating… I hope so...
", "messages": [ { "id": "88835", "postDate": "08/07/2015 16:38:49", "content": "As a suggestion from Tim Head, just tested my model using the synthetic data from the ipython notebook http://t.co/kF40NpHHIz with interesting results
\n\nFor the minimal value of KS the model obtains
\n\nAUC =0.9984177, AUC_R=0.9652983, KS=0.14805
\n\nHalfway to the optimal values obtained by completely ignoring the X1 variable
\n\nAUC=0.98223417, AUC_R=0.98136828, KS=0.04544
\n\nSo there is still room for improvement.
\n\nThis methodology doesn’t try to identify control data, but introduce confusion in the training samples when their variables match control data easily identified as Real data or MonteCarlo.
\n\nIn this sense, probably is not cheating… I hope so...
", "rawMarkdown": "As a suggestion from Tim Head, just tested my model using the synthetic data from the ipython notebook http://t.co/kF40NpHHIz with interesting results\r\n\r\n - There is an optimal value for the regularization parameter (yep, there is one) that minimizes the KS in the control channel data (with a KS value far from 0) (See KS.png)\r\n - When decreasing the KS, the AUC_R measured over real data increase up to an optimal value corresponding to the minimum of KS\r\n - AUC over simulated data and AUC _R are positively correlated, but with a bias (see AUC_AUCR.png) Not sure how estimate this bias using the current data.\r\n\r\nFor the minimal value of KS the model obtains\r\n\r\n**AUC =0.9984177, AUC_R=0.9652983, KS=0.14805**\r\n\r\nHalfway to the optimal values obtained by completely ignoring the X1 variable\r\n\r\n**AUC=0.98223417, AUC_R=0.98136828, KS=0.04544**\r\n\r\nSo there is still room for improvement. \r\n\r\nThis methodology doesn’t try to identify control data, but introduce confusion in the training samples when their variables match control data easily identified as Real data or MonteCarlo.\r\n\r\nIn this sense, probably is not cheating… I hope so...", "votes": null }, { "id": "88867", "postDate": "08/08/2015 00:17:48", "content": "[quote=Vicens Gaitan;88835]\nIn this sense, probably is not cheating… I hope so...\n[/quote]
\n\nI think the rules are very clear now, to me, it goes like this:
\n\nCan you apply the same methodology if I give you a new dataset without ‘check_correlation.csv' and ‘check_agreement.csv' ?
\n\nNo? then its cheating, rules states your solution must be without any usage of those files.
\n\nYes? great then, you did a great work from the physics point of view, not just ML.
", "rawMarkdown": "[quote=Vicens Gaitan;88835]\r\nIn this sense, probably is not cheating… I hope so...\r\n[/quote]\r\n\r\nI think the [rules][1] are very clear now, to me, it goes like this:\r\n\r\nCan you apply the same methodology if I give you a new dataset without ‘check_correlation.csv' and ‘check_agreement.csv' ? \r\n\r\nNo? then its cheating, rules states your solution must be **without *any* usage** of those files.\r\n\r\nYes? great then, you did a great work from the physics point of view, not just ML.\r\n\r\n\r\n [1]: https://www.kaggle.com/c/flavours-of-physics/forums/t/15735/important-training-dataset-reminder", "votes": null }, { "id": "88946", "postDate": "08/09/2015 10:58:51", "content": "My point is that using the control data in the right way , even against the rules, can be useful for physics data analysis, in selecting in a non supervised way, discriminating features for the signal channel that are not able to separate real data and Monte Carlo.
\n\nThis is exactly what we do when decide to not use SPDhits beacause the KS in the agreement check is too high when included as a feature, but by chance we discover than including ratios of SDPhits over other variables, still retain some discriminating power while allowing to pass the KS check.
\n\nMaybe there is a subtle question: to use the agreement test as an 'oracle' for the KS index, or to have direct acces to the individual events in the test_agreement sample. In my opinion both aproachs are equivalent, being the second more efficient from a computational point of view.
\n\n\n\n\nNo? then its cheating, rules states your solution must be without any usage of those files.
\n
Admins, must be considered against the rules to calculate the KS of the check_agreement during the training process? Or to do feature selection tacking into account this check?
\n\nIf the answer is yes, then the only way is lucky random search :)
", "rawMarkdown": "My point is that using the control data in the *right* way , even against the rules, can be useful for physics data analysis, in selecting in a non supervised way, discriminating features for the signal channel that are not able to separate real data and Monte Carlo.\r\n\r\nThis is exactly what we do when decide to not use SPDhits beacause the KS in the agreement check is too high when included as a feature, but by chance we discover than including ratios of SDPhits over other variables, still retain some discriminating power while allowing to pass the KS check.\r\n\r\nMaybe there is a subtle question: to use the agreement test as an 'oracle' for the KS index, or to have direct acces to the individual events in the test_agreement sample. In my opinion both aproachs are equivalent, being the second more efficient from a computational point of view.\r\n\r\n>No? then its cheating, rules states your solution must be **without *any* usage** of those files.\r\n\r\nAdmins, must be considered against the rules to calculate the KS of the check_agreement **during** the training process? Or to do feature selection tacking into account this check?\r\n\r\nIf the answer is yes, then the only way is lucky random search :)", "votes": null }, { "id": "88956", "postDate": "08/09/2015 14:20:37", "content": "[quote]Admins, must be considered against the rules to calculate the KS of the check_agreement during the training process? Or to do feature selection tacking into account this check?
\n\nIf the answer is yes, then the only way is lucky random search :)[/quote]
\n\nBoth preliminary feature selection driven by the KS data, and checking the KS score during training (even without training on KS data), are strictly equivalent to training on the KS data (identical results, and identical process from an information-theoretical point of view. The differences are purely cosmetic).
\n\nSo if you aren't allowed to train on KS data, then neither should you be allowed to use the KS data for feature selection or for controlling that your training stays within the KS boundaries. Otherwise, well that's a big loophole you can exploit for fun (and possibly profit).
\n\n[quote]My point is that using the control data in the right way , even against the rules, can be useful for physics data analysis, in selecting in a non supervised way, discriminating features for the signal channel that are not able to separate real data and Monte Carlo.[/quote]
\n\nIn theory yes, but in practice it is strictly impossible to tell apart features that are useful for physics (good signal discrimination and do not pick up simulation artifacts) from features that are simply good at picking up simulation artifacts while minimizing the KS score. The two are absolutely equivalent (from an information-theoretical point of view) given the data we have.
\n\nSuch features are what you get when you train jointly on the training data and the KS data. There are many other, equivalent methods of using the KS data to obtain these features. Anyway, these features are disallowed in the competition, seemingly.
\n\nThe theory would apply if and only if the KS test was controlling for the usage of features that just pick up simulation artifacts. But in practice, the KS test does not test for that at all, so your search for good features is merely a search for ways to "cheat" the KS test.
", "rawMarkdown": "[quote]Admins, must be considered against the rules to calculate the KS of the check_agreement during the training process? Or to do feature selection tacking into account this check?\r\n\r\nIf the answer is yes, then the only way is lucky random search :)[/quote]\r\n\r\nBoth preliminary feature selection driven by the KS data, and checking the KS score during training (even without training on KS data), are strictly equivalent to training on the KS data (identical results, and identical process from an information-theoretical point of view. The differences are purely cosmetic).\r\n\r\nSo if you aren't allowed to train on KS data, then neither should you be allowed to use the KS data for feature selection or for controlling that your training stays within the KS boundaries. Otherwise, well that's a big loophole you can exploit for fun (and possibly profit).\r\n\r\n[quote]My point is that using the control data in the right way , even against the rules, can be useful for physics data analysis, in selecting in a non supervised way, discriminating features for the signal channel that are not able to separate real data and Monte Carlo.[/quote]\r\n\r\nIn theory yes, but in practice it is strictly impossible to tell apart features that are useful for physics (good signal discrimination *and* do not pick up simulation artifacts) from features that are simply good at picking up simulation artifacts while minimizing the KS score. The two are absolutely equivalent (from an information-theoretical point of view) given the data we have. \r\n\r\nSuch features are what you get when you train jointly on the training data and the KS data. There are many other, equivalent methods of using the KS data to obtain these features. Anyway, these features are disallowed in the competition, seemingly.\r\n\r\nThe theory would apply if and only if the KS test was controlling for the usage of features that just pick up simulation artifacts. But in practice, the KS test does not test for that at all, so your search for good features is merely a search for ways to \"cheat\" the KS test.", "votes": null }, { "id": "88957", "postDate": "08/09/2015 14:24:33", "content": "Then, game over? :(
", "rawMarkdown": "Then, game over? :(", "votes": null }, { "id": "88958", "postDate": "08/09/2015 14:32:13", "content": "\n\n\nThen, game over? :(
\n
Yes. You mention "lucky random search" as the last resort, but if this random search is followed by a KS test and the ROC AUC score test (i.e. if you make a submission), then it is not random anymore, and becomes equivalent again to the methods we've been discussing.
\n\nRandom search is just a way to search a solution space, just like gradient descent. When you train jointly on the training data and the KS data, you are doing gradient descent in the solution space; when you test solutions at random, that is an inefficient way to do the exact same thing.
", "rawMarkdown": "> Then, game over? :(\r\n\r\nYes. You mention \"lucky random search\" as the last resort, but if this random search is followed by a KS test and the ROC AUC score test (i.e. if you make a submission), then it is not random anymore, and becomes equivalent again to the methods we've been discussing. \r\n\r\nRandom search is just a way to search a solution space, just like gradient descent. When you train jointly on the training data and the KS data, you are doing gradient descent in the solution space; when you test solutions at random, that is an inefficient way to do the *exact same* thing.", "votes": null }, { "id": "89035", "postDate": "08/10/2015 16:09:28", "content": "Hi Vicens,
\n\nIndeed what you’ve done seems to be quite interesting! Although it is not compliant with the rules for the main prize, it certainly makes sense to apply for ‘Special Physics prize' and physicists would like to have a closer look at your approach!
", "rawMarkdown": "Hi Vicens, \r\n\r\nIndeed what you’ve done seems to be quite interesting! Although it is not compliant with the rules for the main prize, it certainly makes sense to apply for ‘Special Physics prize' and physicists would like to have a closer look at your approach!", "votes": null } ], "comments": [ { "id": 88867, "author_name": "carloshuertas", "author_url": "", "post_date": "08/08/2015 00:17:48", "content": "[quote=Vicens Gaitan;88835]\nIn this sense, probably is not cheating… I hope so...\n[/quote]
\n\nI think the rules are very clear now, to me, it goes like this:
\n\nCan you apply the same methodology if I give you a new dataset without ‘check_correlation.csv' and ‘check_agreement.csv' ?
\n\nNo? then its cheating, rules states your solution must be without any usage of those files.
\n\nYes? great then, you did a great work from the physics point of view, not just ML.
", "votes": null, "replies": [] }, { "id": 88946, "author_name": "vicensgaitan", "author_url": "", "post_date": "08/09/2015 10:58:51", "content": "My point is that using the control data in the right way , even against the rules, can be useful for physics data analysis, in selecting in a non supervised way, discriminating features for the signal channel that are not able to separate real data and Monte Carlo.
\n\nThis is exactly what we do when decide to not use SPDhits beacause the KS in the agreement check is too high when included as a feature, but by chance we discover than including ratios of SDPhits over other variables, still retain some discriminating power while allowing to pass the KS check.
\n\nMaybe there is a subtle question: to use the agreement test as an 'oracle' for the KS index, or to have direct acces to the individual events in the test_agreement sample. In my opinion both aproachs are equivalent, being the second more efficient from a computational point of view.
\n\n\n\n\nNo? then its cheating, rules states your solution must be without any usage of those files.
\n
Admins, must be considered against the rules to calculate the KS of the check_agreement during the training process? Or to do feature selection tacking into account this check?
\n\nIf the answer is yes, then the only way is lucky random search :)
", "votes": null, "replies": [] }, { "id": 88956, "author_name": "fchollet", "author_url": "", "post_date": "08/09/2015 14:20:37", "content": "[quote]Admins, must be considered against the rules to calculate the KS of the check_agreement during the training process? Or to do feature selection tacking into account this check?
\n\nIf the answer is yes, then the only way is lucky random search :)[/quote]
\n\nBoth preliminary feature selection driven by the KS data, and checking the KS score during training (even without training on KS data), are strictly equivalent to training on the KS data (identical results, and identical process from an information-theoretical point of view. The differences are purely cosmetic).
\n\nSo if you aren't allowed to train on KS data, then neither should you be allowed to use the KS data for feature selection or for controlling that your training stays within the KS boundaries. Otherwise, well that's a big loophole you can exploit for fun (and possibly profit).
\n\n[quote]My point is that using the control data in the right way , even against the rules, can be useful for physics data analysis, in selecting in a non supervised way, discriminating features for the signal channel that are not able to separate real data and Monte Carlo.[/quote]
\n\nIn theory yes, but in practice it is strictly impossible to tell apart features that are useful for physics (good signal discrimination and do not pick up simulation artifacts) from features that are simply good at picking up simulation artifacts while minimizing the KS score. The two are absolutely equivalent (from an information-theoretical point of view) given the data we have.
\n\nSuch features are what you get when you train jointly on the training data and the KS data. There are many other, equivalent methods of using the KS data to obtain these features. Anyway, these features are disallowed in the competition, seemingly.
\n\nThe theory would apply if and only if the KS test was controlling for the usage of features that just pick up simulation artifacts. But in practice, the KS test does not test for that at all, so your search for good features is merely a search for ways to "cheat" the KS test.
", "votes": null, "replies": [] }, { "id": 88957, "author_name": "vicensgaitan", "author_url": "", "post_date": "08/09/2015 14:24:33", "content": "Then, game over? :(
", "votes": null, "replies": [] }, { "id": 88958, "author_name": "fchollet", "author_url": "", "post_date": "08/09/2015 14:32:13", "content": "\n\n\nThen, game over? :(
\n
Yes. You mention "lucky random search" as the last resort, but if this random search is followed by a KS test and the ROC AUC score test (i.e. if you make a submission), then it is not random anymore, and becomes equivalent again to the methods we've been discussing.
\n\nRandom search is just a way to search a solution space, just like gradient descent. When you train jointly on the training data and the KS data, you are doing gradient descent in the solution space; when you test solutions at random, that is an inefficient way to do the exact same thing.
", "votes": null, "replies": [] }, { "id": 89035, "author_name": "anaderi", "author_url": "", "post_date": "08/10/2015 16:09:28", "content": "Hi Vicens,
\n\nIndeed what you’ve done seems to be quite interesting! Although it is not compliant with the rules for the main prize, it certainly makes sense to apply for ‘Special Physics prize' and physicists would like to have a closer look at your approach!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "88835": "As a suggestion from Tim Head, just tested my model using the synthetic data from the ipython notebook http://t.co/kF40NpHHIz with interesting results\r\n\r\n - There is an optimal value for the regularization parameter (yep, there is one) that minimizes the KS in the control channel data (with a KS value far from 0) (See KS.png)\r\n - When decreasing the KS, the AUC_R measured over real data increase up to an optimal value corresponding to the minimum of KS\r\n - AUC over simulated data and AUC _R are positively correlated, but with a bias (see AUC_AUCR.png) Not sure how estimate this bias using the current data.\r\n\r\nFor the minimal value of KS the model obtains\r\n\r\n**AUC =0.9984177, AUC_R=0.9652983, KS=0.14805**\r\n\r\nHalfway to the optimal values obtained by completely ignoring the X1 variable\r\n\r\n**AUC=0.98223417, AUC_R=0.98136828, KS=0.04544**\r\n\r\nSo there is still room for improvement. \r\n\r\nThis methodology doesn’t try to identify control data, but introduce confusion in the training samples when their variables match control data easily identified as Real data or MonteCarlo.\r\n\r\nIn this sense, probably is not cheating… I hope so...", "88867": "[quote=Vicens Gaitan;88835]\r\nIn this sense, probably is not cheating… I hope so...\r\n[/quote]\r\n\r\nI think the [rules][1] are very clear now, to me, it goes like this:\r\n\r\nCan you apply the same methodology if I give you a new dataset without ‘check_correlation.csv' and ‘check_agreement.csv' ? \r\n\r\nNo? then its cheating, rules states your solution must be **without *any* usage** of those files.\r\n\r\nYes? great then, you did a great work from the physics point of view, not just ML.\r\n\r\n\r\n [1]: https://www.kaggle.com/c/flavours-of-physics/forums/t/15735/important-training-dataset-reminder", "88946": "My point is that using the control data in the *right* way , even against the rules, can be useful for physics data analysis, in selecting in a non supervised way, discriminating features for the signal channel that are not able to separate real data and Monte Carlo.\r\n\r\nThis is exactly what we do when decide to not use SPDhits beacause the KS in the agreement check is too high when included as a feature, but by chance we discover than including ratios of SDPhits over other variables, still retain some discriminating power while allowing to pass the KS check.\r\n\r\nMaybe there is a subtle question: to use the agreement test as an 'oracle' for the KS index, or to have direct acces to the individual events in the test_agreement sample. In my opinion both aproachs are equivalent, being the second more efficient from a computational point of view.\r\n\r\n>No? then its cheating, rules states your solution must be **without *any* usage** of those files.\r\n\r\nAdmins, must be considered against the rules to calculate the KS of the check_agreement **during** the training process? Or to do feature selection tacking into account this check?\r\n\r\nIf the answer is yes, then the only way is lucky random search :)", "88956": "[quote]Admins, must be considered against the rules to calculate the KS of the check_agreement during the training process? Or to do feature selection tacking into account this check?\r\n\r\nIf the answer is yes, then the only way is lucky random search :)[/quote]\r\n\r\nBoth preliminary feature selection driven by the KS data, and checking the KS score during training (even without training on KS data), are strictly equivalent to training on the KS data (identical results, and identical process from an information-theoretical point of view. The differences are purely cosmetic).\r\n\r\nSo if you aren't allowed to train on KS data, then neither should you be allowed to use the KS data for feature selection or for controlling that your training stays within the KS boundaries. Otherwise, well that's a big loophole you can exploit for fun (and possibly profit).\r\n\r\n[quote]My point is that using the control data in the right way , even against the rules, can be useful for physics data analysis, in selecting in a non supervised way, discriminating features for the signal channel that are not able to separate real data and Monte Carlo.[/quote]\r\n\r\nIn theory yes, but in practice it is strictly impossible to tell apart features that are useful for physics (good signal discrimination *and* do not pick up simulation artifacts) from features that are simply good at picking up simulation artifacts while minimizing the KS score. The two are absolutely equivalent (from an information-theoretical point of view) given the data we have. \r\n\r\nSuch features are what you get when you train jointly on the training data and the KS data. There are many other, equivalent methods of using the KS data to obtain these features. Anyway, these features are disallowed in the competition, seemingly.\r\n\r\nThe theory would apply if and only if the KS test was controlling for the usage of features that just pick up simulation artifacts. But in practice, the KS test does not test for that at all, so your search for good features is merely a search for ways to \"cheat\" the KS test.", "88957": "Then, game over? :(", "88958": "> Then, game over? :(\r\n\r\nYes. You mention \"lucky random search\" as the last resort, but if this random search is followed by a KS test and the ROC AUC score test (i.e. if you make a submission), then it is not random anymore, and becomes equivalent again to the methods we've been discussing. \r\n\r\nRandom search is just a way to search a solution space, just like gradient descent. When you train jointly on the training data and the KS data, you are doing gradient descent in the solution space; when you test solutions at random, that is an inefficient way to do the *exact same* thing.", "89035": "Hi Vicens, \r\n\r\nIndeed what you’ve done seems to be quite interesting! Although it is not compliant with the rules for the main prize, it certainly makes sense to apply for ‘Special Physics prize' and physicists would like to have a closer look at your approach!" }, "source": "meta" }