Hello,
\n\nfirst, excuse, please, my bad English. Second, excuse, please, my brevity - I really didn't expected that I will place so high, so I have lots of work to complete my solution description, which is required to be published today (October 14th).\nI wish to thank CERN and organizers for this exciting competition. I wish to congratulate the winning team 'Go Polar Bears' and Alexander Gramolin who placed second - we have had great fight in the past hours of the competition and he managed to beat me :-).
\n\nExcuse, please, poor quality of my source code. I really didn't expected such good result, so I didn't expected that I will be asked to publish my code. And there is not enough time to clean it up. So, here it is (it's crazy mixture of three programming languages):\nthird_place.zip\n. Installation instruction are in the INSTALL.txt file.
\n\nI'm going to write description of my solution and hope to publish it on this forum until midnight UTC.
\n\nEdit: description of my model is now packed in the third_place.zip file.
\n\nEdit2: I simplified my solution to the level where it can be published as script. It scores 0.999594 as a "Post-Deadline entry" on private leaderboard (~third place).
", "messages": [ { "id": "96128", "postDate": "10/14/2015 17:15:38", "content": "Hello,
\n\nfirst, excuse, please, my bad English. Second, excuse, please, my brevity - I really didn't expected that I will place so high, so I have lots of work to complete my solution description, which is required to be published today (October 14th).\nI wish to thank CERN and organizers for this exciting competition. I wish to congratulate the winning team 'Go Polar Bears' and Alexander Gramolin who placed second - we have had great fight in the past hours of the competition and he managed to beat me :-).
\n\nExcuse, please, poor quality of my source code. I really didn't expected such good result, so I didn't expected that I will be asked to publish my code. And there is not enough time to clean it up. So, here it is (it's crazy mixture of three programming languages):\nthird_place.zip\n. Installation instruction are in the INSTALL.txt file.
\n\nI'm going to write description of my solution and hope to publish it on this forum until midnight UTC.
\n\nEdit: description of my model is now packed in the third_place.zip file.
\n\nEdit2: I simplified my solution to the level where it can be published as script. It scores 0.999594 as a "Post-Deadline entry" on private leaderboard (~third place).
", "rawMarkdown": "Hello,\r\n\r\nfirst, excuse, please, my bad English. Second, excuse, please, my brevity - I really didn't expected that I will place so high, so I have lots of work to complete my solution description, which is required to be published today (October 14th).\r\nI wish to thank CERN and organizers for this exciting competition. I wish to congratulate the winning team 'Go Polar Bears' and Alexander Gramolin who placed second - we have had great fight in the past hours of the competition and he managed to beat me :-). \r\n\r\nExcuse, please, poor quality of my source code. I really didn't expected such good result, so I didn't expected that I will be asked to publish my code. And there is not enough time to clean it up. So, here it is (it's crazy mixture of three programming languages):\r\n[third_place.zip][1]\r\n. Installation instruction are in the INSTALL.txt file.\r\n\r\nI'm going to write description of my solution and hope to publish it on this forum until midnight UTC.\r\n\r\nEdit: description of my model is now packed in the third_place.zip file.\r\n\r\nEdit2: I simplified my solution to the level where it can be published as [script][2]. It scores 0.999594 as a \"Post-Deadline entry\" on private leaderboard (~third place).\r\n\r\n\r\n [1]: http://jslavicek.cz/third_place.zip\r\n [2]: https://www.kaggle.com/josefslavicek/flavours-of-physics/simplified-version-of-my-solution", "votes": null }, { "id": "96133", "postDate": "10/14/2015 18:36:35", "content": "Hello Josef,
\n\nIt's nice to hear from you, thanks. I guess that such high scores achieved in the leaderboard were surprising for many of us and for the Organizers too. ;)
\n\nUnfortunately, I have serious doubts that the winning models (including my own) can be useful for real physics... As far as I understand now, any usage of the reconstructed invariant mass as a feature (what I did) or usage the agreement/correlation data sets while training the classifier (you did it, right?) makes the model physically meaningless. Unfortunately, the competition wasn't designed perfectly. There are at least two independent ways to calculate the invariant mass of the mother particle using the features provided. I didn't understand earlier that it's a bad idea because I believed in the correlation test.
\n\nI realize that the Organizers are probably unhappy with the results, but I think that they should finally make some statement regarding the situation. After all, one of the purposes of the competition, as I understand, was to test the metrics designed to judge if the classifier is physically meaningful. If so, the negative result is also important. Another positive thing is that it was a lot of fun to work on this problem. We can probably do better next time, so let's draw the right lessons from this experience.
", "rawMarkdown": "Hello Josef,\r\n\r\nIt's nice to hear from you, thanks. I guess that such high scores achieved in the leaderboard were surprising for many of us and for the Organizers too. ;)\r\n\r\nUnfortunately, I have serious doubts that the winning models (including my own) can be useful for real physics... As far as I understand now, any usage of the reconstructed invariant mass as a feature (what I did) or usage the agreement/correlation data sets while training the classifier (you did it, right?) makes the model physically meaningless. Unfortunately, the competition wasn't designed perfectly. There are at least two independent ways to calculate the invariant mass of the mother particle using the features provided. I didn't understand earlier that it's a bad idea because I believed in the correlation test.\r\n\r\nI realize that the Organizers are probably unhappy with the results, but I think that they should finally make some statement regarding the situation. After all, one of the purposes of the competition, as I understand, was to test the metrics designed to judge if the classifier is physically meaningful. If so, the negative result is also important. Another positive thing is that it was a lot of fun to work on this problem. We can probably do better next time, so let's draw the right lessons from this experience.", "votes": null }, { "id": "96135", "postDate": "10/14/2015 18:56:09", "content": "Congratulations to winners.\nAlexander, is there way to figure out physical soundness of a model given its predictions? I'd like to verify my own (e.g. statistically), but I'm not a physicist and have no idea how to accomplish it correctly.
", "rawMarkdown": "Congratulations to winners.\r\nAlexander, is there way to figure out physical soundness of a model given its predictions? I'd like to verify my own (e.g. statistically), but I'm not a physicist and have no idea how to accomplish it correctly.", "votes": null }, { "id": "96138", "postDate": "10/14/2015 19:12:39", "content": "I would say that the good signs of a physically meaningful model are the following:
\n\nThese criteria are also obscure, of course. I encourage you and other strong participants to upload your models and let the Organizers judge about their physical soundness.
", "rawMarkdown": "I would say that the good signs of a physically meaningful model are the following:\r\n\r\n 1. you didn't use the agreement and correlation data sets while training the classifier;\r\n 2. you didn't try to reconstruct the invariant mass of the mother particle and to use it in any way;\r\n 3. you passed the agreement and correlation tests without fine tuning of the model and preferably not performing the tests too many times;\r\n 4. you didn't try to distinguish between the signal and the control channels, which is most likely true if you didn't look at the agreement data set and didn't calculate the invariant mass.\r\n\r\nThese criteria are also obscure, of course. I encourage you and other strong participants to upload your models and let the Organizers judge about their physical soundness.", "votes": null }, { "id": "96143", "postDate": "10/14/2015 19:47:24", "content": "[quote=Alexander Gramolin;96138]
\n\nI would say that the good signs of a physically meaningful model are the following:
\n\nThese criteria are also obscure, of course. I encourage you and other strong participants to upload your models and let the Organizers judge about their physical soundness.
\n\n[/quote]\nCongratulations to both teams from upstairs, your results are amazing. And, I think any solution should satisfy all these 4 criteria, otherwise it would be disqualified.
\n\nMy thought of these control channel tests is that, surely it is not perfectly designed since: 1. it gives some artificial data leakage 2. the real life calibration/control channel is much more complicated that this, and 3. I really don't like the CvM control channel which may mislead competitors for having some useless models on invariant mass. The 3rd point comes from reconstruction of invariant mass, as Alexander mentioned. Using this dataset, one can reconstruct invariant mass (just energy conservation, that is it, anyone knows special relativity can calculate it) and creates some artificial dependence of mass. A physical sounds model should not depend much on the mass spectrum, and the admin uses CvM test to limit this dependence, which may not be a good idea: a model which fails in CvM test can be ensembled with other models for 'smoothing' the CvM value and eventually pass the test, but it doesn't make sense. Since the independence on mass is important for the real life experiment, I don't think this CvM test can be a good idea. As to KS test, I think it is fine, simulation needs calibration, it is OK to ensemble models for passing Ks test, and it doesn't hurt physics much.
", "rawMarkdown": "[quote=Alexander Gramolin;96138]\r\n\r\nI would say that the good signs of a physically meaningful model are the following:\r\n\r\n 1. you didn't use the agreement and correlation data sets while training the classifier;\r\n 2. you didn't try to reconstruct the invariant mass of the mother particle and to use it in any way;\r\n 3. you passed the agreement and correlation tests without fine tuning of the model and preferably not performing the tests too many times;\r\n 4. you didn't try to distinguish between the signal and the control channels, which is most likely true if you didn't look at the agreement data set and didn't calculate the invariant mass.\r\n\r\nThese criteria are also obscure, of course. I encourage you and other strong participants to upload your models and let the Organizers judge about their physical soundness.\r\n\r\n[/quote]\r\nCongratulations to both teams from upstairs, your results are amazing. And, I think any solution should satisfy all these 4 criteria, otherwise it would be disqualified. \r\n\r\nMy thought of these control channel tests is that, surely it is not perfectly designed since: 1. it gives some artificial data leakage 2. the real life calibration/control channel is much more complicated that this, and 3. I really don't like the CvM control channel which may mislead competitors for having some useless models on invariant mass. The 3rd point comes from reconstruction of invariant mass, as Alexander mentioned. Using this dataset, one can reconstruct invariant mass (just energy conservation, that is it, anyone knows special relativity can calculate it) and creates some artificial dependence of mass. A physical sounds model should not depend much on the mass spectrum, and the admin uses CvM test to limit this dependence, which may not be a good idea: a model which fails in CvM test can be ensembled with other models for 'smoothing' the CvM value and eventually pass the test, but it doesn't make sense. Since the independence on mass is important for the real life experiment, I don't think this CvM test can be a good idea. As to KS test, I think it is fine, simulation needs calibration, it is OK to ensemble models for passing Ks test, and it doesn't hurt physics much.", "votes": null }, { "id": "96147", "postDate": "10/14/2015 20:09:38", "content": "Hi All,
\n\nI approached this problem from the pure machine learning prospective (I don't know nothing about particle physics). So, beyond dropping some features to pass the agreement test, I constructed meta features for both train and test using 15 different models. Then, I trained the train data ( with meta features added) using XGBoost, RF and UGBC. I tuned parameters for these models using Hyperopt package with 3-fold cross validation. Finally, I made an ensemble using the following formula:
\n\npreds=0.3 * (xgb_preds^0.65 * rf_preds^0.35) + 0.7 * ugb_preds
\n\nI tuned the weights using Hyperopt with 3-fold CV. In addition, I bagged all models in the ensemble: XGB - 250 times, RF - 130 times, UGBC (very slow!) - 30 times. This ensemble (without bagging) produced CV score of 0.994635. However, this model did not pass the correlation test by a tiny margin. Since, I did not have time to tune it more, I had to find (a possibly worse scoring) model which passes correlation test for sure and ensemble it with my model. I took the model from the public LB (score 0.987511), modified it a little bit and ensembled with my model using the formula:
\n\npreds_final = preds_LB^0.45 * preds_my_model^0.55
\n\nwhich gave me my final score.
\n\nTo conclude, I believe that my model has a lot of potential for improvement. I entered this competition only 4 weeks ago, and I spent more than half of this time getting familiar with data, setting, rules, and etc. Consequently, my model was produced just in one shot. It's a pity that I didn't have much time to:
\n\nOverall, I want to thank both the Kaggle team and guys from CERN for this competition. It was definitely fun to participate in it!
", "rawMarkdown": "Hi All,\r\n\r\nI approached this problem from the pure machine learning prospective (I don't know nothing about particle physics). So, beyond dropping some features to pass the agreement test, I constructed meta features for both train and test using 15 different models. Then, I trained the train data ( with meta features added) using XGBoost, RF and UGBC. I tuned parameters for these models using Hyperopt package with 3-fold cross validation. Finally, I made an ensemble using the following formula:\r\n\r\npreds=0.3 * (xgb_preds^0.65 * rf_preds^0.35) + 0.7 * ugb_preds\r\n\r\nI tuned the weights using Hyperopt with 3-fold CV. In addition, I bagged all models in the ensemble: XGB - 250 times, RF - 130 times, UGBC (very slow!) - 30 times. This ensemble (without bagging) produced CV score of 0.994635. However, this model did not pass the correlation test by a tiny margin. Since, I did not have time to tune it more, I had to find (a possibly worse scoring) model which passes correlation test for sure and ensemble it with my model. I took the model from the public LB (score 0.987511), modified it a little bit and ensembled with my model using the formula:\r\n\r\n preds_final = preds_LB^0.45 * preds_my_model^0.55\r\n\r\nwhich gave me my final score.\r\n\r\nTo conclude, I believe that my model has a lot of potential for improvement. I entered this competition only 4 weeks ago, and I spent more than half of this time getting familiar with data, setting, rules, and etc. Consequently, my model was produced just in one shot. It's a pity that I didn't have much time to:\r\n\r\n1. Select the optimal set of both features and metafeatures\r\n2. Select optimal models for meta features construction\r\n3. Play with more models on the ensembling stage.\r\n\r\nOverall, I want to thank both the Kaggle team and guys from CERN for this competition. It was definitely fun to participate in it!", "votes": null }, { "id": "96163", "postDate": "10/14/2015 21:48:52", "content": "[quote=Alexander Gramolin;96133]\nUnfortunately, I have serious doubts that the winning models (including my own) can be useful for real physics... As far as I understand now, any usage of the reconstructed invariant mass as a feature (what I did) or usage the agreement/correlation data sets while training the classifier (you did it, right?) makes the model physically meaningless. \n[/quote]
\n\nYes, my model uses reconstructed mass. I added it, because my xgbtree started accidentally in some configuration show impressive CV score of 0.9995 and at the same time it started to fail desperately in correlation test. I found a way to combine results from several models to make it pass the correlation test (not knowing that doing it is wrong from physical viewpoint). I deduced that poor results in correlation test means that the xgbtree just found a way to internally reconstruct the mass and use it to make signal predictions. So I trained linear regression to estimate the mass (I didn't know that the mass can be simply computed) and added this estimated mass as another input to my xgbtree. It improved the CV score to 0.9997.
\n\nOn the other hand, I'm using the agreement/correlation data really only to determine, if my solution will pass or fail in the tests - which of course is too a way to leak the information from the files into my model.
", "rawMarkdown": "[quote=Alexander Gramolin;96133]\r\nUnfortunately, I have serious doubts that the winning models (including my own) can be useful for real physics... As far as I understand now, any usage of the reconstructed invariant mass as a feature (what I did) or usage the agreement/correlation data sets while training the classifier (you did it, right?) makes the model physically meaningless. \r\n[/quote]\r\n\r\nYes, my model uses reconstructed mass. I added it, because my xgbtree started accidentally in some configuration show impressive CV score of 0.9995 and at the same time it started to fail desperately in correlation test. I found a way to combine results from several models to make it pass the correlation test (not knowing that doing it is wrong from physical viewpoint). I deduced that poor results in correlation test means that the xgbtree just found a way to internally reconstruct the mass and use it to make signal predictions. So I trained linear regression to estimate the mass (I didn't know that the mass can be simply computed) and added this estimated mass as another input to my xgbtree. It improved the CV score to 0.9997. \r\n\r\nOn the other hand, I'm using the agreement/correlation data really only to determine, if my solution will pass or fail in the tests - which of course is too a way to leak the information from the files into my model.", "votes": null }, { "id": "96167", "postDate": "10/14/2015 22:17:57", "content": "I would love to hear detailed explanation from organizers why calculated mass used as a feature together with other features is not acceptable from the physical point of view. I understand why mass alone can't be use as good predictor, but I am still struggling with understanding why it's still bad to have it combined with other features (provided that cvm is as required < 0.002).
", "rawMarkdown": "I would love to hear detailed explanation from organizers why calculated mass used as a feature together with other features is not acceptable from the physical point of view. I understand why mass alone can't be use as good predictor, but I am still struggling with understanding why it's still bad to have it combined with other features (provided that cvm is as required < 0.002).", "votes": null }, { "id": "96308", "postDate": "10/15/2015 16:18:29", "content": "[quote=Alexander Gramolin;96138]
\n\nI would say that the good signs of a physically meaningful model are the following:
\n\nThese criteria are also obscure, of course. I encourage you and other strong participants to upload your models and let the Organizers judge about their physical soundness.
\n\n[/quote]
\n\nI still don't understand why training (direct or indirect on) on control channel considered wrong. In the end we look for good performance on both channels so why not just mix them for training? I didn't do that, just curious and still trying to find some statistical procedure to assert physical soundness of a model.
\n\nLet's consider more sophisticated setup than direct training on control channel. Train model for maximum performance only on signal channel. At this stage it won't pass agreement test. Then adjust the model via certain procedure in such a way so it did not change too much but learn to pass KS test . Now compare before and after and make sure that its performance on both channels changed only slightly and to the comparable degree. Say, AUC was 0.98/0.97 on signal/control channel; and only slightly degraded to 0.97/0.95 after adjustment for KS. What we just achieved: the model a) performs well on both channels, and b) difference between each sample distribution is small. Isn't a) and b) enough to say the model is physically sound?
", "rawMarkdown": "[quote=Alexander Gramolin;96138]\r\n\r\nI would say that the good signs of a physically meaningful model are the following:\r\n\r\n 1. you didn't use the agreement and correlation data sets while training the classifier;\r\n 2. you didn't try to reconstruct the invariant mass of the mother particle and to use it in any way;\r\n 3. you passed the agreement and correlation tests without fine tuning of the model and preferably not performing the tests too many times;\r\n 4. you didn't try to distinguish between the signal and the control channels, which is most likely true if you didn't look at the agreement data set and didn't calculate the invariant mass.\r\n\r\nThese criteria are also obscure, of course. I encourage you and other strong participants to upload your models and let the Organizers judge about their physical soundness.\r\n\r\n[/quote]\r\n\r\nI still don't understand why training (direct or indirect on) on control channel considered wrong. In the end we look for good performance on both channels so why not just mix them for training? I didn't do that, just curious and still trying to find some **statistical procedure** to assert physical soundness of a model.\r\n\r\nLet's consider more sophisticated setup than direct training on control channel. Train model for maximum performance only on signal channel. At this stage it won't pass agreement test. Then adjust the model via *certain procedure* in such a way so it did not change *too much* but learn to pass KS test . Now compare before and after and make sure that its performance on both channels changed only slightly and to the comparable degree. Say, AUC was 0.98/0.97 on signal/control channel; and only slightly degraded to 0.97/0.95 after adjustment for KS. What we just achieved: the model a) performs well on **both** channels, and b) difference between each sample distribution is small. Isn't a) and b) enough to say the model is physically sound?", "votes": null }, { "id": "96314", "postDate": "10/15/2015 16:58:41", "content": "[quote=rakhlin;96308]
\n\nI still don't understand why training (direct or indirect on) on control channel considered wrong. In the end we look for good performance on both channels so why not just mix them for training? I didn't do that, just curious and still trying to find some statistical procedure to assert physical soundness of a model.
\n\nLet's consider more sophisticated setup than direct training on control channel. Train model for maximum performance only on signal channel. At this stage it won't pass agreement test. Then adjust the model via certain procedure in such a way so it did not change too much but learn to pass KS test . Now compare before and after and make sure that its performance on both channels changed only slightly and to the comparable degree. Say, AUC was 0.98/0.97 on signal/control channel; and only slightly degraded to 0.97/0.95 after adjustment for KS. What we just achieved: the model a) performs well on both channels, and b) difference between each sample distribution is small. Isn't a) and b) enough to say the model is physically sound?
\n\n[/quote]
\n\nThe only reason why we need the control channel is to be able to check that the tested model is not sensitive to the imperfections of the Monte Carlo simulation in which the signal events were obtained. We wouldn't need it if we had both signal and background simulated, but in this competition we actually have real background and, of course, simulated signal.
\n\nThe correlation/agreement tests provided by the organizers are actually good statistical procedures already, but the issue is that you can still trick them if you use them wrong. In theory, you shouldn't even know about existence of the control channel and still be able to pass the agreement test. Real life is more complicated, of course. Even if you aren't using the agreement data set directly, you can learn it well by performing the agreement test many times and adjusting your model correspondingly. This is also a learning process, though not a machine one.
", "rawMarkdown": "[quote=rakhlin;96308]\r\n\r\nI still don't understand why training (direct or indirect on) on control channel considered wrong. In the end we look for good performance on both channels so why not just mix them for training? I didn't do that, just curious and still trying to find some **statistical procedure** to assert physical soundness of a model.\r\n\r\nLet's consider more sophisticated setup than direct training on control channel. Train model for maximum performance only on signal channel. At this stage it won't pass agreement test. Then adjust the model via *certain procedure* in such a way so it did not change *too much* but learn to pass KS test . Now compare before and after and make sure that its performance on both channels changed only slightly and to the comparable degree. Say, AUC was 0.98/0.97 on signal/control channel; and only slightly degraded to 0.97/0.95 after adjustment for KS. What we just achieved: the model a) performs well on **both** channels, and b) difference between each sample distribution is small. Isn't a) and b) enough to say the model is physically sound?\r\n\r\n[/quote]\r\n\r\nThe only reason why we need the control channel is to be able to check that the tested model is not sensitive to the imperfections of the Monte Carlo simulation in which the signal events were obtained. We wouldn't need it if we had both signal and background simulated, but in this competition we actually have real background and, of course, simulated signal.\r\n\r\nThe correlation/agreement tests provided by the organizers are actually good statistical procedures already, but the issue is that you can still trick them if you use them wrong. In theory, you shouldn't even know about existence of the control channel and still be able to pass the agreement test. Real life is more complicated, of course. Even if you aren't using the agreement data set directly, you can learn it well by performing the agreement test many times and adjusting your model correspondingly. This is also a learning process, though not a machine one.", "votes": null }, { "id": "96315", "postDate": "10/15/2015 17:07:16", "content": "Congratulations to all the winners and thanks to Kaggle &CERN for organizing this competition. \nLooking forward to more discussions, code sharing & tips from everyone. Thank you.
", "rawMarkdown": "Congratulations to all the winners and thanks to Kaggle &CERN for organizing this competition. \r\nLooking forward to more discussions, code sharing & tips from everyone. Thank you.", "votes": null }, { "id": "96318", "postDate": "10/15/2015 17:21:29", "content": "[quote=Alexander Gramolin;96314]\nEven if you aren't using the agreement data set directly, you can learn it well by performing the agreement test many times and adjusting your model correspondingly. This is also a learning process, though not a machine one.[/quote]\nSo what's wrong with this, as far as resulted model performs well on both channels and satisfies KS? Moreover, quite probably it would be close to the one obtained without agreement set. Why not consider this as a guarantee in mathematical/statistical sense?
", "rawMarkdown": "[quote=Alexander Gramolin;96314]\r\nEven if you aren't using the agreement data set directly, you can learn it well by performing the agreement test many times and adjusting your model correspondingly. This is also a learning process, though not a machine one.[/quote]\r\nSo what's wrong with this, as far as resulted model performs well on both channels and satisfies KS? Moreover, quite probably it would be close to the one obtained without agreement set. Why not consider this as a guarantee in mathematical/statistical sense?", "votes": null }, { "id": "96323", "postDate": "10/15/2015 17:37:39", "content": "[quote=rakhlin;96318]
\n\nSo what's wrong with this, as far as resulted model performs well on both channels and satisfies KS? Moreover, quite probably it would be close to the one obtained without agreement set. Why not consider this as a guarantee in mathematical/statistical sense?
\n\n[/quote]
\n\nYou just compromise the test and make it useless. Then you need another independent control channel to test your model again. :) The agreement test is auxiliary and shouldn't be used actively (the less the better) in the training process.
", "rawMarkdown": "[quote=rakhlin;96318]\r\n\r\nSo what's wrong with this, as far as resulted model performs well on both channels and satisfies KS? Moreover, quite probably it would be close to the one obtained without agreement set. Why not consider this as a guarantee in mathematical/statistical sense?\r\n\r\n[/quote]\r\n\r\nYou just compromise the test and make it useless. Then you need another independent control channel to test your model again. :) The agreement test is auxiliary and shouldn't be used actively (the less the better) in the training process.", "votes": null }, { "id": "96328", "postDate": "10/15/2015 18:14:00", "content": "[quote=Alexander Gramolin;96323]\nYou just compromise the test and make it useless. Then you need another independent control channel to test your model again. :) The agreement test is auxiliary and shouldn't be used actively (the less the better) in the training process.\n[/quote]\nI am not a physicist, but ML theory gives us certain statistical guaranties against compromised data, i.e. data used to build a model. If data is 'large enough' and model is 'small enough' it guarantees model generalization in statistical sense. As a rule of thumb data must be at least 10 times bigger than number of degrees of freedom of a model. This is not a scientific rule, rather practical consideration, but there is theory behind.
", "rawMarkdown": "[quote=Alexander Gramolin;96323]\r\nYou just compromise the test and make it useless. Then you need another independent control channel to test your model again. :) The agreement test is auxiliary and shouldn't be used actively (the less the better) in the training process.\r\n[/quote]\r\nI am not a physicist, but ML theory gives us certain statistical guaranties against compromised data, i.e. data used to build a model. If data is 'large enough' and model is 'small enough' it guarantees model generalization in statistical sense. As a rule of thumb data must be at least 10 times bigger than number of degrees of freedom of a model. This is not a scientific rule, rather practical consideration, but there is theory behind.", "votes": null }, { "id": "96330", "postDate": "10/15/2015 18:28:47", "content": "[quote=rakhlin;96328]
\n\nI am not a physicist, but ML theory gives us certain statistical guaranties against compromised data, i.e. data used to build a model. If data is 'large enough' and model is 'small enough' it guarantees model generalization in statistical sense. As a rule of thumb data must be at least 10 times bigger than number of degrees of freedom of a model. This is not a scientific rule, rather practical consideration, but there is theory behind.
\n\n[/quote]
\n\nWell, I agree that there should be some way to describe the process of data compromising quantitatively. But in general it's good to not overuse both the agreement and the correlation tests. Keep in mind also that particle physicists try to be conservative in their estimates of systematic uncertainties of measurements. Nobody wants to discover false decays.
", "rawMarkdown": "[quote=rakhlin;96328]\r\n\r\nI am not a physicist, but ML theory gives us certain statistical guaranties against compromised data, i.e. data used to build a model. If data is 'large enough' and model is 'small enough' it guarantees model generalization in statistical sense. As a rule of thumb data must be at least 10 times bigger than number of degrees of freedom of a model. This is not a scientific rule, rather practical consideration, but there is theory behind.\r\n\r\n[/quote]\r\n\r\nWell, I agree that there should be some way to describe the process of data compromising quantitatively. But in general it's good to not overuse both the agreement and the correlation tests. Keep in mind also that particle physicists try to be conservative in their estimates of systematic uncertainties of measurements. Nobody wants to discover false decays.", "votes": null } ], "comments": [ { "id": 96133, "author_name": "gramolin", "author_url": "", "post_date": "10/14/2015 18:36:35", "content": "Hello Josef,
\n\nIt's nice to hear from you, thanks. I guess that such high scores achieved in the leaderboard were surprising for many of us and for the Organizers too. ;)
\n\nUnfortunately, I have serious doubts that the winning models (including my own) can be useful for real physics... As far as I understand now, any usage of the reconstructed invariant mass as a feature (what I did) or usage the agreement/correlation data sets while training the classifier (you did it, right?) makes the model physically meaningless. Unfortunately, the competition wasn't designed perfectly. There are at least two independent ways to calculate the invariant mass of the mother particle using the features provided. I didn't understand earlier that it's a bad idea because I believed in the correlation test.
\n\nI realize that the Organizers are probably unhappy with the results, but I think that they should finally make some statement regarding the situation. After all, one of the purposes of the competition, as I understand, was to test the metrics designed to judge if the classifier is physically meaningful. If so, the negative result is also important. Another positive thing is that it was a lot of fun to work on this problem. We can probably do better next time, so let's draw the right lessons from this experience.
", "votes": null, "replies": [] }, { "id": 96135, "author_name": "rakhlin", "author_url": "", "post_date": "10/14/2015 18:56:09", "content": "Congratulations to winners.\nAlexander, is there way to figure out physical soundness of a model given its predictions? I'd like to verify my own (e.g. statistically), but I'm not a physicist and have no idea how to accomplish it correctly.
", "votes": null, "replies": [] }, { "id": 96138, "author_name": "gramolin", "author_url": "", "post_date": "10/14/2015 19:12:39", "content": "I would say that the good signs of a physically meaningful model are the following:
\n\nThese criteria are also obscure, of course. I encourage you and other strong participants to upload your models and let the Organizers judge about their physical soundness.
", "votes": null, "replies": [] }, { "id": 96143, "author_name": "phunter", "author_url": "", "post_date": "10/14/2015 19:47:24", "content": "[quote=Alexander Gramolin;96138]
\n\nI would say that the good signs of a physically meaningful model are the following:
\n\nThese criteria are also obscure, of course. I encourage you and other strong participants to upload your models and let the Organizers judge about their physical soundness.
\n\n[/quote]\nCongratulations to both teams from upstairs, your results are amazing. And, I think any solution should satisfy all these 4 criteria, otherwise it would be disqualified.
\n\nMy thought of these control channel tests is that, surely it is not perfectly designed since: 1. it gives some artificial data leakage 2. the real life calibration/control channel is much more complicated that this, and 3. I really don't like the CvM control channel which may mislead competitors for having some useless models on invariant mass. The 3rd point comes from reconstruction of invariant mass, as Alexander mentioned. Using this dataset, one can reconstruct invariant mass (just energy conservation, that is it, anyone knows special relativity can calculate it) and creates some artificial dependence of mass. A physical sounds model should not depend much on the mass spectrum, and the admin uses CvM test to limit this dependence, which may not be a good idea: a model which fails in CvM test can be ensembled with other models for 'smoothing' the CvM value and eventually pass the test, but it doesn't make sense. Since the independence on mass is important for the real life experiment, I don't think this CvM test can be a good idea. As to KS test, I think it is fine, simulation needs calibration, it is OK to ensemble models for passing Ks test, and it doesn't hurt physics much.
", "votes": null, "replies": [] }, { "id": 96147, "author_name": "anton79", "author_url": "", "post_date": "10/14/2015 20:09:38", "content": "Hi All,
\n\nI approached this problem from the pure machine learning prospective (I don't know nothing about particle physics). So, beyond dropping some features to pass the agreement test, I constructed meta features for both train and test using 15 different models. Then, I trained the train data ( with meta features added) using XGBoost, RF and UGBC. I tuned parameters for these models using Hyperopt package with 3-fold cross validation. Finally, I made an ensemble using the following formula:
\n\npreds=0.3 * (xgb_preds^0.65 * rf_preds^0.35) + 0.7 * ugb_preds
\n\nI tuned the weights using Hyperopt with 3-fold CV. In addition, I bagged all models in the ensemble: XGB - 250 times, RF - 130 times, UGBC (very slow!) - 30 times. This ensemble (without bagging) produced CV score of 0.994635. However, this model did not pass the correlation test by a tiny margin. Since, I did not have time to tune it more, I had to find (a possibly worse scoring) model which passes correlation test for sure and ensemble it with my model. I took the model from the public LB (score 0.987511), modified it a little bit and ensembled with my model using the formula:
\n\npreds_final = preds_LB^0.45 * preds_my_model^0.55
\n\nwhich gave me my final score.
\n\nTo conclude, I believe that my model has a lot of potential for improvement. I entered this competition only 4 weeks ago, and I spent more than half of this time getting familiar with data, setting, rules, and etc. Consequently, my model was produced just in one shot. It's a pity that I didn't have much time to:
\n\nOverall, I want to thank both the Kaggle team and guys from CERN for this competition. It was definitely fun to participate in it!
", "votes": null, "replies": [] }, { "id": 96163, "author_name": "josefslavicek", "author_url": "", "post_date": "10/14/2015 21:48:52", "content": "[quote=Alexander Gramolin;96133]\nUnfortunately, I have serious doubts that the winning models (including my own) can be useful for real physics... As far as I understand now, any usage of the reconstructed invariant mass as a feature (what I did) or usage the agreement/correlation data sets while training the classifier (you did it, right?) makes the model physically meaningless. \n[/quote]
\n\nYes, my model uses reconstructed mass. I added it, because my xgbtree started accidentally in some configuration show impressive CV score of 0.9995 and at the same time it started to fail desperately in correlation test. I found a way to combine results from several models to make it pass the correlation test (not knowing that doing it is wrong from physical viewpoint). I deduced that poor results in correlation test means that the xgbtree just found a way to internally reconstruct the mass and use it to make signal predictions. So I trained linear regression to estimate the mass (I didn't know that the mass can be simply computed) and added this estimated mass as another input to my xgbtree. It improved the CV score to 0.9997.
\n\nOn the other hand, I'm using the agreement/correlation data really only to determine, if my solution will pass or fail in the tests - which of course is too a way to leak the information from the files into my model.
", "votes": null, "replies": [] }, { "id": 96167, "author_name": "chalmi", "author_url": "", "post_date": "10/14/2015 22:17:57", "content": "I would love to hear detailed explanation from organizers why calculated mass used as a feature together with other features is not acceptable from the physical point of view. I understand why mass alone can't be use as good predictor, but I am still struggling with understanding why it's still bad to have it combined with other features (provided that cvm is as required < 0.002).
", "votes": null, "replies": [] }, { "id": 96308, "author_name": "rakhlin", "author_url": "", "post_date": "10/15/2015 16:18:29", "content": "[quote=Alexander Gramolin;96138]
\n\nI would say that the good signs of a physically meaningful model are the following:
\n\nThese criteria are also obscure, of course. I encourage you and other strong participants to upload your models and let the Organizers judge about their physical soundness.
\n\n[/quote]
\n\nI still don't understand why training (direct or indirect on) on control channel considered wrong. In the end we look for good performance on both channels so why not just mix them for training? I didn't do that, just curious and still trying to find some statistical procedure to assert physical soundness of a model.
\n\nLet's consider more sophisticated setup than direct training on control channel. Train model for maximum performance only on signal channel. At this stage it won't pass agreement test. Then adjust the model via certain procedure in such a way so it did not change too much but learn to pass KS test . Now compare before and after and make sure that its performance on both channels changed only slightly and to the comparable degree. Say, AUC was 0.98/0.97 on signal/control channel; and only slightly degraded to 0.97/0.95 after adjustment for KS. What we just achieved: the model a) performs well on both channels, and b) difference between each sample distribution is small. Isn't a) and b) enough to say the model is physically sound?
", "votes": null, "replies": [] }, { "id": 96314, "author_name": "gramolin", "author_url": "", "post_date": "10/15/2015 16:58:41", "content": "[quote=rakhlin;96308]
\n\nI still don't understand why training (direct or indirect on) on control channel considered wrong. In the end we look for good performance on both channels so why not just mix them for training? I didn't do that, just curious and still trying to find some statistical procedure to assert physical soundness of a model.
\n\nLet's consider more sophisticated setup than direct training on control channel. Train model for maximum performance only on signal channel. At this stage it won't pass agreement test. Then adjust the model via certain procedure in such a way so it did not change too much but learn to pass KS test . Now compare before and after and make sure that its performance on both channels changed only slightly and to the comparable degree. Say, AUC was 0.98/0.97 on signal/control channel; and only slightly degraded to 0.97/0.95 after adjustment for KS. What we just achieved: the model a) performs well on both channels, and b) difference between each sample distribution is small. Isn't a) and b) enough to say the model is physically sound?
\n\n[/quote]
\n\nThe only reason why we need the control channel is to be able to check that the tested model is not sensitive to the imperfections of the Monte Carlo simulation in which the signal events were obtained. We wouldn't need it if we had both signal and background simulated, but in this competition we actually have real background and, of course, simulated signal.
\n\nThe correlation/agreement tests provided by the organizers are actually good statistical procedures already, but the issue is that you can still trick them if you use them wrong. In theory, you shouldn't even know about existence of the control channel and still be able to pass the agreement test. Real life is more complicated, of course. Even if you aren't using the agreement data set directly, you can learn it well by performing the agreement test many times and adjusting your model correspondingly. This is also a learning process, though not a machine one.
", "votes": null, "replies": [] }, { "id": 96315, "author_name": "pythonomic", "author_url": "", "post_date": "10/15/2015 17:07:16", "content": "Congratulations to all the winners and thanks to Kaggle &CERN for organizing this competition. \nLooking forward to more discussions, code sharing & tips from everyone. Thank you.
", "votes": null, "replies": [] }, { "id": 96318, "author_name": "rakhlin", "author_url": "", "post_date": "10/15/2015 17:21:29", "content": "[quote=Alexander Gramolin;96314]\nEven if you aren't using the agreement data set directly, you can learn it well by performing the agreement test many times and adjusting your model correspondingly. This is also a learning process, though not a machine one.[/quote]\nSo what's wrong with this, as far as resulted model performs well on both channels and satisfies KS? Moreover, quite probably it would be close to the one obtained without agreement set. Why not consider this as a guarantee in mathematical/statistical sense?
", "votes": null, "replies": [] }, { "id": 96323, "author_name": "gramolin", "author_url": "", "post_date": "10/15/2015 17:37:39", "content": "[quote=rakhlin;96318]
\n\nSo what's wrong with this, as far as resulted model performs well on both channels and satisfies KS? Moreover, quite probably it would be close to the one obtained without agreement set. Why not consider this as a guarantee in mathematical/statistical sense?
\n\n[/quote]
\n\nYou just compromise the test and make it useless. Then you need another independent control channel to test your model again. :) The agreement test is auxiliary and shouldn't be used actively (the less the better) in the training process.
", "votes": null, "replies": [] }, { "id": 96328, "author_name": "rakhlin", "author_url": "", "post_date": "10/15/2015 18:14:00", "content": "[quote=Alexander Gramolin;96323]\nYou just compromise the test and make it useless. Then you need another independent control channel to test your model again. :) The agreement test is auxiliary and shouldn't be used actively (the less the better) in the training process.\n[/quote]\nI am not a physicist, but ML theory gives us certain statistical guaranties against compromised data, i.e. data used to build a model. If data is 'large enough' and model is 'small enough' it guarantees model generalization in statistical sense. As a rule of thumb data must be at least 10 times bigger than number of degrees of freedom of a model. This is not a scientific rule, rather practical consideration, but there is theory behind.
", "votes": null, "replies": [] }, { "id": 96330, "author_name": "gramolin", "author_url": "", "post_date": "10/15/2015 18:28:47", "content": "[quote=rakhlin;96328]
\n\nI am not a physicist, but ML theory gives us certain statistical guaranties against compromised data, i.e. data used to build a model. If data is 'large enough' and model is 'small enough' it guarantees model generalization in statistical sense. As a rule of thumb data must be at least 10 times bigger than number of degrees of freedom of a model. This is not a scientific rule, rather practical consideration, but there is theory behind.
\n\n[/quote]
\n\nWell, I agree that there should be some way to describe the process of data compromising quantitatively. But in general it's good to not overuse both the agreement and the correlation tests. Keep in mind also that particle physicists try to be conservative in their estimates of systematic uncertainties of measurements. Nobody wants to discover false decays.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "96128": "Hello,\r\n\r\nfirst, excuse, please, my bad English. Second, excuse, please, my brevity - I really didn't expected that I will place so high, so I have lots of work to complete my solution description, which is required to be published today (October 14th).\r\nI wish to thank CERN and organizers for this exciting competition. I wish to congratulate the winning team 'Go Polar Bears' and Alexander Gramolin who placed second - we have had great fight in the past hours of the competition and he managed to beat me :-). \r\n\r\nExcuse, please, poor quality of my source code. I really didn't expected such good result, so I didn't expected that I will be asked to publish my code. And there is not enough time to clean it up. So, here it is (it's crazy mixture of three programming languages):\r\n[third_place.zip][1]\r\n. Installation instruction are in the INSTALL.txt file.\r\n\r\nI'm going to write description of my solution and hope to publish it on this forum until midnight UTC.\r\n\r\nEdit: description of my model is now packed in the third_place.zip file.\r\n\r\nEdit2: I simplified my solution to the level where it can be published as [script][2]. It scores 0.999594 as a \"Post-Deadline entry\" on private leaderboard (~third place).\r\n\r\n\r\n [1]: http://jslavicek.cz/third_place.zip\r\n [2]: https://www.kaggle.com/josefslavicek/flavours-of-physics/simplified-version-of-my-solution", "96133": "Hello Josef,\r\n\r\nIt's nice to hear from you, thanks. I guess that such high scores achieved in the leaderboard were surprising for many of us and for the Organizers too. ;)\r\n\r\nUnfortunately, I have serious doubts that the winning models (including my own) can be useful for real physics... As far as I understand now, any usage of the reconstructed invariant mass as a feature (what I did) or usage the agreement/correlation data sets while training the classifier (you did it, right?) makes the model physically meaningless. Unfortunately, the competition wasn't designed perfectly. There are at least two independent ways to calculate the invariant mass of the mother particle using the features provided. I didn't understand earlier that it's a bad idea because I believed in the correlation test.\r\n\r\nI realize that the Organizers are probably unhappy with the results, but I think that they should finally make some statement regarding the situation. After all, one of the purposes of the competition, as I understand, was to test the metrics designed to judge if the classifier is physically meaningful. If so, the negative result is also important. Another positive thing is that it was a lot of fun to work on this problem. We can probably do better next time, so let's draw the right lessons from this experience.", "96135": "Congratulations to winners.\r\nAlexander, is there way to figure out physical soundness of a model given its predictions? I'd like to verify my own (e.g. statistically), but I'm not a physicist and have no idea how to accomplish it correctly.", "96138": "I would say that the good signs of a physically meaningful model are the following:\r\n\r\n 1. you didn't use the agreement and correlation data sets while training the classifier;\r\n 2. you didn't try to reconstruct the invariant mass of the mother particle and to use it in any way;\r\n 3. you passed the agreement and correlation tests without fine tuning of the model and preferably not performing the tests too many times;\r\n 4. you didn't try to distinguish between the signal and the control channels, which is most likely true if you didn't look at the agreement data set and didn't calculate the invariant mass.\r\n\r\nThese criteria are also obscure, of course. I encourage you and other strong participants to upload your models and let the Organizers judge about their physical soundness.", "96143": "[quote=Alexander Gramolin;96138]\r\n\r\nI would say that the good signs of a physically meaningful model are the following:\r\n\r\n 1. you didn't use the agreement and correlation data sets while training the classifier;\r\n 2. you didn't try to reconstruct the invariant mass of the mother particle and to use it in any way;\r\n 3. you passed the agreement and correlation tests without fine tuning of the model and preferably not performing the tests too many times;\r\n 4. you didn't try to distinguish between the signal and the control channels, which is most likely true if you didn't look at the agreement data set and didn't calculate the invariant mass.\r\n\r\nThese criteria are also obscure, of course. I encourage you and other strong participants to upload your models and let the Organizers judge about their physical soundness.\r\n\r\n[/quote]\r\nCongratulations to both teams from upstairs, your results are amazing. And, I think any solution should satisfy all these 4 criteria, otherwise it would be disqualified. \r\n\r\nMy thought of these control channel tests is that, surely it is not perfectly designed since: 1. it gives some artificial data leakage 2. the real life calibration/control channel is much more complicated that this, and 3. I really don't like the CvM control channel which may mislead competitors for having some useless models on invariant mass. The 3rd point comes from reconstruction of invariant mass, as Alexander mentioned. Using this dataset, one can reconstruct invariant mass (just energy conservation, that is it, anyone knows special relativity can calculate it) and creates some artificial dependence of mass. A physical sounds model should not depend much on the mass spectrum, and the admin uses CvM test to limit this dependence, which may not be a good idea: a model which fails in CvM test can be ensembled with other models for 'smoothing' the CvM value and eventually pass the test, but it doesn't make sense. Since the independence on mass is important for the real life experiment, I don't think this CvM test can be a good idea. As to KS test, I think it is fine, simulation needs calibration, it is OK to ensemble models for passing Ks test, and it doesn't hurt physics much.", "96147": "Hi All,\r\n\r\nI approached this problem from the pure machine learning prospective (I don't know nothing about particle physics). So, beyond dropping some features to pass the agreement test, I constructed meta features for both train and test using 15 different models. Then, I trained the train data ( with meta features added) using XGBoost, RF and UGBC. I tuned parameters for these models using Hyperopt package with 3-fold cross validation. Finally, I made an ensemble using the following formula:\r\n\r\npreds=0.3 * (xgb_preds^0.65 * rf_preds^0.35) + 0.7 * ugb_preds\r\n\r\nI tuned the weights using Hyperopt with 3-fold CV. In addition, I bagged all models in the ensemble: XGB - 250 times, RF - 130 times, UGBC (very slow!) - 30 times. This ensemble (without bagging) produced CV score of 0.994635. However, this model did not pass the correlation test by a tiny margin. Since, I did not have time to tune it more, I had to find (a possibly worse scoring) model which passes correlation test for sure and ensemble it with my model. I took the model from the public LB (score 0.987511), modified it a little bit and ensembled with my model using the formula:\r\n\r\n preds_final = preds_LB^0.45 * preds_my_model^0.55\r\n\r\nwhich gave me my final score.\r\n\r\nTo conclude, I believe that my model has a lot of potential for improvement. I entered this competition only 4 weeks ago, and I spent more than half of this time getting familiar with data, setting, rules, and etc. Consequently, my model was produced just in one shot. It's a pity that I didn't have much time to:\r\n\r\n1. Select the optimal set of both features and metafeatures\r\n2. Select optimal models for meta features construction\r\n3. Play with more models on the ensembling stage.\r\n\r\nOverall, I want to thank both the Kaggle team and guys from CERN for this competition. It was definitely fun to participate in it!", "96163": "[quote=Alexander Gramolin;96133]\r\nUnfortunately, I have serious doubts that the winning models (including my own) can be useful for real physics... As far as I understand now, any usage of the reconstructed invariant mass as a feature (what I did) or usage the agreement/correlation data sets while training the classifier (you did it, right?) makes the model physically meaningless. \r\n[/quote]\r\n\r\nYes, my model uses reconstructed mass. I added it, because my xgbtree started accidentally in some configuration show impressive CV score of 0.9995 and at the same time it started to fail desperately in correlation test. I found a way to combine results from several models to make it pass the correlation test (not knowing that doing it is wrong from physical viewpoint). I deduced that poor results in correlation test means that the xgbtree just found a way to internally reconstruct the mass and use it to make signal predictions. So I trained linear regression to estimate the mass (I didn't know that the mass can be simply computed) and added this estimated mass as another input to my xgbtree. It improved the CV score to 0.9997. \r\n\r\nOn the other hand, I'm using the agreement/correlation data really only to determine, if my solution will pass or fail in the tests - which of course is too a way to leak the information from the files into my model.", "96167": "I would love to hear detailed explanation from organizers why calculated mass used as a feature together with other features is not acceptable from the physical point of view. I understand why mass alone can't be use as good predictor, but I am still struggling with understanding why it's still bad to have it combined with other features (provided that cvm is as required < 0.002).", "96308": "[quote=Alexander Gramolin;96138]\r\n\r\nI would say that the good signs of a physically meaningful model are the following:\r\n\r\n 1. you didn't use the agreement and correlation data sets while training the classifier;\r\n 2. you didn't try to reconstruct the invariant mass of the mother particle and to use it in any way;\r\n 3. you passed the agreement and correlation tests without fine tuning of the model and preferably not performing the tests too many times;\r\n 4. you didn't try to distinguish between the signal and the control channels, which is most likely true if you didn't look at the agreement data set and didn't calculate the invariant mass.\r\n\r\nThese criteria are also obscure, of course. I encourage you and other strong participants to upload your models and let the Organizers judge about their physical soundness.\r\n\r\n[/quote]\r\n\r\nI still don't understand why training (direct or indirect on) on control channel considered wrong. In the end we look for good performance on both channels so why not just mix them for training? I didn't do that, just curious and still trying to find some **statistical procedure** to assert physical soundness of a model.\r\n\r\nLet's consider more sophisticated setup than direct training on control channel. Train model for maximum performance only on signal channel. At this stage it won't pass agreement test. Then adjust the model via *certain procedure* in such a way so it did not change *too much* but learn to pass KS test . Now compare before and after and make sure that its performance on both channels changed only slightly and to the comparable degree. Say, AUC was 0.98/0.97 on signal/control channel; and only slightly degraded to 0.97/0.95 after adjustment for KS. What we just achieved: the model a) performs well on **both** channels, and b) difference between each sample distribution is small. Isn't a) and b) enough to say the model is physically sound?", "96314": "[quote=rakhlin;96308]\r\n\r\nI still don't understand why training (direct or indirect on) on control channel considered wrong. In the end we look for good performance on both channels so why not just mix them for training? I didn't do that, just curious and still trying to find some **statistical procedure** to assert physical soundness of a model.\r\n\r\nLet's consider more sophisticated setup than direct training on control channel. Train model for maximum performance only on signal channel. At this stage it won't pass agreement test. Then adjust the model via *certain procedure* in such a way so it did not change *too much* but learn to pass KS test . Now compare before and after and make sure that its performance on both channels changed only slightly and to the comparable degree. Say, AUC was 0.98/0.97 on signal/control channel; and only slightly degraded to 0.97/0.95 after adjustment for KS. What we just achieved: the model a) performs well on **both** channels, and b) difference between each sample distribution is small. Isn't a) and b) enough to say the model is physically sound?\r\n\r\n[/quote]\r\n\r\nThe only reason why we need the control channel is to be able to check that the tested model is not sensitive to the imperfections of the Monte Carlo simulation in which the signal events were obtained. We wouldn't need it if we had both signal and background simulated, but in this competition we actually have real background and, of course, simulated signal.\r\n\r\nThe correlation/agreement tests provided by the organizers are actually good statistical procedures already, but the issue is that you can still trick them if you use them wrong. In theory, you shouldn't even know about existence of the control channel and still be able to pass the agreement test. Real life is more complicated, of course. Even if you aren't using the agreement data set directly, you can learn it well by performing the agreement test many times and adjusting your model correspondingly. This is also a learning process, though not a machine one.", "96315": "Congratulations to all the winners and thanks to Kaggle &CERN for organizing this competition. \r\nLooking forward to more discussions, code sharing & tips from everyone. Thank you.", "96318": "[quote=Alexander Gramolin;96314]\r\nEven if you aren't using the agreement data set directly, you can learn it well by performing the agreement test many times and adjusting your model correspondingly. This is also a learning process, though not a machine one.[/quote]\r\nSo what's wrong with this, as far as resulted model performs well on both channels and satisfies KS? Moreover, quite probably it would be close to the one obtained without agreement set. Why not consider this as a guarantee in mathematical/statistical sense?", "96323": "[quote=rakhlin;96318]\r\n\r\nSo what's wrong with this, as far as resulted model performs well on both channels and satisfies KS? Moreover, quite probably it would be close to the one obtained without agreement set. Why not consider this as a guarantee in mathematical/statistical sense?\r\n\r\n[/quote]\r\n\r\nYou just compromise the test and make it useless. Then you need another independent control channel to test your model again. :) The agreement test is auxiliary and shouldn't be used actively (the less the better) in the training process.", "96328": "[quote=Alexander Gramolin;96323]\r\nYou just compromise the test and make it useless. Then you need another independent control channel to test your model again. :) The agreement test is auxiliary and shouldn't be used actively (the less the better) in the training process.\r\n[/quote]\r\nI am not a physicist, but ML theory gives us certain statistical guaranties against compromised data, i.e. data used to build a model. If data is 'large enough' and model is 'small enough' it guarantees model generalization in statistical sense. As a rule of thumb data must be at least 10 times bigger than number of degrees of freedom of a model. This is not a scientific rule, rather practical consideration, but there is theory behind.", "96330": "[quote=rakhlin;96328]\r\n\r\nI am not a physicist, but ML theory gives us certain statistical guaranties against compromised data, i.e. data used to build a model. If data is 'large enough' and model is 'small enough' it guarantees model generalization in statistical sense. As a rule of thumb data must be at least 10 times bigger than number of degrees of freedom of a model. This is not a scientific rule, rather practical consideration, but there is theory behind.\r\n\r\n[/quote]\r\n\r\nWell, I agree that there should be some way to describe the process of data compromising quantitatively. But in general it's good to not overuse both the agreement and the correlation tests. Keep in mind also that particle physicists try to be conservative in their estimates of systematic uncertainties of measurements. Nobody wants to discover false decays." }, "source": "meta" }