Source code and brief description of the solution ranked second on the Private Leaderboard can be found at https://github.com/gramolin/flavours-of-physics
\n\nI'd like to congratulate the Go Polar Bears team and Josef Slavicek who also achieved very high scores. I'm very grateful to the Organizers for the opportunity to work on an exciting problem and to analyze real data from LHCb. Here are some thoughts regarding the technical details of the competition which I'd like to share:
\n\nDespite all this, please don't stop your efforts to bring particle physicists and machine learning people together. We look forward to the next Kaggle competition from CERN!
", "messages": [ { "id": "96577", "postDate": "10/18/2015 18:33:04", "content": "Source code and brief description of the solution ranked second on the Private Leaderboard can be found at https://github.com/gramolin/flavours-of-physics
\n\nI'd like to congratulate the Go Polar Bears team and Josef Slavicek who also achieved very high scores. I'm very grateful to the Organizers for the opportunity to work on an exciting problem and to analyze real data from LHCb. Here are some thoughts regarding the technical details of the competition which I'd like to share:
\n\nDespite all this, please don't stop your efforts to bring particle physicists and machine learning people together. We look forward to the next Kaggle competition from CERN!
", "rawMarkdown": "Source code and brief description of the solution ranked second on the Private Leaderboard can be found at https://github.com/gramolin/flavours-of-physics\r\n\r\nI'd like to congratulate the Go Polar Bears team and Josef Slavicek who also achieved very high scores. I'm very grateful to the Organizers for the opportunity to work on an exciting problem and to analyze real data from LHCb. Here are some thoughts regarding the technical details of the competition which I'd like to share:\r\n\r\n 1. I realize that it was difficult to perfectly combine real and simulated data and make them indistinguishable from one another. By removing real background events from the region of invariant masses around the tau mass you, of course, made sure that there are no potential decays of tau to three muons among them. At the same time, you introduced a serious vulnerability, because now mass can be used to perfectly discriminate between signal and background.\r\n 2. It was probably a bad idea to provide the agreement and correlation datasets to participants. Instead, I think it would be enough to just let them check the KS and CvM scores while making submissions (limited number of times). This would ensure that they don't overuse the tests.\r\n 3. For me personally, it was sad that you didn't provide azimuthal (phi) angles of the particles. This limited our ability to use kinematics to analyze events. At the same time, it didn't prevent us to calculate the mass of the mother particle anyway.\r\n\r\nDespite all this, please don't stop your efforts to bring particle physicists and machine learning people together. We look forward to the next Kaggle competition from CERN!", "votes": null }, { "id": "96606", "postDate": "10/19/2015 03:24:30", "content": "An xgboost-only solution, nice to see the true power of xgboost!
", "rawMarkdown": "An xgboost-only solution, nice to see the true power of xgboost!", "votes": null }, { "id": "96862", "postDate": "10/20/2015 20:12:27", "content": "This is excellent; a 2 model ensemble proving that there is still room for smart feature engineering rather than brute force 100/1000 model ensembles + meta ensemblers + stacked ensembles.
", "rawMarkdown": "This is excellent; a 2 model ensemble proving that there is still room for smart feature engineering rather than brute force 100/1000 model ensembles + meta ensemblers + stacked ensembles.", "votes": null }, { "id": "96877", "postDate": "10/21/2015 02:02:28", "content": "Just a quick question, I see the invariant mass of the mother particle is calculated. How does the model with invariant mass pass the CvM test? Thanks.
", "rawMarkdown": "Just a quick question, I see the invariant mass of the mother particle is calculated. How does the model with invariant mass pass the CvM test? Thanks.", "votes": null }, { "id": "96896", "postDate": "10/21/2015 07:38:55", "content": "[quote=phunter]
\n\nJust a quick question, I see the invariant mass of the mother particle is calculated. How does the model with invariant mass pass the CvM test? Thanks.
\n\n[/quote]
\n\nVicens Gaitan performed a nice analysis of the mass gap vulnerability: https://www.kaggle.com/vicensgaitan/flavours-of-physics/clipping-spreading
\n\nI think that here we have a similar situation. I combined a strong (kinematic) and a weak (geometric) classifiers to pass the correlation test. In addition, I used the XGBoost parameter scale_pos_weight=5 which, I guess, is somewhat equivalent to raising the predicted probabilities to some power.
\n\nThough the idea of combining a geometric and a kinematic classifiers can actually be useful, it's impossible to fairly test it with the datasets provided. I wish we didn't have that bad mass gap in the background (which probably could be filled with simulated events)...
", "rawMarkdown": "[quote=phunter]\r\n\r\nJust a quick question, I see the invariant mass of the mother particle is calculated. How does the model with invariant mass pass the CvM test? Thanks.\r\n\r\n[/quote]\r\n\r\nVicens Gaitan performed a nice analysis of the mass gap vulnerability: https://www.kaggle.com/vicensgaitan/flavours-of-physics/clipping-spreading\r\n\r\nI think that here we have a similar situation. I combined a strong (kinematic) and a weak (geometric) classifiers to pass the correlation test. In addition, I used the XGBoost parameter scale_pos_weight=5 which, I guess, is somewhat equivalent to raising the predicted probabilities to some power.\r\n\r\nThough the idea of combining a geometric and a kinematic classifiers can actually be useful, it's impossible to fairly test it with the datasets provided. I wish we didn't have that bad mass gap in the background (which probably could be filled with simulated events)...", "votes": null }, { "id": "97009", "postDate": "10/21/2015 21:19:50", "content": "@Alexander Gramolin I see, thanks. I had similar observation that, kinematic features (without explicit mass or energy features) can make xgboost fail in the KS test and I am afraid the reason is that, xgboost pickups some complicated rules which implicitly use energy, mass or other combinations which are not physical sounds, since xgboost is a strong classifier.
\n\nscale_pos_weight for the training set should be around 3.8 which is the scale of negative vs positive, giving more weight to positive samples may help xgboost finding the good rules.
\n\nAs to the question if weighted combination of a strong (kinematic with invaraint mass) and a weak (geometric) classifiers can be useful for the real life (aka physical sounds), I think we should see admins' opinion, maybe this search channel is robust with invariant mass so using mass is fine?
", "rawMarkdown": "Alexander Gramolin I see, thanks. I had similar observation that, kinematic features (without explicit mass or energy features) can make xgboost fail in the KS test and I am afraid the reason is that, xgboost pickups some complicated rules which implicitly use energy, mass or other combinations which are not physical sounds, since xgboost is a strong classifier. \r\n\r\nscale_pos_weight for the training set should be around 3.8 which is the scale of negative vs positive, giving more weight to positive samples may help xgboost finding the good rules.\r\n\r\nAs to the question if weighted combination of a strong (kinematic with invaraint mass) and a weak (geometric) classifiers can be useful for the real life (aka physical sounds), I think we should see admins' opinion, maybe this search channel is robust with invariant mass so using mass is fine?", "votes": null }, { "id": "97028", "postDate": "10/22/2015 00:33:55", "content": "xgboost just doing its work well. From a perspective of statistical inference failure on a distribution other than that was used for training means nothing (just as success).
", "rawMarkdown": "xgboost just doing its work well. From a perspective of statistical inference failure on a distribution other than that was used for training means nothing (just as success).", "votes": null }, { "id": "97281", "postDate": "10/25/2015 02:58:40", "content": "Congrats and thank you for sharing
", "rawMarkdown": "Congrats and thank you for sharing", "votes": null } ], "comments": [ { "id": 96606, "author_name": "phunter", "author_url": "", "post_date": "10/19/2015 03:24:30", "content": "An xgboost-only solution, nice to see the true power of xgboost!
", "votes": null, "replies": [] }, { "id": 96862, "author_name": "guidotapia", "author_url": "", "post_date": "10/20/2015 20:12:27", "content": "This is excellent; a 2 model ensemble proving that there is still room for smart feature engineering rather than brute force 100/1000 model ensembles + meta ensemblers + stacked ensembles.
", "votes": null, "replies": [] }, { "id": 96877, "author_name": "phunter", "author_url": "", "post_date": "10/21/2015 02:02:28", "content": "Just a quick question, I see the invariant mass of the mother particle is calculated. How does the model with invariant mass pass the CvM test? Thanks.
", "votes": null, "replies": [] }, { "id": 96896, "author_name": "gramolin", "author_url": "", "post_date": "10/21/2015 07:38:55", "content": "[quote=phunter]
\n\nJust a quick question, I see the invariant mass of the mother particle is calculated. How does the model with invariant mass pass the CvM test? Thanks.
\n\n[/quote]
\n\nVicens Gaitan performed a nice analysis of the mass gap vulnerability: https://www.kaggle.com/vicensgaitan/flavours-of-physics/clipping-spreading
\n\nI think that here we have a similar situation. I combined a strong (kinematic) and a weak (geometric) classifiers to pass the correlation test. In addition, I used the XGBoost parameter scale_pos_weight=5 which, I guess, is somewhat equivalent to raising the predicted probabilities to some power.
\n\nThough the idea of combining a geometric and a kinematic classifiers can actually be useful, it's impossible to fairly test it with the datasets provided. I wish we didn't have that bad mass gap in the background (which probably could be filled with simulated events)...
", "votes": null, "replies": [] }, { "id": 97009, "author_name": "phunter", "author_url": "", "post_date": "10/21/2015 21:19:50", "content": "@Alexander Gramolin I see, thanks. I had similar observation that, kinematic features (without explicit mass or energy features) can make xgboost fail in the KS test and I am afraid the reason is that, xgboost pickups some complicated rules which implicitly use energy, mass or other combinations which are not physical sounds, since xgboost is a strong classifier.
\n\nscale_pos_weight for the training set should be around 3.8 which is the scale of negative vs positive, giving more weight to positive samples may help xgboost finding the good rules.
\n\nAs to the question if weighted combination of a strong (kinematic with invaraint mass) and a weak (geometric) classifiers can be useful for the real life (aka physical sounds), I think we should see admins' opinion, maybe this search channel is robust with invariant mass so using mass is fine?
", "votes": null, "replies": [] }, { "id": 97028, "author_name": "rakhlin", "author_url": "", "post_date": "10/22/2015 00:33:55", "content": "xgboost just doing its work well. From a perspective of statistical inference failure on a distribution other than that was used for training means nothing (just as success).
", "votes": null, "replies": [] }, { "id": 97281, "author_name": "aiijoe", "author_url": "", "post_date": "10/25/2015 02:58:40", "content": "Congrats and thank you for sharing
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "96577": "Source code and brief description of the solution ranked second on the Private Leaderboard can be found at https://github.com/gramolin/flavours-of-physics\r\n\r\nI'd like to congratulate the Go Polar Bears team and Josef Slavicek who also achieved very high scores. I'm very grateful to the Organizers for the opportunity to work on an exciting problem and to analyze real data from LHCb. Here are some thoughts regarding the technical details of the competition which I'd like to share:\r\n\r\n 1. I realize that it was difficult to perfectly combine real and simulated data and make them indistinguishable from one another. By removing real background events from the region of invariant masses around the tau mass you, of course, made sure that there are no potential decays of tau to three muons among them. At the same time, you introduced a serious vulnerability, because now mass can be used to perfectly discriminate between signal and background.\r\n 2. It was probably a bad idea to provide the agreement and correlation datasets to participants. Instead, I think it would be enough to just let them check the KS and CvM scores while making submissions (limited number of times). This would ensure that they don't overuse the tests.\r\n 3. For me personally, it was sad that you didn't provide azimuthal (phi) angles of the particles. This limited our ability to use kinematics to analyze events. At the same time, it didn't prevent us to calculate the mass of the mother particle anyway.\r\n\r\nDespite all this, please don't stop your efforts to bring particle physicists and machine learning people together. We look forward to the next Kaggle competition from CERN!", "96606": "An xgboost-only solution, nice to see the true power of xgboost!", "96862": "This is excellent; a 2 model ensemble proving that there is still room for smart feature engineering rather than brute force 100/1000 model ensembles + meta ensemblers + stacked ensembles.", "96877": "Just a quick question, I see the invariant mass of the mother particle is calculated. How does the model with invariant mass pass the CvM test? Thanks.", "96896": "[quote=phunter]\r\n\r\nJust a quick question, I see the invariant mass of the mother particle is calculated. How does the model with invariant mass pass the CvM test? Thanks.\r\n\r\n[/quote]\r\n\r\nVicens Gaitan performed a nice analysis of the mass gap vulnerability: https://www.kaggle.com/vicensgaitan/flavours-of-physics/clipping-spreading\r\n\r\nI think that here we have a similar situation. I combined a strong (kinematic) and a weak (geometric) classifiers to pass the correlation test. In addition, I used the XGBoost parameter scale_pos_weight=5 which, I guess, is somewhat equivalent to raising the predicted probabilities to some power.\r\n\r\nThough the idea of combining a geometric and a kinematic classifiers can actually be useful, it's impossible to fairly test it with the datasets provided. I wish we didn't have that bad mass gap in the background (which probably could be filled with simulated events)...", "97009": "Alexander Gramolin I see, thanks. I had similar observation that, kinematic features (without explicit mass or energy features) can make xgboost fail in the KS test and I am afraid the reason is that, xgboost pickups some complicated rules which implicitly use energy, mass or other combinations which are not physical sounds, since xgboost is a strong classifier. \r\n\r\nscale_pos_weight for the training set should be around 3.8 which is the scale of negative vs positive, giving more weight to positive samples may help xgboost finding the good rules.\r\n\r\nAs to the question if weighted combination of a strong (kinematic with invaraint mass) and a weak (geometric) classifiers can be useful for the real life (aka physical sounds), I think we should see admins' opinion, maybe this search channel is robust with invariant mass so using mass is fine?", "97028": "xgboost just doing its work well. From a perspective of statistical inference failure on a distribution other than that was used for training means nothing (just as success).", "97281": "Congrats and thank you for sharing" }, "source": "meta" }