I am not sure if this was intended by organizer but features included almost complete information about mass of mother particle. Using elementary equations of special theory of relativity I was able to get mass with absolute mean error ~7 and using estimators reduced this further to ~6. (Anyone get better estimations?) As we all know mass was strongly correlated with signal. Of course organizers wanted something, as I understand, beyond mass. But I don't understand why they excluded mass from test data if this was possible to calculate almost exactly anyway. In my opinion mass should be included in features and there should be no correlation test. This is true that mass could be confusing, but this was only one feature. And classifier based only on mass was far worse than good classifier based on mass and all other features. So the goal of this competition to find something which is better than mass would have been achieved anyway. To win you must have anyway invented something much better than mass alone. I used calculated mass as a feature but combined with other classifier that used features that were as much unrelated with mass as possible - that improved results but still allowed to pass correlation test, which I will argue was only unnecessary obstacle in this task (taking into account that organizer wanted to escape from mass!).
\n\nAnd one more remark. Assuming mass is confusing we should have also data that proves this. And this data should be used in training and test. In the presented data mass wasn't confusing at all, it was the strongest feature.
\n\nI wonder if winner also used calculated mass? Please write here your thoughts about usage of mass in this competition.
", "messages": [ { "id": "95932", "postDate": "10/13/2015 00:52:35", "content": "I am not sure if this was intended by organizer but features included almost complete information about mass of mother particle. Using elementary equations of special theory of relativity I was able to get mass with absolute mean error ~7 and using estimators reduced this further to ~6. (Anyone get better estimations?) As we all know mass was strongly correlated with signal. Of course organizers wanted something, as I understand, beyond mass. But I don't understand why they excluded mass from test data if this was possible to calculate almost exactly anyway. In my opinion mass should be included in features and there should be no correlation test. This is true that mass could be confusing, but this was only one feature. And classifier based only on mass was far worse than good classifier based on mass and all other features. So the goal of this competition to find something which is better than mass would have been achieved anyway. To win you must have anyway invented something much better than mass alone. I used calculated mass as a feature but combined with other classifier that used features that were as much unrelated with mass as possible - that improved results but still allowed to pass correlation test, which I will argue was only unnecessary obstacle in this task (taking into account that organizer wanted to escape from mass!).
\n\nAnd one more remark. Assuming mass is confusing we should have also data that proves this. And this data should be used in training and test. In the presented data mass wasn't confusing at all, it was the strongest feature.
\n\nI wonder if winner also used calculated mass? Please write here your thoughts about usage of mass in this competition.
", "rawMarkdown": "I am not sure if this was intended by organizer but features included almost complete information about mass of mother particle. Using elementary equations of special theory of relativity I was able to get mass with absolute mean error ~7 and using estimators reduced this further to ~6. (Anyone get better estimations?) As we all know mass was strongly correlated with signal. Of course organizers wanted something, as I understand, beyond mass. But I don't understand why they excluded mass from test data if this was possible to calculate almost exactly anyway. In my opinion mass should be included in features and there should be no correlation test. This is true that mass could be confusing, but this was only one feature. And classifier based only on mass was far worse than good classifier based on mass and all other features. So the goal of this competition to find something which is better than mass would have been achieved anyway. To win you must have anyway invented something much better than mass alone. I used calculated mass as a feature but combined with other classifier that used features that were as much unrelated with mass as possible - that improved results but still allowed to pass correlation test, which I will argue was only unnecessary obstacle in this task (taking into account that organizer wanted to escape from mass!).\r\n\r\nAnd one more remark. Assuming mass is confusing we should have also data that proves this. And this data should be used in training and test. In the presented data mass wasn't confusing at all, it was the strongest feature.\r\n\r\nI wonder if winner also used calculated mass? Please write here your thoughts about usage of mass in this competition.", "votes": null }, { "id": "95948", "postDate": "10/13/2015 06:04:36", "content": "I was able to calculate the mass to within +/-1 % of the given mass value for 95% of the training dataset from relativistic kinematic (see https://www.kaggle.com/c/flavours-of-physics/forums/t/16783/quick-reference-of-hep-kinematics-calculation ). This assumed the muon mass for all three decay products which may not be valid in all cases. But then I failed / did not spend enough effort on figuring out how to pass the Cramer-van-Mises test. But at least it was interesting to see the mass distributions in the 'agreement test' sample (which was at masses higher than the training set) and the correlation test sample (which is nicely covering the signal region and sidebands).
\n\nTraditionally, the mass variable is a very important one in the search for new phenomena in particle physics. If one can show that an excess over known backgrounds are concentrated at a certain mass (i.e. produced by a new, yet undiscovered unknown particle), this adds a lot of credibility to the result.
\n\nNow what one wants to avoid is to 'shape the background' by e.g. preferentially selecting events in one mass region (for example around the tau mass 1777 MeV/c^2) which then form a peak, faking evidence for a new physics phenomenon. You could e.g. easily train a classifier based on the mass only, apply it to a distribution which is flat in mass (e.g. background as recorded with the detector) and you should get a peak in the mass spectrum without the presence of any new particle.
", "rawMarkdown": "I was able to calculate the mass to within +/-1 % of the given mass value for 95% of the training dataset from relativistic kinematic (see https://www.kaggle.com/c/flavours-of-physics/forums/t/16783/quick-reference-of-hep-kinematics-calculation ). This assumed the muon mass for all three decay products which may not be valid in all cases. But then I failed / did not spend enough effort on figuring out how to pass the Cramer-van-Mises test. But at least it was interesting to see the mass distributions in the 'agreement test' sample (which was at masses higher than the training set) and the correlation test sample (which is nicely covering the signal region and sidebands).\r\n\r\nTraditionally, the mass variable is a very important one in the search for new phenomena in particle physics. If one can show that an excess over known backgrounds are concentrated at a certain mass (i.e. produced by a new, yet undiscovered unknown particle), this adds a lot of credibility to the result.\r\n\r\nNow what one wants to avoid is to 'shape the background' by e.g. preferentially selecting events in one mass region (for example around the tau mass 1777 MeV/c^2) which then form a peak, faking evidence for a new physics phenomenon. You could e.g. easily train a classifier based on the mass only, apply it to a distribution which is flat in mass (e.g. background as recorded with the detector) and you should get a peak in the mass spectrum without the presence of any new particle.", "votes": null }, { "id": "95950", "postDate": "10/13/2015 06:25:43", "content": "In principle, we need another angle information of each event to fully reconstruct the mass. Unfortunately it is missing in the dataset. Maybe the CERN host hides this angle data on purpose. Instead, I try to use pt to effectively get the mother particle mass as follows
\n\nmu=105.7 # muon mass
\n\ndef reconstructed_mass_by_pt(df):
\n\n list=[sqrt(df['p0_pt']*df['p0_pt']+mu*mu), \n\n sqrt(df['p1_pt']*df['p1_pt']+mu*mu),\n\n sqrt(df['p2_pt']*df['p2_pt']+mu*mu) ]\n\n return sqrt(abs(sum(list)*sum(list)-df['pt']*df['pt']))\nSome of the difference values between sum(list)*sum(list) and df['pt']*df['pt'] are negative due to the fluctuation. That is why I choose to use the absolute values. This feature of the effective mass turns out to be very useful to distinguish signals and backgrounds (see figure in the attachment). This mass feature will result in the fail of passing the test. In order to pass the Cramer-van-Mises test, another stable classifier is required to smooth the final result. For example, xgboost (with the mass feature) + rf (without the mass feature) is a good choice.
", "rawMarkdown": "In principle, we need another angle information of each event to fully reconstruct the mass. Unfortunately it is missing in the dataset. Maybe the CERN host hides this angle data on purpose. Instead, I try to use pt to effectively get the mother particle mass as follows\r\n\r\nmu=105.7 # muon mass\r\n\r\n\r\ndef reconstructed_mass_by_pt(df):\r\n\r\n list=[sqrt(df['p0_pt']*df['p0_pt']+mu*mu), \r\n\r\n sqrt(df['p1_pt']*df['p1_pt']+mu*mu),\r\n\r\n sqrt(df['p2_pt']*df['p2_pt']+mu*mu) ]\r\n\r\n return sqrt(abs(sum(list)*sum(list)-df['pt']*df['pt']))\r\n \r\nSome of the difference values between sum(list)*sum(list) and df['pt']*df['pt'] are negative due to the fluctuation. That is why I choose to use the absolute values. This feature of the effective mass turns out to be very useful to distinguish signals and backgrounds (see figure in the attachment). This mass feature will result in the fail of passing the test. In order to pass the Cramer-van-Mises test, another stable classifier is required to smooth the final result. For example, xgboost (with the mass feature) + rf (without the mass feature) is a good choice.", "votes": null }, { "id": "95954", "postDate": "10/13/2015 07:14:53", "content": "Hm ... I got this in a bit longer way, but probably equivalent.
\n\ntotal_pz = sqrt(p0_p * p0_p - p0_pt * p0_pt) + sqrt(p0_p * p0_p - p1_pt * p1_pt) + sqrt(p0_p * p0_p - p2_pt * p2_pt)\np = sqrt(pt*pt + total_pz*total_pz)\nE0 = sqrt(p0_p*p0_p + 11164.0356)\nE1 = sqrt(p1_p*p1_p + 11164.0356)\nE2 = sqrt(p2_p*p2_p + 11164.0356)\nE = E0 + E1 + E2 \nm = sqrt(E*E - p*p)\n\n\n
And of course I also noticed that data for agreement test could be easily filtered by mass. There was a temptation to use it ;-) But I choose to develop physically sound classifier - and filtering agreement test data would result in overfit by using some aspects of features that were related to simulation v. reality recognition. And as such this would have no physial meaning at all. Instead I developed second classifier based on features that don't distinguish simulated decays from real decays (and also where not corralled too much with mass) and joined this with the classifier that was based sollely on mass. Best results that I got in this method was 0.999225 and it passed both tests without any tricks. The only thing is I needed to choose a small coefficient alpha in final prediction
\n\nY = alpha * [prediction based on mass] + (1 - alpha) * [prediction based on special set of features] \n\n\n
to avoid high score on Cramer-van-Mises test - but I think this is perfectly valid.
", "rawMarkdown": "Hm ... I got this in a bit longer way, but probably equivalent.\r\n\r\n\r\ntotal_pz = sqrt(p0_p * p0_p - p0_pt * p0_pt) + sqrt(p0_p * p0_p - p1_pt * p1_pt) + sqrt(p0_p * p0_p - p2_pt * p2_pt)\r\np = sqrt(pt*pt + total_pz*total_pz)\r\nE0 = sqrt(p0_p*p0_p + 11164.0356)\r\nE1 = sqrt(p1_p*p1_p + 11164.0356)\r\nE2 = sqrt(p2_p*p2_p + 11164.0356)\r\nE = E0 + E1 + E2 \r\nm = sqrt(E*E - p*p)\r\n\r\n\r\nAnd of course I also noticed that data for agreement test could be easily filtered by mass. There was a temptation to use it ;-) But I choose to develop physically sound classifier - and filtering agreement test data would result in overfit by using some aspects of features that were related to simulation v. reality recognition. And as such this would have no physial meaning at all. Instead I developed second classifier based on features that don't distinguish simulated decays from real decays (and also where not corralled too much with mass) and joined this with the classifier that was based sollely on mass. Best results that I got in this method was 0.999225 and it passed both tests without any tricks. The only thing is I needed to choose a small coefficient alpha in final prediction \r\n
\r\nY = alpha * [prediction based on mass] + (1 - alpha) * [prediction based on special set of features] \r\n\r\nto avoid high score on Cramer-van-Mises test - but I think this is perfectly valid.", "votes": null }, { "id": "95955", "postDate": "10/13/2015 07:25:11", "content": "
[quote=Andre Holzner;95948]
\n\nNow what one wants to avoid is to 'shape the background' by e.g. preferentially selecting events in one mass region (for example around the tau mass 1777 MeV/c^2) which then form a peak, faking evidence for a new physics phenomenon. You could e.g. easily train a classifier based on the mass only, apply it to a distribution which is flat in mass (e.g. background as recorded with the detector) and you should get a peak in the mass spectrum without the presence of any new particle.
\n\n[/quote]
\n\nBut why this didn't happen in our data set? I mean in our data set mass was a very good predictor with very small number of FPs. If in reality you can get this calculated mass at level of around 1777 MeV/c^2 and still this is not desired decay, such data should be presented in this competition. Then it would be clear that you can not relay too much on mass. But anyway mass would be strong feature.
", "rawMarkdown": "[quote=Andre Holzner;95948]\r\n\r\nNow what one wants to avoid is to 'shape the background' by e.g. preferentially selecting events in one mass region (for example around the tau mass 1777 MeV/c^2) which then form a peak, faking evidence for a new physics phenomenon. You could e.g. easily train a classifier based on the mass only, apply it to a distribution which is flat in mass (e.g. background as recorded with the detector) and you should get a peak in the mass spectrum without the presence of any new particle.\r\n\r\n[/quote]\r\n\r\nBut why this didn't happen in our data set? I mean in our data set mass was a very good predictor with very small number of FPs. If in reality you can get this calculated mass at level of around 1777 MeV/c^2 and still this is not desired decay, such data should be presented in this competition. Then it would be clear that you can not relay too much on mass. But anyway mass would be strong feature.", "votes": null }, { "id": "95973", "postDate": "10/13/2015 09:29:44", "content": "In practice, the mass will be used in a likelihood ratio estimator on a subset of the observed data which pass a threshold on the classifier's output.
\n\nI have uploaded a script here https://www.kaggle.com/holzner/flavours-of-physics/candidate-mass/ \nto illustrate the 'estimated' mass distributions.
\n\nIf you plot the mass directly on the training set, you'll notice that there are no background events in the range 1757.. 1798 MeV, so from the background in the training dataset it's impossible to create a spurious peak at the tau mass but in data recorded with the detector for example you easily could.
", "rawMarkdown": "In practice, the mass will be used in a likelihood ratio estimator on a subset of the observed data which pass a threshold on the classifier's output.\r\n\r\nI have uploaded a script here https://www.kaggle.com/holzner/flavours-of-physics/candidate-mass/ \r\nto illustrate the 'estimated' mass distributions. \r\n\r\nIf you plot the mass directly on the training set, you'll notice that there are no background events in the range 1757.. 1798 MeV, so from the background in the training dataset it's impossible to create a spurious peak at the tau mass but in data recorded with the detector for example you easily could.", "votes": null }, { "id": "95976", "postDate": "10/13/2015 09:42:01", "content": "[quote=Andre Holzner;95973]
\n\nIn practice, the mass will be used in a likelihood ratio estimator on a subset of the observed data which pass a threshold on the classifier's output.\n...\nIf you plot the mass directly on the training set, you'll notice that there are no background events in the range 1757.. 1798 MeV, so from the background in the training dataset it's impossible to create a spurious peak at the tau mass but in data recorded with the detector for example you easily could.
\n\n[/quote]
\n\nPerhaps I miss something, but I still don't understand why not to add such data recorded with detector to data set and then train this together with mass information. It would be probably better than doing this in 2 stages. Usually joining conditions in ML is better than treating them separately. Even on provided data set mass alone was not as good as mass + other features.
", "rawMarkdown": "[quote=Andre Holzner;95973]\r\n\r\nIn practice, the mass will be used in a likelihood ratio estimator on a subset of the observed data which pass a threshold on the classifier's output.\r\n...\r\nIf you plot the mass directly on the training set, you'll notice that there are no background events in the range 1757.. 1798 MeV, so from the background in the training dataset it's impossible to create a spurious peak at the tau mass but in data recorded with the detector for example you easily could.\r\n\r\n[/quote]\r\n\r\nPerhaps I miss something, but I still don't understand why not to add such data recorded with detector to data set and then train this together with mass information. It would be probably better than doing this in 2 stages. Usually joining conditions in ML is better than treating them separately. Even on provided data set mass alone was not as good as mass + other features.", "votes": null }, { "id": "95980", "postDate": "10/13/2015 10:42:04", "content": "[quote=Michal Wojcik;95976]\nPerhaps I miss something, but I still don't understand why not to add such data recorded with detector to data set and then train this together with mass information. It would be probably better than doing this in 2 stages. Usually joining conditions in ML is better than treating them separately. Even on provided data set mass alone was not as good as mass + other features.\n[/quote]
\n\nI fully agree that a combined classifier (using all available information) is expected to perform better than splitting this into two stages.
\n\nHowever, in practice:
\n\nmass is 'fundamental' :-) . Physics phenomena at these microscopic scales are modeled by the exchange ('interaction') of particles among particles. Each particle has a mass. Evidence for a new physics phenomena is best supported if one can quote a mass for it.
data recorded from the detector does not come from the same n-dimensional probability density as the simulation (on which one trains the classifiers, although I think in this competition at least the background was probably recorded detector data). Simulation of the known interactions ('background') is only possible with a limited precision, simulation of the interaction of the particles produced in the collision with the detector elements also can't be simulated perfectly and is computationally very expensive. Given the fact that one is typically looking for a few tens of collisions in a several billion recorded ones per year, one has to find ways to work with the limited precision of the simulation. One way is use regions in variable space which have no expected signal to demonstrate the understanding of the background events above which one is looking for a signal. The 'mass sidebands' (mass regions away from the tau mass) are the best candidate to do this in a search like tau -> 3 mu .
false positive results ('fake discoveries') are much worse for reputation and credibility than false negative results ('missed discoveries'). You may have heard of the 'faster than light neutrino anomaly' but an example (where a mass is reconstructed like in this competition) is this one: http://www.science20.com/quantum_diaries_survivor/death_dijet_anomaly-130783 . A subtle effect on the correction (calibration) of the energies of particles (similar to the p{0,1,2}_p variables) recorded in the detector caused excitement (3.2 standard deviations above background).
I hope this clarifies it a bit.
", "rawMarkdown": "[quote=Michal Wojcik;95976]\r\nPerhaps I miss something, but I still don't understand why not to add such data recorded with detector to data set and then train this together with mass information. It would be probably better than doing this in 2 stages. Usually joining conditions in ML is better than treating them separately. Even on provided data set mass alone was not as good as mass + other features.\r\n[/quote]\r\n\r\nI fully agree that a combined classifier (using all available information) is expected to perform better than splitting this into two stages.\r\n\r\nHowever, in practice:\r\n\r\n - mass *is* 'fundamental' :-) . Physics phenomena at these microscopic scales are modeled by the exchange ('interaction') of particles among particles. Each particle has a mass. Evidence for a new physics phenomena is best supported if one can quote a mass for it.\r\n\r\n - data recorded from the detector does not come from the same n-dimensional probability density as the simulation (on which one trains the classifiers, although I think in this competition at least the background was probably recorded detector data). Simulation of the known interactions ('background') is only possible with a limited precision, simulation of the interaction of the particles produced in the collision with the detector elements also can't be simulated perfectly and is computationally very expensive. Given the fact that one is typically looking for a few tens of collisions in a several billion recorded ones per year, one has to find ways to work with the limited precision of the simulation. One way is use regions in variable space which have no expected signal to demonstrate the understanding of the background events above which one is looking for a signal. The 'mass sidebands' (mass regions away from the tau mass) are the best candidate to do this in a search like tau -> 3 mu . \r\n\r\n - false positive results ('fake discoveries') are much worse for reputation and credibility than false negative results ('missed discoveries'). You may have heard of the 'faster than light neutrino anomaly' but an example (where a mass is reconstructed like in this competition) is this one: http://www.science20.com/quantum_diaries_survivor/death_dijet_anomaly-130783 . A subtle effect on the correction (calibration) of the energies of particles (similar to the p{0,1,2}_p variables) recorded in the detector caused excitement (3.2 standard deviations above background). \r\n\r\nI hope this clarifies it a bit.", "votes": null }, { "id": "95981", "postDate": "10/13/2015 11:34:40", "content": "Yes mass is fundamental. But here we are talking about quite a different thing. Mass calculated as a feature from pt, p{0,1,2}_p and p{0,1,2}_pt variables. We already know that this feature sometimes differs from real mass because of some unrealistic assumptions in calculations. In fact we may forget this is a mass. This is just a feature M. But as a feature it is proved to be useful in a model (at least on the provided data set). So is there any practical reason for not using this M feature in model for this, as I understand, first line filter? (And of course using in model doesn't mean relaying solely on M). As I described my solution is using a linear combination on some exponential function from M, exp((M - 1777)^2/base) - with a very small factor like 0.1 and other classifier, which is based on some combination of features that try to hide absolute values - to be mass uncorrelated. Is there any particular reason of not using such linear combination in this pre-filter and using only the part independent on feature M? I got the part of false alarms - but there is still a hope that other features will "brake" this alarm (after all the cvm of final classifier with mass anyway is less than 0.002).
\n\nAs I understand we are talking only about pre-filtering, nobody will claim discovery based on this classifier results, but only will search more deeply in the selected results as I understand.
\n\nBTW, could you or admins provide some sample of data where mass is really faking predictions? That would be good to see how the designed classifier will behave in such case.
\n\nAnd this part is interesting:
\n\n\n\n\nOne way is use regions in variable space which have no expected signal to demonstrate the understanding of the background events above which one is looking for a signal. The 'mass sidebands' (mass regions away from the tau mass) are the best candidate to do this in a search like tau -> 3 mu .
\n
Can you explain why is that? I mean it is rather counter-intuitive.
", "rawMarkdown": "Yes mass is fundamental. But here we are talking about quite a different thing. Mass calculated as a feature from pt, p{0,1,2}_p and p{0,1,2}_pt variables. We already know that this feature sometimes differs from real mass because of some unrealistic assumptions in calculations. In fact we may forget this is a mass. This is just a feature M. But as a feature it is proved to be useful in a model (at least on the provided data set). So is there any practical reason for not using this M feature in model for this, as I understand, first line filter? (And of course using in model doesn't mean relaying solely on M). As I described my solution is using a linear combination on some exponential function from M, exp((M - 1777)^2/base) - with a very small factor like 0.1 and other classifier, which is based on some combination of features that try to hide absolute values - to be mass uncorrelated. Is there any particular reason of not using such linear combination in this pre-filter and using only the part independent on feature M? I got the part of false alarms - but there is still a hope that other features will \"brake\" this alarm (after all the cvm of final classifier with mass anyway is less than 0.002). \r\n\r\nAs I understand we are talking only about pre-filtering, nobody will claim discovery based on this classifier results, but only will search more deeply in the selected results as I understand. \r\n\r\nBTW, could you or admins provide some sample of data where mass is really faking predictions? That would be good to see how the designed classifier will behave in such case. \r\n\r\nAnd this part is interesting:\r\n\r\n> One way is use regions in variable space which have no expected signal to demonstrate the understanding of the background events above which one is looking for a signal. The 'mass sidebands' (mass regions away from the tau mass) are the best candidate to do this in a search like tau -> 3 mu .\r\n\r\nCan you explain why is that? I mean it is rather counter-intuitive.", "votes": null }, { "id": "95996", "postDate": "10/13/2015 15:49:04", "content": "I had similar calculation for the 'pseudo' transverse invariant mass and it helped about +0.005 AUC, but I realized it could fail the CvM test. I think CvM test is for preventing the bias to the invariant mass and the model needs to be more generalized, because HEP has look-elsewhere effect that the full mass spectrum should be watched https://en.wikipedia.org/wiki/Look-elsewhere_effect thus a bias to mass peak may not be physical sound and may not help in the real life. The same story happened during the Higgs boson search where some team found a peak at 115 GeV and it turned out a statistical fluctuation and a artificial mass peak fit, so introducing a bias on invariant mass is not a good idea. So I chose not to play with CvM test, and left these mass features alone.
\n\nHowever, the trick for passing CvM test was good: I used the same trick for passing KS test. I had models with SPDhits which couldn't pass KS test for sure because SPDhits were not well simulated and needed calibration, but other models like RF were very OK with no SPDhits, so ensemble these models under the constraint of KS score can maximize the AUC. It becomes an optimization problem.
\n\n[quote=Liucheng Wang;95950]
\n\nIn principle, we need another angle information of each event to fully reconstruct the mass. Unfortunately it is missing in the dataset. Maybe the CERN host hides this angle data on purpose. Instead, I try to use pt to effectively get the mother particle mass as follows
\n\nmu=105.7 # muon mass
\n\ndef reconstructed_mass_by_pt(df):
\n\n list=[sqrt(df['p0_pt']*df['p0_pt']+mu*mu), \n\n sqrt(df['p1_pt']*df['p1_pt']+mu*mu),\n\n sqrt(df['p2_pt']*df['p2_pt']+mu*mu) ]\n\n return sqrt(abs(sum(list)*sum(list)-df['pt']*df['pt']))\nSome of the difference values between sum(list)*sum(list) and df['pt']*df['pt'] are negative due to the fluctuation. That is why I choose to use the absolute values. This feature of the effective mass turns out to be very useful to distinguish signals and backgrounds (see figure in the attachment). This mass feature will result in the fail of passing the test. In order to pass the Cramer-van-Mises test, another stable classifier is required to smooth the final result. For example, xgboost (with the mass feature) + rf (without the mass feature) is a good choice.
\n\n[/quote]
", "rawMarkdown": "I had similar calculation for the 'pseudo' transverse invariant mass and it helped about +0.005 AUC, but I realized it could fail the CvM test. I think CvM test is for preventing the bias to the invariant mass and the model needs to be more generalized, because HEP has look-elsewhere effect that the full mass spectrum should be watched https://en.wikipedia.org/wiki/Look-elsewhere_effect thus a bias to mass peak may not be physical sound and may not help in the real life. The same story happened during the Higgs boson search where some team found a peak at 115 GeV and it turned out a statistical fluctuation and a artificial mass peak fit, so introducing a bias on invariant mass is not a good idea. So I chose not to play with CvM test, and left these mass features alone.\r\n\r\nHowever, the trick for passing CvM test was good: I used the same trick for passing KS test. I had models with SPDhits which couldn't pass KS test for sure because SPDhits were not well simulated and needed calibration, but other models like RF were very OK with no SPDhits, so ensemble these models under the constraint of KS score can maximize the AUC. It becomes an optimization problem.\r\n\r\n[quote=Liucheng Wang;95950]\r\n\r\nIn principle, we need another angle information of each event to fully reconstruct the mass. Unfortunately it is missing in the dataset. Maybe the CERN host hides this angle data on purpose. Instead, I try to use pt to effectively get the mother particle mass as follows\r\n\r\nmu=105.7 # muon mass\r\n\r\n\r\ndef reconstructed_mass_by_pt(df):\r\n\r\n list=[sqrt(df['p0_pt']*df['p0_pt']+mu*mu), \r\n\r\n sqrt(df['p1_pt']*df['p1_pt']+mu*mu),\r\n\r\n sqrt(df['p2_pt']*df['p2_pt']+mu*mu) ]\r\n\r\n return sqrt(abs(sum(list)*sum(list)-df['pt']*df['pt']))\r\n \r\nSome of the difference values between sum(list)*sum(list) and df['pt']*df['pt'] are negative due to the fluctuation. That is why I choose to use the absolute values. This feature of the effective mass turns out to be very useful to distinguish signals and backgrounds (see figure in the attachment). This mass feature will result in the fail of passing the test. In order to pass the Cramer-van-Mises test, another stable classifier is required to smooth the final result. For example, xgboost (with the mass feature) + rf (without the mass feature) is a good choice.\r\n\r\n\r\n[/quote]", "votes": null }, { "id": "96245", "postDate": "10/15/2015 08:34:05", "content": "(disclaimer: I'm not a competition admin)
\n\n[quote=Michal Wojcik;95981]\nAnd this part is interesting:
\n\n\n\n\nOne way is use regions in variable space which have no expected signal to demonstrate the understanding of the background events above which one is looking for a signal. The 'mass sidebands' (mass regions away from the tau mass) are the best candidate to do this in a search like tau -> 3 mu .
\n
Can you explain why is that? I mean it is rather counter-intuitive.
\n\n[/quote]
\n\nMass is rather used as a 'last line' filter. It is very powerful indeed but one has to demonstrate that the probability of selecting events (rows in the dataset) is independent of mass.
\n\nIn the end, one runs a hypothesis test of
\n\nvs.
\n\nOften a likelihood ratio is used. For a simple 'number of events (collisions) with a classifier output greater than a given threshold' this is:
\n\nQ = Probability(number of observed events | expected number of signal + background events) / Probability(number of observed events | expected number of background events)
\n\nHow to obtain the number of expected background events from data ?
\n\nWhat is done in searches like these that one assumes e.g. everything with a mass below 1757 MeV and everything with a mass above 1798 is background (there is no signal there). One needs however to know also the background from 1757 to 1798 MeV. This is done by interpolating between these two mass points. The obvious choice is to assume that the background follows a flat distribution between m = 1757 and 1798 MeV. The CvM test was meant to ensure this.
\n\nIf the classifier is not independent of mass for background events, this assumption is not valid any more and one can't estimate the number of background events this way.
", "rawMarkdown": "(disclaimer: I'm not a competition admin)\r\n\r\n[quote=Michal Wojcik;95981]\r\nAnd this part is interesting:\r\n\r\n> One way is use regions in variable space which have no expected signal to demonstrate the understanding of the background events above which one is looking for a signal. The 'mass sidebands' (mass regions away from the tau mass) are the best candidate to do this in a search like tau -> 3 mu .\r\n\r\nCan you explain why is that? I mean it is rather counter-intuitive.\r\n\r\n[/quote]\r\n\r\nMass is rather used as a 'last line' filter. It is very powerful indeed but one has to demonstrate that the probability of selecting events (rows in the dataset) is independent of mass. \r\n\r\nIn the end, one runs a hypothesis test of\r\n\r\n - 'background only present in recorded data' \r\n\r\nvs. \r\n\r\n - 'signal + background present in recorded data'\r\n\r\nOften a likelihood ratio is used. For a simple 'number of events (collisions) with a classifier output greater than a given threshold' this is:\r\n\r\n Q = Probability(number of observed events | expected number of signal + background events) / Probability(number of observed events | expected number of background events)\r\n\r\n - the number of observed (recorded with the detector) events passing the classifier threshold is obvious to obtain\r\n - the number of expected signal events passing the classifier threshold is obtained from simulation\r\n - the important point is the number of background events: it could also be obtained from simulation but this has large uncertainties (limited statistics and imperfect modelling) so in practice it is also obtained from recorded data.\r\n\r\nHow to obtain the number of expected background events from data ? \r\n\r\nWhat is done in searches like these that one assumes e.g. everything with a mass below 1757 MeV and everything with a mass above 1798 is background (there is no signal there). One needs however to know also the background from 1757 to 1798 MeV. This is done by interpolating between these two mass points. The obvious choice is to assume that the background follows a flat distribution between m = 1757 and 1798 MeV. The CvM test was meant to ensure this. \r\n\r\nIf the classifier is not independent of mass for background events, this assumption is not valid any more and one can't estimate the number of background events this way.", "votes": null } ], "comments": [ { "id": 95948, "author_name": "holzner", "author_url": "", "post_date": "10/13/2015 06:04:36", "content": "I was able to calculate the mass to within +/-1 % of the given mass value for 95% of the training dataset from relativistic kinematic (see https://www.kaggle.com/c/flavours-of-physics/forums/t/16783/quick-reference-of-hep-kinematics-calculation ). This assumed the muon mass for all three decay products which may not be valid in all cases. But then I failed / did not spend enough effort on figuring out how to pass the Cramer-van-Mises test. But at least it was interesting to see the mass distributions in the 'agreement test' sample (which was at masses higher than the training set) and the correlation test sample (which is nicely covering the signal region and sidebands).
\n\nTraditionally, the mass variable is a very important one in the search for new phenomena in particle physics. If one can show that an excess over known backgrounds are concentrated at a certain mass (i.e. produced by a new, yet undiscovered unknown particle), this adds a lot of credibility to the result.
\n\nNow what one wants to avoid is to 'shape the background' by e.g. preferentially selecting events in one mass region (for example around the tau mass 1777 MeV/c^2) which then form a peak, faking evidence for a new physics phenomenon. You could e.g. easily train a classifier based on the mass only, apply it to a distribution which is flat in mass (e.g. background as recorded with the detector) and you should get a peak in the mass spectrum without the presence of any new particle.
", "votes": null, "replies": [] }, { "id": 95950, "author_name": "liuchengwang", "author_url": "", "post_date": "10/13/2015 06:25:43", "content": "In principle, we need another angle information of each event to fully reconstruct the mass. Unfortunately it is missing in the dataset. Maybe the CERN host hides this angle data on purpose. Instead, I try to use pt to effectively get the mother particle mass as follows
\n\nmu=105.7 # muon mass
\n\ndef reconstructed_mass_by_pt(df):
\n\n list=[sqrt(df['p0_pt']*df['p0_pt']+mu*mu), \n\n sqrt(df['p1_pt']*df['p1_pt']+mu*mu),\n\n sqrt(df['p2_pt']*df['p2_pt']+mu*mu) ]\n\n return sqrt(abs(sum(list)*sum(list)-df['pt']*df['pt']))\nSome of the difference values between sum(list)*sum(list) and df['pt']*df['pt'] are negative due to the fluctuation. That is why I choose to use the absolute values. This feature of the effective mass turns out to be very useful to distinguish signals and backgrounds (see figure in the attachment). This mass feature will result in the fail of passing the test. In order to pass the Cramer-van-Mises test, another stable classifier is required to smooth the final result. For example, xgboost (with the mass feature) + rf (without the mass feature) is a good choice.
", "votes": null, "replies": [] }, { "id": 95954, "author_name": "chalmi", "author_url": "", "post_date": "10/13/2015 07:14:53", "content": "Hm ... I got this in a bit longer way, but probably equivalent.
\n\ntotal_pz = sqrt(p0_p * p0_p - p0_pt * p0_pt) + sqrt(p0_p * p0_p - p1_pt * p1_pt) + sqrt(p0_p * p0_p - p2_pt * p2_pt)\np = sqrt(pt*pt + total_pz*total_pz)\nE0 = sqrt(p0_p*p0_p + 11164.0356)\nE1 = sqrt(p1_p*p1_p + 11164.0356)\nE2 = sqrt(p2_p*p2_p + 11164.0356)\nE = E0 + E1 + E2 \nm = sqrt(E*E - p*p)\n\n\n
And of course I also noticed that data for agreement test could be easily filtered by mass. There was a temptation to use it ;-) But I choose to develop physically sound classifier - and filtering agreement test data would result in overfit by using some aspects of features that were related to simulation v. reality recognition. And as such this would have no physial meaning at all. Instead I developed second classifier based on features that don't distinguish simulated decays from real decays (and also where not corralled too much with mass) and joined this with the classifier that was based sollely on mass. Best results that I got in this method was 0.999225 and it passed both tests without any tricks. The only thing is I needed to choose a small coefficient alpha in final prediction
\n\nY = alpha * [prediction based on mass] + (1 - alpha) * [prediction based on special set of features] \n\n\n
to avoid high score on Cramer-van-Mises test - but I think this is perfectly valid.
", "votes": null, "replies": [] }, { "id": 95955, "author_name": "chalmi", "author_url": "", "post_date": "10/13/2015 07:25:11", "content": "[quote=Andre Holzner;95948]
\n\nNow what one wants to avoid is to 'shape the background' by e.g. preferentially selecting events in one mass region (for example around the tau mass 1777 MeV/c^2) which then form a peak, faking evidence for a new physics phenomenon. You could e.g. easily train a classifier based on the mass only, apply it to a distribution which is flat in mass (e.g. background as recorded with the detector) and you should get a peak in the mass spectrum without the presence of any new particle.
\n\n[/quote]
\n\nBut why this didn't happen in our data set? I mean in our data set mass was a very good predictor with very small number of FPs. If in reality you can get this calculated mass at level of around 1777 MeV/c^2 and still this is not desired decay, such data should be presented in this competition. Then it would be clear that you can not relay too much on mass. But anyway mass would be strong feature.
", "votes": null, "replies": [] }, { "id": 95973, "author_name": "holzner", "author_url": "", "post_date": "10/13/2015 09:29:44", "content": "In practice, the mass will be used in a likelihood ratio estimator on a subset of the observed data which pass a threshold on the classifier's output.
\n\nI have uploaded a script here https://www.kaggle.com/holzner/flavours-of-physics/candidate-mass/ \nto illustrate the 'estimated' mass distributions.
\n\nIf you plot the mass directly on the training set, you'll notice that there are no background events in the range 1757.. 1798 MeV, so from the background in the training dataset it's impossible to create a spurious peak at the tau mass but in data recorded with the detector for example you easily could.
", "votes": null, "replies": [] }, { "id": 95976, "author_name": "chalmi", "author_url": "", "post_date": "10/13/2015 09:42:01", "content": "[quote=Andre Holzner;95973]
\n\nIn practice, the mass will be used in a likelihood ratio estimator on a subset of the observed data which pass a threshold on the classifier's output.\n...\nIf you plot the mass directly on the training set, you'll notice that there are no background events in the range 1757.. 1798 MeV, so from the background in the training dataset it's impossible to create a spurious peak at the tau mass but in data recorded with the detector for example you easily could.
\n\n[/quote]
\n\nPerhaps I miss something, but I still don't understand why not to add such data recorded with detector to data set and then train this together with mass information. It would be probably better than doing this in 2 stages. Usually joining conditions in ML is better than treating them separately. Even on provided data set mass alone was not as good as mass + other features.
", "votes": null, "replies": [] }, { "id": 95980, "author_name": "holzner", "author_url": "", "post_date": "10/13/2015 10:42:04", "content": "[quote=Michal Wojcik;95976]\nPerhaps I miss something, but I still don't understand why not to add such data recorded with detector to data set and then train this together with mass information. It would be probably better than doing this in 2 stages. Usually joining conditions in ML is better than treating them separately. Even on provided data set mass alone was not as good as mass + other features.\n[/quote]
\n\nI fully agree that a combined classifier (using all available information) is expected to perform better than splitting this into two stages.
\n\nHowever, in practice:
\n\nmass is 'fundamental' :-) . Physics phenomena at these microscopic scales are modeled by the exchange ('interaction') of particles among particles. Each particle has a mass. Evidence for a new physics phenomena is best supported if one can quote a mass for it.
data recorded from the detector does not come from the same n-dimensional probability density as the simulation (on which one trains the classifiers, although I think in this competition at least the background was probably recorded detector data). Simulation of the known interactions ('background') is only possible with a limited precision, simulation of the interaction of the particles produced in the collision with the detector elements also can't be simulated perfectly and is computationally very expensive. Given the fact that one is typically looking for a few tens of collisions in a several billion recorded ones per year, one has to find ways to work with the limited precision of the simulation. One way is use regions in variable space which have no expected signal to demonstrate the understanding of the background events above which one is looking for a signal. The 'mass sidebands' (mass regions away from the tau mass) are the best candidate to do this in a search like tau -> 3 mu .
false positive results ('fake discoveries') are much worse for reputation and credibility than false negative results ('missed discoveries'). You may have heard of the 'faster than light neutrino anomaly' but an example (where a mass is reconstructed like in this competition) is this one: http://www.science20.com/quantum_diaries_survivor/death_dijet_anomaly-130783 . A subtle effect on the correction (calibration) of the energies of particles (similar to the p{0,1,2}_p variables) recorded in the detector caused excitement (3.2 standard deviations above background).
I hope this clarifies it a bit.
", "votes": null, "replies": [] }, { "id": 95981, "author_name": "chalmi", "author_url": "", "post_date": "10/13/2015 11:34:40", "content": "Yes mass is fundamental. But here we are talking about quite a different thing. Mass calculated as a feature from pt, p{0,1,2}_p and p{0,1,2}_pt variables. We already know that this feature sometimes differs from real mass because of some unrealistic assumptions in calculations. In fact we may forget this is a mass. This is just a feature M. But as a feature it is proved to be useful in a model (at least on the provided data set). So is there any practical reason for not using this M feature in model for this, as I understand, first line filter? (And of course using in model doesn't mean relaying solely on M). As I described my solution is using a linear combination on some exponential function from M, exp((M - 1777)^2/base) - with a very small factor like 0.1 and other classifier, which is based on some combination of features that try to hide absolute values - to be mass uncorrelated. Is there any particular reason of not using such linear combination in this pre-filter and using only the part independent on feature M? I got the part of false alarms - but there is still a hope that other features will "brake" this alarm (after all the cvm of final classifier with mass anyway is less than 0.002).
\n\nAs I understand we are talking only about pre-filtering, nobody will claim discovery based on this classifier results, but only will search more deeply in the selected results as I understand.
\n\nBTW, could you or admins provide some sample of data where mass is really faking predictions? That would be good to see how the designed classifier will behave in such case.
\n\nAnd this part is interesting:
\n\n\n\n\nOne way is use regions in variable space which have no expected signal to demonstrate the understanding of the background events above which one is looking for a signal. The 'mass sidebands' (mass regions away from the tau mass) are the best candidate to do this in a search like tau -> 3 mu .
\n
Can you explain why is that? I mean it is rather counter-intuitive.
", "votes": null, "replies": [] }, { "id": 95996, "author_name": "phunter", "author_url": "", "post_date": "10/13/2015 15:49:04", "content": "I had similar calculation for the 'pseudo' transverse invariant mass and it helped about +0.005 AUC, but I realized it could fail the CvM test. I think CvM test is for preventing the bias to the invariant mass and the model needs to be more generalized, because HEP has look-elsewhere effect that the full mass spectrum should be watched https://en.wikipedia.org/wiki/Look-elsewhere_effect thus a bias to mass peak may not be physical sound and may not help in the real life. The same story happened during the Higgs boson search where some team found a peak at 115 GeV and it turned out a statistical fluctuation and a artificial mass peak fit, so introducing a bias on invariant mass is not a good idea. So I chose not to play with CvM test, and left these mass features alone.
\n\nHowever, the trick for passing CvM test was good: I used the same trick for passing KS test. I had models with SPDhits which couldn't pass KS test for sure because SPDhits were not well simulated and needed calibration, but other models like RF were very OK with no SPDhits, so ensemble these models under the constraint of KS score can maximize the AUC. It becomes an optimization problem.
\n\n[quote=Liucheng Wang;95950]
\n\nIn principle, we need another angle information of each event to fully reconstruct the mass. Unfortunately it is missing in the dataset. Maybe the CERN host hides this angle data on purpose. Instead, I try to use pt to effectively get the mother particle mass as follows
\n\nmu=105.7 # muon mass
\n\ndef reconstructed_mass_by_pt(df):
\n\n list=[sqrt(df['p0_pt']*df['p0_pt']+mu*mu), \n\n sqrt(df['p1_pt']*df['p1_pt']+mu*mu),\n\n sqrt(df['p2_pt']*df['p2_pt']+mu*mu) ]\n\n return sqrt(abs(sum(list)*sum(list)-df['pt']*df['pt']))\nSome of the difference values between sum(list)*sum(list) and df['pt']*df['pt'] are negative due to the fluctuation. That is why I choose to use the absolute values. This feature of the effective mass turns out to be very useful to distinguish signals and backgrounds (see figure in the attachment). This mass feature will result in the fail of passing the test. In order to pass the Cramer-van-Mises test, another stable classifier is required to smooth the final result. For example, xgboost (with the mass feature) + rf (without the mass feature) is a good choice.
\n\n[/quote]
", "votes": null, "replies": [] }, { "id": 96245, "author_name": "holzner", "author_url": "", "post_date": "10/15/2015 08:34:05", "content": "(disclaimer: I'm not a competition admin)
\n\n[quote=Michal Wojcik;95981]\nAnd this part is interesting:
\n\n\n\n\nOne way is use regions in variable space which have no expected signal to demonstrate the understanding of the background events above which one is looking for a signal. The 'mass sidebands' (mass regions away from the tau mass) are the best candidate to do this in a search like tau -> 3 mu .
\n
Can you explain why is that? I mean it is rather counter-intuitive.
\n\n[/quote]
\n\nMass is rather used as a 'last line' filter. It is very powerful indeed but one has to demonstrate that the probability of selecting events (rows in the dataset) is independent of mass.
\n\nIn the end, one runs a hypothesis test of
\n\nvs.
\n\nOften a likelihood ratio is used. For a simple 'number of events (collisions) with a classifier output greater than a given threshold' this is:
\n\nQ = Probability(number of observed events | expected number of signal + background events) / Probability(number of observed events | expected number of background events)
\n\nHow to obtain the number of expected background events from data ?
\n\nWhat is done in searches like these that one assumes e.g. everything with a mass below 1757 MeV and everything with a mass above 1798 is background (there is no signal there). One needs however to know also the background from 1757 to 1798 MeV. This is done by interpolating between these two mass points. The obvious choice is to assume that the background follows a flat distribution between m = 1757 and 1798 MeV. The CvM test was meant to ensure this.
\n\nIf the classifier is not independent of mass for background events, this assumption is not valid any more and one can't estimate the number of background events this way.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "95932": "I am not sure if this was intended by organizer but features included almost complete information about mass of mother particle. Using elementary equations of special theory of relativity I was able to get mass with absolute mean error ~7 and using estimators reduced this further to ~6. (Anyone get better estimations?) As we all know mass was strongly correlated with signal. Of course organizers wanted something, as I understand, beyond mass. But I don't understand why they excluded mass from test data if this was possible to calculate almost exactly anyway. In my opinion mass should be included in features and there should be no correlation test. This is true that mass could be confusing, but this was only one feature. And classifier based only on mass was far worse than good classifier based on mass and all other features. So the goal of this competition to find something which is better than mass would have been achieved anyway. To win you must have anyway invented something much better than mass alone. I used calculated mass as a feature but combined with other classifier that used features that were as much unrelated with mass as possible - that improved results but still allowed to pass correlation test, which I will argue was only unnecessary obstacle in this task (taking into account that organizer wanted to escape from mass!).\r\n\r\nAnd one more remark. Assuming mass is confusing we should have also data that proves this. And this data should be used in training and test. In the presented data mass wasn't confusing at all, it was the strongest feature.\r\n\r\nI wonder if winner also used calculated mass? Please write here your thoughts about usage of mass in this competition.", "95948": "I was able to calculate the mass to within +/-1 % of the given mass value for 95% of the training dataset from relativistic kinematic (see https://www.kaggle.com/c/flavours-of-physics/forums/t/16783/quick-reference-of-hep-kinematics-calculation ). This assumed the muon mass for all three decay products which may not be valid in all cases. But then I failed / did not spend enough effort on figuring out how to pass the Cramer-van-Mises test. But at least it was interesting to see the mass distributions in the 'agreement test' sample (which was at masses higher than the training set) and the correlation test sample (which is nicely covering the signal region and sidebands).\r\n\r\nTraditionally, the mass variable is a very important one in the search for new phenomena in particle physics. If one can show that an excess over known backgrounds are concentrated at a certain mass (i.e. produced by a new, yet undiscovered unknown particle), this adds a lot of credibility to the result.\r\n\r\nNow what one wants to avoid is to 'shape the background' by e.g. preferentially selecting events in one mass region (for example around the tau mass 1777 MeV/c^2) which then form a peak, faking evidence for a new physics phenomenon. You could e.g. easily train a classifier based on the mass only, apply it to a distribution which is flat in mass (e.g. background as recorded with the detector) and you should get a peak in the mass spectrum without the presence of any new particle.", "95950": "In principle, we need another angle information of each event to fully reconstruct the mass. Unfortunately it is missing in the dataset. Maybe the CERN host hides this angle data on purpose. Instead, I try to use pt to effectively get the mother particle mass as follows\r\n\r\nmu=105.7 # muon mass\r\n\r\n\r\ndef reconstructed_mass_by_pt(df):\r\n\r\n list=[sqrt(df['p0_pt']*df['p0_pt']+mu*mu), \r\n\r\n sqrt(df['p1_pt']*df['p1_pt']+mu*mu),\r\n\r\n sqrt(df['p2_pt']*df['p2_pt']+mu*mu) ]\r\n\r\n return sqrt(abs(sum(list)*sum(list)-df['pt']*df['pt']))\r\n \r\nSome of the difference values between sum(list)*sum(list) and df['pt']*df['pt'] are negative due to the fluctuation. That is why I choose to use the absolute values. This feature of the effective mass turns out to be very useful to distinguish signals and backgrounds (see figure in the attachment). This mass feature will result in the fail of passing the test. In order to pass the Cramer-van-Mises test, another stable classifier is required to smooth the final result. For example, xgboost (with the mass feature) + rf (without the mass feature) is a good choice.", "95954": "Hm ... I got this in a bit longer way, but probably equivalent.\r\n\r\n\r\ntotal_pz = sqrt(p0_p * p0_p - p0_pt * p0_pt) + sqrt(p0_p * p0_p - p1_pt * p1_pt) + sqrt(p0_p * p0_p - p2_pt * p2_pt)\r\np = sqrt(pt*pt + total_pz*total_pz)\r\nE0 = sqrt(p0_p*p0_p + 11164.0356)\r\nE1 = sqrt(p1_p*p1_p + 11164.0356)\r\nE2 = sqrt(p2_p*p2_p + 11164.0356)\r\nE = E0 + E1 + E2 \r\nm = sqrt(E*E - p*p)\r\n\r\n\r\nAnd of course I also noticed that data for agreement test could be easily filtered by mass. There was a temptation to use it ;-) But I choose to develop physically sound classifier - and filtering agreement test data would result in overfit by using some aspects of features that were related to simulation v. reality recognition. And as such this would have no physial meaning at all. Instead I developed second classifier based on features that don't distinguish simulated decays from real decays (and also where not corralled too much with mass) and joined this with the classifier that was based sollely on mass. Best results that I got in this method was 0.999225 and it passed both tests without any tricks. The only thing is I needed to choose a small coefficient alpha in final prediction \r\n
\r\nY = alpha * [prediction based on mass] + (1 - alpha) * [prediction based on special set of features] \r\n\r\nto avoid high score on Cramer-van-Mises test - but I think this is perfectly valid.", "95955": "[quote=Andre Holzner;95948]\r\n\r\nNow what one wants to avoid is to 'shape the background' by e.g. preferentially selecting events in one mass region (for example around the tau mass 1777 MeV/c^2) which then form a peak, faking evidence for a new physics phenomenon. You could e.g. easily train a classifier based on the mass only, apply it to a distribution which is flat in mass (e.g. background as recorded with the detector) and you should get a peak in the mass spectrum without the presence of any new particle.\r\n\r\n[/quote]\r\n\r\nBut why this didn't happen in our data set? I mean in our data set mass was a very good predictor with very small number of FPs. If in reality you can get this calculated mass at level of around 1777 MeV/c^2 and still this is not desired decay, such data should be presented in this competition. Then it would be clear that you can not relay too much on mass. But anyway mass would be strong feature.", "95973": "In practice, the mass will be used in a likelihood ratio estimator on a subset of the observed data which pass a threshold on the classifier's output.\r\n\r\nI have uploaded a script here https://www.kaggle.com/holzner/flavours-of-physics/candidate-mass/ \r\nto illustrate the 'estimated' mass distributions. \r\n\r\nIf you plot the mass directly on the training set, you'll notice that there are no background events in the range 1757.. 1798 MeV, so from the background in the training dataset it's impossible to create a spurious peak at the tau mass but in data recorded with the detector for example you easily could.", "95976": "[quote=Andre Holzner;95973]\r\n\r\nIn practice, the mass will be used in a likelihood ratio estimator on a subset of the observed data which pass a threshold on the classifier's output.\r\n...\r\nIf you plot the mass directly on the training set, you'll notice that there are no background events in the range 1757.. 1798 MeV, so from the background in the training dataset it's impossible to create a spurious peak at the tau mass but in data recorded with the detector for example you easily could.\r\n\r\n[/quote]\r\n\r\nPerhaps I miss something, but I still don't understand why not to add such data recorded with detector to data set and then train this together with mass information. It would be probably better than doing this in 2 stages. Usually joining conditions in ML is better than treating them separately. Even on provided data set mass alone was not as good as mass + other features.", "95980": "[quote=Michal Wojcik;95976]\r\nPerhaps I miss something, but I still don't understand why not to add such data recorded with detector to data set and then train this together with mass information. It would be probably better than doing this in 2 stages. Usually joining conditions in ML is better than treating them separately. Even on provided data set mass alone was not as good as mass + other features.\r\n[/quote]\r\n\r\nI fully agree that a combined classifier (using all available information) is expected to perform better than splitting this into two stages.\r\n\r\nHowever, in practice:\r\n\r\n - mass *is* 'fundamental' :-) . Physics phenomena at these microscopic scales are modeled by the exchange ('interaction') of particles among particles. Each particle has a mass. Evidence for a new physics phenomena is best supported if one can quote a mass for it.\r\n\r\n - data recorded from the detector does not come from the same n-dimensional probability density as the simulation (on which one trains the classifiers, although I think in this competition at least the background was probably recorded detector data). Simulation of the known interactions ('background') is only possible with a limited precision, simulation of the interaction of the particles produced in the collision with the detector elements also can't be simulated perfectly and is computationally very expensive. Given the fact that one is typically looking for a few tens of collisions in a several billion recorded ones per year, one has to find ways to work with the limited precision of the simulation. One way is use regions in variable space which have no expected signal to demonstrate the understanding of the background events above which one is looking for a signal. The 'mass sidebands' (mass regions away from the tau mass) are the best candidate to do this in a search like tau -> 3 mu . \r\n\r\n - false positive results ('fake discoveries') are much worse for reputation and credibility than false negative results ('missed discoveries'). You may have heard of the 'faster than light neutrino anomaly' but an example (where a mass is reconstructed like in this competition) is this one: http://www.science20.com/quantum_diaries_survivor/death_dijet_anomaly-130783 . A subtle effect on the correction (calibration) of the energies of particles (similar to the p{0,1,2}_p variables) recorded in the detector caused excitement (3.2 standard deviations above background). \r\n\r\nI hope this clarifies it a bit.", "95981": "Yes mass is fundamental. But here we are talking about quite a different thing. Mass calculated as a feature from pt, p{0,1,2}_p and p{0,1,2}_pt variables. We already know that this feature sometimes differs from real mass because of some unrealistic assumptions in calculations. In fact we may forget this is a mass. This is just a feature M. But as a feature it is proved to be useful in a model (at least on the provided data set). So is there any practical reason for not using this M feature in model for this, as I understand, first line filter? (And of course using in model doesn't mean relaying solely on M). As I described my solution is using a linear combination on some exponential function from M, exp((M - 1777)^2/base) - with a very small factor like 0.1 and other classifier, which is based on some combination of features that try to hide absolute values - to be mass uncorrelated. Is there any particular reason of not using such linear combination in this pre-filter and using only the part independent on feature M? I got the part of false alarms - but there is still a hope that other features will \"brake\" this alarm (after all the cvm of final classifier with mass anyway is less than 0.002). \r\n\r\nAs I understand we are talking only about pre-filtering, nobody will claim discovery based on this classifier results, but only will search more deeply in the selected results as I understand. \r\n\r\nBTW, could you or admins provide some sample of data where mass is really faking predictions? That would be good to see how the designed classifier will behave in such case. \r\n\r\nAnd this part is interesting:\r\n\r\n> One way is use regions in variable space which have no expected signal to demonstrate the understanding of the background events above which one is looking for a signal. The 'mass sidebands' (mass regions away from the tau mass) are the best candidate to do this in a search like tau -> 3 mu .\r\n\r\nCan you explain why is that? I mean it is rather counter-intuitive.", "95996": "I had similar calculation for the 'pseudo' transverse invariant mass and it helped about +0.005 AUC, but I realized it could fail the CvM test. I think CvM test is for preventing the bias to the invariant mass and the model needs to be more generalized, because HEP has look-elsewhere effect that the full mass spectrum should be watched https://en.wikipedia.org/wiki/Look-elsewhere_effect thus a bias to mass peak may not be physical sound and may not help in the real life. The same story happened during the Higgs boson search where some team found a peak at 115 GeV and it turned out a statistical fluctuation and a artificial mass peak fit, so introducing a bias on invariant mass is not a good idea. So I chose not to play with CvM test, and left these mass features alone.\r\n\r\nHowever, the trick for passing CvM test was good: I used the same trick for passing KS test. I had models with SPDhits which couldn't pass KS test for sure because SPDhits were not well simulated and needed calibration, but other models like RF were very OK with no SPDhits, so ensemble these models under the constraint of KS score can maximize the AUC. It becomes an optimization problem.\r\n\r\n[quote=Liucheng Wang;95950]\r\n\r\nIn principle, we need another angle information of each event to fully reconstruct the mass. Unfortunately it is missing in the dataset. Maybe the CERN host hides this angle data on purpose. Instead, I try to use pt to effectively get the mother particle mass as follows\r\n\r\nmu=105.7 # muon mass\r\n\r\n\r\ndef reconstructed_mass_by_pt(df):\r\n\r\n list=[sqrt(df['p0_pt']*df['p0_pt']+mu*mu), \r\n\r\n sqrt(df['p1_pt']*df['p1_pt']+mu*mu),\r\n\r\n sqrt(df['p2_pt']*df['p2_pt']+mu*mu) ]\r\n\r\n return sqrt(abs(sum(list)*sum(list)-df['pt']*df['pt']))\r\n \r\nSome of the difference values between sum(list)*sum(list) and df['pt']*df['pt'] are negative due to the fluctuation. That is why I choose to use the absolute values. This feature of the effective mass turns out to be very useful to distinguish signals and backgrounds (see figure in the attachment). This mass feature will result in the fail of passing the test. In order to pass the Cramer-van-Mises test, another stable classifier is required to smooth the final result. For example, xgboost (with the mass feature) + rf (without the mass feature) is a good choice.\r\n\r\n\r\n[/quote]", "96245": "(disclaimer: I'm not a competition admin)\r\n\r\n[quote=Michal Wojcik;95981]\r\nAnd this part is interesting:\r\n\r\n> One way is use regions in variable space which have no expected signal to demonstrate the understanding of the background events above which one is looking for a signal. The 'mass sidebands' (mass regions away from the tau mass) are the best candidate to do this in a search like tau -> 3 mu .\r\n\r\nCan you explain why is that? I mean it is rather counter-intuitive.\r\n\r\n[/quote]\r\n\r\nMass is rather used as a 'last line' filter. It is very powerful indeed but one has to demonstrate that the probability of selecting events (rows in the dataset) is independent of mass. \r\n\r\nIn the end, one runs a hypothesis test of\r\n\r\n - 'background only present in recorded data' \r\n\r\nvs. \r\n\r\n - 'signal + background present in recorded data'\r\n\r\nOften a likelihood ratio is used. For a simple 'number of events (collisions) with a classifier output greater than a given threshold' this is:\r\n\r\n Q = Probability(number of observed events | expected number of signal + background events) / Probability(number of observed events | expected number of background events)\r\n\r\n - the number of observed (recorded with the detector) events passing the classifier threshold is obvious to obtain\r\n - the number of expected signal events passing the classifier threshold is obtained from simulation\r\n - the important point is the number of background events: it could also be obtained from simulation but this has large uncertainties (limited statistics and imperfect modelling) so in practice it is also obtained from recorded data.\r\n\r\nHow to obtain the number of expected background events from data ? \r\n\r\nWhat is done in searches like these that one assumes e.g. everything with a mass below 1757 MeV and everything with a mass above 1798 is background (there is no signal there). One needs however to know also the background from 1757 to 1798 MeV. This is done by interpolating between these two mass points. The obvious choice is to assume that the background follows a flat distribution between m = 1757 and 1798 MeV. The CvM test was meant to ensure this. \r\n\r\nIf the classifier is not independent of mass for background events, this assumption is not valid any more and one can't estimate the number of background events this way." }, "source": "meta" }