I watched the introduction video and read the documentation, but still I do not understand this. I can calculate the test locally, however it is still a black-box to me.
\n\nIf its possible I would like a bit more information such as:
\n\nIf my submission failed the test, it is because I did ______________________ or because I did not _______________________
\n\nAs now I am just trying random things until it passes the test.
", "messages": [ { "id": "86436", "postDate": "07/22/2015 08:05:37", "content": "I watched the introduction video and read the documentation, but still I do not understand this. I can calculate the test locally, however it is still a black-box to me.
\n\nIf its possible I would like a bit more information such as:
\n\nIf my submission failed the test, it is because I did ______________________ or because I did not _______________________
\n\nAs now I am just trying random things until it passes the test.
", "rawMarkdown": "I watched the introduction video and read the documentation, but still I do not understand this. I can calculate the test locally, however it is still a black-box to me.\r\n\r\nIf its possible I would like a bit more information such as:\r\n\r\nIf my submission failed the test, it is because I did __________________________ or because I did not ___________________________\r\n\r\nAs now I am just trying random things until it passes the test.", "votes": null }, { "id": "86451", "postDate": "07/22/2015 09:59:30", "content": "I think the idea of the test (which I too find difficult to satisfy!) is that we should only be modelling effects that occur in real life, not just the simulated data.
\n\nUnfortunately, there is no real life data that we can look at for this event! The closest we have is real life data from a similar event, together with simulated data. We are hoping that if our predictions are not sensitive to whether the data were simulated or real for the control data, then it won't be for the real data either.
\n\nTo answer your actual question, if your submission failed the test, it is because your features and/or modelling technique were picking up inaccuracies in the simulation rather than genuine signal. Finding ways not to do this is probably the hardest part of the competition, but you have the control dataset to help you.
\n\nI expect to face a similar problem for the mass test as well.
", "rawMarkdown": "I think the idea of the test (which I too find difficult to satisfy!) is that we should only be modelling effects that occur in real life, not just the simulated data.\r\n\r\nUnfortunately, there *is* no real life data that we can look at for this event! The closest we have is real life data from a similar event, together with simulated data. We are hoping that if our predictions are not sensitive to whether the data were simulated or real for the control data, then it won't be for the real data either.\r\n\r\nTo answer your actual question, if your submission failed the test, it is because your features and/or modelling technique were picking up inaccuracies in the simulation rather than genuine signal. Finding ways not to do this is probably the hardest part of the competition, but you have the control dataset to help you.\r\n\r\nI expect to face a similar problem for the mass test as well.", "votes": null }, { "id": "86485", "postDate": "07/22/2015 15:45:41", "content": "Got it, thx, I have not faced problems with the correlation test yet, well.... it seems that is time to fight the agreement test.
", "rawMarkdown": "Got it, thx, I have not faced problems with the correlation test yet, well.... it seems that is time to fight the agreement test.", "votes": null }, { "id": "86580", "postDate": "07/23/2015 07:49:52", "content": "The "agreement test" makes the whole challenge rather nonsensical imho. Such a constraint makes the challenge depart from "machine learning" into an entirely new discipline. Maybe we could call it "machine learning that is allowed to learn some correlations but not some others". One real-world application beyond physics would be "politically correct machine learning", by the way.
\n\nThe only way we can bring this challenge back into the realm of Productive Work would to find a way to clean up the data ourselves, i.e. transform it in a way that erases simulation patterns (which the organizers should have done beforehand, obviously...). Removing the field 'SPDhits' is one first step, but it appears not to be sufficient.
", "rawMarkdown": "The \"agreement test\" makes the whole challenge rather nonsensical imho. Such a constraint makes the challenge depart from \"machine learning\" into an entirely new discipline. Maybe we could call it \"machine learning that is allowed to learn some correlations but not some others\". One real-world application beyond physics would be \"politically correct machine learning\", by the way.\r\n\r\nThe only way we can bring this challenge back into the realm of Productive Work would to find a way to clean up the data ourselves, i.e. transform it in a way that erases simulation patterns (which the organizers should have done beforehand, obviously...). Removing the field 'SPDhits' is one first step, but it appears not to be sufficient.", "votes": null }, { "id": "86585", "postDate": "07/23/2015 08:43:06", "content": "Hi there.
\n\nRegarding the comment "which the organizers should have done beforehand, obviously...":
\n\nThis is a genuine issue in high energy physics. The simulation of high-energy particle collisions and the simulation of detector response to those collisions is extremely complicated and relies on many approximations where the underlying physical processes are not yet well measured. Correcting the simulation to precisely match the data is a very time consuming and non-trivial task, as making one field agree often makes others disagree.
\n\nIn a pure machine learning competition the goal may be to maximise the performance metric on the hidden dataset, but ultimately this isn't a useful metric in a physics analysis, particularly in which the decay we are looking for is not yet known to exist. In a typical analysis of this kind in HEP, the vast majority of the time is not spent on training a machine learning algorithm, it is spent determining if the algorithm you have trained behaves as it should on the real data.
\n\nIn this instance the SPDhits variable is known not to have precisely the same shape in data and simulation, but SPDhits should be the same between the control data and the tau->3mu signal, hence, if your algorithm uses SPDhits it needs to do so in a way that is performant on the control data. Personally, I'm very interested in seeing if anyone comes up with a way to use this variable without failing the test.....
", "rawMarkdown": "Hi there. \r\n\r\nRegarding the comment \"which the organizers should have done beforehand, obviously...\": \r\n\r\nThis is a genuine issue in high energy physics. The simulation of high-energy particle collisions and the simulation of detector response to those collisions is extremely complicated and relies on many approximations where the underlying physical processes are not yet well measured. Correcting the simulation to precisely match the data is a very time consuming and non-trivial task, as making one field agree often makes others disagree. \r\n\r\nIn a pure machine learning competition the goal may be to maximise the performance metric on the hidden dataset, but ultimately this isn't a useful metric in a physics analysis, particularly in which the decay we are looking for is not yet known to exist. In a typical analysis of this kind in HEP, the vast majority of the time is not spent on training a machine learning algorithm, it is spent determining if the algorithm you have trained behaves as it should on the real data. \r\n\r\nIn this instance the SPDhits variable is known not to have precisely the same shape in data and simulation, but SPDhits should be the same between the control data and the tau->3mu signal, hence, if your algorithm uses SPDhits it needs to do so in a way that is performant on the control data. Personally, I'm very interested in seeing if anyone comes up with a way to use this variable without failing the test.....", "votes": null }, { "id": "88803", "postDate": "08/07/2015 11:14:33", "content": "Hi,
\n\nindeed, running with only check_agreement.csv, I found that most of variables do not have consistent** distributions between data (asking signal=1, applying weights) and MC (asking signal=0, applying weight actually they are 1). Do you guys see large disagreements there? thanks!
\n\n** consistent criteria: scipy.stats.ks_2samp( data, mc ) < 0.09
", "rawMarkdown": "Hi,\r\n\r\nindeed, running with only check_agreement.csv, I found that most of variables do not have consistent** distributions between data (asking signal=1, applying weights) and MC (asking signal=0, applying weight actually they are 1). Do you guys see large disagreements there? thanks!\r\n\r\n** consistent criteria: scipy.stats.ks_2samp( data, mc ) < 0.09", "votes": null }, { "id": "89042", "postDate": "08/10/2015 17:09:00", "content": "Hi, Xiaohu! Notes about your checking:
\n\nHi Tatiana,
\n\nThanks for the reminding. Indeed, weights were missing in the ks calculation. After introducing them, most of variables have reasonable MC/Data agreement except a few.
", "rawMarkdown": "Hi Tatiana,\r\n\r\nThanks for the reminding. Indeed, weights were missing in the ks calculation. After introducing them, most of variables have reasonable MC/Data agreement except a few.", "votes": null } ], "comments": [ { "id": 86451, "author_name": "jamesdlawrence", "author_url": "", "post_date": "07/22/2015 09:59:30", "content": "I think the idea of the test (which I too find difficult to satisfy!) is that we should only be modelling effects that occur in real life, not just the simulated data.
\n\nUnfortunately, there is no real life data that we can look at for this event! The closest we have is real life data from a similar event, together with simulated data. We are hoping that if our predictions are not sensitive to whether the data were simulated or real for the control data, then it won't be for the real data either.
\n\nTo answer your actual question, if your submission failed the test, it is because your features and/or modelling technique were picking up inaccuracies in the simulation rather than genuine signal. Finding ways not to do this is probably the hardest part of the competition, but you have the control dataset to help you.
\n\nI expect to face a similar problem for the mass test as well.
", "votes": null, "replies": [] }, { "id": 86485, "author_name": "carloshuertas", "author_url": "", "post_date": "07/22/2015 15:45:41", "content": "Got it, thx, I have not faced problems with the correlation test yet, well.... it seems that is time to fight the agreement test.
", "votes": null, "replies": [] }, { "id": 86580, "author_name": "fchollet", "author_url": "", "post_date": "07/23/2015 07:49:52", "content": "The "agreement test" makes the whole challenge rather nonsensical imho. Such a constraint makes the challenge depart from "machine learning" into an entirely new discipline. Maybe we could call it "machine learning that is allowed to learn some correlations but not some others". One real-world application beyond physics would be "politically correct machine learning", by the way.
\n\nThe only way we can bring this challenge back into the realm of Productive Work would to find a way to clean up the data ourselves, i.e. transform it in a way that erases simulation patterns (which the organizers should have done beforehand, obviously...). Removing the field 'SPDhits' is one first step, but it appears not to be sufficient.
", "votes": null, "replies": [] }, { "id": 86585, "author_name": "conorfitzpatrick", "author_url": "", "post_date": "07/23/2015 08:43:06", "content": "Hi there.
\n\nRegarding the comment "which the organizers should have done beforehand, obviously...":
\n\nThis is a genuine issue in high energy physics. The simulation of high-energy particle collisions and the simulation of detector response to those collisions is extremely complicated and relies on many approximations where the underlying physical processes are not yet well measured. Correcting the simulation to precisely match the data is a very time consuming and non-trivial task, as making one field agree often makes others disagree.
\n\nIn a pure machine learning competition the goal may be to maximise the performance metric on the hidden dataset, but ultimately this isn't a useful metric in a physics analysis, particularly in which the decay we are looking for is not yet known to exist. In a typical analysis of this kind in HEP, the vast majority of the time is not spent on training a machine learning algorithm, it is spent determining if the algorithm you have trained behaves as it should on the real data.
\n\nIn this instance the SPDhits variable is known not to have precisely the same shape in data and simulation, but SPDhits should be the same between the control data and the tau->3mu signal, hence, if your algorithm uses SPDhits it needs to do so in a way that is performant on the control data. Personally, I'm very interested in seeing if anyone comes up with a way to use this variable without failing the test.....
", "votes": null, "replies": [] }, { "id": 88803, "author_name": "xiaohu", "author_url": "", "post_date": "08/07/2015 11:14:33", "content": "Hi,
\n\nindeed, running with only check_agreement.csv, I found that most of variables do not have consistent** distributions between data (asking signal=1, applying weights) and MC (asking signal=0, applying weight actually they are 1). Do you guys see large disagreements there? thanks!
\n\n** consistent criteria: scipy.stats.ks_2samp( data, mc ) < 0.09
", "votes": null, "replies": [] }, { "id": 89042, "author_name": "tataantares", "author_url": "", "post_date": "08/10/2015 17:09:00", "content": "Hi, Xiaohu! Notes about your checking:
\n\nHi Tatiana,
\n\nThanks for the reminding. Indeed, weights were missing in the ks calculation. After introducing them, most of variables have reasonable MC/Data agreement except a few.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "86436": "I watched the introduction video and read the documentation, but still I do not understand this. I can calculate the test locally, however it is still a black-box to me.\r\n\r\nIf its possible I would like a bit more information such as:\r\n\r\nIf my submission failed the test, it is because I did __________________________ or because I did not ___________________________\r\n\r\nAs now I am just trying random things until it passes the test.", "86451": "I think the idea of the test (which I too find difficult to satisfy!) is that we should only be modelling effects that occur in real life, not just the simulated data.\r\n\r\nUnfortunately, there *is* no real life data that we can look at for this event! The closest we have is real life data from a similar event, together with simulated data. We are hoping that if our predictions are not sensitive to whether the data were simulated or real for the control data, then it won't be for the real data either.\r\n\r\nTo answer your actual question, if your submission failed the test, it is because your features and/or modelling technique were picking up inaccuracies in the simulation rather than genuine signal. Finding ways not to do this is probably the hardest part of the competition, but you have the control dataset to help you.\r\n\r\nI expect to face a similar problem for the mass test as well.", "86485": "Got it, thx, I have not faced problems with the correlation test yet, well.... it seems that is time to fight the agreement test.", "86580": "The \"agreement test\" makes the whole challenge rather nonsensical imho. Such a constraint makes the challenge depart from \"machine learning\" into an entirely new discipline. Maybe we could call it \"machine learning that is allowed to learn some correlations but not some others\". One real-world application beyond physics would be \"politically correct machine learning\", by the way.\r\n\r\nThe only way we can bring this challenge back into the realm of Productive Work would to find a way to clean up the data ourselves, i.e. transform it in a way that erases simulation patterns (which the organizers should have done beforehand, obviously...). Removing the field 'SPDhits' is one first step, but it appears not to be sufficient.", "86585": "Hi there. \r\n\r\nRegarding the comment \"which the organizers should have done beforehand, obviously...\": \r\n\r\nThis is a genuine issue in high energy physics. The simulation of high-energy particle collisions and the simulation of detector response to those collisions is extremely complicated and relies on many approximations where the underlying physical processes are not yet well measured. Correcting the simulation to precisely match the data is a very time consuming and non-trivial task, as making one field agree often makes others disagree. \r\n\r\nIn a pure machine learning competition the goal may be to maximise the performance metric on the hidden dataset, but ultimately this isn't a useful metric in a physics analysis, particularly in which the decay we are looking for is not yet known to exist. In a typical analysis of this kind in HEP, the vast majority of the time is not spent on training a machine learning algorithm, it is spent determining if the algorithm you have trained behaves as it should on the real data. \r\n\r\nIn this instance the SPDhits variable is known not to have precisely the same shape in data and simulation, but SPDhits should be the same between the control data and the tau->3mu signal, hence, if your algorithm uses SPDhits it needs to do so in a way that is performant on the control data. Personally, I'm very interested in seeing if anyone comes up with a way to use this variable without failing the test.....", "88803": "Hi,\r\n\r\nindeed, running with only check_agreement.csv, I found that most of variables do not have consistent** distributions between data (asking signal=1, applying weights) and MC (asking signal=0, applying weight actually they are 1). Do you guys see large disagreements there? thanks!\r\n\r\n** consistent criteria: scipy.stats.ks_2samp( data, mc ) < 0.09", "89042": "Hi, Xiaohu! Notes about your checking:\r\n\r\n* data has label \"0\" and arbitrary weights (they are positive and negative!) and Monte Carlo has label \"1\" and all weights equal to 1.\r\n* you should calculate KS using the weights (they play a curtail role), otherwise computed KS-value will be wrong (as it is now).", "89052": "Hi Tatiana,\r\n\r\nThanks for the reminding. Indeed, weights were missing in the ks calculation. After introducing them, most of variables have reasonable MC/Data agreement except a few." }, "source": "meta" }