One of the most common machine learning problems is to find the best predictive model using training data that gives the best predictions on the test data. In the High Energy Physics (HEP) data analysis a machine learning problem is usually formulated a bit differently (it requires a bit of sophistication for both the problem statement and the restrictions). There are several reasons for that.
\n\nThe main reason is that in HEP you verify one of two hypotheses (H_0: decay happens, or H_1: it doesn't) and machine learning is just a step used only for data selection.
\n\nImagine we want to train a classifier to differentiate between healthy and infected students. For training we use two samples: infected students from the previous year (as 'signal') and healthy people around the city (as 'background') — we take people outside of the school to avoid possible contamination of our training samples.
\n\nFrom the common machine learning perspective you train a classifier to predict if a particular student is infected or not. From the HEP analysis perspective you are interested in the amount of infected students (say, if there are significantly larger amount of infected students than it is allowed, the school should be quarantined). So physicists use classifiers' output as a discrimination variable between the signal (infected person) and the background (healthy one) and select the data with the output greater than certain threshold. Then they estimate the expected amount of healthy students in the selected data and compare it with the actual amount of students within the selection ("observed" amount).
\n\nFor instance, you expected 7-10 healthy persons and observed 20. In this case you claim that the data contains a number of infected people because observed number of "healthy" people is significantly larger than you expect and the school should be quarantined (or you can claim a discovery in physics term). To choose the best classifier we use weighted area under ROC curve metric.
The second reason is that you have to check for the absence of the correlation between the mass and the classifier's predictions. In our example, feature
\n\nF = "distance of student's home from the school X",
is analogous to the event mass. If our classifier is correlated with F it might mark all persons from the school X as 'infected' and healthy for others (since in our training sample all infected people are coming from the school X and all healthy people outside of it). But we cannot claim quarantine based on this selection, since it is obviously biased. To avoid such an error we introduce the correlation test.
We cannot add 'signal' labels to real data events, so it cannot be used for training of a classifier. Background events for the training set are taken outside of the signal mass region, because our signal decay has a specific mass range and events outside of this region for sure contain no signal. Signal events can be only simulated. A simulation is not perfect because we cannot simulate all possible processes and we use some approximations which lead to non-perfect signal data.
\n\nSpeaking in terms of infected/healthy students, the information about the infected students we take from the previous year records (analogous to simulated data) and the information about the healthy ones - outside of the school X (analogues to real events outside of the signal mass region).
\n\nHere is a list to summarize types of data and its origin:
\n\nsignal:
\n\nbackground:
\n\nWe train a classifier on the simulated signal data versus the background real data (it can also be simulated, but that's a different story). To claim a discovery we want to be sure that our selection preserves as many as possible signal events, thus for each False Positive Rate (FPR) our True Positive Rate (TPR) is the highest possible. The classifier can pick not only the difference between the signal and the background, but between the simulated and the real data as well (or not only between infected and healthy people but between this and the previous year data). If this happens TPR for the fixed FPR will be overestimated.
\n\nAs a result, we need to apply calibration procedure to estimate TPR of a classifier for the real signal data. One of the ways to calibrate TPR values (or just check if it is needed indeed) is to see the classifier's performance on the data that is similar to our decay, but is very well known. It is called 'control channel', which happens frequently and it is close by its physical properties to the signal decay.
\n\nA control channel is an analogous to the data about students from another (much bigger) city.
\n\ncontrol signal:
\n\ncontrol background:
\n\nFor the calibration procedure physicists assume that the TPR_{real data} / TPR_{MC} will be the same in the control and the signal channels. Thus, for a given FPR:
\n\nTPR_{real data}(signal) = TPR_{real data}(control) / TPR_{MC}(control) * TPR_{MC}(signal) (1) — calibration equation.
\n\nSince some features are not well-simulated, this ratio:
\n\nTPR_{real data}(control) / TPR_{MC}(control)
\n\ncould be very small, as a result your true TPR for the real signal will be also very small.
\n\nAs a solution for the problem physicists do reweighting of data to make MC and real data consistent. If reweighting is applied only to the control channel, it will violate the equality (1), so the same reweighting should be applied to the signal channel to restore (1). We don't cover reweighting in details here, but essentially it is a procedure to assign certain weights to some events so weighted distributions from one dataset would be similar to the distributions from another one.
\n\nIn the physical analysis you should choose the FPR points which depend on the hypotheses testing statistics. Thus you should calibrate the TPR specific points depending on your statistics. In the competition we would like to find a classifier which is optimal for any hypotheses testing statistics, so we want to avoid calibration procedure. That is why, KS test is implemented to make sure that
\n\nTPR_{real data}(control) / TPR_{MC}(control) ~ 1 (2)
\n\nIf this approximation is true it is fair to compare the different classifiers without any calibration.
Classifiers' systematic error. Physicists result always includes the systematic and statistic errors. To be precise we should rewrite equation (1):
\n\nTPR_{real data}(signal) = TPR_{real data}(control) / TPR_{MC}(control) * TPR_{MC}(signal) + systematic_error + statistic_error (3)
\n\nIn some cases, the systematic errors are negligible and we can compare models as is.\nBut in the real world the control channel MC and the signal channel MC are not completely similar — it is an example of the data systematic error, to be more precise, this is a systematic error of the physical difference in the nature between the channels but it can be reduced (this could be checked, for example, by comparing predictions PDF for MC samples in the control and the signal channels). \nDuring your model training you can introduce other kinds of the systematic errors which might break (1). This case would be impossible to check using the regular competition platform. So for the competition you should produce models that don't break (1) by adding new systematic errors.
Some examples:
\n\nWhen you simply train a classifier and perform feature selection or parameters optimization you might introduce some systematic error, but in the most cases it is negligible, for example, if you treat your signal and control channels equally.
When you train a classifier to distinguish the signal channel and the control channel, this model will dramatically increase a systematic error introducing huge difference between the signal channel and the control channel.
To sum up:
\n\nIn the HEP data analysis the main ML task is not to make prediction for each individual event, but to estimate total amount of interesting enough events within the given region.
In the HEP data analysis several restrictions on the classifier exist, which are rather complicated and require description of such aspects as mass correlation, MC/real agreement, control channel assumptions, calibration and all systematics estimations. The systematic error estimation cannot be calculated within the competition framework (and cannot be included as a part of the metric) since it is very dependent on the way you work with the data. So for the competition we simplified restrictions for the sake of understanding and performance.
Your solutions should not break the assumption (1) regarding the control and the signal channels. Otherwise an estimation of the classifier performance would require reweighting (or another kind of magic) which is also very dependent on the data preprocessing.
Tanya, Andrey, Marcin, Eugene
", "messages": [ { "id": "89569", "postDate": "08/17/2015 15:41:28", "content": "One of the most common machine learning problems is to find the best predictive model using training data that gives the best predictions on the test data. In the High Energy Physics (HEP) data analysis a machine learning problem is usually formulated a bit differently (it requires a bit of sophistication for both the problem statement and the restrictions). There are several reasons for that.
\n\nThe main reason is that in HEP you verify one of two hypotheses (H_0: decay happens, or H_1: it doesn't) and machine learning is just a step used only for data selection.
\n\nImagine we want to train a classifier to differentiate between healthy and infected students. For training we use two samples: infected students from the previous year (as 'signal') and healthy people around the city (as 'background') — we take people outside of the school to avoid possible contamination of our training samples.
\n\nFrom the common machine learning perspective you train a classifier to predict if a particular student is infected or not. From the HEP analysis perspective you are interested in the amount of infected students (say, if there are significantly larger amount of infected students than it is allowed, the school should be quarantined). So physicists use classifiers' output as a discrimination variable between the signal (infected person) and the background (healthy one) and select the data with the output greater than certain threshold. Then they estimate the expected amount of healthy students in the selected data and compare it with the actual amount of students within the selection ("observed" amount).
\n\nFor instance, you expected 7-10 healthy persons and observed 20. In this case you claim that the data contains a number of infected people because observed number of "healthy" people is significantly larger than you expect and the school should be quarantined (or you can claim a discovery in physics term). To choose the best classifier we use weighted area under ROC curve metric.
The second reason is that you have to check for the absence of the correlation between the mass and the classifier's predictions. In our example, feature
\n\nF = "distance of student's home from the school X",
is analogous to the event mass. If our classifier is correlated with F it might mark all persons from the school X as 'infected' and healthy for others (since in our training sample all infected people are coming from the school X and all healthy people outside of it). But we cannot claim quarantine based on this selection, since it is obviously biased. To avoid such an error we introduce the correlation test.
We cannot add 'signal' labels to real data events, so it cannot be used for training of a classifier. Background events for the training set are taken outside of the signal mass region, because our signal decay has a specific mass range and events outside of this region for sure contain no signal. Signal events can be only simulated. A simulation is not perfect because we cannot simulate all possible processes and we use some approximations which lead to non-perfect signal data.
\n\nSpeaking in terms of infected/healthy students, the information about the infected students we take from the previous year records (analogous to simulated data) and the information about the healthy ones - outside of the school X (analogues to real events outside of the signal mass region).
\n\nHere is a list to summarize types of data and its origin:
\n\nsignal:
\n\nbackground:
\n\nWe train a classifier on the simulated signal data versus the background real data (it can also be simulated, but that's a different story). To claim a discovery we want to be sure that our selection preserves as many as possible signal events, thus for each False Positive Rate (FPR) our True Positive Rate (TPR) is the highest possible. The classifier can pick not only the difference between the signal and the background, but between the simulated and the real data as well (or not only between infected and healthy people but between this and the previous year data). If this happens TPR for the fixed FPR will be overestimated.
\n\nAs a result, we need to apply calibration procedure to estimate TPR of a classifier for the real signal data. One of the ways to calibrate TPR values (or just check if it is needed indeed) is to see the classifier's performance on the data that is similar to our decay, but is very well known. It is called 'control channel', which happens frequently and it is close by its physical properties to the signal decay.
\n\nA control channel is an analogous to the data about students from another (much bigger) city.
\n\ncontrol signal:
\n\ncontrol background:
\n\nFor the calibration procedure physicists assume that the TPR_{real data} / TPR_{MC} will be the same in the control and the signal channels. Thus, for a given FPR:
\n\nTPR_{real data}(signal) = TPR_{real data}(control) / TPR_{MC}(control) * TPR_{MC}(signal) (1) — calibration equation.
\n\nSince some features are not well-simulated, this ratio:
\n\nTPR_{real data}(control) / TPR_{MC}(control)
\n\ncould be very small, as a result your true TPR for the real signal will be also very small.
\n\nAs a solution for the problem physicists do reweighting of data to make MC and real data consistent. If reweighting is applied only to the control channel, it will violate the equality (1), so the same reweighting should be applied to the signal channel to restore (1). We don't cover reweighting in details here, but essentially it is a procedure to assign certain weights to some events so weighted distributions from one dataset would be similar to the distributions from another one.
\n\nIn the physical analysis you should choose the FPR points which depend on the hypotheses testing statistics. Thus you should calibrate the TPR specific points depending on your statistics. In the competition we would like to find a classifier which is optimal for any hypotheses testing statistics, so we want to avoid calibration procedure. That is why, KS test is implemented to make sure that
\n\nTPR_{real data}(control) / TPR_{MC}(control) ~ 1 (2)
\n\nIf this approximation is true it is fair to compare the different classifiers without any calibration.
Classifiers' systematic error. Physicists result always includes the systematic and statistic errors. To be precise we should rewrite equation (1):
\n\nTPR_{real data}(signal) = TPR_{real data}(control) / TPR_{MC}(control) * TPR_{MC}(signal) + systematic_error + statistic_error (3)
\n\nIn some cases, the systematic errors are negligible and we can compare models as is.\nBut in the real world the control channel MC and the signal channel MC are not completely similar — it is an example of the data systematic error, to be more precise, this is a systematic error of the physical difference in the nature between the channels but it can be reduced (this could be checked, for example, by comparing predictions PDF for MC samples in the control and the signal channels). \nDuring your model training you can introduce other kinds of the systematic errors which might break (1). This case would be impossible to check using the regular competition platform. So for the competition you should produce models that don't break (1) by adding new systematic errors.
Some examples:
\n\nWhen you simply train a classifier and perform feature selection or parameters optimization you might introduce some systematic error, but in the most cases it is negligible, for example, if you treat your signal and control channels equally.
When you train a classifier to distinguish the signal channel and the control channel, this model will dramatically increase a systematic error introducing huge difference between the signal channel and the control channel.
To sum up:
\n\nIn the HEP data analysis the main ML task is not to make prediction for each individual event, but to estimate total amount of interesting enough events within the given region.
In the HEP data analysis several restrictions on the classifier exist, which are rather complicated and require description of such aspects as mass correlation, MC/real agreement, control channel assumptions, calibration and all systematics estimations. The systematic error estimation cannot be calculated within the competition framework (and cannot be included as a part of the metric) since it is very dependent on the way you work with the data. So for the competition we simplified restrictions for the sake of understanding and performance.
Your solutions should not break the assumption (1) regarding the control and the signal channels. Otherwise an estimation of the classifier performance would require reweighting (or another kind of magic) which is also very dependent on the data preprocessing.
Tanya, Andrey, Marcin, Eugene
", "rawMarkdown": "One of the most common machine learning problems is to find the best predictive model using training data that gives the best predictions on the test data. In the High Energy Physics (HEP) data analysis a machine learning problem is usually formulated a bit differently (it requires a bit of sophistication for both the problem statement and the restrictions). There are several reasons for that.\r\n\r\n1. The main reason is that in HEP you **verify one of two hypotheses** (H_0: decay happens, or H_1: it doesn't) and machine learning is just a step used only for data selection.\r\n \r\n Imagine we want to train a classifier to differentiate between **healthy** and **infected** students. For training we use two samples: infected students from the previous year (as 'signal') and healthy people around the city (as 'background') — we take people outside of the school to avoid possible contamination of our training samples.\r\n\r\n From the common machine learning perspective you train a classifier to predict if a particular student is infected or not. From the HEP analysis perspective you are interested in the amount of infected students (say, if there are significantly larger amount of infected students than it is allowed, the school should be quarantined). So physicists use classifiers' output as a discrimination variable between the signal (infected person) and the background (healthy one) and select the data with the output greater than certain threshold. Then they estimate the **expected** amount of healthy students in the selected data and compare it with the **actual** amount of students within the selection (\"observed\" amount). \r\n\r\n For instance, you **expected 7-10 healthy** persons and **observed 20**. In this case you claim that the data contains a number of infected people because observed number of \"healthy\" people is significantly larger than you expect and the school should be quarantined (or you can claim a discovery in physics term). To choose the best classifier we use weighted area under ROC curve metric. \r\n\r\n2. The second reason is that you have to check for the **absence of the correlation** between the mass and the classifier's predictions. In our example, feature\r\n\r\n `F = \"distance of student's home from the school X\"`,\r\n\r\n is analogous to the event mass. If our classifier is correlated with `F` it might mark all persons from the school X as 'infected' and healthy for others (since in our training sample all infected people are coming from the school X and all healthy people outside of it). But we cannot claim quarantine based on this selection, since it is obviously biased. To avoid such an error we introduce the correlation test. \r\n\r\n3. We cannot add 'signal' labels to real data events, so it cannot be used for training of a classifier. Background events for the training set are taken outside of the signal mass region, because our signal decay has a specific mass range and events outside of this region for sure contain no signal. Signal events can be only simulated. A simulation is not perfect because we cannot simulate all possible processes and we use some approximations which lead to non-perfect signal data. \r\n\r\n Speaking in terms of infected/healthy students, the information about the infected students we take from the previous year records (analogous to simulated data) and the information about the healthy ones - outside of the school X (analogues to real events outside of the signal mass region). \r\n\r\n Here is a list to summarize types of data and its origin:\r\n\r\n **signal:**\r\n - **HEP:** simulated, the decay mass region (MC)\r\n - **Flu:** infected students records, last year, the school X \r\n\r\n **background:**\r\n - **HEP:** real, outside of the decay mass region\r\n - **Flu:** healthy students records outside of the school X, this year \r\n\r\n We train a classifier on the simulated signal data versus the background real data (it can also be simulated, but that's a different story). To claim a discovery we want to be sure that our selection preserves as many as possible signal events, thus for each False Positive Rate (**FPR**) our True Positive Rate (**TPR**) is the highest possible. The classifier can pick not only the difference between the signal and the background, but between the simulated and the real data as well (or not only between infected and healthy people but between this and the previous year data). If this happens TPR for the fixed FPR **will be overestimated**. \r\n\r\n As a result, we need to apply **calibration procedure** to estimate TPR of a classifier for the real signal data. One of the ways to calibrate TPR values (or just check if it is needed indeed) is to see the classifier's performance on the data that is similar to our decay, but is very well known. It is called **'control channel'**, which happens frequently and it is close by its physical properties to the signal decay. \r\n\r\n A control channel is an analogous to the data about students from another (much bigger) city.\r\n\r\n **control signal:**\r\n\r\n - **HEP:** real+simulated of another decay\r\n - **Flu:** infected students records from another city, this and the previous year\r\n\r\n **control background:**\r\n\r\n - **HEP:** real events of another decay\r\n - **Flu:** healthy students records from another city\r\n\r\n For the calibration procedure physicists assume that the **TPR_{real data} / TPR_{MC}** will be the same in the control and the signal channels. Thus, for a given FPR:\r\n\r\n **TPR_{real data}(signal) = TPR_{real data}(control) / TPR_{MC}(control) * TPR_{MC}(signal)** (1) — calibration equation.\r\n\r\n Since some features are not well-simulated, this ratio:\r\n\r\n **TPR_{real data}(control) / TPR_{MC}(control)**\r\n\r\n could be very small, as a result your true TPR for the real signal will be also very small. \r\n\r\n As a solution for the problem physicists do reweighting of data to make MC and real data consistent. If reweighting is applied only to the control channel, it will violate the equality (1), so the same reweighting should be applied to the signal channel to restore (1). We don't cover reweighting in details here, but essentially it is a procedure to assign certain weights to some events so weighted distributions from one dataset would be similar to the distributions from another one.\r\n\r\n In the physical analysis you should choose the FPR points which depend on the hypotheses testing statistics. Thus you should calibrate the TPR specific points depending on your statistics. In the competition we would like to find a classifier which is optimal for any hypotheses testing statistics, so we want to avoid calibration procedure. That is why, **KS test** is implemented to make sure that\r\n\r\n **TPR_{real data}(control) / TPR_{MC}(control) ~ 1** (2) \r\n \r\n **If this approximation is true it is fair to compare the different classifiers without any calibration.** \r\n\r\n4. **Classifiers' systematic error.** Physicists result always includes the systematic and statistic errors. To be precise we should rewrite equation (1):\r\n\r\n TPR_{real data}(signal) = TPR_{real data}(control) / TPR_{MC}(control) * TPR_{MC}(signal) + **systematic_error + statistic_error** (3)\r\n\r\n In some cases, the systematic errors are negligible and we can compare models as is.\r\nBut in the real world the control channel MC and the signal channel MC are not completely similar — it is an example of the data systematic error, to be more precise, this is a systematic error of the physical difference in the nature between the channels but it can be reduced (this could be checked, for example, by comparing predictions PDF for MC samples in the control and the signal channels). \r\nDuring your model training you can introduce other kinds of the systematic errors which might break (1). This case would be impossible to check using the regular competition platform. So for the competition you should produce models that **don't break (1)** by adding new systematic errors.\r\n\r\n\r\n**Some examples:**\r\n\r\n1. When you simply train a classifier and perform feature selection or parameters optimization you might introduce some systematic error, but in the most cases it is negligible, for example, if you treat your signal and control channels equally.\r\n\r\n2. When you train a classifier to distinguish the signal channel and the control channel, this model will dramatically increase a systematic error introducing huge difference between the signal channel and the control channel.\r\n\r\n**To sum up:**\r\n\r\n1. In the HEP data analysis the main ML task is not to make prediction for each individual event, but to estimate total amount of interesting enough events within the given region.\r\n\r\n2. In the HEP data analysis several restrictions on the classifier exist, which are rather complicated and require description of such aspects as mass correlation, MC/real agreement, control channel assumptions, calibration and all systematics estimations. The systematic error estimation cannot be calculated within the competition framework (and cannot be included as a part of the metric) since it is very dependent on the way you work with the data. So for the competition we simplified restrictions for the sake of understanding and performance.\r\n\r\n3. Your solutions should not break the assumption (1) regarding the control and the signal channels. Otherwise an estimation of the classifier performance would require reweighting (or another kind of magic) which is also very dependent on the data preprocessing.\r\n\r\nTanya, Andrey, Marcin, Eugene", "votes": null }, { "id": "91302", "postDate": "09/02/2015 06:21:19", "content": "Oh, this post is informative. Some of them have never been heard before even I have a high energy physics background.
", "rawMarkdown": "Oh, this post is informative. Some of them have never been heard before even I have a high energy physics background.", "votes": null }, { "id": "2108411", "postDate": "01/20/2023 14:18:27", "content": "Why is this discussion deserted? HEP is an excellent topic for discussion.
", "rawMarkdown": "Why is this discussion deserted? HEP is an excellent topic for discussion.", "votes": null } ], "comments": [ { "id": 2108411, "author_name": "donaldlaishram", "author_url": "", "post_date": "01/20/2023 14:18:27", "content": "Why is this discussion deserted? HEP is an excellent topic for discussion.
", "votes": null, "replies": [] }, { "id": 91302, "author_name": "liuchengwang", "author_url": "", "post_date": "09/02/2015 06:21:19", "content": "Oh, this post is informative. Some of them have never been heard before even I have a high energy physics background.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "89569": "One of the most common machine learning problems is to find the best predictive model using training data that gives the best predictions on the test data. In the High Energy Physics (HEP) data analysis a machine learning problem is usually formulated a bit differently (it requires a bit of sophistication for both the problem statement and the restrictions). There are several reasons for that.\r\n\r\n1. The main reason is that in HEP you **verify one of two hypotheses** (H_0: decay happens, or H_1: it doesn't) and machine learning is just a step used only for data selection.\r\n \r\n Imagine we want to train a classifier to differentiate between **healthy** and **infected** students. For training we use two samples: infected students from the previous year (as 'signal') and healthy people around the city (as 'background') — we take people outside of the school to avoid possible contamination of our training samples.\r\n\r\n From the common machine learning perspective you train a classifier to predict if a particular student is infected or not. From the HEP analysis perspective you are interested in the amount of infected students (say, if there are significantly larger amount of infected students than it is allowed, the school should be quarantined). So physicists use classifiers' output as a discrimination variable between the signal (infected person) and the background (healthy one) and select the data with the output greater than certain threshold. Then they estimate the **expected** amount of healthy students in the selected data and compare it with the **actual** amount of students within the selection (\"observed\" amount). \r\n\r\n For instance, you **expected 7-10 healthy** persons and **observed 20**. In this case you claim that the data contains a number of infected people because observed number of \"healthy\" people is significantly larger than you expect and the school should be quarantined (or you can claim a discovery in physics term). To choose the best classifier we use weighted area under ROC curve metric. \r\n\r\n2. The second reason is that you have to check for the **absence of the correlation** between the mass and the classifier's predictions. In our example, feature\r\n\r\n `F = \"distance of student's home from the school X\"`,\r\n\r\n is analogous to the event mass. If our classifier is correlated with `F` it might mark all persons from the school X as 'infected' and healthy for others (since in our training sample all infected people are coming from the school X and all healthy people outside of it). But we cannot claim quarantine based on this selection, since it is obviously biased. To avoid such an error we introduce the correlation test. \r\n\r\n3. We cannot add 'signal' labels to real data events, so it cannot be used for training of a classifier. Background events for the training set are taken outside of the signal mass region, because our signal decay has a specific mass range and events outside of this region for sure contain no signal. Signal events can be only simulated. A simulation is not perfect because we cannot simulate all possible processes and we use some approximations which lead to non-perfect signal data. \r\n\r\n Speaking in terms of infected/healthy students, the information about the infected students we take from the previous year records (analogous to simulated data) and the information about the healthy ones - outside of the school X (analogues to real events outside of the signal mass region). \r\n\r\n Here is a list to summarize types of data and its origin:\r\n\r\n **signal:**\r\n - **HEP:** simulated, the decay mass region (MC)\r\n - **Flu:** infected students records, last year, the school X \r\n\r\n **background:**\r\n - **HEP:** real, outside of the decay mass region\r\n - **Flu:** healthy students records outside of the school X, this year \r\n\r\n We train a classifier on the simulated signal data versus the background real data (it can also be simulated, but that's a different story). To claim a discovery we want to be sure that our selection preserves as many as possible signal events, thus for each False Positive Rate (**FPR**) our True Positive Rate (**TPR**) is the highest possible. The classifier can pick not only the difference between the signal and the background, but between the simulated and the real data as well (or not only between infected and healthy people but between this and the previous year data). If this happens TPR for the fixed FPR **will be overestimated**. \r\n\r\n As a result, we need to apply **calibration procedure** to estimate TPR of a classifier for the real signal data. One of the ways to calibrate TPR values (or just check if it is needed indeed) is to see the classifier's performance on the data that is similar to our decay, but is very well known. It is called **'control channel'**, which happens frequently and it is close by its physical properties to the signal decay. \r\n\r\n A control channel is an analogous to the data about students from another (much bigger) city.\r\n\r\n **control signal:**\r\n\r\n - **HEP:** real+simulated of another decay\r\n - **Flu:** infected students records from another city, this and the previous year\r\n\r\n **control background:**\r\n\r\n - **HEP:** real events of another decay\r\n - **Flu:** healthy students records from another city\r\n\r\n For the calibration procedure physicists assume that the **TPR_{real data} / TPR_{MC}** will be the same in the control and the signal channels. Thus, for a given FPR:\r\n\r\n **TPR_{real data}(signal) = TPR_{real data}(control) / TPR_{MC}(control) * TPR_{MC}(signal)** (1) — calibration equation.\r\n\r\n Since some features are not well-simulated, this ratio:\r\n\r\n **TPR_{real data}(control) / TPR_{MC}(control)**\r\n\r\n could be very small, as a result your true TPR for the real signal will be also very small. \r\n\r\n As a solution for the problem physicists do reweighting of data to make MC and real data consistent. If reweighting is applied only to the control channel, it will violate the equality (1), so the same reweighting should be applied to the signal channel to restore (1). We don't cover reweighting in details here, but essentially it is a procedure to assign certain weights to some events so weighted distributions from one dataset would be similar to the distributions from another one.\r\n\r\n In the physical analysis you should choose the FPR points which depend on the hypotheses testing statistics. Thus you should calibrate the TPR specific points depending on your statistics. In the competition we would like to find a classifier which is optimal for any hypotheses testing statistics, so we want to avoid calibration procedure. That is why, **KS test** is implemented to make sure that\r\n\r\n **TPR_{real data}(control) / TPR_{MC}(control) ~ 1** (2) \r\n \r\n **If this approximation is true it is fair to compare the different classifiers without any calibration.** \r\n\r\n4. **Classifiers' systematic error.** Physicists result always includes the systematic and statistic errors. To be precise we should rewrite equation (1):\r\n\r\n TPR_{real data}(signal) = TPR_{real data}(control) / TPR_{MC}(control) * TPR_{MC}(signal) + **systematic_error + statistic_error** (3)\r\n\r\n In some cases, the systematic errors are negligible and we can compare models as is.\r\nBut in the real world the control channel MC and the signal channel MC are not completely similar — it is an example of the data systematic error, to be more precise, this is a systematic error of the physical difference in the nature between the channels but it can be reduced (this could be checked, for example, by comparing predictions PDF for MC samples in the control and the signal channels). \r\nDuring your model training you can introduce other kinds of the systematic errors which might break (1). This case would be impossible to check using the regular competition platform. So for the competition you should produce models that **don't break (1)** by adding new systematic errors.\r\n\r\n\r\n**Some examples:**\r\n\r\n1. When you simply train a classifier and perform feature selection or parameters optimization you might introduce some systematic error, but in the most cases it is negligible, for example, if you treat your signal and control channels equally.\r\n\r\n2. When you train a classifier to distinguish the signal channel and the control channel, this model will dramatically increase a systematic error introducing huge difference between the signal channel and the control channel.\r\n\r\n**To sum up:**\r\n\r\n1. In the HEP data analysis the main ML task is not to make prediction for each individual event, but to estimate total amount of interesting enough events within the given region.\r\n\r\n2. In the HEP data analysis several restrictions on the classifier exist, which are rather complicated and require description of such aspects as mass correlation, MC/real agreement, control channel assumptions, calibration and all systematics estimations. The systematic error estimation cannot be calculated within the competition framework (and cannot be included as a part of the metric) since it is very dependent on the way you work with the data. So for the competition we simplified restrictions for the sake of understanding and performance.\r\n\r\n3. Your solutions should not break the assumption (1) regarding the control and the signal channels. Otherwise an estimation of the classifier performance would require reweighting (or another kind of magic) which is also very dependent on the data preprocessing.\r\n\r\nTanya, Andrey, Marcin, Eugene", "91302": "Oh, this post is informative. Some of them have never been heard before even I have a high energy physics background.", "2108411": "Why is this discussion deserted? HEP is an excellent topic for discussion." }, "source": "meta" }