My submission received a 0.02 on the correlation test, which is above the 0.002 threshold. I did not include mass during training, so how is it possible that I could fail this test? Any ideas on how this happens outside of using mass during training?
", "messages": [ { "id": "86911", "postDate": "07/25/2015 06:48:27", "content": "My submission received a 0.02 on the correlation test, which is above the 0.002 threshold. I did not include mass during training, so how is it possible that I could fail this test? Any ideas on how this happens outside of using mass during training?
", "rawMarkdown": "My submission received a 0.02 on the correlation test, which is above the 0.002 threshold. I did not include mass during training, so how is it possible that I could fail this test? Any ideas on how this happens outside of using mass during training?", "votes": null }, { "id": "86919", "postDate": "07/25/2015 08:02:22", "content": "There are many possibilities how a classifier can peak up a mass correlation. The most common way is when you include momentum information in your training. If you look at eq2. (appendix A) in https://storage.googleapis.com/kaggle-competitions/kaggle/4488/media/lhcb_description_official.pdf you will see that the mass can be calculated when you know the momentum and energy of the particles. I would start by checking this variables.
", "rawMarkdown": "There are many possibilities how a classifier can peak up a mass correlation. The most common way is when you include momentum information in your training. If you look at eq2. (appendix A) in https://kaggle2.blob.core.windows.net/competitions/kaggle/4488/media/lhcb_description_official.pdf you will see that the mass can be calculated when you know the momentum and energy of the particles. I would start by checking this variables.", "votes": null }, { "id": "86930", "postDate": "07/25/2015 11:12:18", "content": "I don't understand how is it possible that a good solution may pass the correlation test.\nAs I see in the data, the mass and the signal are strongly dependent. Any good solution gives prediction close to true signal, therefore such prediction must be also dependent on the mass.
", "rawMarkdown": "I don't understand how is it possible that a good solution may pass the correlation test.\r\nAs I see in the data, the mass and the signal are strongly dependent. Any good solution gives prediction close to true signal, therefore such prediction must be also dependent on the mass.", "votes": null }, { "id": "86954", "postDate": "07/25/2015 15:57:50", "content": "That is exactly what I was thinking. If mass is related to signal and can be inferred from the other properties, wouldn't that be a good thing? I can see how training using the mass is unreliable, but inferring it from the rest of the data seems like a strength to me. Any physicists/admins have a comment on this that can help us understand the implications of this a little more?
", "rawMarkdown": "That is exactly what I was thinking. If mass is related to signal and can be inferred from the other properties, wouldn't that be a good thing? I can see how training using the mass is unreliable, but inferring it from the rest of the data seems like a strength to me. Any physicists/admins have a comment on this that can help us understand the implications of this a little more?", "votes": null }, { "id": "86968", "postDate": "07/25/2015 20:30:20", "content": "You are in theory correct. In a perfect world one could put the mass inside the classifier and get the prediction. The problem arises when one needs to check that the classifier is correct. You then need to apply the classifier to a channel that you know well. In our case this is Ds->phi pi. Mass of a tau lepton is 1777 MeV/c^2, while the mass of a Ds meson is 1968 MeV/c^2, so your classifier would not accept any Ds events and you couldn't test your classifier on data.
\n\nFurthermore, in our MC the mass resolution is not perfectly modeled. We are correcting for this when calculating the efficiency using data driven methods. So putting mass in as it is will give you also the wrong answer because of this effect.
\n\nI understand that this is a painful check but we really need it.
\n\nThere are a lot more reasons not to put it in, but i think these tow should do. If not please let me know and will try explaining further :)
\n\nCheers,\nMarcin
", "rawMarkdown": "You are in theory correct. In a perfect world one could put the mass inside the classifier and get the prediction. The problem arises when one needs to check that the classifier is correct. You then need to apply the classifier to a channel that you know well. In our case this is Ds->phi pi. Mass of a tau lepton is 1777 MeV/c^2, while the mass of a Ds meson is 1968 MeV/c^2, so your classifier would not accept any Ds events and you couldn't test your classifier on data.\r\n\r\nFurthermore, in our MC the mass resolution is not perfectly modeled. We are correcting for this when calculating the efficiency using data driven methods. So putting mass in as it is will give you also the wrong answer because of this effect. \r\n\r\nI understand that this is a painful check but we really need it.\r\n\r\nThere are a lot more reasons not to put it in, but i think these tow should do. If not please let me know and will try explaining further :)\r\n\r\nCheers,\r\nMarcin", "votes": null }, { "id": "86982", "postDate": "07/25/2015 23:35:25", "content": "Hi, Victor!
\n\nIndeed, there is huge correlation between event mass and label. But we are interested in the absence of the correlation between classifier predictions and event mass for background (zero labeled) events.
\n\nLet me explain this in two examples:
\n\nfirst classifier returns random values from [0., 0.5] for all events outside of tau mass region and values from [0.5, 1.] for events inside the tau mass region.
second classifier returns values from [0., 0.5] for all background events and [0.5, 1.] for all signal events.
Both have very high quality (in our tests), however the first one is useless, because we have many background events in tau mass region and we are unable to estimate their number, while the second is perfect.
", "rawMarkdown": "Hi, Victor!\r\n\r\nIndeed, there is huge correlation between event mass and label. But we are interested in the absence of the correlation between classifier predictions and event mass for **background (zero labeled)** events. \r\n\r\nLet me explain this in two examples:\r\n\r\n1. first classifier returns random values from [0., 0.5] for all events outside of tau mass region and values from [0.5, 1.] for events inside the tau mass region.\r\n\r\n2. second classifier returns values from [0., 0.5] for all background events and [0.5, 1.] for all signal events.\r\n\r\nBoth have very high quality (in our tests), however the first one is useless, because we have many background events in tau mass region and we are unable to estimate their number, while the second is perfect.", "votes": null } ], "comments": [ { "id": 86919, "author_name": "albercik1905", "author_url": "", "post_date": "07/25/2015 08:02:22", "content": "There are many possibilities how a classifier can peak up a mass correlation. The most common way is when you include momentum information in your training. If you look at eq2. (appendix A) in https://storage.googleapis.com/kaggle-competitions/kaggle/4488/media/lhcb_description_official.pdf you will see that the mass can be calculated when you know the momentum and energy of the particles. I would start by checking this variables.
", "votes": null, "replies": [] }, { "id": 86930, "author_name": "nedelko", "author_url": "", "post_date": "07/25/2015 11:12:18", "content": "I don't understand how is it possible that a good solution may pass the correlation test.\nAs I see in the data, the mass and the signal are strongly dependent. Any good solution gives prediction close to true signal, therefore such prediction must be also dependent on the mass.
", "votes": null, "replies": [] }, { "id": 86954, "author_name": "jstaker7", "author_url": "", "post_date": "07/25/2015 15:57:50", "content": "That is exactly what I was thinking. If mass is related to signal and can be inferred from the other properties, wouldn't that be a good thing? I can see how training using the mass is unreliable, but inferring it from the rest of the data seems like a strength to me. Any physicists/admins have a comment on this that can help us understand the implications of this a little more?
", "votes": null, "replies": [] }, { "id": 86968, "author_name": "albercik1905", "author_url": "", "post_date": "07/25/2015 20:30:20", "content": "You are in theory correct. In a perfect world one could put the mass inside the classifier and get the prediction. The problem arises when one needs to check that the classifier is correct. You then need to apply the classifier to a channel that you know well. In our case this is Ds->phi pi. Mass of a tau lepton is 1777 MeV/c^2, while the mass of a Ds meson is 1968 MeV/c^2, so your classifier would not accept any Ds events and you couldn't test your classifier on data.
\n\nFurthermore, in our MC the mass resolution is not perfectly modeled. We are correcting for this when calculating the efficiency using data driven methods. So putting mass in as it is will give you also the wrong answer because of this effect.
\n\nI understand that this is a painful check but we really need it.
\n\nThere are a lot more reasons not to put it in, but i think these tow should do. If not please let me know and will try explaining further :)
\n\nCheers,\nMarcin
", "votes": null, "replies": [] }, { "id": 86982, "author_name": "tataantares", "author_url": "", "post_date": "07/25/2015 23:35:25", "content": "Hi, Victor!
\n\nIndeed, there is huge correlation between event mass and label. But we are interested in the absence of the correlation between classifier predictions and event mass for background (zero labeled) events.
\n\nLet me explain this in two examples:
\n\nfirst classifier returns random values from [0., 0.5] for all events outside of tau mass region and values from [0.5, 1.] for events inside the tau mass region.
second classifier returns values from [0., 0.5] for all background events and [0.5, 1.] for all signal events.
Both have very high quality (in our tests), however the first one is useless, because we have many background events in tau mass region and we are unable to estimate their number, while the second is perfect.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "86911": "My submission received a 0.02 on the correlation test, which is above the 0.002 threshold. I did not include mass during training, so how is it possible that I could fail this test? Any ideas on how this happens outside of using mass during training?", "86919": "There are many possibilities how a classifier can peak up a mass correlation. The most common way is when you include momentum information in your training. If you look at eq2. (appendix A) in https://kaggle2.blob.core.windows.net/competitions/kaggle/4488/media/lhcb_description_official.pdf you will see that the mass can be calculated when you know the momentum and energy of the particles. I would start by checking this variables.", "86930": "I don't understand how is it possible that a good solution may pass the correlation test.\r\nAs I see in the data, the mass and the signal are strongly dependent. Any good solution gives prediction close to true signal, therefore such prediction must be also dependent on the mass.", "86954": "That is exactly what I was thinking. If mass is related to signal and can be inferred from the other properties, wouldn't that be a good thing? I can see how training using the mass is unreliable, but inferring it from the rest of the data seems like a strength to me. Any physicists/admins have a comment on this that can help us understand the implications of this a little more?", "86968": "You are in theory correct. In a perfect world one could put the mass inside the classifier and get the prediction. The problem arises when one needs to check that the classifier is correct. You then need to apply the classifier to a channel that you know well. In our case this is Ds->phi pi. Mass of a tau lepton is 1777 MeV/c^2, while the mass of a Ds meson is 1968 MeV/c^2, so your classifier would not accept any Ds events and you couldn't test your classifier on data.\r\n\r\nFurthermore, in our MC the mass resolution is not perfectly modeled. We are correcting for this when calculating the efficiency using data driven methods. So putting mass in as it is will give you also the wrong answer because of this effect. \r\n\r\nI understand that this is a painful check but we really need it.\r\n\r\nThere are a lot more reasons not to put it in, but i think these tow should do. If not please let me know and will try explaining further :)\r\n\r\nCheers,\r\nMarcin", "86982": "Hi, Victor!\r\n\r\nIndeed, there is huge correlation between event mass and label. But we are interested in the absence of the correlation between classifier predictions and event mass for **background (zero labeled)** events. \r\n\r\nLet me explain this in two examples:\r\n\r\n1. first classifier returns random values from [0., 0.5] for all events outside of tau mass region and values from [0.5, 1.] for events inside the tau mass region.\r\n\r\n2. second classifier returns values from [0., 0.5] for all background events and [0.5, 1.] for all signal events.\r\n\r\nBoth have very high quality (in our tests), however the first one is useless, because we have many background events in tau mass region and we are unable to estimate their number, while the second is perfect." }, "source": "meta" }