{ "id": 16018, "title": "TO Kaggle Admins: handling probabilities greater than 1 and perhaps negative", "url": "/competitions/grasp-and-lift-eeg-detection/discussion/16018", "author_name": "", "post_date": "2015-08-19T05:22:46.093Z", "votes": null, "comment_count": 2, "views": 392, "content": "

Hi Kaggle admins,

\n\n

I inadvertently submitted submission with probabilites greater than 1. I fixed the issue after the fact and it scored the same. It looks like the scoring function is scaling the probabilites before AUC computation. So, no harm here

\n\n

So, I can't help but to ask what happens if someone inadvertently submits negative probabilites? This kind of mistake happened in past competitions. Are probabilites checked to see if they are less than zero before scoring?

\n\n

[EDIT] I am sure I am not the only one that submitted p>1 as the neon code shared here produced those probabilites.

\n\n

[EDIT2] The theory of AUC computation doesn't care about absolute values but about the relative rank between positive and negative samples and adjusts the threshold accordingly. For evaluation of a test set it might not matter much. But, I am not sure about the specific implementation by kaggle and that is my question is to the admins. Moreover, if someone tries to ensemble different models, the scale of probability becomes an issue. \nIn an actual online prediction you only have a probabilty for one test sample just like you don't have future data and it would be hard how to interprete that output 'probability' if it is not standard probability in [0,1]. So, it would be better to restrict the definition of probability to its stricter sense of ϵ [1,0] and total probability adding up to 1.

\n\n

Thanks!

", "messages": [ { "id": "89753", "postDate": "08/19/2015 05:22:46", "content": "

Hi Kaggle admins,

\n\n

I inadvertently submitted submission with probabilites greater than 1. I fixed the issue after the fact and it scored the same. It looks like the scoring function is scaling the probabilites before AUC computation. So, no harm here

\n\n

So, I can't help but to ask what happens if someone inadvertently submits negative probabilites? This kind of mistake happened in past competitions. Are probabilites checked to see if they are less than zero before scoring?

\n\n

[EDIT] I am sure I am not the only one that submitted p>1 as the neon code shared here produced those probabilites.

\n\n

[EDIT2] The theory of AUC computation doesn't care about absolute values but about the relative rank between positive and negative samples and adjusts the threshold accordingly. For evaluation of a test set it might not matter much. But, I am not sure about the specific implementation by kaggle and that is my question is to the admins. Moreover, if someone tries to ensemble different models, the scale of probability becomes an issue. \nIn an actual online prediction you only have a probabilty for one test sample just like you don't have future data and it would be hard how to interprete that output 'probability' if it is not standard probability in [0,1]. So, it would be better to restrict the definition of probability to its stricter sense of ϵ [1,0] and total probability adding up to 1.

\n\n

Thanks!

", "rawMarkdown": "Hi Kaggle admins, \r\n\r\nI inadvertently submitted submission with probabilites greater than 1. I fixed the issue after the fact and it scored the same. It looks like the scoring function is scaling the probabilites before AUC computation. So, no harm here\r\n\r\nSo, I can't help but to ask what happens if someone inadvertently submits negative probabilites? This kind of mistake happened in past competitions. Are probabilites checked to see if they are less than zero before scoring?\r\n\r\n[EDIT] I am sure I am not the only one that submitted p>1 as the neon code shared here produced those probabilites.\r\n\r\n[EDIT2] The theory of AUC computation doesn't care about absolute values but about the relative rank between positive and negative samples and adjusts the threshold accordingly. For evaluation of a test set it might not matter much. But, I am not sure about the specific implementation by kaggle and that is my question is to the admins. Moreover, if someone tries to ensemble different models, the scale of probability becomes an issue. \r\nIn an actual online prediction you only have a probabilty for one test sample just like you don't have future data and it would be hard how to interprete that output 'probability' if it is not standard probability in [0,1]. So, it would be better to restrict the definition of probability to its stricter sense of ϵ [1,0] and total probability adding up to 1.\r\n\r\nThanks!", "votes": null }, { "id": "89763", "postDate": "08/19/2015 08:08:55", "content": "

Disclaimer: I'm not 100% sure on this one, since I'm not familiar with the AUC implementation Kaggle uses.

\n\n

But the way AU-ROC is usually calculated, the scaling of the scores doesn't matter at all. It implicitly uses all (relevant) thresholds between positive and negative classes. So in principle, they don't have to be probabilities and they don't event have to be positive. But all your models should operate on the same scale (see this post).

", "rawMarkdown": "**Disclaimer: I'm not 100% sure on this one, since I'm not familiar with the AUC implementation Kaggle uses.**\r\n\r\nBut the way AU-ROC is usually calculated, the scaling of the scores doesn't matter at all. It implicitly uses all (relevant) thresholds between positive and negative classes. So in principle, they don't have to be probabilities and they don't event have to be positive. But all your models should operate on the same scale (see [this post][1]). \r\n\r\n\r\n [1]: https://www.kaggle.com/c/grasp-and-lift-eeg-detection/forums/t/16011/one-more-question-about-scoring", "votes": null }, { "id": "89769", "postDate": "08/19/2015 08:50:42", "content": "

@Deep as Klaus said then scale should not matter. A good post about AUC can be found her: http://fastml.com/what-you-wanted-to-know-about-auc/

", "rawMarkdown": "Deep as Klaus said then scale should not matter. A good post about AUC can be found her: http://fastml.com/what-you-wanted-to-know-about-auc/", "votes": null } ], "comments": [ { "id": 89763, "author_name": "qwlouse", "author_url": "", "post_date": "08/19/2015 08:08:55", "content": "

Disclaimer: I'm not 100% sure on this one, since I'm not familiar with the AUC implementation Kaggle uses.

\n\n

But the way AU-ROC is usually calculated, the scaling of the scores doesn't matter at all. It implicitly uses all (relevant) thresholds between positive and negative classes. So in principle, they don't have to be probabilities and they don't event have to be positive. But all your models should operate on the same scale (see this post).

", "votes": null, "replies": [] }, { "id": 89769, "author_name": "", "author_url": "", "post_date": "08/19/2015 08:50:42", "content": "

@Deep as Klaus said then scale should not matter. A good post about AUC can be found her: http://fastml.com/what-you-wanted-to-know-about-auc/

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "89753": "Hi Kaggle admins, \r\n\r\nI inadvertently submitted submission with probabilites greater than 1. I fixed the issue after the fact and it scored the same. It looks like the scoring function is scaling the probabilites before AUC computation. So, no harm here\r\n\r\nSo, I can't help but to ask what happens if someone inadvertently submits negative probabilites? This kind of mistake happened in past competitions. Are probabilites checked to see if they are less than zero before scoring?\r\n\r\n[EDIT] I am sure I am not the only one that submitted p>1 as the neon code shared here produced those probabilites.\r\n\r\n[EDIT2] The theory of AUC computation doesn't care about absolute values but about the relative rank between positive and negative samples and adjusts the threshold accordingly. For evaluation of a test set it might not matter much. But, I am not sure about the specific implementation by kaggle and that is my question is to the admins. Moreover, if someone tries to ensemble different models, the scale of probability becomes an issue. \r\nIn an actual online prediction you only have a probabilty for one test sample just like you don't have future data and it would be hard how to interprete that output 'probability' if it is not standard probability in [0,1]. So, it would be better to restrict the definition of probability to its stricter sense of ϵ [1,0] and total probability adding up to 1.\r\n\r\nThanks!", "89763": "**Disclaimer: I'm not 100% sure on this one, since I'm not familiar with the AUC implementation Kaggle uses.**\r\n\r\nBut the way AU-ROC is usually calculated, the scaling of the scores doesn't matter at all. It implicitly uses all (relevant) thresholds between positive and negative classes. So in principle, they don't have to be probabilities and they don't event have to be positive. But all your models should operate on the same scale (see [this post][1]). \r\n\r\n\r\n [1]: https://www.kaggle.com/c/grasp-and-lift-eeg-detection/forums/t/16011/one-more-question-about-scoring", "89769": "Deep as Klaus said then scale should not matter. A good post about AUC can be found her: http://fastml.com/what-you-wanted-to-know-about-auc/" }, "source": "meta" }