{ "id": 12358, "title": "Rescaling and replacing probabilities: which first?", "url": "/competitions/datasciencebowl/discussion/12358", "author_name": "", "post_date": "2015-01-29T09:30:04.033Z", "votes": null, "comment_count": 5, "views": 1494, "content": "

In the evaluation page it says:

\n

"submitted probabilities for a given image ... are rescaled prior to being scored"

\n

and

\n

"predicted probabilities are replaced with max(min(p,1−10−15),10−15)"

\n

My question:

\n

Which one is done first?

\n

Thanks

", "messages": [ { "id": "63299", "postDate": "01/29/2015 09:30:04", "content": "

In the evaluation page it says:

\n

"submitted probabilities for a given image ... are rescaled prior to being scored"

\n

and

\n

"predicted probabilities are replaced with max(min(p,1−10−15),10−15)"

\n

My question:

\n

Which one is done first?

\n

Thanks

", "rawMarkdown": "", "votes": null }, { "id": "63300", "postDate": "01/29/2015 09:43:30", "content": "

Re-scaling *should* be done first.

\n

The other way around would not make sense or achieve the stated purpose.

", "rawMarkdown": "", "votes": null }, { "id": "63302", "postDate": "01/29/2015 10:23:30", "content": "

[quote=Neil Slater;63300]

\n

Re-scaling *should* be done first.

\n

The other way around would not make sense or achieve the stated purpose.

\n

[/quote]

\n

I agree but that would mean you could get a sum of probabilities slightly higher than 1. (Max 120*1e-15 higher) Which is probably of no importance.

\n

More important, if I choose not to normalize myself and submit values higher than 1, I would want to know how the score is calculated exactly.

", "rawMarkdown": "", "votes": null }, { "id": "63308", "postDate": "01/29/2015 14:55:23", "content": "

The capping is done before the normalization.

", "rawMarkdown": "", "votes": null }, { "id": "63313", "postDate": "01/29/2015 15:17:27", "content": "

[quote=Jules;63302]

\n

[quote=Neil Slater;63300]

\n

Re-scaling *should* be done first.

\n

The other way around would not make sense or achieve the stated purpose.

\n

[/quote]

\n

I agree but that would mean you could get a sum of probabilities slightly higher than 1. (Max 120*1e-15 higher) Which is probably of no importance.

\n

More important, if I choose not to normalize myself and submit values higher than 1, I would want to know how the score is calculated exactly.

\n

[/quote]

\n

The scaling provides self-consistent probabilities for a set of exclusive classes.

\n

Whilst I'd say that technically the scoring limits for input into the log function are based on your probabilities, using the capping to limit maximum penalty for exceptionally inaccurate ones. Manipulations of your probabilities for scoring purposes do not have to maintain consistency, and don't even have to take into account all your predictions (although they do for the scoring metric used here)

", "rawMarkdown": "", "votes": null }, { "id": "63314", "postDate": "01/29/2015 15:42:17", "content": "

[quote=William Cukierski;63308]

\n

The capping is done before the normalization.

\n

[/quote]

\n

Thanks

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 63300, "author_name": "slobo777", "author_url": "", "post_date": "01/29/2015 09:43:30", "content": "

Re-scaling *should* be done first.

\n

The other way around would not make sense or achieve the stated purpose.

", "votes": null, "replies": [] }, { "id": 63302, "author_name": "julesvanligtenberg", "author_url": "", "post_date": "01/29/2015 10:23:30", "content": "

[quote=Neil Slater;63300]

\n

Re-scaling *should* be done first.

\n

The other way around would not make sense or achieve the stated purpose.

\n

[/quote]

\n

I agree but that would mean you could get a sum of probabilities slightly higher than 1. (Max 120*1e-15 higher) Which is probably of no importance.

\n

More important, if I choose not to normalize myself and submit values higher than 1, I would want to know how the score is calculated exactly.

", "votes": null, "replies": [] }, { "id": 63308, "author_name": "wcukierski", "author_url": "", "post_date": "01/29/2015 14:55:23", "content": "

The capping is done before the normalization.

", "votes": null, "replies": [] }, { "id": 63313, "author_name": "slobo777", "author_url": "", "post_date": "01/29/2015 15:17:27", "content": "

[quote=Jules;63302]

\n

[quote=Neil Slater;63300]

\n

Re-scaling *should* be done first.

\n

The other way around would not make sense or achieve the stated purpose.

\n

[/quote]

\n

I agree but that would mean you could get a sum of probabilities slightly higher than 1. (Max 120*1e-15 higher) Which is probably of no importance.

\n

More important, if I choose not to normalize myself and submit values higher than 1, I would want to know how the score is calculated exactly.

\n

[/quote]

\n

The scaling provides self-consistent probabilities for a set of exclusive classes.

\n

Whilst I'd say that technically the scoring limits for input into the log function are based on your probabilities, using the capping to limit maximum penalty for exceptionally inaccurate ones. Manipulations of your probabilities for scoring purposes do not have to maintain consistency, and don't even have to take into account all your predictions (although they do for the scoring metric used here)

", "votes": null, "replies": [] }, { "id": 63314, "author_name": "julesvanligtenberg", "author_url": "", "post_date": "01/29/2015 15:42:17", "content": "

[quote=William Cukierski;63308]

\n

The capping is done before the normalization.

\n

[/quote]

\n

Thanks

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "63299": "", "63300": "", "63302": "", "63308": "", "63313": "", "63314": "" }, "source": "meta" }