{ "id": 5216, "title": "Submission Format / Evaluation", "url": "/competitions/accelerometer-biometric-competition/discussion/5216", "author_name": "", "post_date": "2013-07-28T11:46:23.930Z", "votes": null, "comment_count": 4, "views": 1913, "content": "

According to the description of the submission format, we can enter any real number as a prediction.  Does that mean we can use 0 as meaning we have absolute certainty in a non-match and 1 as absolute certainty in a match?

\n

I was a bit confused by the negative values in the example.

\n

 

\n

Quote from the submission example:

\n

"Each line of your submission should contain an QuestionId and a
prediction, IsTrue. Note that you may submit any real-valued number as a
prediction,  since AUC is only sensitive to the ranking.
sampleSubmission.csv shows a representative valid submission.  The
format looks like this:
QuestionId,IsTrue
1,0
2,0.3
3,99999
4,-0.8
etc..."

\n

 

\n

Any help is appreciated.

", "messages": [ { "id": "27728", "postDate": "07/28/2013 11:46:23", "content": "

According to the description of the submission format, we can enter any real number as a prediction.  Does that mean we can use 0 as meaning we have absolute certainty in a non-match and 1 as absolute certainty in a match?

\n

I was a bit confused by the negative values in the example.

\n

 

\n

Quote from the submission example:

\n

"Each line of your submission should contain an QuestionId and a
prediction, IsTrue. Note that you may submit any real-valued number as a
prediction,  since AUC is only sensitive to the ranking.
sampleSubmission.csv shows a representative valid submission.  The
format looks like this:
QuestionId,IsTrue
1,0
2,0.3
3,99999
4,-0.8
etc..."

\n

 

\n

Any help is appreciated.

", "rawMarkdown": "", "votes": null }, { "id": "27729", "postDate": "07/28/2013 12:02:57", "content": "

Since the evaluation is based on AUC you are free to put any number in the -infinity to +infinity range as long as you make sure that higher numbers correspond with higher chance of something being true.

\n

So, you are OK to output 0 when you are absolutely certain that something is not true and 1 when you are absolutely certain that something is true and any number in between reflecting your certainty. But, I would recommend giving probabilistic outputs over binary outputs as AUC penalizes models with binary outputs.

\n

 

\n

 

", "rawMarkdown": "", "votes": null }, { "id": "27736", "postDate": "07/29/2013 03:33:28", "content": "

Excellent. Thank you for the quick reply!

", "rawMarkdown": "", "votes": null }, { "id": "28749", "postDate": "08/15/2013 18:05:00", "content": "", "rawMarkdown": "", "votes": null }, { "id": "28750", "postDate": "08/15/2013 18:06:08", "content": "

[quote=Sashikanth Dareddy;27729]

\n

Since the evaluation is based on AUC you are free to put any number in the -infinity to +infinity range as long as you make sure that higher numbers correspond with higher chance of something being true.

\n

So, you are OK to output 0 when you are absolutely certain that something is not true and 1 when you are absolutely certain that something is true and any number in between reflecting your certainty. But, I would recommend giving probabilistic outputs over binary outputs as AUC penalizes models with binary outputs.

\n

[/quote]

\n

 

\n

Since AUC is between -inf and +inf, I think for a person to use binary values, it would make more sense to use -1 and +1. Or am I missing something here?

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 27729, "author_name": "sashikanthdareddy", "author_url": "", "post_date": "07/28/2013 12:02:57", "content": "

Since the evaluation is based on AUC you are free to put any number in the -infinity to +infinity range as long as you make sure that higher numbers correspond with higher chance of something being true.

\n

So, you are OK to output 0 when you are absolutely certain that something is not true and 1 when you are absolutely certain that something is true and any number in between reflecting your certainty. But, I would recommend giving probabilistic outputs over binary outputs as AUC penalizes models with binary outputs.

\n

 

\n

 

", "votes": null, "replies": [] }, { "id": 27736, "author_name": "convalytics", "author_url": "", "post_date": "07/29/2013 03:33:28", "content": "

Excellent. Thank you for the quick reply!

", "votes": null, "replies": [] }, { "id": 28749, "author_name": "", "author_url": "", "post_date": "08/15/2013 18:05:00", "content": "", "votes": null, "replies": [] }, { "id": 28750, "author_name": "", "author_url": "", "post_date": "08/15/2013 18:06:08", "content": "

[quote=Sashikanth Dareddy;27729]

\n

Since the evaluation is based on AUC you are free to put any number in the -infinity to +infinity range as long as you make sure that higher numbers correspond with higher chance of something being true.

\n

So, you are OK to output 0 when you are absolutely certain that something is not true and 1 when you are absolutely certain that something is true and any number in between reflecting your certainty. But, I would recommend giving probabilistic outputs over binary outputs as AUC penalizes models with binary outputs.

\n

[/quote]

\n

 

\n

Since AUC is between -inf and +inf, I think for a person to use binary values, it would make more sense to use -1 and +1. Or am I missing something here?

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "27728": "", "27729": "", "27736": "", "28749": "", "28750": "" }, "source": "meta" }