{ "id": 5610, "title": "Need Clearification for Scoring", "url": "/competitions/facebook-recruiting-iii-keyword-extraction/discussion/5610", "author_name": "", "post_date": "2013-09-01T08:48:58.813Z", "votes": null, "comment_count": 7, "views": 3278, "content": "

In the evaluation section it says

\n

"To receive credit, the tag you predict must be an exact match"

\n

In the case of the correct tags is "javascript html css", if we produce the prediction of "javascript", is it counted as 1 true positive, or as 1 negative? And if the prediction is "javascript html css", the evaluation result is it 1 true positive or 3 positives? Are the tags produced evaluated separately for each question, or the combination is evaluated?

\n

 

", "messages": [ { "id": "29905", "postDate": "09/01/2013 08:48:58", "content": "

In the evaluation section it says

\n

"To receive credit, the tag you predict must be an exact match"

\n

In the case of the correct tags is "javascript html css", if we produce the prediction of "javascript", is it counted as 1 true positive, or as 1 negative? And if the prediction is "javascript html css", the evaluation result is it 1 true positive or 3 positives? Are the tags produced evaluated separately for each question, or the combination is evaluated?

\n

 

", "rawMarkdown": "", "votes": null }, { "id": "29910", "postDate": "09/01/2013 09:11:49", "content": "

[quote=weidong Liang;29905]

\n

In the evaluation section it says

\n

"To receive credit, the tag you predict must be an exact match"

\n

In the case of the correct tags is "javascript html css", if we produce the prediction of "javascript", is it counted as 1 true positive, or as 1 negative? And if the prediction is "javascript html css", the evaluation result is it 1 true positive or 3 positives? Are the tags produced evaluated separately for each question, or the combination is evaluated?

\n

[/quote]

\n

 

\n

Since F1-Score is calculated for ever individual class and only then is the mean taken. Therefore, it is highly likely that the strings in the Tags column will be split and the individual tags used for scoring. 

", "rawMarkdown": "", "votes": null }, { "id": "29931", "postDate": "09/01/2013 14:10:53", "content": "

The case where you predict 'javascript', you should get one true positive, zero false positives and two false negatives. In the second case if you predict 'javascript html css', you will get three true positives, zero false negatives and zero false positives.  Thus on these observations you will get a precisions of 1 / (1 + 0) = 1 and 3 / (3 + 0) = 1. You will get recalls of 1 / (1 + 2) = 1/3 and 3 / (3 + 0) = 1.  

", "rawMarkdown": "", "votes": null }, { "id": "32241", "postDate": "10/11/2013 04:44:48", "content": "

Hi,

\n

There are two different answers here that actually raise an interesting question. I re-read the evaluation details and cannot find the answer. (initially I had the same intuition as Kenny)

\n

Are the tp, fp, tn, fn, computed across each "class" (i.e. each tag) or across each post ?

\n

Best, 
Lorenzo

", "rawMarkdown": "", "votes": null }, { "id": "32245", "postDate": "10/11/2013 07:31:29", "content": "

I am considering mean across each post and validation is differentially consistent so far.

", "rawMarkdown": "", "votes": null }, { "id": "32254", "postDate": "10/11/2013 15:50:23", "content": "

As cross-validation of Kenny:

\n

The way I calculate the F1-score is to split the outcome variable, giving the actual amount of tags that have to be predicted. I just sum all of these, then given, say case 100, if I predict "javascript" and it is in ["javascript", "css", "html"], then I count it as a true positive. If you do not also predict "css" and "html", these will get added to the false negative list.

\n

 

\n

Precision = true positive / total amount of your predictions =  1

\n

Recall      = true positive / total amount of (unobserved) tags that should be predicted = 1/3

\n

 

\n

F1-score = 2 * (p * r / (p + r) = 1 * (1 * 1/3 / (1 + 1/3)) = 0.4

\n

 

\n

Based on cross-validation of train/test, I would say this is correct.

", "rawMarkdown": "", "votes": null }, { "id": "32298", "postDate": "10/12/2013 22:37:55", "content": "

[quote=weidong Liang;29905]

\n

"To receive credit, the tag you predict must be an exact match"

\n

In the case of the correct tags is "javascript html css", if we produce the prediction of "javascript", 

\n

[/quote]

\n

 

\n

The (set of) tags on each item are evaluated separately. Thus your example counts as 1 true positive + 2 false negatives.

\n

The "exact match" refers to each tag, not the set, so if you predicted "javascript-library" instead of "javascript", that would also be a negative, since the (sister) tag isn't an exact match.

\n

 

\n

As to how we determine what is/isn't a sister tag, see also my question about we can legally automatically determine which tags are synonymous/related?

", "rawMarkdown": "", "votes": null }, { "id": "33695", "postDate": "11/14/2013 08:30:52", "content": "

What about the converse situation where I have predicted too many tags? Say for example the correct output is ["javascript"] and I have calculated ["javascript", "java", "js", "html", "css"], then would it mean I have 1 tp, 4 fp, 0tn, 0fn? 

\n

So to clarify is it correct that any tags that aren't predicted are counted as false negatives (fn) and any extra tags that are predicted are counted as false positives (fp)?

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 29910, "author_name": "rohits", "author_url": "", "post_date": "09/01/2013 09:11:49", "content": "

[quote=weidong Liang;29905]

\n

In the evaluation section it says

\n

"To receive credit, the tag you predict must be an exact match"

\n

In the case of the correct tags is "javascript html css", if we produce the prediction of "javascript", is it counted as 1 true positive, or as 1 negative? And if the prediction is "javascript html css", the evaluation result is it 1 true positive or 3 positives? Are the tags produced evaluated separately for each question, or the combination is evaluated?

\n

[/quote]

\n

 

\n

Since F1-Score is calculated for ever individual class and only then is the mean taken. Therefore, it is highly likely that the strings in the Tags column will be split and the individual tags used for scoring. 

", "votes": null, "replies": [] }, { "id": 29931, "author_name": "darrelkj", "author_url": "", "post_date": "09/01/2013 14:10:53", "content": "

The case where you predict 'javascript', you should get one true positive, zero false positives and two false negatives. In the second case if you predict 'javascript html css', you will get three true positives, zero false negatives and zero false positives.  Thus on these observations you will get a precisions of 1 / (1 + 0) = 1 and 3 / (3 + 0) = 1. You will get recalls of 1 / (1 + 2) = 1/3 and 3 / (3 + 0) = 1.  

", "votes": null, "replies": [] }, { "id": 32241, "author_name": "lorenzolucido", "author_url": "", "post_date": "10/11/2013 04:44:48", "content": "

Hi,

\n

There are two different answers here that actually raise an interesting question. I re-read the evaluation details and cannot find the answer. (initially I had the same intuition as Kenny)

\n

Are the tp, fp, tn, fn, computed across each "class" (i.e. each tag) or across each post ?

\n

Best, 
Lorenzo

", "votes": null, "replies": [] }, { "id": 32245, "author_name": "emiliogortizg", "author_url": "", "post_date": "10/11/2013 07:31:29", "content": "

I am considering mean across each post and validation is differentially consistent so far.

", "votes": null, "replies": [] }, { "id": 32254, "author_name": "pvkooten", "author_url": "", "post_date": "10/11/2013 15:50:23", "content": "

As cross-validation of Kenny:

\n

The way I calculate the F1-score is to split the outcome variable, giving the actual amount of tags that have to be predicted. I just sum all of these, then given, say case 100, if I predict "javascript" and it is in ["javascript", "css", "html"], then I count it as a true positive. If you do not also predict "css" and "html", these will get added to the false negative list.

\n

 

\n

Precision = true positive / total amount of your predictions =  1

\n

Recall      = true positive / total amount of (unobserved) tags that should be predicted = 1/3

\n

 

\n

F1-score = 2 * (p * r / (p + r) = 1 * (1 * 1/3 / (1 + 1/3)) = 0.4

\n

 

\n

Based on cross-validation of train/test, I would say this is correct.

", "votes": null, "replies": [] }, { "id": 32298, "author_name": "smcinerney", "author_url": "", "post_date": "10/12/2013 22:37:55", "content": "

[quote=weidong Liang;29905]

\n

"To receive credit, the tag you predict must be an exact match"

\n

In the case of the correct tags is "javascript html css", if we produce the prediction of "javascript", 

\n

[/quote]

\n

 

\n

The (set of) tags on each item are evaluated separately. Thus your example counts as 1 true positive + 2 false negatives.

\n

The "exact match" refers to each tag, not the set, so if you predicted "javascript-library" instead of "javascript", that would also be a negative, since the (sister) tag isn't an exact match.

\n

 

\n

As to how we determine what is/isn't a sister tag, see also my question about we can legally automatically determine which tags are synonymous/related?

", "votes": null, "replies": [] }, { "id": 33695, "author_name": "matthewthornton", "author_url": "", "post_date": "11/14/2013 08:30:52", "content": "

What about the converse situation where I have predicted too many tags? Say for example the correct output is ["javascript"] and I have calculated ["javascript", "java", "js", "html", "css"], then would it mean I have 1 tp, 4 fp, 0tn, 0fn? 

\n

So to clarify is it correct that any tags that aren't predicted are counted as false negatives (fn) and any extra tags that are predicted are counted as false positives (fp)?

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "29905": "", "29910": "", "29931": "", "32241": "", "32245": "", "32254": "", "32298": "", "33695": "" }, "source": "meta" }