{ "id": 6362, "title": "clarify the evaluation of predicted synonymous tags", "url": "/competitions/facebook-recruiting-iii-keyword-extraction/discussion/6362", "author_name": "", "post_date": "2013-11-21T16:09:14.600Z", "votes": null, "comment_count": 1, "views": 777, "content": "

Hi,

\n

I am not finding the passage describing the penalization of predicted tags that are ("subjectively") synonyms very clear.  Particularly, I'm not fully following point 3 in the evaluation description [1]:

\n

To receive credit, the tag you predict must be an exact match, regardless of whether you believe the tags are synonyms. Why aren't synonyms counted? ...

\n

3. Participants are equally penalized for predicting a synonym of a correct tag, so the task can be framed as not only predicting a tag, but also modeling the distribution(s) of potential synonyms

\n

 

\n

The wording, "equally penalized" is throwing me off here.  For example, if I were to predict the tag "python" for a sample where one of the actual tags was "python2.7", then just to be clear, since this is not an exact match, is it considered as wrong ("equally penalized") as a predicted tag of something that is completely arbitrary/random?  This is how I think it is too be understood, but just want to ask to be completely sure.

\n

 

\n

[1] http://www.kaggle.com/c/facebook-recruiting-iii-keyword-extraction/details/evaluation

\n

 

\n

 

", "messages": [ { "id": "35003", "postDate": "11/21/2013 16:09:14", "content": "

Hi,

\n

I am not finding the passage describing the penalization of predicted tags that are ("subjectively") synonyms very clear.  Particularly, I'm not fully following point 3 in the evaluation description [1]:

\n

To receive credit, the tag you predict must be an exact match, regardless of whether you believe the tags are synonyms. Why aren't synonyms counted? ...

\n

3. Participants are equally penalized for predicting a synonym of a correct tag, so the task can be framed as not only predicting a tag, but also modeling the distribution(s) of potential synonyms

\n

 

\n

The wording, "equally penalized" is throwing me off here.  For example, if I were to predict the tag "python" for a sample where one of the actual tags was "python2.7", then just to be clear, since this is not an exact match, is it considered as wrong ("equally penalized") as a predicted tag of something that is completely arbitrary/random?  This is how I think it is too be understood, but just want to ask to be completely sure.

\n

 

\n

[1] http://www.kaggle.com/c/facebook-recruiting-iii-keyword-extraction/details/evaluation

\n

 

\n

 

", "rawMarkdown": "", "votes": null }, { "id": "35005", "postDate": "11/21/2013 16:43:08", "content": "

[quote=plessthan05;35003]

\n

For example, if I were to predict the tag "python" for a sample where one of the actual tags was "python2.7", then just to be clear, since this is not an exact match, is it considered as wrong ("equally penalized") as a predicted tag of something that is completely arbitrary/random?  This is how I think it is too be understood, but just want to ask to be completely sure.

\n

[/quote]

\n

This is correct.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 35005, "author_name": "wcukierski", "author_url": "", "post_date": "11/21/2013 16:43:08", "content": "

[quote=plessthan05;35003]

\n

For example, if I were to predict the tag "python" for a sample where one of the actual tags was "python2.7", then just to be clear, since this is not an exact match, is it considered as wrong ("equally penalized") as a predicted tag of something that is completely arbitrary/random?  This is how I think it is too be understood, but just want to ask to be completely sure.

\n

[/quote]

\n

This is correct.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "35003": "", "35005": "" }, "source": "meta" }