{
  "id": 6308,
  "title": "Evaluation Function",
  "url": "/competitions/facebook-recruiting-iii-keyword-extraction/discussion/6308",
  "author_name": "Nikhil Agarwal",
  "post_date": "2013-11-14T04:59:36.143000",
  "votes": 0,
  "comment_count": 4,
  "views": 1527,
  "content": "<p>Hi,</p>\n<p>I think I am computing the F1 score incorrectly. Can the admin's clarify the equation used to compute F1 score.</p>\n<p>This is how I have implemented it:</p>\n<p><code>def computeF1(guessSet, realSet):</code></p>\n<p><code></code><code>tp = len(guessSet.intersection(realSet))&nbsp;&nbsp; #Predicted Positives</code></p>\n<p><code>predPositive = len(guessSet)&nbsp;&nbsp; #Total Predictions</code></p>\n<p><code>actualPositive = len(realSet)&nbsp;&nbsp;&nbsp; #Total Real Keywords</code></p>\n<p><code>if tp == 0:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</code><code>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </code></p>\n<p><code>&nbsp;&nbsp;&nbsp;&nbsp; return 0&nbsp;&nbsp;&nbsp;</code><code></code></p>\n<p><code>p = tp/float(predPositive)&nbsp;&nbsp;&nbsp;</code><code></code></p>\n<p><code>r = tp/float(actualPositive)&nbsp;&nbsp;&nbsp;</code><code></code></p>\n<p><code>return 2*p*r/(p+r)</code></p>",
  "messages": [
    {
      "id": 35356,
      "postDate": "2013-11-27T15:08:04.273Z",
      "content": "<p>Follow the wiki link on the evaluation page:</p>\n<p style=\"padding-left: 30px\">For this metric, each row of the solution file represents a list of items that are &quot;correct&quot; for that row. The submission file lists the predicted items. We compute the F1 metric for each row: The &quot;true positives&quot; are the intersection of the two lists, false positives are predicted items that aren't real, and false negatives are real items that aren't predicted.</p>\n<p>It's computing the precision and recall for each row and averaging the individual F-scores. I believe this is the micro F-score?</p>",
      "votes": 1
    },
    {
      "id": 35409,
      "postDate": "2013-11-28T00:53:51.700Z",
      "content": "<p>Thanks guys, I finally managed to reproduce the eval function !</p>"
    },
    {
      "id": 35353,
      "postDate": "2013-11-27T14:50:58.377Z",
      "content": "<p>I would support this. The evaluation page is quite unclear. Would you mind give more indications on how to compute the F1-score ? (mean across each class, across each samples, macro, micro...)</p>\n<p>&nbsp;</p>\n<p>Thanks !</p>\n<p>&nbsp;</p>"
    },
    {
      "id": 33693,
      "postDate": "2013-11-14T04:59:36.143Z",
      "content": "<p>Hi,</p>\n<p>I think I am computing the F1 score incorrectly. Can the admin's clarify the equation used to compute F1 score.</p>\n<p>This is how I have implemented it:</p>\n<p><code>def computeF1(guessSet, realSet):</code></p>\n<p><code></code><code>tp = len(guessSet.intersection(realSet))&nbsp;&nbsp; #Predicted Positives</code></p>\n<p><code>predPositive = len(guessSet)&nbsp;&nbsp; #Total Predictions</code></p>\n<p><code>actualPositive = len(realSet)&nbsp;&nbsp;&nbsp; #Total Real Keywords</code></p>\n<p><code>if tp == 0:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</code><code>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </code></p>\n<p><code>&nbsp;&nbsp;&nbsp;&nbsp; return 0&nbsp;&nbsp;&nbsp;</code><code></code></p>\n<p><code>p = tp/float(predPositive)&nbsp;&nbsp;&nbsp;</code><code></code></p>\n<p><code>r = tp/float(actualPositive)&nbsp;&nbsp;&nbsp;</code><code></code></p>\n<p><code>return 2*p*r/(p+r)</code></p>"
    },
    {
      "id": 35381,
      "postDate": "2013-11-27T21:28:56.690Z",
      "rawMarkdown": "",
      "votes": 1,
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 35356,
      "author_name": "Will Cukierski",
      "author_url": "",
      "post_date": "2013-11-27T15:08:04.273000",
      "content": "<p>Follow the wiki link on the evaluation page:</p>\n<p style=\"padding-left: 30px\">For this metric, each row of the solution file represents a list of items that are &quot;correct&quot; for that row. The submission file lists the predicted items. We compute the F1 metric for each row: The &quot;true positives&quot; are the intersection of the two lists, false positives are predicted items that aren't real, and false negatives are real items that aren't predicted.</p>\n<p>It's computing the precision and recall for each row and averaging the individual F-scores. I believe this is the micro F-score?</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 35409,
      "author_name": "Lorenzo Lucido",
      "author_url": "",
      "post_date": "2013-11-28T00:53:51.700000",
      "content": "<p>Thanks guys, I finally managed to reproduce the eval function !</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 35353,
      "author_name": "Lorenzo Lucido",
      "author_url": "",
      "post_date": "2013-11-27T14:50:58.377000",
      "content": "<p>I would support this. The evaluation page is quite unclear. Would you mind give more indications on how to compute the F1-score ? (mean across each class, across each samples, macro, micro...)</p>\n<p>&nbsp;</p>\n<p>Thanks !</p>\n<p>&nbsp;</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 35381,
      "author_name": "",
      "author_url": "",
      "post_date": "2013-11-27T21:28:56.690000",
      "content": "",
      "votes": 1,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "35356": "",
    "35409": "",
    "35353": "",
    "33693": "",
    "35381": ""
  }
}