{
  "id": 2002,
  "title": "Alternate explanation of Mean Average Precision",
  "url": "/competitions/FacebookRecruiting/discussion/2002",
  "author_name": "",
  "post_date": "2012-06-06T20:06:50.887Z",
  "votes": 55,
  "comment_count": 8,
  "views": 12257,
  "content": "<p>The&nbsp;provided&nbsp;&nbsp;<a href=\"http://www.kaggle.com/c/FacebookRecruiting/details/Evaluation\">explanation</a>&nbsp;is fine and all, but I will explain it differently here. (I would love to hear any improvements/corrections!)</p>\r\n<p>Consider one row of test.csv. You have a source_node, and you have to predict an ordered list of up to ten destination_nodes. Facebook and Kaggle know, for that source_node, some number of correct destination_nodes.</p>\r\n<p>For this row (one list of predictions) we do a sum and then divide to standardize.</p>\r\n<p>For every prediction, is it correct? If it isn't correct, you get no points for that prediction. If it is correct, you get a number of points equal to the number of correct predictions up to and including this one, divided by the position of this prediction\r\n in the list. For example:</p>\r\n<table border=\"0\">\r\n<tbody>\r\n<tr>\r\n<td><strong>Prediction&nbsp;</strong></td>\r\n<td><strong>&nbsp;Correctness&nbsp;</strong></td>\r\n<td><strong>&nbsp;Points</strong></td>\r\n</tr>\r\n<tr>\r\n<td>1</td>\r\n<td>wrong</td>\r\n<td>none</td>\r\n</tr>\r\n<tr>\r\n<td>2</td>\r\n<td>right</td>\r\n<td>1 / 2</td>\r\n</tr>\r\n<tr>\r\n<td>3</td>\r\n<td>right</td>\r\n<td>2 / 3</td>\r\n</tr>\r\n<tr>\r\n<td>4</td>\r\n<td>wrong</td>\r\n<td>none</td>\r\n</tr>\r\n<tr>\r\n<td>5</td>\r\n<td>right</td>\r\n<td>3 / 5</td>\r\n</tr>\r\n<tr>\r\n<td>6</td>\r\n<td>wrong</td>\r\n<td>none</td>\r\n</tr>\r\n<tr>\r\n<td>7</td>\r\n<td>wrong</td>\r\n<td>none</td>\r\n</tr>\r\n<tr>\r\n<td>8</td>\r\n<td>wrong</td>\r\n<td>none</td>\r\n</tr>\r\n<tr>\r\n<td>9</td>\r\n<td>right</td>\r\n<td>4 / 9</td>\r\n</tr>\r\n<tr>\r\n<td>10</td>\r\n<td>wrong</td>\r\n<td>none</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<p>Note that order matters; if the correct answers in this example had been in positions 1, 2, 3, and 4, the sum at this point would be 4.</p>\r\n<p>The number you divide by is the number of points possible. This is the lesser of ten (the most you can predict) and the number of actual correct answers that exist. For this source_node, maybe there are only four possible correct predictions. Then divide\r\n by four. This makes the maximum possible average precision for every line of the test set equal to one. So average precision is the &quot;average&quot; of the &quot;precision&quot; at every position in the list of predictions. (Oh - and if there are no correct answers or you\r\n make no predictions, then the average precision is just zero.)</p>\r\n<ul>\r\n<li>Is it bad to submit a lot of predictions for every test source_node? No. There is no harm in using all ten guesses per test source_node, even if that source_node has fewer than ten correct destination_nodes. However, since order matters, it is important\r\n to put your best guesses first. </li><li>Does every test source_node have correct destinatin_nodes? It seems likely, but is nowhere promised. It doesn't particularly matter, but if there are no correct answers, then everybody gets a zero for that row. All the test source_nodes at least exist somewhere\r\n in the training data, but not necessarily as source_nodes. </li><li>Does order really matter? Okay: if all your predictions are correct, then order doesn't matter. The only bad thing is having an incorrect prediction before a correct prediction.\r\n</li></ul>\r\n<p>The &quot;Mean&quot; in &quot;Mean Average Precision&quot; is just how all the individual (per test data set row) average precision scores get combined. Mean means mean.</p>\r\n<p>&nbsp;</p>\r\n<p>More thoughts?</p>\r\n<p>&nbsp;</p>\r\n<p>Also: my code to implement this:&nbsp;<a href=\"https://gist.github.com/2891017\">https://gist.github.com/2891017</a></p>",
  "messages": [
    {
      "id": "11430",
      "postDate": "06/06/2012 20:06:50",
      "content": "<p>The&nbsp;provided&nbsp;&nbsp;<a href=\"http://www.kaggle.com/c/FacebookRecruiting/details/Evaluation\">explanation</a>&nbsp;is fine and all, but I will explain it differently here. (I would love to hear any improvements/corrections!)</p>\r\n<p>Consider one row of test.csv. You have a source_node, and you have to predict an ordered list of up to ten destination_nodes. Facebook and Kaggle know, for that source_node, some number of correct destination_nodes.</p>\r\n<p>For this row (one list of predictions) we do a sum and then divide to standardize.</p>\r\n<p>For every prediction, is it correct? If it isn't correct, you get no points for that prediction. If it is correct, you get a number of points equal to the number of correct predictions up to and including this one, divided by the position of this prediction\r\n in the list. For example:</p>\r\n<table border=\"0\">\r\n<tbody>\r\n<tr>\r\n<td><strong>Prediction&nbsp;</strong></td>\r\n<td><strong>&nbsp;Correctness&nbsp;</strong></td>\r\n<td><strong>&nbsp;Points</strong></td>\r\n</tr>\r\n<tr>\r\n<td>1</td>\r\n<td>wrong</td>\r\n<td>none</td>\r\n</tr>\r\n<tr>\r\n<td>2</td>\r\n<td>right</td>\r\n<td>1 / 2</td>\r\n</tr>\r\n<tr>\r\n<td>3</td>\r\n<td>right</td>\r\n<td>2 / 3</td>\r\n</tr>\r\n<tr>\r\n<td>4</td>\r\n<td>wrong</td>\r\n<td>none</td>\r\n</tr>\r\n<tr>\r\n<td>5</td>\r\n<td>right</td>\r\n<td>3 / 5</td>\r\n</tr>\r\n<tr>\r\n<td>6</td>\r\n<td>wrong</td>\r\n<td>none</td>\r\n</tr>\r\n<tr>\r\n<td>7</td>\r\n<td>wrong</td>\r\n<td>none</td>\r\n</tr>\r\n<tr>\r\n<td>8</td>\r\n<td>wrong</td>\r\n<td>none</td>\r\n</tr>\r\n<tr>\r\n<td>9</td>\r\n<td>right</td>\r\n<td>4 / 9</td>\r\n</tr>\r\n<tr>\r\n<td>10</td>\r\n<td>wrong</td>\r\n<td>none</td>\r\n</tr>\r\n</tbody>\r\n</table>\r\n<p>Note that order matters; if the correct answers in this example had been in positions 1, 2, 3, and 4, the sum at this point would be 4.</p>\r\n<p>The number you divide by is the number of points possible. This is the lesser of ten (the most you can predict) and the number of actual correct answers that exist. For this source_node, maybe there are only four possible correct predictions. Then divide\r\n by four. This makes the maximum possible average precision for every line of the test set equal to one. So average precision is the &quot;average&quot; of the &quot;precision&quot; at every position in the list of predictions. (Oh - and if there are no correct answers or you\r\n make no predictions, then the average precision is just zero.)</p>\r\n<ul>\r\n<li>Is it bad to submit a lot of predictions for every test source_node? No. There is no harm in using all ten guesses per test source_node, even if that source_node has fewer than ten correct destination_nodes. However, since order matters, it is important\r\n to put your best guesses first. </li><li>Does every test source_node have correct destinatin_nodes? It seems likely, but is nowhere promised. It doesn't particularly matter, but if there are no correct answers, then everybody gets a zero for that row. All the test source_nodes at least exist somewhere\r\n in the training data, but not necessarily as source_nodes. </li><li>Does order really matter? Okay: if all your predictions are correct, then order doesn't matter. The only bad thing is having an incorrect prediction before a correct prediction.\r\n</li></ul>\r\n<p>The &quot;Mean&quot; in &quot;Mean Average Precision&quot; is just how all the individual (per test data set row) average precision scores get combined. Mean means mean.</p>\r\n<p>&nbsp;</p>\r\n<p>More thoughts?</p>\r\n<p>&nbsp;</p>\r\n<p>Also: my code to implement this:&nbsp;<a href=\"https://gist.github.com/2891017\">https://gist.github.com/2891017</a></p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "11447",
      "postDate": "06/07/2012 19:13:49",
      "content": "<p>Thanks for the intuitive explanation! I was having a bit of a hard time getting my head around this metric.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33338",
      "postDate": "11/06/2013 22:09:01",
      "content": "<p>This is a great explanation, but I have a quick question I'm hoping someone can help with. &nbsp;For users with no destination nodes (no &quot;right answers&quot;), it AP undefined? &nbsp;Thus, would those cases be excluded from the calculation of MAP?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "43009",
      "postDate": "04/25/2014 15:43:51",
      "content": "<p>Hello,</p>\n<p>Thank you for your explanation !</p>\n<p>You write &quot;Does order really matter? Okay: if all your predictions are correct, then order doesn't matter...[]&quot;</p>\n<p>Nevertheless if you have the right order it's better (like for bet for horse's race :-)). Do you know a metric used which does not give the same score if you have all the responses ordered than non ordered (but no wrong prediction) ? You could imagine performing MAP@k by evaluating precision at each step, but, at a given step, you could compare your prediction list with the given prediction ordered list at this step (not all the list, only the list known at this step). In your method, all prediction possible are known, and you compare with the whole set. The way I explain is, align the same number of item in each list (results and prediction) and compare.</p>\n\n<p>BR.</p>\n<p>PH.Simon</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "155240",
      "postDate": "01/10/2017 08:31:34",
      "content": "<p>Awesome explanation. Thanks a lot.</p>",
      "rawMarkdown": "Awesome explanation. Thanks a lot.",
      "votes": null
    },
    {
      "id": "166618",
      "postDate": "03/10/2017 11:13:22",
      "content": "<p>I am new to this and just been learning about MAP. I have been thinking about how many predictions to submit.</p>\n\n<p>Do you divide by the minimum of a) The number of actual correct answers that exist and b) The number of answers you ARE ALLOWED to submit.</p>\n\n<p>or</p>\n\n<p>Do you divide by the minimum of a) The number of actual correct answers that exist and c) The number of answers you ACTUALLY submit.</p>\n\n<p>If it were the latter, then I am imagine some game playing / optimisations where you would only submit the answers that you were really sure about.</p>\n\n<p>If the former, then you are right - nothing to be lost by submitting your whole allocation of guesses.</p>\n\n<p>Also, I don't really understand why you get a zero when there are not possibly correct answers. Seems a bit arbitrary. Why not 0.5 or 1.0?</p>",
      "rawMarkdown": "I am new to this and just been learning about MAP. I have been thinking about how many predictions to submit.\n\nDo you divide by the minimum of a) The number of actual correct answers that exist and b) The number of answers you ARE ALLOWED to submit.\n\nor\n\nDo you divide by the minimum of a) The number of actual correct answers that exist and c) The number of answers you ACTUALLY submit.\n\nIf it were the latter, then I am imagine some game playing / optimisations where you would only submit the answers that you were really sure about.\n\nIf the former, then you are right - nothing to be lost by submitting your whole allocation of guesses.\n\nAlso, I don't really understand why you get a zero when there are not possibly correct answers. Seems a bit arbitrary. Why not 0.5 or 1.0?",
      "votes": null
    },
    {
      "id": "166746",
      "postDate": "03/10/2017 21:11:55",
      "content": "<blockquote>\n  <p>the minimum of a) The number of actual correct answers that exist and\n  b) The number of answers you ARE ALLOWED to submit</p>\n</blockquote>",
      "rawMarkdown": "> the minimum of a) The number of actual correct answers that exist and\n> b) The number of answers you ARE ALLOWED to submit",
      "votes": null
    },
    {
      "id": "169860",
      "postDate": "03/23/2017 02:42:48",
      "content": "<p>Marvelous! Thanks for sharing! </p>",
      "rawMarkdown": "Marvelous! Thanks for sharing!",
      "votes": null
    },
    {
      "id": "170153",
      "postDate": "03/24/2017 10:04:07",
      "content": "<p>Thank you for great explanation.</p>",
      "rawMarkdown": "Thank you for great explanation.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 11447,
      "author_name": "randomforestfanatic",
      "author_url": "",
      "post_date": "06/07/2012 19:13:49",
      "content": "<p>Thanks for the intuitive explanation! I was having a bit of a hard time getting my head around this metric.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33338,
      "author_name": "justjoe",
      "author_url": "",
      "post_date": "11/06/2013 22:09:01",
      "content": "<p>This is a great explanation, but I have a quick question I'm hoping someone can help with. &nbsp;For users with no destination nodes (no &quot;right answers&quot;), it AP undefined? &nbsp;Thus, would those cases be excluded from the calculation of MAP?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 43009,
      "author_name": "anotherone0",
      "author_url": "",
      "post_date": "04/25/2014 15:43:51",
      "content": "<p>Hello,</p>\n<p>Thank you for your explanation !</p>\n<p>You write &quot;Does order really matter? Okay: if all your predictions are correct, then order doesn't matter...[]&quot;</p>\n<p>Nevertheless if you have the right order it's better (like for bet for horse's race :-)). Do you know a metric used which does not give the same score if you have all the responses ordered than non ordered (but no wrong prediction) ? You could imagine performing MAP@k by evaluating precision at each step, but, at a given step, you could compare your prediction list with the given prediction ordered list at this step (not all the list, only the list known at this step). In your method, all prediction possible are known, and you compare with the whole set. The way I explain is, align the same number of item in each list (results and prediction) and compare.</p>\n\n<p>BR.</p>\n<p>PH.Simon</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 155240,
      "author_name": "aknirala",
      "author_url": "",
      "post_date": "01/10/2017 08:31:34",
      "content": "<p>Awesome explanation. Thanks a lot.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 166618,
      "author_name": "jinkos",
      "author_url": "",
      "post_date": "03/10/2017 11:13:22",
      "content": "<p>I am new to this and just been learning about MAP. I have been thinking about how many predictions to submit.</p>\n\n<p>Do you divide by the minimum of a) The number of actual correct answers that exist and b) The number of answers you ARE ALLOWED to submit.</p>\n\n<p>or</p>\n\n<p>Do you divide by the minimum of a) The number of actual correct answers that exist and c) The number of answers you ACTUALLY submit.</p>\n\n<p>If it were the latter, then I am imagine some game playing / optimisations where you would only submit the answers that you were really sure about.</p>\n\n<p>If the former, then you are right - nothing to be lost by submitting your whole allocation of guesses.</p>\n\n<p>Also, I don't really understand why you get a zero when there are not possibly correct answers. Seems a bit arbitrary. Why not 0.5 or 1.0?</p>",
      "votes": null,
      "replies": [
        {
          "id": 166746,
          "author_name": "ajschumacher",
          "author_url": "",
          "post_date": "03/10/2017 21:11:55",
          "content": "<blockquote>\n  <p>the minimum of a) The number of actual correct answers that exist and\n  b) The number of answers you ARE ALLOWED to submit</p>\n</blockquote>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 169860,
      "author_name": "bitzhangxi",
      "author_url": "",
      "post_date": "03/23/2017 02:42:48",
      "content": "<p>Marvelous! Thanks for sharing! </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 170153,
      "author_name": "samlinlinlin",
      "author_url": "",
      "post_date": "03/24/2017 10:04:07",
      "content": "<p>Thank you for great explanation.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "11430": "",
    "11447": "",
    "33338": "",
    "43009": "",
    "155240": "Awesome explanation. Thanks a lot.",
    "166618": "I am new to this and just been learning about MAP. I have been thinking about how many predictions to submit.\n\nDo you divide by the minimum of a) The number of actual correct answers that exist and b) The number of answers you ARE ALLOWED to submit.\n\nor\n\nDo you divide by the minimum of a) The number of actual correct answers that exist and c) The number of answers you ACTUALLY submit.\n\nIf it were the latter, then I am imagine some game playing / optimisations where you would only submit the answers that you were really sure about.\n\nIf the former, then you are right - nothing to be lost by submitting your whole allocation of guesses.\n\nAlso, I don't really understand why you get a zero when there are not possibly correct answers. Seems a bit arbitrary. Why not 0.5 or 1.0?",
    "166746": "> the minimum of a) The number of actual correct answers that exist and\n> b) The number of answers you ARE ALLOWED to submit",
    "169860": "Marvelous! Thanks for sharing!",
    "170153": "Thank you for great explanation."
  },
  "source": "meta"
}