{
  "id": 3857,
  "title": "Evaluating solutions in the final phase",
  "url": "/competitions/event-recommendation-engine-challenge/discussion/3857",
  "author_name": "",
  "post_date": "2013-02-17T15:11:51.027Z",
  "votes": null,
  "comment_count": 1,
  "views": 1741,
  "content": "<p>Hi,</p>\r\n<p>I am just wondering what the best strategy is to evaluate solutions now that we don't have access to the leaderboard. I split the training data in training and validation sets (possibly using cross-validation), with no overlap in the users between the two\r\n sets.</p>\r\n<p><span style=\"line-height:1.4em\">I evaluate the error using:&nbsp;</span><span style=\"line-height:1.4em\">mapk(k=120, actual=liked_events, predicted=recommended_events) where:</span></p>\r\n<ul>\r\n<li><span style=\"line-height:1.4em\">liked_events is a list of vectors of events that users in the validation set were interested.</span>\r\n</li><li><span style=\"line-height:1.4em\">recommended_events is a list of all the events user visited, sorted&nbsp;&nbsp;from those the user would be most-interested in to those the user would be least-interested in.&nbsp;</span>\r\n</li><li><span style=\"line-height:1.4em\">mapk is the R function&nbsp;<a href=\"https://github.com/benhamner/Metrics/blob/master/R/R/metrics.r#L145\">mean average precision at k</a></span>\r\n</li></ul>\r\n<p>Is this correct?</p>",
  "messages": [
    {
      "id": "20550",
      "postDate": "02/17/2013 15:11:51",
      "content": "<p>Hi,</p>\r\n<p>I am just wondering what the best strategy is to evaluate solutions now that we don't have access to the leaderboard. I split the training data in training and validation sets (possibly using cross-validation), with no overlap in the users between the two\r\n sets.</p>\r\n<p><span style=\"line-height:1.4em\">I evaluate the error using:&nbsp;</span><span style=\"line-height:1.4em\">mapk(k=120, actual=liked_events, predicted=recommended_events) where:</span></p>\r\n<ul>\r\n<li><span style=\"line-height:1.4em\">liked_events is a list of vectors of events that users in the validation set were interested.</span>\r\n</li><li><span style=\"line-height:1.4em\">recommended_events is a list of all the events user visited, sorted&nbsp;&nbsp;from those the user would be most-interested in to those the user would be least-interested in.&nbsp;</span>\r\n</li><li><span style=\"line-height:1.4em\">mapk is the R function&nbsp;<a href=\"https://github.com/benhamner/Metrics/blob/master/R/R/metrics.r#L145\">mean average precision at k</a></span>\r\n</li></ul>\r\n<p>Is this correct?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "20629",
      "postDate": "02/20/2013 14:38:15",
      "content": "<p>Sorry, at the time I wrote this, I didn't realize that the ground truth data for the public leaderboard was released, which makes things easier.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 20629,
      "author_name": "ayy69366",
      "author_url": "",
      "post_date": "02/20/2013 14:38:15",
      "content": "<p>Sorry, at the time I wrote this, I didn't realize that the ground truth data for the public leaderboard was released, which makes things easier.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "20550": "",
    "20629": ""
  },
  "source": "meta"
}