{
  "id": 6960,
  "title": "Training Set Generalization?",
  "url": "/competitions/galaxy-zoo-the-galaxy-challenge/discussion/6960",
  "author_name": "",
  "post_date": "2014-01-29T09:21:46.187Z",
  "votes": null,
  "comment_count": 2,
  "views": 1242,
  "content": "<p>Has anyone else had problems with the performance on the held-out training set data differing widely from the test set? &nbsp;I've been training on 40k images and evaluating on 20k, then making test set predictions that are dramatically worse. &nbsp;The models aren't particularly good, but they should still beat the central pixel benchmark (random forest with a grid of spatial histograms). &nbsp;RMSE on the 20k holdout is 0.1, but 0.18 on the test set.</p>\n<p>&nbsp;</p>\n<p>Is there something obvious that I'm missing? &nbsp;Any advice would be appreciated.</p>",
  "messages": [
    {
      "id": "38086",
      "postDate": "01/29/2014 09:21:46",
      "content": "<p>Has anyone else had problems with the performance on the held-out training set data differing widely from the test set? &nbsp;I've been training on 40k images and evaluating on 20k, then making test set predictions that are dramatically worse. &nbsp;The models aren't particularly good, but they should still beat the central pixel benchmark (random forest with a grid of spatial histograms). &nbsp;RMSE on the 20k holdout is 0.1, but 0.18 on the test set.</p>\n<p>&nbsp;</p>\n<p>Is there something obvious that I'm missing? &nbsp;Any advice would be appreciated.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "38118",
      "postDate": "01/29/2014 19:52:59",
      "content": "<p>You probably have a bug somewhere. I see differences of 0.001 between my submission results and my validation set with known predictions.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "38149",
      "postDate": "01/30/2014 07:40:24",
      "content": "<p>Haha yeah it was really bad - turns out I was running a completely different registration flow for my prediction function. &nbsp;:( but now I don't suck at least.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 38118,
      "author_name": "delcacho",
      "author_url": "",
      "post_date": "01/29/2014 19:52:59",
      "content": "<p>You probably have a bug somewhere. I see differences of 0.001 between my submission results and my validation set with known predictions.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 38149,
      "author_name": "elliotkulakow",
      "author_url": "",
      "post_date": "01/30/2014 07:40:24",
      "content": "<p>Haha yeah it was really bad - turns out I was running a completely different registration flow for my prediction function. &nbsp;:( but now I don't suck at least.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "38086": "",
    "38118": "",
    "38149": ""
  },
  "source": "meta"
}