{
  "id": 4486,
  "title": "Reproducibility Error",
  "url": "/competitions/cause-effect-pairs/discussion/4486",
  "author_name": "",
  "post_date": "2013-05-02T15:26:18.990Z",
  "votes": null,
  "comment_count": 2,
  "views": 1027,
  "content": "<p>Hi, I have a strange problem so I thought I'd share. I had written an R script with a GBM, a GLM and a Random Forest. Random Forest always produced the best results 88.406 on leaderboard. I thought I'd best tidy up my code so I removed the GBM and GLM from\r\n the script. The random seed has been set the same for both scripts. However, when I now run the script with just the Random Forest, my score is 88.385. I've looked at the variables set for GBM and GLM (ntree, minobs, depth, lambda) and I can't see a clash\r\n in the wording with anything from Random Forest. I know it's not a big change and for the competition, it doesn't really matter - but I was wondering what it could be. Any ideas?</p>\r\n<p>&nbsp;</p>\r\n<p>EDIT: And for the first training tree, the MSE difference is 2%</p>",
  "messages": [
    {
      "id": "23814",
      "postDate": "05/02/2013 15:26:18",
      "content": "<p>Hi, I have a strange problem so I thought I'd share. I had written an R script with a GBM, a GLM and a Random Forest. Random Forest always produced the best results 88.406 on leaderboard. I thought I'd best tidy up my code so I removed the GBM and GLM from\r\n the script. The random seed has been set the same for both scripts. However, when I now run the script with just the Random Forest, my score is 88.385. I've looked at the variables set for GBM and GLM (ntree, minobs, depth, lambda) and I can't see a clash\r\n in the wording with anything from Random Forest. I know it's not a big change and for the competition, it doesn't really matter - but I was wondering what it could be. Any ideas?</p>\r\n<p>&nbsp;</p>\r\n<p>EDIT: And for the first training tree, the MSE difference is 2%</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "23815",
      "postDate": "05/02/2013 15:37:53",
      "content": "<p><span style=\"line-height:1.4em\">I'd guess the removed sections used the random generator, so the Random Forest not longer starts from the very same seed.</span></p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "23816",
      "postDate": "05/02/2013 15:44:05",
      "content": "<p>yes - that's what I thought. It must reset the seed the same way though as I can reproduce the results for each individual script. It must leave the seed different - haven't come across this before as usually averages work the best.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 23815,
      "author_name": "jay670790",
      "author_url": "",
      "post_date": "05/02/2013 15:37:53",
      "content": "<p><span style=\"line-height:1.4em\">I'd guess the removed sections used the random generator, so the Random Forest not longer starts from the very same seed.</span></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 23816,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "05/02/2013 15:44:05",
      "content": "<p>yes - that's what I thought. It must reset the seed the same way though as I can reproduce the results for each individual script. It must leave the seed different - haven't come across this before as usually averages work the best.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "23814": "",
    "23815": "",
    "23816": ""
  },
  "source": "meta"
}