{
  "id": 14426,
  "title": "Help a noob in R : ]",
  "url": "/competitions/facebook-recruiting-iv-human-or-bot/discussion/14426",
  "author_name": "",
  "post_date": "2015-05-29T02:14:21.953Z",
  "votes": null,
  "comment_count": 5,
  "views": 1230,
  "content": "<p>Hi all,</p>\n\n<p>Been working through this competition slowly (literally). I've been having some problems getting my R code streamlined and it probably takes me about 45 minutes to get just 15 random forest trees... which tbh is very frustrating when that scores about 0.7 and 50% of the people on the LB are scoring 0.9 :S</p>\n\n<p>Things I have tried:</p>\n\n<p>*Using FREAD (saw it in an earlier post and was a very good suggestion!)</p>\n\n<p>*Running multiple random forests and then combining them for a larger forest of forests (crashes my computer when I try to add my 15 trees together...)</p>\n\n<p>*Running 64 bit R and setting memory limit as high as I can...</p>\n\n<p>I had focused on just trying to feed my trees better data but maybe I need to do something different for a dataset this size?&nbsp;</p>\n\n<p>Either way, any help would be appreciated so I don't have to bang my head against the wall / wait haplessly for the next 10 days till I see final solutions.</p>\n\n<p>Thx!</p>",
  "messages": [
    {
      "id": "80328",
      "postDate": "05/29/2015 02:14:21",
      "content": "<p>Hi all,</p>\n\n<p>Been working through this competition slowly (literally). I've been having some problems getting my R code streamlined and it probably takes me about 45 minutes to get just 15 random forest trees... which tbh is very frustrating when that scores about 0.7 and 50% of the people on the LB are scoring 0.9 :S</p>\n\n<p>Things I have tried:</p>\n\n<p>*Using FREAD (saw it in an earlier post and was a very good suggestion!)</p>\n\n<p>*Running multiple random forests and then combining them for a larger forest of forests (crashes my computer when I try to add my 15 trees together...)</p>\n\n<p>*Running 64 bit R and setting memory limit as high as I can...</p>\n\n<p>I had focused on just trying to feed my trees better data but maybe I need to do something different for a dataset this size?&nbsp;</p>\n\n<p>Either way, any help would be appreciated so I don't have to bang my head against the wall / wait haplessly for the next 10 days till I see final solutions.</p>\n\n<p>Thx!</p>",
      "rawMarkdown": "Hi all,\n\nBeen working through this competition slowly (literally). I've been having some problems getting my R code streamlined and it probably takes me about 45 minutes to get just 15 random forest trees... which tbh is very frustrating when that scores about 0.7 and 50% of the people on the LB are scoring 0.9 :S\n\nThings I have tried:\n\n*Using FREAD (saw it in an earlier post and was a very good suggestion!)\n\n*Running multiple random forests and then combining them for a larger forest of forests (crashes my computer when I try to add my 15 trees together...)\n\n*Running 64 bit R and setting memory limit as high as I can...\n\nI had focused on just trying to feed my trees better data but maybe I need to do something different for a dataset this size?&nbsp;\n\nEither way, any help would be appreciated so I don't have to bang my head against the wall / wait haplessly for the next 10 days till I see final solutions.\n\nThx!",
      "votes": null
    },
    {
      "id": "80360",
      "postDate": "05/29/2015 14:04:54",
      "content": "<p>What are the specs on your computer? &nbsp;If they're pretty low and you're having difficulty coding a workaround there is always the option of using an Rstudio AMI (&nbsp;http://www.louisaslett.com/RStudio_AMI/ ) which you can use to run R code on EC2 on amazon web services to get more computing power and/or memory.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "80420",
      "postDate": "05/29/2015 22:04:17",
      "content": "<p>It sounds like you might benefit from reducing the number of nodes in each tree.&nbsp; In the randomForest package this can be set with the &quot;maxnodes&quot; option.&nbsp; I'd recommend starting with something small, say 20, and working up from there.</p>\n<p>Another consideration is whether you actually need to combine the forests.&nbsp; If you only need the predictions, then perhaps scoring them individually and then averaging the resulting scores would be sufficient.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "80476",
      "postDate": "05/30/2015 20:35:59",
      "content": "<p>You first have to figure out what part of your code is taking so long. Loading the data from disk? Computing the features? Actually fitting the random forest? Find your bottleneck first and then you have a clear target for optimization.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "80486",
      "postDate": "05/31/2015 00:24:05",
      "content": "<p>I have the equivalent of an i3 processor and 8 gb of ram.&nbsp;</p>\n<p>Loading the &quot;bids&quot; using FREAD takes me about 50 seconds.</p>\n<p>Merging bids with train is in the order of 3-5 minutes.</p>\n<p>Running a RF on all the data takes a really long time, like 40 minutes, and I only can't crank that many trees out of it.</p>\n<p>I was able to speed up my process a bit by NULLing all the datasets I wasn't using and trying to close everything except for R for the runs.&nbsp;</p>\n<p>I think I saw that running a production code is faster out of R Studio so when I start to get a model that tests well maybe I will try to move it into base R...?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "80500",
      "postDate": "05/31/2015 04:03:59",
      "content": "<p>You might want to try the caret implementation which can easily parallalize (with doSNOW or simillar packages) while training the data for randomforest. This package has a dedicated website and here it is http://topepo.github.io/caret/index.html . Give it a go if you have not and see if this improves the run-time.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 80360,
      "author_name": "jessekolb",
      "author_url": "",
      "post_date": "05/29/2015 14:04:54",
      "content": "<p>What are the specs on your computer? &nbsp;If they're pretty low and you're having difficulty coding a workaround there is always the option of using an Rstudio AMI (&nbsp;http://www.louisaslett.com/RStudio_AMI/ ) which you can use to run R code on EC2 on amazon web services to get more computing power and/or memory.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 80420,
      "author_name": "speculation",
      "author_url": "",
      "post_date": "05/29/2015 22:04:17",
      "content": "<p>It sounds like you might benefit from reducing the number of nodes in each tree.&nbsp; In the randomForest package this can be set with the &quot;maxnodes&quot; option.&nbsp; I'd recommend starting with something small, say 20, and working up from there.</p>\n<p>Another consideration is whether you actually need to combine the forests.&nbsp; If you only need the predictions, then perhaps scoring them individually and then averaging the resulting scores would be sufficient.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 80476,
      "author_name": "cfhlearns",
      "author_url": "",
      "post_date": "05/30/2015 20:35:59",
      "content": "<p>You first have to figure out what part of your code is taking so long. Loading the data from disk? Computing the features? Actually fitting the random forest? Find your bottleneck first and then you have a clear target for optimization.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 80486,
      "author_name": "bruttoblotto",
      "author_url": "",
      "post_date": "05/31/2015 00:24:05",
      "content": "<p>I have the equivalent of an i3 processor and 8 gb of ram.&nbsp;</p>\n<p>Loading the &quot;bids&quot; using FREAD takes me about 50 seconds.</p>\n<p>Merging bids with train is in the order of 3-5 minutes.</p>\n<p>Running a RF on all the data takes a really long time, like 40 minutes, and I only can't crank that many trees out of it.</p>\n<p>I was able to speed up my process a bit by NULLing all the datasets I wasn't using and trying to close everything except for R for the runs.&nbsp;</p>\n<p>I think I saw that running a production code is faster out of R Studio so when I start to get a model that tests well maybe I will try to move it into base R...?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 80500,
      "author_name": "daretoshare",
      "author_url": "",
      "post_date": "05/31/2015 04:03:59",
      "content": "<p>You might want to try the caret implementation which can easily parallalize (with doSNOW or simillar packages) while training the data for randomforest. This package has a dedicated website and here it is http://topepo.github.io/caret/index.html . Give it a go if you have not and see if this improves the run-time.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "80328": "Hi all,\n\nBeen working through this competition slowly (literally). I've been having some problems getting my R code streamlined and it probably takes me about 45 minutes to get just 15 random forest trees... which tbh is very frustrating when that scores about 0.7 and 50% of the people on the LB are scoring 0.9 :S\n\nThings I have tried:\n\n*Using FREAD (saw it in an earlier post and was a very good suggestion!)\n\n*Running multiple random forests and then combining them for a larger forest of forests (crashes my computer when I try to add my 15 trees together...)\n\n*Running 64 bit R and setting memory limit as high as I can...\n\nI had focused on just trying to feed my trees better data but maybe I need to do something different for a dataset this size?&nbsp;\n\nEither way, any help would be appreciated so I don't have to bang my head against the wall / wait haplessly for the next 10 days till I see final solutions.\n\nThx!",
    "80360": "",
    "80420": "",
    "80476": "",
    "80486": "",
    "80500": ""
  },
  "source": "meta"
}