{
  "id": 12520,
  "title": "scikit-learn Random Forest memory problem",
  "url": "/competitions/datasciencebowl/discussion/12520",
  "author_name": "",
  "post_date": "2015-02-15T10:00:45.733Z",
  "votes": 1,
  "comment_count": 10,
  "views": 6220,
  "content": "<p>I'm running&nbsp;scikit-learn Random Forest with&nbsp;python 3.4 in windows 7 64-bit. I have this very simple model:</p>\n<p>RF = ExtraTreesClassifier(n_estimators=500)</p>\n<p>The dataset has &nbsp;30,000 x 500 elements. Even with no parallel processing, the memory usage creeps up to 16 GB within a minute! I'm wondering why there is so much memory usage.</p>\n<p>My overall code is very similar to the code in the tutorial -- I even took out the cross validations to make it &quot;leaner&quot;.</p>\n<p>Any suggestions?</p>\n<p>Thanks.&nbsp;</p>",
  "messages": [
    {
      "id": "64262",
      "postDate": "02/15/2015 10:00:45",
      "content": "<p>I'm running&nbsp;scikit-learn Random Forest with&nbsp;python 3.4 in windows 7 64-bit. I have this very simple model:</p>\n<p>RF = ExtraTreesClassifier(n_estimators=500)</p>\n<p>The dataset has &nbsp;30,000 x 500 elements. Even with no parallel processing, the memory usage creeps up to 16 GB within a minute! I'm wondering why there is so much memory usage.</p>\n<p>My overall code is very similar to the code in the tutorial -- I even took out the cross validations to make it &quot;leaner&quot;.</p>\n<p>Any suggestions?</p>\n<p>Thanks.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64264",
      "postDate": "02/15/2015 10:06:37",
      "content": "<p>which version are of scikit-learn are you using? In 0.15.2, they improved RF and it doesnt copy the dataset. try 0.15.2 or 0.16-git</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64265",
      "postDate": "02/15/2015 10:14:37",
      "content": "<p>Thanks. I'm using&nbsp;0.15.2. I too read that RF memory management was improved in this version. I'm very puzzled. The memory usage steady ramps up &nbsp;to my PC's max memory.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64374",
      "postDate": "02/17/2015 00:09:03",
      "content": "<p>Try decreasing the n_jobs value for the classifier. I read on another Kaggle forum that the memory usage was proportional to&nbsp;2*n_jobs*sizeof(X). This solution worked for me.</p>\n<p>See:<br><a href=\"https://www.kaggle.com/c/bluebook-for-bulldozers/forums/t/4091/scikit-learn-multi-core-processing/24747\">https://www.kaggle.com/c/bluebook-for-bulldozers/forums/t/4091/scikit-learn-multi-core-processing/24747<br><br></a>Edit: Just noticed that you are running without parallel processing, so my comment can safely be ignored.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64375",
      "postDate": "02/17/2015 00:16:37",
      "content": "<p>Thanks for the effort though :) I just needed to decrease&nbsp;n_estimators and control max_depth. It's kind of trial and error to find the optimum numbers for the PC's capacity. &nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "99580",
      "postDate": "11/26/2015 23:04:13",
      "content": "<p>Hi Yakamoz,</p>\n\n<p>Does the problem occur during fitting of the forest or when you use the model to predict ? </p>\n\n<p>As for me, no memory problem when I fit the forest, but when I used it to predict on a large dataset, the memory usage creeps up. I ended up slicing the dataset (using numpy &quot;vsplit&quot;), predict the result for each slice and stack the final results (using numpy vstack)</p>\n\n<p>This way I am able to use much bigger forests.</p>",
      "rawMarkdown": "Hi Yakamoz,\r\n\r\nDoes the problem occur during fitting of the forest or when you use the model to predict ? \r\n\r\nAs for me, no memory problem when I fit the forest, but when I used it to predict on a large dataset, the memory usage creeps up. I ended up slicing the dataset (using numpy \"vsplit\"), predict the result for each slice and stack the final results (using numpy vstack)\r\n\r\nThis way I am able to use much bigger forests.",
      "votes": null
    },
    {
      "id": "99825",
      "postDate": "11/30/2015 18:26:48",
      "content": "<p>Hi Thomas,</p>\n\n<p>The problem was during training. But it's fine now. At the time I was using too deep of a forest (max_depth). </p>",
      "rawMarkdown": "Hi Thomas,\r\n\r\nThe problem was during training. But it's fine now. At the time I was using too deep of a forest (max_depth).",
      "votes": null
    },
    {
      "id": "138762",
      "postDate": "10/10/2016 20:29:47",
      "content": "<p>Any advice on memory management for building a forest with a big ( say, 500,000x1000 ) dataset ?</p>",
      "rawMarkdown": "Any advice on memory management for building a forest with a big ( say, 500,000x1000 ) dataset ?",
      "votes": null
    },
    {
      "id": "184259",
      "postDate": "05/21/2017 06:10:46",
      "content": "<p>Still the same...I am getting the memory error with RF</p>",
      "rawMarkdown": "Still the same...I am getting the memory error with RF",
      "votes": null
    },
    {
      "id": "280027",
      "postDate": "02/09/2018 05:57:33",
      "content": "<p>@Yakamoz, I'm getting this issue with scikit learn 0.19, have you been ale to resolve yours?</p>",
      "rawMarkdown": "Yakamoz, I'm getting this issue with scikit learn 0.19, have you been ale to resolve yours?",
      "votes": null
    },
    {
      "id": "899677",
      "postDate": "06/24/2020 11:34:49",
      "content": "<p>The memory can skyrocket because the extra trees classifier will build a full tree. In case large and complex data, the depth can be very large. I was running the <a href=\"https://mljar.com/blog/random-forest-memory/\">experiment</a> about memory consumption for Random Forest and what helps is setting the <code>max_depth</code> of the tree. In my case, I reduced the size of the Random Forest <code>66</code> times and improve the performance by <code>4%</code> by limiting the tree depth from <code>42</code> to <code>6</code>.</p>",
      "rawMarkdown": "The memory can skyrocket because the extra trees classifier will build a full tree. In case large and complex data, the depth can be very large. I was running the [experiment](https://mljar.com/blog/random-forest-memory/) about memory consumption for Random Forest and what helps is setting the `max_depth` of the tree. In my case, I reduced the size of the Random Forest `66` times and improve the performance by `4%` by limiting the tree depth from `42` to `6`.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 64264,
      "author_name": "abhishek",
      "author_url": "",
      "post_date": "02/15/2015 10:06:37",
      "content": "<p>which version are of scikit-learn are you using? In 0.15.2, they improved RF and it doesnt copy the dataset. try 0.15.2 or 0.16-git</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64265,
      "author_name": "yakamoz",
      "author_url": "",
      "post_date": "02/15/2015 10:14:37",
      "content": "<p>Thanks. I'm using&nbsp;0.15.2. I too read that RF memory management was improved in this version. I'm very puzzled. The memory usage steady ramps up &nbsp;to my PC's max memory.&nbsp;</p>",
      "votes": null,
      "replies": [
        {
          "id": 184259,
          "author_name": "kathir16101992",
          "author_url": "",
          "post_date": "05/21/2017 06:10:46",
          "content": "<p>Still the same...I am getting the memory error with RF</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 64374,
      "author_name": "thekannman",
      "author_url": "",
      "post_date": "02/17/2015 00:09:03",
      "content": "<p>Try decreasing the n_jobs value for the classifier. I read on another Kaggle forum that the memory usage was proportional to&nbsp;2*n_jobs*sizeof(X). This solution worked for me.</p>\n<p>See:<br><a href=\"https://www.kaggle.com/c/bluebook-for-bulldozers/forums/t/4091/scikit-learn-multi-core-processing/24747\">https://www.kaggle.com/c/bluebook-for-bulldozers/forums/t/4091/scikit-learn-multi-core-processing/24747<br><br></a>Edit: Just noticed that you are running without parallel processing, so my comment can safely be ignored.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64375,
      "author_name": "yakamoz",
      "author_url": "",
      "post_date": "02/17/2015 00:16:37",
      "content": "<p>Thanks for the effort though :) I just needed to decrease&nbsp;n_estimators and control max_depth. It's kind of trial and error to find the optimum numbers for the PC's capacity. &nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 99580,
      "author_name": "tartieret",
      "author_url": "",
      "post_date": "11/26/2015 23:04:13",
      "content": "<p>Hi Yakamoz,</p>\n\n<p>Does the problem occur during fitting of the forest or when you use the model to predict ? </p>\n\n<p>As for me, no memory problem when I fit the forest, but when I used it to predict on a large dataset, the memory usage creeps up. I ended up slicing the dataset (using numpy &quot;vsplit&quot;), predict the result for each slice and stack the final results (using numpy vstack)</p>\n\n<p>This way I am able to use much bigger forests.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 99825,
      "author_name": "yakamoz",
      "author_url": "",
      "post_date": "11/30/2015 18:26:48",
      "content": "<p>Hi Thomas,</p>\n\n<p>The problem was during training. But it's fine now. At the time I was using too deep of a forest (max_depth). </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 138762,
      "author_name": "kevinroy",
      "author_url": "",
      "post_date": "10/10/2016 20:29:47",
      "content": "<p>Any advice on memory management for building a forest with a big ( say, 500,000x1000 ) dataset ?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 280027,
      "author_name": "nabiil",
      "author_url": "",
      "post_date": "02/09/2018 05:57:33",
      "content": "<p>@Yakamoz, I'm getting this issue with scikit learn 0.19, have you been ale to resolve yours?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 899677,
      "author_name": "mt77pp",
      "author_url": "",
      "post_date": "06/24/2020 11:34:49",
      "content": "<p>The memory can skyrocket because the extra trees classifier will build a full tree. In case large and complex data, the depth can be very large. I was running the <a href=\"https://mljar.com/blog/random-forest-memory/\">experiment</a> about memory consumption for Random Forest and what helps is setting the <code>max_depth</code> of the tree. In my case, I reduced the size of the Random Forest <code>66</code> times and improve the performance by <code>4%</code> by limiting the tree depth from <code>42</code> to <code>6</code>.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "64262": "",
    "64264": "",
    "64265": "",
    "64374": "",
    "64375": "",
    "99580": "Hi Yakamoz,\r\n\r\nDoes the problem occur during fitting of the forest or when you use the model to predict ? \r\n\r\nAs for me, no memory problem when I fit the forest, but when I used it to predict on a large dataset, the memory usage creeps up. I ended up slicing the dataset (using numpy \"vsplit\"), predict the result for each slice and stack the final results (using numpy vstack)\r\n\r\nThis way I am able to use much bigger forests.",
    "99825": "Hi Thomas,\r\n\r\nThe problem was during training. But it's fine now. At the time I was using too deep of a forest (max_depth).",
    "138762": "Any advice on memory management for building a forest with a big ( say, 500,000x1000 ) dataset ?",
    "184259": "Still the same...I am getting the memory error with RF",
    "280027": "Yakamoz, I'm getting this issue with scikit learn 0.19, have you been ale to resolve yours?",
    "899677": "The memory can skyrocket because the extra trees classifier will build a full tree. In case large and complex data, the depth can be very large. I was running the [experiment](https://mljar.com/blog/random-forest-memory/) about memory consumption for Random Forest and what helps is setting the `max_depth` of the tree. In my case, I reduced the size of the Random Forest `66` times and improve the performance by `4%` by limiting the tree depth from `42` to `6`."
  },
  "source": "meta"
}