{
  "id": 21288,
  "title": "How to better parallelize xgboost? ",
  "url": "/competitions/facebook-v-predicting-check-ins/discussion/21288",
  "author_name": "",
  "post_date": "2016-05-28T21:19:24.477Z",
  "votes": null,
  "comment_count": 5,
  "views": 789,
  "content": "<p>Hi guys:</p>\n\n<p>I'm running python xgboost on a pretty hefty machine, but somehow I'm not able to fully use its power - from the &quot;top&quot; command I see only 1 core is being used (to 100%), but for the others not so much.</p>\n\n<p>I used scikit-learn's random forest module in the past on this same machine and I was able to see multiple cores being used to 100% from &quot;top&quot; command.</p>\n\n<p>Here are my parameters for xgb model:</p>\n\n<pre><code>param = {}\nparam['objective'] = 'multi:softprob' \nparam['eta'] = 0.05 \n\nparam['max_depth'] = 6 \nparam['silent'] = 1 \n\nparam['nthread'] = -1 # perhaps I set this wrong?\n\nparam['eval_metric'] = 'merror'\nparam['num_class'] = overall_num_of_classes \n</code></pre>\n\n<p>I've also attached a screenshot of the output from my &quot;top&quot; command. Thanks in advance for your kind help!</p>",
  "messages": [
    {
      "id": "121693",
      "postDate": "05/28/2016 21:19:24",
      "content": "<p>Hi guys:</p>\n\n<p>I'm running python xgboost on a pretty hefty machine, but somehow I'm not able to fully use its power - from the &quot;top&quot; command I see only 1 core is being used (to 100%), but for the others not so much.</p>\n\n<p>I used scikit-learn's random forest module in the past on this same machine and I was able to see multiple cores being used to 100% from &quot;top&quot; command.</p>\n\n<p>Here are my parameters for xgb model:</p>\n\n<pre><code>param = {}\nparam['objective'] = 'multi:softprob' \nparam['eta'] = 0.05 \n\nparam['max_depth'] = 6 \nparam['silent'] = 1 \n\nparam['nthread'] = -1 # perhaps I set this wrong?\n\nparam['eval_metric'] = 'merror'\nparam['num_class'] = overall_num_of_classes \n</code></pre>\n\n<p>I've also attached a screenshot of the output from my &quot;top&quot; command. Thanks in advance for your kind help!</p>",
      "rawMarkdown": "Hi guys:\r\n\r\nI'm running python xgboost on a pretty hefty machine, but somehow I'm not able to fully use its power - from the \"top\" command I see only 1 core is being used (to 100%), but for the others not so much.\r\n\r\nI used scikit-learn's random forest module in the past on this same machine and I was able to see multiple cores being used to 100% from \"top\" command.\r\n\r\nHere are my parameters for xgb model:\r\n\r\n    param = {}\r\n    param['objective'] = 'multi:softprob' \r\n    param['eta'] = 0.05 \r\n    \r\n    param['max_depth'] = 6 \r\n    param['silent'] = 1 \r\n    \r\n    param['nthread'] = -1 # perhaps I set this wrong?\r\n    \r\n    param['eval_metric'] = 'merror'\r\n    param['num_class'] = overall_num_of_classes \r\n\r\nI've also attached a screenshot of the output from my \"top\" command. Thanks in advance for your kind help!",
      "votes": null
    },
    {
      "id": "121706",
      "postDate": "05/28/2016 23:55:22",
      "content": "<p>[quote=jasoos;121693]</p>\n\n<p>Hi guys:</p>\n\n<p>I'm running python xgboost on a pretty hefty machine, but somehow I'm not able to fully use its power - from the &quot;top&quot; command I see only 1 core is being used (to 100%), but for the others not so much.</p>\n\n<p>I used scikit-learn's random forest module in the past on this same machine and I was able to see multiple cores being used to 100% from &quot;top&quot; command.</p>\n\n<p>Here are my parameters for xgb model:</p>\n\n<pre><code>param = {}\nparam['objective'] = 'multi:softprob' \nparam['eta'] = 0.05 \n\nparam['max_depth'] = 6 \nparam['silent'] = 1 \n\nparam['nthread'] = -1 # perhaps I set this wrong?\n\nparam['eval_metric'] = 'merror'\nparam['num_class'] = overall_num_of_classes \n</code></pre>\n\n<p>I've also attached a screenshot of the output from my &quot;top&quot; command. Thanks in advance for your kind help!</p>\n\n<p>[/quote]\nHow many classes do you have for the multi:softprob? \nIf you use it as is, one threthhold you have is to at least record a vector or probability which is not \nparallelized</p>",
      "rawMarkdown": "[quote=jasoos;121693]\r\n\r\nHi guys:\r\n\r\nI'm running python xgboost on a pretty hefty machine, but somehow I'm not able to fully use its power - from the \"top\" command I see only 1 core is being used (to 100%), but for the others not so much.\r\n\r\nI used scikit-learn's random forest module in the past on this same machine and I was able to see multiple cores being used to 100% from \"top\" command.\r\n\r\nHere are my parameters for xgb model:\r\n\r\n    param = {}\r\n    param['objective'] = 'multi:softprob' \r\n    param['eta'] = 0.05 \r\n    \r\n    param['max_depth'] = 6 \r\n    param['silent'] = 1 \r\n    \r\n    param['nthread'] = -1 # perhaps I set this wrong?\r\n    \r\n    param['eval_metric'] = 'merror'\r\n    param['num_class'] = overall_num_of_classes \r\n\r\nI've also attached a screenshot of the output from my \"top\" command. Thanks in advance for your kind help!\r\n\r\n[/quote]\r\nHow many classes do you have for the multi:softprob? \r\nIf you use it as is, one threthhold you have is to at least record a vector or probability which is not \r\nparallelized",
      "votes": null
    },
    {
      "id": "121708",
      "postDate": "05/29/2016 00:09:25",
      "content": "<p>Hey @w_________g, thanks for your post.</p>\n\n<p>I'm using the grid approach and for each each grid I have about ~700 classes/place_id within. In one particular grid which I checked, the parameter - </p>\n\n<pre><code>param['num_class'] = overall_num_of_classes # this is 660 for one of my grids\n</code></pre>\n\n<p>which uses the &quot;overall_num_of_classes&quot; variable is set to 660. Is this what you were asking for?</p>\n\n<p>Also I'm not sure whether I understand &quot;one threshold you have is to at least record a vector or probability which is not parallelized&quot; - do you mind elaborate a bit please?</p>\n\n<p>I'm just learning to use xgboost, first timer; thanks again!</p>",
      "rawMarkdown": "Hey @w_________g, thanks for your post.\r\n\r\nI'm using the grid approach and for each each grid I have about ~700 classes/place_id within. In one particular grid which I checked, the parameter - \r\n\r\n    param['num_class'] = overall_num_of_classes # this is 660 for one of my grids\r\n\r\nwhich uses the \"overall_num_of_classes\" variable is set to 660. Is this what you were asking for?\r\n\r\nAlso I'm not sure whether I understand \"one threshold you have is to at least record a vector or probability which is not parallelized\" - do you mind elaborate a bit please?\r\n\r\nI'm just learning to use xgboost, first timer; thanks again!",
      "votes": null
    },
    {
      "id": "121709",
      "postDate": "05/29/2016 00:17:29",
      "content": "<p>I remember that you have to do something differently for multi-threaded xgboost if you're an OS X user. At least for R, not sure about python.</p>",
      "rawMarkdown": "I remember that you have to do something differently for multi-threaded xgboost if you're an OS X user. At least for R, not sure about python.",
      "votes": null
    },
    {
      "id": "121711",
      "postDate": "05/29/2016 00:37:10",
      "content": "<p>Thanks @StT</p>\n\n<p>Just to clarify, I'm using Anaconda Python 2.7 on a 36 core 60G RAM machine running Amazon Linux as OS (similar to CentOS/RHEL).</p>\n\n<p>c4.8xlarge in AWS terms - it's a little expensive :) so would like to increase parallelism as much as possible.</p>",
      "rawMarkdown": "Thanks @StT\r\n\r\nJust to clarify, I'm using Anaconda Python 2.7 on a 36 core 60G RAM machine running Amazon Linux as OS (similar to CentOS/RHEL).\r\n\r\nc4.8xlarge in AWS terms - it's a little expensive :) so would like to increase parallelism as much as possible.",
      "votes": null
    },
    {
      "id": "121721",
      "postDate": "05/29/2016 02:56:20",
      "content": "<p>[quote=jasoos;121708]</p>\n\n<p>Hey @w_________g, thanks for your post.</p>\n\n<p>I'm using the grid approach and for each each grid I have about ~700 classes/place_id within. In one particular grid which I checked, the parameter - </p>\n\n<pre><code>param['num_class'] = overall_num_of_classes # this is 660 for one of my grids\n</code></pre>\n\n<p>which uses the &quot;overall_num_of_classes&quot; variable is set to 660. Is this what you were asking for?\nyes</p>\n\n<p>Also I'm not sure whether I understand &quot;one threshold you have is to at least record a vector or probability which is not parallelized&quot; - do you mind elaborate a bit please?\nI was talking about large number of classes, may not apply to the case of 660</p>\n\n<p>I'm just learning to use xgboost, first timer; thanks again!</p>\n\n<p>[/quote]</p>\n\n<p>For the grid method, you can do embarrassingly parallel by putting different blocks on different threads</p>",
      "rawMarkdown": "[quote=jasoos;121708]\r\n\r\nHey @w_________g, thanks for your post.\r\n\r\nI'm using the grid approach and for each each grid I have about ~700 classes/place_id within. In one particular grid which I checked, the parameter - \r\n\r\n    param['num_class'] = overall_num_of_classes # this is 660 for one of my grids\r\n\r\nwhich uses the \"overall_num_of_classes\" variable is set to 660. Is this what you were asking for?\r\nyes\r\n\r\nAlso I'm not sure whether I understand \"one threshold you have is to at least record a vector or probability which is not parallelized\" - do you mind elaborate a bit please?\r\nI was talking about large number of classes, may not apply to the case of 660\r\n\r\nI'm just learning to use xgboost, first timer; thanks again!\r\n\r\n[/quote]\r\n\r\nFor the grid method, you can do embarrassingly parallel by putting different blocks on different threads",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 121706,
      "author_name": "weizhezhang",
      "author_url": "",
      "post_date": "05/28/2016 23:55:22",
      "content": "<p>[quote=jasoos;121693]</p>\n\n<p>Hi guys:</p>\n\n<p>I'm running python xgboost on a pretty hefty machine, but somehow I'm not able to fully use its power - from the &quot;top&quot; command I see only 1 core is being used (to 100%), but for the others not so much.</p>\n\n<p>I used scikit-learn's random forest module in the past on this same machine and I was able to see multiple cores being used to 100% from &quot;top&quot; command.</p>\n\n<p>Here are my parameters for xgb model:</p>\n\n<pre><code>param = {}\nparam['objective'] = 'multi:softprob' \nparam['eta'] = 0.05 \n\nparam['max_depth'] = 6 \nparam['silent'] = 1 \n\nparam['nthread'] = -1 # perhaps I set this wrong?\n\nparam['eval_metric'] = 'merror'\nparam['num_class'] = overall_num_of_classes \n</code></pre>\n\n<p>I've also attached a screenshot of the output from my &quot;top&quot; command. Thanks in advance for your kind help!</p>\n\n<p>[/quote]\nHow many classes do you have for the multi:softprob? \nIf you use it as is, one threthhold you have is to at least record a vector or probability which is not \nparallelized</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 121708,
      "author_name": "howardxie",
      "author_url": "",
      "post_date": "05/29/2016 00:09:25",
      "content": "<p>Hey @w_________g, thanks for your post.</p>\n\n<p>I'm using the grid approach and for each each grid I have about ~700 classes/place_id within. In one particular grid which I checked, the parameter - </p>\n\n<pre><code>param['num_class'] = overall_num_of_classes # this is 660 for one of my grids\n</code></pre>\n\n<p>which uses the &quot;overall_num_of_classes&quot; variable is set to 660. Is this what you were asking for?</p>\n\n<p>Also I'm not sure whether I understand &quot;one threshold you have is to at least record a vector or probability which is not parallelized&quot; - do you mind elaborate a bit please?</p>\n\n<p>I'm just learning to use xgboost, first timer; thanks again!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 121709,
      "author_name": "fireiceearth",
      "author_url": "",
      "post_date": "05/29/2016 00:17:29",
      "content": "<p>I remember that you have to do something differently for multi-threaded xgboost if you're an OS X user. At least for R, not sure about python.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 121711,
      "author_name": "howardxie",
      "author_url": "",
      "post_date": "05/29/2016 00:37:10",
      "content": "<p>Thanks @StT</p>\n\n<p>Just to clarify, I'm using Anaconda Python 2.7 on a 36 core 60G RAM machine running Amazon Linux as OS (similar to CentOS/RHEL).</p>\n\n<p>c4.8xlarge in AWS terms - it's a little expensive :) so would like to increase parallelism as much as possible.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 121721,
      "author_name": "weizhezhang",
      "author_url": "",
      "post_date": "05/29/2016 02:56:20",
      "content": "<p>[quote=jasoos;121708]</p>\n\n<p>Hey @w_________g, thanks for your post.</p>\n\n<p>I'm using the grid approach and for each each grid I have about ~700 classes/place_id within. In one particular grid which I checked, the parameter - </p>\n\n<pre><code>param['num_class'] = overall_num_of_classes # this is 660 for one of my grids\n</code></pre>\n\n<p>which uses the &quot;overall_num_of_classes&quot; variable is set to 660. Is this what you were asking for?\nyes</p>\n\n<p>Also I'm not sure whether I understand &quot;one threshold you have is to at least record a vector or probability which is not parallelized&quot; - do you mind elaborate a bit please?\nI was talking about large number of classes, may not apply to the case of 660</p>\n\n<p>I'm just learning to use xgboost, first timer; thanks again!</p>\n\n<p>[/quote]</p>\n\n<p>For the grid method, you can do embarrassingly parallel by putting different blocks on different threads</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "121693": "Hi guys:\r\n\r\nI'm running python xgboost on a pretty hefty machine, but somehow I'm not able to fully use its power - from the \"top\" command I see only 1 core is being used (to 100%), but for the others not so much.\r\n\r\nI used scikit-learn's random forest module in the past on this same machine and I was able to see multiple cores being used to 100% from \"top\" command.\r\n\r\nHere are my parameters for xgb model:\r\n\r\n    param = {}\r\n    param['objective'] = 'multi:softprob' \r\n    param['eta'] = 0.05 \r\n    \r\n    param['max_depth'] = 6 \r\n    param['silent'] = 1 \r\n    \r\n    param['nthread'] = -1 # perhaps I set this wrong?\r\n    \r\n    param['eval_metric'] = 'merror'\r\n    param['num_class'] = overall_num_of_classes \r\n\r\nI've also attached a screenshot of the output from my \"top\" command. Thanks in advance for your kind help!",
    "121706": "[quote=jasoos;121693]\r\n\r\nHi guys:\r\n\r\nI'm running python xgboost on a pretty hefty machine, but somehow I'm not able to fully use its power - from the \"top\" command I see only 1 core is being used (to 100%), but for the others not so much.\r\n\r\nI used scikit-learn's random forest module in the past on this same machine and I was able to see multiple cores being used to 100% from \"top\" command.\r\n\r\nHere are my parameters for xgb model:\r\n\r\n    param = {}\r\n    param['objective'] = 'multi:softprob' \r\n    param['eta'] = 0.05 \r\n    \r\n    param['max_depth'] = 6 \r\n    param['silent'] = 1 \r\n    \r\n    param['nthread'] = -1 # perhaps I set this wrong?\r\n    \r\n    param['eval_metric'] = 'merror'\r\n    param['num_class'] = overall_num_of_classes \r\n\r\nI've also attached a screenshot of the output from my \"top\" command. Thanks in advance for your kind help!\r\n\r\n[/quote]\r\nHow many classes do you have for the multi:softprob? \r\nIf you use it as is, one threthhold you have is to at least record a vector or probability which is not \r\nparallelized",
    "121708": "Hey @w_________g, thanks for your post.\r\n\r\nI'm using the grid approach and for each each grid I have about ~700 classes/place_id within. In one particular grid which I checked, the parameter - \r\n\r\n    param['num_class'] = overall_num_of_classes # this is 660 for one of my grids\r\n\r\nwhich uses the \"overall_num_of_classes\" variable is set to 660. Is this what you were asking for?\r\n\r\nAlso I'm not sure whether I understand \"one threshold you have is to at least record a vector or probability which is not parallelized\" - do you mind elaborate a bit please?\r\n\r\nI'm just learning to use xgboost, first timer; thanks again!",
    "121709": "I remember that you have to do something differently for multi-threaded xgboost if you're an OS X user. At least for R, not sure about python.",
    "121711": "Thanks @StT\r\n\r\nJust to clarify, I'm using Anaconda Python 2.7 on a 36 core 60G RAM machine running Amazon Linux as OS (similar to CentOS/RHEL).\r\n\r\nc4.8xlarge in AWS terms - it's a little expensive :) so would like to increase parallelism as much as possible.",
    "121721": "[quote=jasoos;121708]\r\n\r\nHey @w_________g, thanks for your post.\r\n\r\nI'm using the grid approach and for each each grid I have about ~700 classes/place_id within. In one particular grid which I checked, the parameter - \r\n\r\n    param['num_class'] = overall_num_of_classes # this is 660 for one of my grids\r\n\r\nwhich uses the \"overall_num_of_classes\" variable is set to 660. Is this what you were asking for?\r\nyes\r\n\r\nAlso I'm not sure whether I understand \"one threshold you have is to at least record a vector or probability which is not parallelized\" - do you mind elaborate a bit please?\r\nI was talking about large number of classes, may not apply to the case of 660\r\n\r\nI'm just learning to use xgboost, first timer; thanks again!\r\n\r\n[/quote]\r\n\r\nFor the grid method, you can do embarrassingly parallel by putting different blocks on different threads"
  },
  "source": "meta"
}