{
  "id": 22736,
  "title": "Xgboost very slow",
  "url": "/competitions/grupo-bimbo-inventory-demand/discussion/22736",
  "author_name": "",
  "post_date": "2016-08-07T05:44:45.810Z",
  "votes": 1,
  "comment_count": 8,
  "views": 1448,
  "content": "<p>Hi,</p>\n\n<p>I am using xgboost to train a linear regression model with following parameter, I have about 7 million samples with ~20 features. My machine has 32 GB Ram with Octa core. It took about 4.5 hours to get a model. This is much longer than some benchmarks i found online. </p>\n\n<p>{'subsample': 0.85, 'silent': False, 'max_depth': 10, 'n_estimators': 75, 'objective': 'reg:linear'} </p>\n\n<p>Are there any tricks to make xgboost fast ? I used python wrapper of xboost. </p>",
  "messages": [
    {
      "id": "130447",
      "postDate": "08/07/2016 05:44:45",
      "content": "<p>Hi,</p>\n\n<p>I am using xgboost to train a linear regression model with following parameter, I have about 7 million samples with ~20 features. My machine has 32 GB Ram with Octa core. It took about 4.5 hours to get a model. This is much longer than some benchmarks i found online. </p>\n\n<p>{'subsample': 0.85, 'silent': False, 'max_depth': 10, 'n_estimators': 75, 'objective': 'reg:linear'} </p>\n\n<p>Are there any tricks to make xgboost fast ? I used python wrapper of xboost. </p>",
      "rawMarkdown": "Hi,\r\n\r\nI am using xgboost to train a linear regression model with following parameter, I have about 7 million samples with ~20 features. My machine has 32 GB Ram with Octa core. It took about 4.5 hours to get a model. This is much longer than some benchmarks i found online. \r\n\r\n\r\n\r\n{'subsample': 0.85, 'silent': False, 'max_depth': 10, 'n_estimators': 75, 'objective': 'reg:linear'} \r\n\r\nAre there any tricks to make xgboost fast ? I used python wrapper of xboost.",
      "votes": null
    },
    {
      "id": "130461",
      "postDate": "08/07/2016 10:10:07",
      "content": "<p>Is your xgboost running multi-core? That's where I think may be causing the slowness.</p>",
      "rawMarkdown": "Is your xgboost running multi-core? That's where I think may be causing the slowness.",
      "votes": null
    },
    {
      "id": "130502",
      "postDate": "08/07/2016 20:21:35",
      "content": "<p>In case you are using python and XGB's sklearn api, earlier I made the mistake of using XGBClassifier instead of XGBRegressor. Both will run, but XGBRegressor is a lot more faster in this task. </p>",
      "rawMarkdown": "In case you are using python and XGB's sklearn api, earlier I made the mistake of using XGBClassifier instead of XGBRegressor. Both will run, but XGBRegressor is a lot more faster in this task.",
      "votes": null
    },
    {
      "id": "130503",
      "postDate": "08/07/2016 20:27:26",
      "content": "<p>I used XGBRegressor and didn't turn off any default parameters about number of threads. I could see the process having more than 20 threads and eating up all my 8 cores to maximum. </p>\n\n<p>Can anyone give me benchmark of how long does it take for training XGboost for a specific parameters,  number of training example and number of features.  So that I can compare mine. </p>",
      "rawMarkdown": "I used XGBRegressor and didn't turn off any default parameters about number of threads. I could see the process having more than 20 threads and eating up all my 8 cores to maximum. \r\n\r\nCan anyone give me benchmark of how long does it take for training XGboost for a specific parameters,  number of training example and number of features.  So that I can compare mine.",
      "votes": null
    },
    {
      "id": "130504",
      "postDate": "08/07/2016 20:52:02",
      "content": "<p>I am getting by with the following simple setting:</p>\n\n<pre><code>xgb.XGBRegressor(max_depth=8, learning_rate=0.30, n_estimators=15,\n                 objective=&quot;reg:linear&quot;, subsample=0.5,\n                 colsample_bytree=0.8, seed=1234)\n</code></pre>\n\n<p>As a benchmark,  I run this with 25 features and weeks 8 and 9 as training data, and the modelling (exclude feature engineering) takes 9 minutes. </p>\n\n<p>I am on i7 4 core (8 thread) and 16GB of RAM </p>\n\n<p>Regards\nYifan</p>",
      "rawMarkdown": "I am getting by with the following simple setting:\r\n\r\n    xgb.XGBRegressor(max_depth=8, learning_rate=0.30, n_estimators=15,\r\n                     objective=\"reg:linear\", subsample=0.5,\r\n                     colsample_bytree=0.8, seed=1234)\r\n\r\nAs a benchmark,  I run this with 25 features and weeks 8 and 9 as training data, and the modelling (exclude feature engineering) takes 9 minutes. \r\n\r\nI am on i7 4 core (8 thread) and 16GB of RAM \r\n\r\nRegards\r\nYifan",
      "votes": null
    },
    {
      "id": "130576",
      "postDate": "08/08/2016 17:05:43",
      "content": "<p>As Yifan's call points out, a higher eta/learning_rate makes things a lot faster.  I'm using .1 myself.</p>",
      "rawMarkdown": "As Yifan's call points out, a higher eta/learning_rate makes things a lot faster.  I'm using .1 myself.",
      "votes": null
    },
    {
      "id": "130582",
      "postDate": "08/08/2016 17:56:54",
      "content": "<p>I still don't have any clue.</p>\n\n<p>For the following set of parameters (very similar to Yifan) my training takes more than 1 hour. Each iteration takes about 3-4 mins.</p>\n\n<pre><code>p['learning_rate'] = [0.3]\np['max_depth'] = [8]\np['n_estimators'] = [30]\np['objective'] = ['reg:linear']\np['colsample_bytree'] = [0.7]\np['silent'] = [False]\np['subsample'] = [0.8]\n</code></pre>\n\n<p>Some verbose information</p>\n\n<pre><code>[10:46:33] E:\\Git\\xgboost\\src\\tree\\updater_prune.cc:74: tree pruning end, 1 roots, 382 extra nodes, 0 pruned nodes, max_depth=8\n[10:49:05] E:\\Git\\xgboost\\src\\tree\\updater_prune.cc:74: tree pruning end, 1 roots, 432 extra nodes, 0 pruned nodes, max_depth=8\n[10:51:26] E:\\Git\\xgboost\\src\\tree\\updater_prune.cc:74: tree pruning end, 1 roots, 456 extra nodes, 0 pruned nodes, max_depth=8\n[10:54:04] E:\\Git\\xgboost\\src\\tree\\updater_prune.cc:74: tree pruning end, 1 roots, 496 extra nodes, 0 pruned nodes, max_depth=8\n</code></pre>\n\n<p>All of the 8 cores are running 100%. Appreciate any help. </p>",
      "rawMarkdown": "I still don't have any clue.\r\n\r\nFor the following set of parameters (very similar to Yifan) my training takes more than 1 hour. Each iteration takes about 3-4 mins.\r\n\r\n    p['learning_rate'] = [0.3]\r\n    p['max_depth'] = [8]\r\n    p['n_estimators'] = [30]\r\n    p['objective'] = ['reg:linear']\r\n    p['colsample_bytree'] = [0.7]\r\n    p['silent'] = [False]\r\n    p['subsample'] = [0.8]\r\n\r\nSome verbose information\r\n\r\n    [10:46:33] E:\\Git\\xgboost\\src\\tree\\updater_prune.cc:74: tree pruning end, 1 roots, 382 extra nodes, 0 pruned nodes, max_depth=8\r\n    [10:49:05] E:\\Git\\xgboost\\src\\tree\\updater_prune.cc:74: tree pruning end, 1 roots, 432 extra nodes, 0 pruned nodes, max_depth=8\r\n    [10:51:26] E:\\Git\\xgboost\\src\\tree\\updater_prune.cc:74: tree pruning end, 1 roots, 456 extra nodes, 0 pruned nodes, max_depth=8\r\n    [10:54:04] E:\\Git\\xgboost\\src\\tree\\updater_prune.cc:74: tree pruning end, 1 roots, 496 extra nodes, 0 pruned nodes, max_depth=8\r\n\r\nAll of the 8 cores are running 100%. Appreciate any help.",
      "votes": null
    },
    {
      "id": "130584",
      "postDate": "08/08/2016 18:34:03",
      "content": "<p>I found the problem. I build the library in debug mode instead of release mode. Now it is really fast :) </p>\n\n<p>Thanks everyone. </p>",
      "rawMarkdown": "I found the problem. I build the library in debug mode instead of release mode. Now it is really fast :) \r\n\r\nThanks everyone.",
      "votes": null
    },
    {
      "id": "652115",
      "postDate": "10/18/2019 11:32:52",
      "content": "<p>Hi can you please explain what is this debug mode and release mode? I am running a XGBClassifier and it is taking ~4 hours to train the model.</p>",
      "rawMarkdown": "Hi can you please explain what is this debug mode and release mode? I am running a XGBClassifier and it is taking ~4 hours to train the model.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 130461,
      "author_name": "qwang88",
      "author_url": "",
      "post_date": "08/07/2016 10:10:07",
      "content": "<p>Is your xgboost running multi-core? That's where I think may be causing the slowness.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 130502,
      "author_name": "yifanxie",
      "author_url": "",
      "post_date": "08/07/2016 20:21:35",
      "content": "<p>In case you are using python and XGB's sklearn api, earlier I made the mistake of using XGBClassifier instead of XGBRegressor. Both will run, but XGBRegressor is a lot more faster in this task. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 130503,
      "author_name": "kumaran5555",
      "author_url": "",
      "post_date": "08/07/2016 20:27:26",
      "content": "<p>I used XGBRegressor and didn't turn off any default parameters about number of threads. I could see the process having more than 20 threads and eating up all my 8 cores to maximum. </p>\n\n<p>Can anyone give me benchmark of how long does it take for training XGboost for a specific parameters,  number of training example and number of features.  So that I can compare mine. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 130504,
      "author_name": "yifanxie",
      "author_url": "",
      "post_date": "08/07/2016 20:52:02",
      "content": "<p>I am getting by with the following simple setting:</p>\n\n<pre><code>xgb.XGBRegressor(max_depth=8, learning_rate=0.30, n_estimators=15,\n                 objective=&quot;reg:linear&quot;, subsample=0.5,\n                 colsample_bytree=0.8, seed=1234)\n</code></pre>\n\n<p>As a benchmark,  I run this with 25 features and weeks 8 and 9 as training data, and the modelling (exclude feature engineering) takes 9 minutes. </p>\n\n<p>I am on i7 4 core (8 thread) and 16GB of RAM </p>\n\n<p>Regards\nYifan</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 130576,
      "author_name": "happycube",
      "author_url": "",
      "post_date": "08/08/2016 17:05:43",
      "content": "<p>As Yifan's call points out, a higher eta/learning_rate makes things a lot faster.  I'm using .1 myself.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 130582,
      "author_name": "kumaran5555",
      "author_url": "",
      "post_date": "08/08/2016 17:56:54",
      "content": "<p>I still don't have any clue.</p>\n\n<p>For the following set of parameters (very similar to Yifan) my training takes more than 1 hour. Each iteration takes about 3-4 mins.</p>\n\n<pre><code>p['learning_rate'] = [0.3]\np['max_depth'] = [8]\np['n_estimators'] = [30]\np['objective'] = ['reg:linear']\np['colsample_bytree'] = [0.7]\np['silent'] = [False]\np['subsample'] = [0.8]\n</code></pre>\n\n<p>Some verbose information</p>\n\n<pre><code>[10:46:33] E:\\Git\\xgboost\\src\\tree\\updater_prune.cc:74: tree pruning end, 1 roots, 382 extra nodes, 0 pruned nodes, max_depth=8\n[10:49:05] E:\\Git\\xgboost\\src\\tree\\updater_prune.cc:74: tree pruning end, 1 roots, 432 extra nodes, 0 pruned nodes, max_depth=8\n[10:51:26] E:\\Git\\xgboost\\src\\tree\\updater_prune.cc:74: tree pruning end, 1 roots, 456 extra nodes, 0 pruned nodes, max_depth=8\n[10:54:04] E:\\Git\\xgboost\\src\\tree\\updater_prune.cc:74: tree pruning end, 1 roots, 496 extra nodes, 0 pruned nodes, max_depth=8\n</code></pre>\n\n<p>All of the 8 cores are running 100%. Appreciate any help. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 130584,
      "author_name": "kumaran5555",
      "author_url": "",
      "post_date": "08/08/2016 18:34:03",
      "content": "<p>I found the problem. I build the library in debug mode instead of release mode. Now it is really fast :) </p>\n\n<p>Thanks everyone. </p>",
      "votes": null,
      "replies": [
        {
          "id": 652115,
          "author_name": "yash912",
          "author_url": "",
          "post_date": "10/18/2019 11:32:52",
          "content": "<p>Hi can you please explain what is this debug mode and release mode? I am running a XGBClassifier and it is taking ~4 hours to train the model.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "130447": "Hi,\r\n\r\nI am using xgboost to train a linear regression model with following parameter, I have about 7 million samples with ~20 features. My machine has 32 GB Ram with Octa core. It took about 4.5 hours to get a model. This is much longer than some benchmarks i found online. \r\n\r\n\r\n\r\n{'subsample': 0.85, 'silent': False, 'max_depth': 10, 'n_estimators': 75, 'objective': 'reg:linear'} \r\n\r\nAre there any tricks to make xgboost fast ? I used python wrapper of xboost.",
    "130461": "Is your xgboost running multi-core? That's where I think may be causing the slowness.",
    "130502": "In case you are using python and XGB's sklearn api, earlier I made the mistake of using XGBClassifier instead of XGBRegressor. Both will run, but XGBRegressor is a lot more faster in this task.",
    "130503": "I used XGBRegressor and didn't turn off any default parameters about number of threads. I could see the process having more than 20 threads and eating up all my 8 cores to maximum. \r\n\r\nCan anyone give me benchmark of how long does it take for training XGboost for a specific parameters,  number of training example and number of features.  So that I can compare mine.",
    "130504": "I am getting by with the following simple setting:\r\n\r\n    xgb.XGBRegressor(max_depth=8, learning_rate=0.30, n_estimators=15,\r\n                     objective=\"reg:linear\", subsample=0.5,\r\n                     colsample_bytree=0.8, seed=1234)\r\n\r\nAs a benchmark,  I run this with 25 features and weeks 8 and 9 as training data, and the modelling (exclude feature engineering) takes 9 minutes. \r\n\r\nI am on i7 4 core (8 thread) and 16GB of RAM \r\n\r\nRegards\r\nYifan",
    "130576": "As Yifan's call points out, a higher eta/learning_rate makes things a lot faster.  I'm using .1 myself.",
    "130582": "I still don't have any clue.\r\n\r\nFor the following set of parameters (very similar to Yifan) my training takes more than 1 hour. Each iteration takes about 3-4 mins.\r\n\r\n    p['learning_rate'] = [0.3]\r\n    p['max_depth'] = [8]\r\n    p['n_estimators'] = [30]\r\n    p['objective'] = ['reg:linear']\r\n    p['colsample_bytree'] = [0.7]\r\n    p['silent'] = [False]\r\n    p['subsample'] = [0.8]\r\n\r\nSome verbose information\r\n\r\n    [10:46:33] E:\\Git\\xgboost\\src\\tree\\updater_prune.cc:74: tree pruning end, 1 roots, 382 extra nodes, 0 pruned nodes, max_depth=8\r\n    [10:49:05] E:\\Git\\xgboost\\src\\tree\\updater_prune.cc:74: tree pruning end, 1 roots, 432 extra nodes, 0 pruned nodes, max_depth=8\r\n    [10:51:26] E:\\Git\\xgboost\\src\\tree\\updater_prune.cc:74: tree pruning end, 1 roots, 456 extra nodes, 0 pruned nodes, max_depth=8\r\n    [10:54:04] E:\\Git\\xgboost\\src\\tree\\updater_prune.cc:74: tree pruning end, 1 roots, 496 extra nodes, 0 pruned nodes, max_depth=8\r\n\r\nAll of the 8 cores are running 100%. Appreciate any help.",
    "130584": "I found the problem. I build the library in debug mode instead of release mode. Now it is really fast :) \r\n\r\nThanks everyone.",
    "652115": "Hi can you please explain what is this debug mode and release mode? I am running a XGBClassifier and it is taking ~4 hours to train the model."
  },
  "source": "meta"
}