{ "id": 23170, "title": "Tuning XGBoost with Caret", "url": "/competitions/grupo-bimbo-inventory-demand/discussion/23170", "author_name": "", "post_date": "2016-08-30T03:09:08.980Z", "votes": null, "comment_count": 8, "views": 3566, "content": "

Hello All,

\n\n

I'm trying to tune XGBoost with the code below. When I do this, the following error shows up.

\n\n
\n

"Error in train.default(x = data.matrix(data_train[wltst, features,\n with = FALSE]), : Metric RMSE not applicable for classification\n models In addition: Warning messages: 1: In\n data.matrix(data_train[wltst, features, with = FALSE]) : NAs\n introduced by coercion 2: In data.matrix(data_train[wltst, features,\n with = FALSE]) : NAs introduced by coercion 3: In\n data.matrix(data_train[wltst, features, with = FALSE]) : NAs\n introduced by coercion"

\n
\n\n

Any help figuring this out is much appreciated.

\n\n###########Code Begins Below\n\n
# set up the cross-validated hyper-parameter search\nxgb_grid_1 = expand.grid(\n  nrounds = 2, \n  max_depth = c(5, 10, 15), \n  eta = c(0.01, 0.001, 0.0001), \n  gamma = c(1, 2, 3), \n  colsample_bytree = c(0.4, 0.7, 1.0), \n  min_child_weight = c(0.5, 1, 1.5)\n)\n\n# pack the training control parameters\nxgb_trcontrol_1 = trainControl(\n  method = "cv",\n  number = 3,  \n  allowParallel = TRUE\n)\n\n\n# train the model for each parameter combination in the grid, \n#   using CV to evaluate\nxgb_train_1 = train(\n  x = data.matrix(data_train[wltst,features,with=FALSE]),\n  y = data.matrix(data_train[wltst,target]),\n  trControl = xgb_trcontrol_1,\n  tuneGrid = xgb_grid_1,\n  method = "xgbLinear"\n)\n
", "messages": [ { "id": "132870", "postDate": "08/30/2016 03:09:08", "content": "

Hello All,

\n\n

I'm trying to tune XGBoost with the code below. When I do this, the following error shows up.

\n\n
\n

"Error in train.default(x = data.matrix(data_train[wltst, features,\n with = FALSE]), : Metric RMSE not applicable for classification\n models In addition: Warning messages: 1: In\n data.matrix(data_train[wltst, features, with = FALSE]) : NAs\n introduced by coercion 2: In data.matrix(data_train[wltst, features,\n with = FALSE]) : NAs introduced by coercion 3: In\n data.matrix(data_train[wltst, features, with = FALSE]) : NAs\n introduced by coercion"

\n
\n\n

Any help figuring this out is much appreciated.

\n\n###########Code Begins Below\n\n
# set up the cross-validated hyper-parameter search\nxgb_grid_1 = expand.grid(\n  nrounds = 2, \n  max_depth = c(5, 10, 15), \n  eta = c(0.01, 0.001, 0.0001), \n  gamma = c(1, 2, 3), \n  colsample_bytree = c(0.4, 0.7, 1.0), \n  min_child_weight = c(0.5, 1, 1.5)\n)\n\n# pack the training control parameters\nxgb_trcontrol_1 = trainControl(\n  method = "cv",\n  number = 3,  \n  allowParallel = TRUE\n)\n\n\n# train the model for each parameter combination in the grid, \n#   using CV to evaluate\nxgb_train_1 = train(\n  x = data.matrix(data_train[wltst,features,with=FALSE]),\n  y = data.matrix(data_train[wltst,target]),\n  trControl = xgb_trcontrol_1,\n  tuneGrid = xgb_grid_1,\n  method = "xgbLinear"\n)\n
", "rawMarkdown": "Hello All,\r\n\r\nI'm trying to tune XGBoost with the code below. When I do this, the following error shows up.\r\n\r\n> \"Error in train.default(x = data.matrix(data_train[wltst, features,\r\n> with = FALSE]), : Metric RMSE not applicable for classification\r\n> models In addition: Warning messages: 1: In\r\n> data.matrix(data_train[wltst, features, with = FALSE]) : NAs\r\n> introduced by coercion 2: In data.matrix(data_train[wltst, features,\r\n> with = FALSE]) : NAs introduced by coercion 3: In\r\n> data.matrix(data_train[wltst, features, with = FALSE]) : NAs\r\n> introduced by coercion\"\r\n\r\nAny help figuring this out is much appreciated.\r\n\r\n#################Code Begins Below########################\r\n\r\n # set up the cross-validated hyper-parameter search\r\n xgb_grid_1 = expand.grid(\r\n nrounds = 2, \r\n max_depth = c(5, 10, 15), \r\n eta = c(0.01, 0.001, 0.0001), \r\n gamma = c(1, 2, 3), \r\n colsample_bytree = c(0.4, 0.7, 1.0), \r\n min_child_weight = c(0.5, 1, 1.5)\r\n )\r\n \r\n # pack the training control parameters\r\n xgb_trcontrol_1 = trainControl(\r\n method = \"cv\",\r\n number = 3, \r\n allowParallel = TRUE\r\n )\r\n \r\n \r\n # train the model for each parameter combination in the grid, \r\n # using CV to evaluate\r\n xgb_train_1 = train(\r\n x = data.matrix(data_train[wltst,features,with=FALSE]),\r\n y = data.matrix(data_train[wltst,target]),\r\n trControl = xgb_trcontrol_1,\r\n tuneGrid = xgb_grid_1,\r\n method = \"xgbLinear\"\r\n )", "votes": null }, { "id": "132876", "postDate": "08/30/2016 04:34:46", "content": "

I see "xgbLinear" but your tuning grid is using parameters meant for xgbTree.

", "rawMarkdown": "I see \"xgbLinear\" but your tuning grid is using parameters meant for xgbTree.", "votes": null }, { "id": "132890", "postDate": "08/30/2016 09:11:25", "content": "

@Lateral Analytics Hello ! your question is teaching me something: it is possible to automatize the exploration of xgboost parameters. Do you know where I can find detailed info on how to set that up ?\nThanks,\nClaude

", "rawMarkdown": "Lateral Analytics Hello ! your question is teaching me something: it is possible to automatize the exploration of xgboost parameters. Do you know where I can find detailed info on how to set that up ?\r\nThanks,\r\nClaude", "votes": null }, { "id": "132901", "postDate": "08/30/2016 10:16:44", "content": "

max kuhn book, he wrote the awesome caret package, just google github for example code on setting it up, you can even do things like list apply across a series of models such as c("gbm", "rf", "earth","nnet"...) and then let caret automatically search the parameters space, however, you will need to set your data up so that missing values are removed and categorical variables are dummied out

", "rawMarkdown": "max kuhn book, he wrote the awesome caret package, just google github for example code on setting it up, you can even do things like list apply across a series of models such as c(\"gbm\", \"rf\", \"earth\",\"nnet\"...) and then let caret automatically search the parameters space, however, you will need to set your data up so that missing values are removed and categorical variables are dummied out", "votes": null }, { "id": "132905", "postDate": "08/30/2016 10:26:45", "content": "
\n

Error in train.default(x = data.matrix(data_train[wltst, features, with = FALSE]), : Metric RMSE not applicable for classification models

\n
\n\n

caret should not support RMSE for classification (it might still run), even if xgboost supports it.

\n\n
\n

Warning messages: 1: In data.matrix(data_train[wltst, features, with = FALSE]) : NAs introduced by coercion 2: In data.matrix(data_train[wltst, features, with = FALSE]) : NAs introduced by coercion 3: In data.matrix(data_train[wltst, features, with = FALSE]) : NAs introduced by coercion"

\n
\n\n

You have character columns in data_train, they are converted as NAs by data.matrix through as.numeric in data.matrix.

\n\n
xgb_grid_1 = expand.grid(\n  nrounds = 2, \n  max_depth = c(5, 10, 15), \n  eta = c(0.01, 0.001, 0.0001), \n  gamma = c(1, 2, 3), \n  colsample_bytree = c(0.4, 0.7, 1.0), \n  min_child_weight = c(0.5, 1, 1.5)\n)\n
\n\n

eta does not to be tuned for the purpose of grid searching. You can keep it constant:

\n\n\n\n

Don't use parallel because you will spawn core^2 threads if you are using also doParallel, which is not what you want (xgboost in caret is supposed to run multithreaded already). It will slow down the learning.

\n\n
xgb_train_1 = train(\n  x = data.matrix(data_train[wltst,features,with=FALSE]),\n  y = data.matrix(data_train[wltst,target]),\n  trControl = xgb_trcontrol_1,\n  tuneGrid = xgb_grid_1,\n  method = "xgbLinear"\n)\n
\n\n

Should be "xgbTree" for the method, and you are using data.matrix on a data.frame/data.table with character columns. To coerce a data.frame/data.table in a matrix using data.matrix, data.matrix checks the type of each column:

\n\n", "rawMarkdown": "> Error in train.default(x = data.matrix(data_train[wltst, features, with = FALSE]), : Metric RMSE not applicable for classification models\r\n\r\ncaret should not support RMSE for classification (it might still run), even if xgboost supports it.\r\n\r\n> Warning messages: 1: In data.matrix(data_train[wltst, features, with = FALSE]) : NAs introduced by coercion 2: In data.matrix(data_train[wltst, features, with = FALSE]) : NAs introduced by coercion 3: In data.matrix(data_train[wltst, features, with = FALSE]) : NAs introduced by coercion\"\r\n\r\nYou have character columns in data_train, they are converted as NAs by data.matrix through as.numeric in data.matrix.\r\n\r\n xgb_grid_1 = expand.grid(\r\n nrounds = 2, \r\n max_depth = c(5, 10, 15), \r\n eta = c(0.01, 0.001, 0.0001), \r\n gamma = c(1, 2, 3), \r\n colsample_bytree = c(0.4, 0.7, 1.0), \r\n min_child_weight = c(0.5, 1, 1.5)\r\n )\r\n\r\neta does not to be tuned for the purpose of grid searching. You can keep it constant:\r\n\r\n* The higher the eta the faster the grid search (but less accurate)\r\n* The lower the eta the slower the grid search (but more accurate)\r\n\r\n xgb_trcontrol_1 = trainControl(\r\n method = \"cv\",\r\n number = 3, \r\n allowParallel = TRUE\r\n )\r\n\r\nDon't use parallel because you will spawn core^2 threads if you are using also doParallel, which is not what you want (xgboost in caret is supposed to run multithreaded already). It will slow down the learning.\r\n\r\n xgb_train_1 = train(\r\n x = data.matrix(data_train[wltst,features,with=FALSE]),\r\n y = data.matrix(data_train[wltst,target]),\r\n trControl = xgb_trcontrol_1,\r\n tuneGrid = xgb_grid_1,\r\n method = \"xgbLinear\"\r\n )\r\n\r\nShould be \"xgbTree\" for the method, and you are using data.matrix on a data.frame/data.table with character columns. To coerce a data.frame/data.table in a matrix using data.matrix, data.matrix checks the type of each column:\r\n\r\n* If it is numeric, it is left as is\r\n* If it is not numeric, it is coerced to numeric using as.numeric (factors/logicals/integers can be coerced to numeric, characters cannot)", "votes": null }, { "id": "133022", "postDate": "08/30/2016 23:23:13", "content": "

Thanks for your help Laurae,

\n\n

I know this competition is almost over, but I'm really trying to make sure I understand how to use Caret with xgBoost for future competitions. My response to your helpful comments is below.

\n\n
\n

I'm not trying to do classification, but regression. I'm not sure how\n to specify that I want regression. Also, in this code I didn't\n specify that the Metric should be "RMSE". Why does Caret make this\n assumption?

\n \n

I understand that turning character columns to NA won't help make the\n model any better, but they should keep it from running. At least I\n think so.

\n \n

ETA is now only a single value. I won't waste time tuning it.

\n \n

allowParallel = TRUE has been changed to allowParallel = FALSE

\n \n

The method has been changed to "xgbTree".

\n
\n\n

When the updated code is run the errors below occur.

\n\n
Error in train.default(x = data.matrix(data_train[wltst, features, with = FALSE]),  : \n  Metric RMSE not applicable for classification models\nIn addition: Warning messages:\n1: In data.matrix(data_train[wltst, features, with = FALSE]) :\n  NAs introduced by coercion\n2: In data.matrix(data_train[wltst, features, with = FALSE]) :\n  NAs introduced by coercion\n3: In data.matrix(data_train[wltst, features, with = FALSE]) :\n  NAs introduced by coercion\n
\n\n

The updated code is below.

\n\n#\n\n

set up the cross-validated hyper-parameter search

\n\n

xgb_grid_1 = expand.grid(\n nrounds = 2, \n max_depth = c(5, 10, 15), \n eta = 0.001, \n gamma = c(1, 2, 3), \n colsample_bytree = c(0.4, 0.7, 1.0), \n min_child_weight = c(0.5, 1, 1.5)\n)

\n\n

pack the training control parameters

\n\n

xgb_trcontrol_1 = trainControl(\n method = "cv",\n number = 3,
\n allowParallel = FALSE\n)

\n\n

train the model for each parameter combination in the grid,

\n\n

using CV to evaluate

\n\n

xgb_train_1 = train(\n x = data.matrix(data_train[wltst,features,with=FALSE]),\n y = data.matrix(data_train[wltst,target]),\n trControl = xgb_trcontrol_1,\n tuneGrid = xgb_grid_1,\n method = "xgbTree"\n)

\n\n

[quote=Laurae;132905]

\n\n
\n

Error in train.default(x = data.matrix(data_train[wltst, features, with = FALSE]), : Metric RMSE not applicable for classification models

\n
\n\n

caret should not support RMSE for classification (it might still run), even if xgboost supports it.

\n\n
\n

Warning messages: 1: In data.matrix(data_train[wltst, features, with = FALSE]) : NAs introduced by coercion 2: In data.matrix(data_train[wltst, features, with = FALSE]) : NAs introduced by coercion 3: In data.matrix(data_train[wltst, features, with = FALSE]) : NAs introduced by coercion"

\n
\n\n

You have character columns in data_train, they are converted as NAs by data.matrix through as.numeric in data.matrix.

\n\n
xgb_grid_1 = expand.grid(\n  nrounds = 2, \n  max_depth = c(5, 10, 15), \n  eta = c(0.01, 0.001, 0.0001), \n  gamma = c(1, 2, 3), \n  colsample_bytree = c(0.4, 0.7, 1.0), \n  min_child_weight = c(0.5, 1, 1.5)\n)\n
\n\n

eta does not to be tuned for the purpose of grid searching. You can keep it constant:

\n\n\n\n

Don't use parallel because you will spawn core^2 threads if you are using also doParallel, which is not what you want (xgboost in caret is supposed to run multithreaded already). It will slow down the learning.

\n\n
xgb_train_1 = train(\n  x = data.matrix(data_train[wltst,features,with=FALSE]),\n  y = data.matrix(data_train[wltst,target]),\n  trControl = xgb_trcontrol_1,\n  tuneGrid = xgb_grid_1,\n  method = "xgbLinear"\n)\n
\n\n

Should be "xgbTree" for the method, and you are using data.matrix on a data.frame/data.table with character columns. To coerce a data.frame/data.table in a matrix using data.matrix, data.matrix checks the type of each column:

\n\n\n\n

[/quote]

", "rawMarkdown": "Thanks for your help Laurae,\r\n\r\nI know this competition is almost over, but I'm really trying to make sure I understand how to use Caret with xgBoost for future competitions. My response to your helpful comments is below.\r\n\r\n> I'm not trying to do classification, but regression. I'm not sure how\r\n> to specify that I want regression. Also, in this code I didn't\r\n> specify that the Metric should be \"RMSE\". Why does Caret make this\r\n> assumption?\r\n> \r\n> I understand that turning character columns to NA won't help make the\r\n> model any better, but they should keep it from running. At least I\r\n> think so.\r\n> \r\n> ETA is now only a single value. I won't waste time tuning it.\r\n> \r\n> allowParallel = TRUE has been changed to allowParallel = FALSE\r\n> \r\n> The method has been changed to \"xgbTree\".\r\n\r\nWhen the updated code is run the errors below occur.\r\n\r\n Error in train.default(x = data.matrix(data_train[wltst, features, with = FALSE]), : \r\n Metric RMSE not applicable for classification models\r\n In addition: Warning messages:\r\n 1: In data.matrix(data_train[wltst, features, with = FALSE]) :\r\n NAs introduced by coercion\r\n 2: In data.matrix(data_train[wltst, features, with = FALSE]) :\r\n NAs introduced by coercion\r\n 3: In data.matrix(data_train[wltst, features, with = FALSE]) :\r\n NAs introduced by coercion\r\n\r\nThe updated code is below.\r\n##########################################\r\n\r\n# set up the cross-validated hyper-parameter search\r\nxgb_grid_1 = expand.grid(\r\n nrounds = 2, \r\n max_depth = c(5, 10, 15), \r\n eta = 0.001, \r\n gamma = c(1, 2, 3), \r\n colsample_bytree = c(0.4, 0.7, 1.0), \r\n min_child_weight = c(0.5, 1, 1.5)\r\n)\r\n\r\n# pack the training control parameters\r\nxgb_trcontrol_1 = trainControl(\r\n method = \"cv\",\r\n number = 3, \r\n allowParallel = FALSE\r\n)\r\n\r\n\r\n# train the model for each parameter combination in the grid, \r\n# using CV to evaluate\r\nxgb_train_1 = train(\r\n x = data.matrix(data_train[wltst,features,with=FALSE]),\r\n y = data.matrix(data_train[wltst,target]),\r\n trControl = xgb_trcontrol_1,\r\n tuneGrid = xgb_grid_1,\r\n method = \"xgbTree\"\r\n)\r\n\r\n\r\n[quote=Laurae;132905]\r\n\r\n> Error in train.default(x = data.matrix(data_train[wltst, features, with = FALSE]), : Metric RMSE not applicable for classification models\r\n\r\ncaret should not support RMSE for classification (it might still run), even if xgboost supports it.\r\n\r\n> Warning messages: 1: In data.matrix(data_train[wltst, features, with = FALSE]) : NAs introduced by coercion 2: In data.matrix(data_train[wltst, features, with = FALSE]) : NAs introduced by coercion 3: In data.matrix(data_train[wltst, features, with = FALSE]) : NAs introduced by coercion\"\r\n\r\nYou have character columns in data_train, they are converted as NAs by data.matrix through as.numeric in data.matrix.\r\n\r\n xgb_grid_1 = expand.grid(\r\n nrounds = 2, \r\n max_depth = c(5, 10, 15), \r\n eta = c(0.01, 0.001, 0.0001), \r\n gamma = c(1, 2, 3), \r\n colsample_bytree = c(0.4, 0.7, 1.0), \r\n min_child_weight = c(0.5, 1, 1.5)\r\n )\r\n\r\neta does not to be tuned for the purpose of grid searching. You can keep it constant:\r\n\r\n* The higher the eta the faster the grid search (but less accurate)\r\n* The lower the eta the slower the grid search (but more accurate)\r\n\r\n xgb_trcontrol_1 = trainControl(\r\n method = \"cv\",\r\n number = 3, \r\n allowParallel = TRUE\r\n )\r\n\r\nDon't use parallel because you will spawn core^2 threads if you are using also doParallel, which is not what you want (xgboost in caret is supposed to run multithreaded already). It will slow down the learning.\r\n\r\n xgb_train_1 = train(\r\n x = data.matrix(data_train[wltst,features,with=FALSE]),\r\n y = data.matrix(data_train[wltst,target]),\r\n trControl = xgb_trcontrol_1,\r\n tuneGrid = xgb_grid_1,\r\n method = \"xgbLinear\"\r\n )\r\n\r\nShould be \"xgbTree\" for the method, and you are using data.matrix on a data.frame/data.table with character columns. To coerce a data.frame/data.table in a matrix using data.matrix, data.matrix checks the type of each column:\r\n\r\n* If it is numeric, it is left as is\r\n* If it is not numeric, it is coerced to numeric using as.numeric (factors/logicals/integers can be coerced to numeric, characters cannot)\r\n\r\n[/quote]", "votes": null }, { "id": "133023", "postDate": "08/30/2016 23:30:26", "content": "

[quote=Lateral Analytics;133022]

\n\n

I'm not trying to do classification, but regression. I'm not sure how\nto specify that I want regression. Also, in this code I didn't\nspecify that the Metric should be "RMSE". Why does Caret make this\nassumption?

\n\n

[/quote]

\n\n

caret does not support RMSE for classification. You need to use a custom metric that replicates RMSE, or you need to tell caret you are using a regression (by not passing factors as label).

", "rawMarkdown": "[quote=Lateral Analytics;133022]\r\n\r\nI'm not trying to do classification, but regression. I'm not sure how\r\nto specify that I want regression. Also, in this code I didn't\r\nspecify that the Metric should be \"RMSE\". Why does Caret make this\r\nassumption?\r\n\r\n[/quote]\r\n\r\ncaret does not support RMSE for classification. You need to use a custom metric that replicates RMSE, or you need to tell caret you are using a regression (by not passing factors as label).", "votes": null }, { "id": "133100", "postDate": "08/31/2016 08:41:35", "content": "

Here is also a nice article about how to tune xgboost automatically with help of the mlr (and mlrMBO) R-package:

\n\n

http://mlr-org.github.io/How-to-win-a-drone-in-20-lines-of-R-code/

\n\n

They won a competition with very few code lines.

\n\n

The mlrMBO package, which currently only is on github can be installed via devtools:

\n\n
install.packages("devtools")\nlibrary(devtools)\ninstall_github("berndbischl/ParamHelpers")\ninstall_github("mlr-org/mlr")\ninstall_github("mlr-org/mlrMBO")\n
", "rawMarkdown": "Here is also a nice article about how to tune xgboost automatically with help of the mlr (and mlrMBO) R-package: \r\n\r\nhttp://mlr-org.github.io/How-to-win-a-drone-in-20-lines-of-R-code/\r\n\r\nThey won a competition with very few code lines.\r\n\r\nThe mlrMBO package, which currently only is on github can be installed via devtools: \r\n\r\n install.packages(\"devtools\")\r\n library(devtools)\r\n install_github(\"berndbischl/ParamHelpers\")\r\n install_github(\"mlr-org/mlr\")\r\n install_github(\"mlr-org/mlrMBO\")", "votes": null }, { "id": "133118", "postDate": "08/31/2016 12:15:49", "content": "

mlrMBO, it`s very useful

", "rawMarkdown": "mlrMBO, it`s very useful", "votes": null } ], "comments": [ { "id": 132876, "author_name": "brandenkmurray", "author_url": "", "post_date": "08/30/2016 04:34:46", "content": "

I see "xgbLinear" but your tuning grid is using parameters meant for xgbTree.

", "votes": null, "replies": [] }, { "id": 132890, "author_name": "cdamp350", "author_url": "", "post_date": "08/30/2016 09:11:25", "content": "

@Lateral Analytics Hello ! your question is teaching me something: it is possible to automatize the exploration of xgboost parameters. Do you know where I can find detailed info on how to set that up ?\nThanks,\nClaude

", "votes": null, "replies": [] }, { "id": 132901, "author_name": "anonemaus", "author_url": "", "post_date": "08/30/2016 10:16:44", "content": "

max kuhn book, he wrote the awesome caret package, just google github for example code on setting it up, you can even do things like list apply across a series of models such as c("gbm", "rf", "earth","nnet"...) and then let caret automatically search the parameters space, however, you will need to set your data up so that missing values are removed and categorical variables are dummied out

", "votes": null, "replies": [] }, { "id": 132905, "author_name": "laurae2", "author_url": "", "post_date": "08/30/2016 10:26:45", "content": "
\n

Error in train.default(x = data.matrix(data_train[wltst, features, with = FALSE]), : Metric RMSE not applicable for classification models

\n
\n\n

caret should not support RMSE for classification (it might still run), even if xgboost supports it.

\n\n
\n

Warning messages: 1: In data.matrix(data_train[wltst, features, with = FALSE]) : NAs introduced by coercion 2: In data.matrix(data_train[wltst, features, with = FALSE]) : NAs introduced by coercion 3: In data.matrix(data_train[wltst, features, with = FALSE]) : NAs introduced by coercion"

\n
\n\n

You have character columns in data_train, they are converted as NAs by data.matrix through as.numeric in data.matrix.

\n\n
xgb_grid_1 = expand.grid(\n  nrounds = 2, \n  max_depth = c(5, 10, 15), \n  eta = c(0.01, 0.001, 0.0001), \n  gamma = c(1, 2, 3), \n  colsample_bytree = c(0.4, 0.7, 1.0), \n  min_child_weight = c(0.5, 1, 1.5)\n)\n
\n\n

eta does not to be tuned for the purpose of grid searching. You can keep it constant:

\n\n\n\n

Don't use parallel because you will spawn core^2 threads if you are using also doParallel, which is not what you want (xgboost in caret is supposed to run multithreaded already). It will slow down the learning.

\n\n
xgb_train_1 = train(\n  x = data.matrix(data_train[wltst,features,with=FALSE]),\n  y = data.matrix(data_train[wltst,target]),\n  trControl = xgb_trcontrol_1,\n  tuneGrid = xgb_grid_1,\n  method = "xgbLinear"\n)\n
\n\n

Should be "xgbTree" for the method, and you are using data.matrix on a data.frame/data.table with character columns. To coerce a data.frame/data.table in a matrix using data.matrix, data.matrix checks the type of each column:

\n\n", "votes": null, "replies": [] }, { "id": 133022, "author_name": "jeremyjesse", "author_url": "", "post_date": "08/30/2016 23:23:13", "content": "

Thanks for your help Laurae,

\n\n

I know this competition is almost over, but I'm really trying to make sure I understand how to use Caret with xgBoost for future competitions. My response to your helpful comments is below.

\n\n
\n

I'm not trying to do classification, but regression. I'm not sure how\n to specify that I want regression. Also, in this code I didn't\n specify that the Metric should be "RMSE". Why does Caret make this\n assumption?

\n \n

I understand that turning character columns to NA won't help make the\n model any better, but they should keep it from running. At least I\n think so.

\n \n

ETA is now only a single value. I won't waste time tuning it.

\n \n

allowParallel = TRUE has been changed to allowParallel = FALSE

\n \n

The method has been changed to "xgbTree".

\n
\n\n

When the updated code is run the errors below occur.

\n\n
Error in train.default(x = data.matrix(data_train[wltst, features, with = FALSE]),  : \n  Metric RMSE not applicable for classification models\nIn addition: Warning messages:\n1: In data.matrix(data_train[wltst, features, with = FALSE]) :\n  NAs introduced by coercion\n2: In data.matrix(data_train[wltst, features, with = FALSE]) :\n  NAs introduced by coercion\n3: In data.matrix(data_train[wltst, features, with = FALSE]) :\n  NAs introduced by coercion\n
\n\n

The updated code is below.

\n\n#\n\n

set up the cross-validated hyper-parameter search

\n\n

xgb_grid_1 = expand.grid(\n nrounds = 2, \n max_depth = c(5, 10, 15), \n eta = 0.001, \n gamma = c(1, 2, 3), \n colsample_bytree = c(0.4, 0.7, 1.0), \n min_child_weight = c(0.5, 1, 1.5)\n)

\n\n

pack the training control parameters

\n\n

xgb_trcontrol_1 = trainControl(\n method = "cv",\n number = 3,
\n allowParallel = FALSE\n)

\n\n

train the model for each parameter combination in the grid,

\n\n

using CV to evaluate

\n\n

xgb_train_1 = train(\n x = data.matrix(data_train[wltst,features,with=FALSE]),\n y = data.matrix(data_train[wltst,target]),\n trControl = xgb_trcontrol_1,\n tuneGrid = xgb_grid_1,\n method = "xgbTree"\n)

\n\n

[quote=Laurae;132905]

\n\n
\n

Error in train.default(x = data.matrix(data_train[wltst, features, with = FALSE]), : Metric RMSE not applicable for classification models

\n
\n\n

caret should not support RMSE for classification (it might still run), even if xgboost supports it.

\n\n
\n

Warning messages: 1: In data.matrix(data_train[wltst, features, with = FALSE]) : NAs introduced by coercion 2: In data.matrix(data_train[wltst, features, with = FALSE]) : NAs introduced by coercion 3: In data.matrix(data_train[wltst, features, with = FALSE]) : NAs introduced by coercion"

\n
\n\n

You have character columns in data_train, they are converted as NAs by data.matrix through as.numeric in data.matrix.

\n\n
xgb_grid_1 = expand.grid(\n  nrounds = 2, \n  max_depth = c(5, 10, 15), \n  eta = c(0.01, 0.001, 0.0001), \n  gamma = c(1, 2, 3), \n  colsample_bytree = c(0.4, 0.7, 1.0), \n  min_child_weight = c(0.5, 1, 1.5)\n)\n
\n\n

eta does not to be tuned for the purpose of grid searching. You can keep it constant:

\n\n\n\n

Don't use parallel because you will spawn core^2 threads if you are using also doParallel, which is not what you want (xgboost in caret is supposed to run multithreaded already). It will slow down the learning.

\n\n
xgb_train_1 = train(\n  x = data.matrix(data_train[wltst,features,with=FALSE]),\n  y = data.matrix(data_train[wltst,target]),\n  trControl = xgb_trcontrol_1,\n  tuneGrid = xgb_grid_1,\n  method = "xgbLinear"\n)\n
\n\n

Should be "xgbTree" for the method, and you are using data.matrix on a data.frame/data.table with character columns. To coerce a data.frame/data.table in a matrix using data.matrix, data.matrix checks the type of each column:

\n\n\n\n

[/quote]

", "votes": null, "replies": [] }, { "id": 133023, "author_name": "laurae2", "author_url": "", "post_date": "08/30/2016 23:30:26", "content": "

[quote=Lateral Analytics;133022]

\n\n

I'm not trying to do classification, but regression. I'm not sure how\nto specify that I want regression. Also, in this code I didn't\nspecify that the Metric should be "RMSE". Why does Caret make this\nassumption?

\n\n

[/quote]

\n\n

caret does not support RMSE for classification. You need to use a custom metric that replicates RMSE, or you need to tell caret you are using a regression (by not passing factors as label).

", "votes": null, "replies": [] }, { "id": 133100, "author_name": "icedragon", "author_url": "", "post_date": "08/31/2016 08:41:35", "content": "

Here is also a nice article about how to tune xgboost automatically with help of the mlr (and mlrMBO) R-package:

\n\n

http://mlr-org.github.io/How-to-win-a-drone-in-20-lines-of-R-code/

\n\n

They won a competition with very few code lines.

\n\n

The mlrMBO package, which currently only is on github can be installed via devtools:

\n\n
install.packages("devtools")\nlibrary(devtools)\ninstall_github("berndbischl/ParamHelpers")\ninstall_github("mlr-org/mlr")\ninstall_github("mlr-org/mlrMBO")\n
", "votes": null, "replies": [] }, { "id": 133118, "author_name": "niks2000", "author_url": "", "post_date": "08/31/2016 12:15:49", "content": "

mlrMBO, it`s very useful

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "132870": "Hello All,\r\n\r\nI'm trying to tune XGBoost with the code below. When I do this, the following error shows up.\r\n\r\n> \"Error in train.default(x = data.matrix(data_train[wltst, features,\r\n> with = FALSE]), : Metric RMSE not applicable for classification\r\n> models In addition: Warning messages: 1: In\r\n> data.matrix(data_train[wltst, features, with = FALSE]) : NAs\r\n> introduced by coercion 2: In data.matrix(data_train[wltst, features,\r\n> with = FALSE]) : NAs introduced by coercion 3: In\r\n> data.matrix(data_train[wltst, features, with = FALSE]) : NAs\r\n> introduced by coercion\"\r\n\r\nAny help figuring this out is much appreciated.\r\n\r\n#################Code Begins Below########################\r\n\r\n # set up the cross-validated hyper-parameter search\r\n xgb_grid_1 = expand.grid(\r\n nrounds = 2, \r\n max_depth = c(5, 10, 15), \r\n eta = c(0.01, 0.001, 0.0001), \r\n gamma = c(1, 2, 3), \r\n colsample_bytree = c(0.4, 0.7, 1.0), \r\n min_child_weight = c(0.5, 1, 1.5)\r\n )\r\n \r\n # pack the training control parameters\r\n xgb_trcontrol_1 = trainControl(\r\n method = \"cv\",\r\n number = 3, \r\n allowParallel = TRUE\r\n )\r\n \r\n \r\n # train the model for each parameter combination in the grid, \r\n # using CV to evaluate\r\n xgb_train_1 = train(\r\n x = data.matrix(data_train[wltst,features,with=FALSE]),\r\n y = data.matrix(data_train[wltst,target]),\r\n trControl = xgb_trcontrol_1,\r\n tuneGrid = xgb_grid_1,\r\n method = \"xgbLinear\"\r\n )", "132876": "I see \"xgbLinear\" but your tuning grid is using parameters meant for xgbTree.", "132890": "Lateral Analytics Hello ! your question is teaching me something: it is possible to automatize the exploration of xgboost parameters. Do you know where I can find detailed info on how to set that up ?\r\nThanks,\r\nClaude", "132901": "max kuhn book, he wrote the awesome caret package, just google github for example code on setting it up, you can even do things like list apply across a series of models such as c(\"gbm\", \"rf\", \"earth\",\"nnet\"...) and then let caret automatically search the parameters space, however, you will need to set your data up so that missing values are removed and categorical variables are dummied out", "132905": "> Error in train.default(x = data.matrix(data_train[wltst, features, with = FALSE]), : Metric RMSE not applicable for classification models\r\n\r\ncaret should not support RMSE for classification (it might still run), even if xgboost supports it.\r\n\r\n> Warning messages: 1: In data.matrix(data_train[wltst, features, with = FALSE]) : NAs introduced by coercion 2: In data.matrix(data_train[wltst, features, with = FALSE]) : NAs introduced by coercion 3: In data.matrix(data_train[wltst, features, with = FALSE]) : NAs introduced by coercion\"\r\n\r\nYou have character columns in data_train, they are converted as NAs by data.matrix through as.numeric in data.matrix.\r\n\r\n xgb_grid_1 = expand.grid(\r\n nrounds = 2, \r\n max_depth = c(5, 10, 15), \r\n eta = c(0.01, 0.001, 0.0001), \r\n gamma = c(1, 2, 3), \r\n colsample_bytree = c(0.4, 0.7, 1.0), \r\n min_child_weight = c(0.5, 1, 1.5)\r\n )\r\n\r\neta does not to be tuned for the purpose of grid searching. You can keep it constant:\r\n\r\n* The higher the eta the faster the grid search (but less accurate)\r\n* The lower the eta the slower the grid search (but more accurate)\r\n\r\n xgb_trcontrol_1 = trainControl(\r\n method = \"cv\",\r\n number = 3, \r\n allowParallel = TRUE\r\n )\r\n\r\nDon't use parallel because you will spawn core^2 threads if you are using also doParallel, which is not what you want (xgboost in caret is supposed to run multithreaded already). It will slow down the learning.\r\n\r\n xgb_train_1 = train(\r\n x = data.matrix(data_train[wltst,features,with=FALSE]),\r\n y = data.matrix(data_train[wltst,target]),\r\n trControl = xgb_trcontrol_1,\r\n tuneGrid = xgb_grid_1,\r\n method = \"xgbLinear\"\r\n )\r\n\r\nShould be \"xgbTree\" for the method, and you are using data.matrix on a data.frame/data.table with character columns. To coerce a data.frame/data.table in a matrix using data.matrix, data.matrix checks the type of each column:\r\n\r\n* If it is numeric, it is left as is\r\n* If it is not numeric, it is coerced to numeric using as.numeric (factors/logicals/integers can be coerced to numeric, characters cannot)", "133022": "Thanks for your help Laurae,\r\n\r\nI know this competition is almost over, but I'm really trying to make sure I understand how to use Caret with xgBoost for future competitions. My response to your helpful comments is below.\r\n\r\n> I'm not trying to do classification, but regression. I'm not sure how\r\n> to specify that I want regression. Also, in this code I didn't\r\n> specify that the Metric should be \"RMSE\". Why does Caret make this\r\n> assumption?\r\n> \r\n> I understand that turning character columns to NA won't help make the\r\n> model any better, but they should keep it from running. At least I\r\n> think so.\r\n> \r\n> ETA is now only a single value. I won't waste time tuning it.\r\n> \r\n> allowParallel = TRUE has been changed to allowParallel = FALSE\r\n> \r\n> The method has been changed to \"xgbTree\".\r\n\r\nWhen the updated code is run the errors below occur.\r\n\r\n Error in train.default(x = data.matrix(data_train[wltst, features, with = FALSE]), : \r\n Metric RMSE not applicable for classification models\r\n In addition: Warning messages:\r\n 1: In data.matrix(data_train[wltst, features, with = FALSE]) :\r\n NAs introduced by coercion\r\n 2: In data.matrix(data_train[wltst, features, with = FALSE]) :\r\n NAs introduced by coercion\r\n 3: In data.matrix(data_train[wltst, features, with = FALSE]) :\r\n NAs introduced by coercion\r\n\r\nThe updated code is below.\r\n##########################################\r\n\r\n# set up the cross-validated hyper-parameter search\r\nxgb_grid_1 = expand.grid(\r\n nrounds = 2, \r\n max_depth = c(5, 10, 15), \r\n eta = 0.001, \r\n gamma = c(1, 2, 3), \r\n colsample_bytree = c(0.4, 0.7, 1.0), \r\n min_child_weight = c(0.5, 1, 1.5)\r\n)\r\n\r\n# pack the training control parameters\r\nxgb_trcontrol_1 = trainControl(\r\n method = \"cv\",\r\n number = 3, \r\n allowParallel = FALSE\r\n)\r\n\r\n\r\n# train the model for each parameter combination in the grid, \r\n# using CV to evaluate\r\nxgb_train_1 = train(\r\n x = data.matrix(data_train[wltst,features,with=FALSE]),\r\n y = data.matrix(data_train[wltst,target]),\r\n trControl = xgb_trcontrol_1,\r\n tuneGrid = xgb_grid_1,\r\n method = \"xgbTree\"\r\n)\r\n\r\n\r\n[quote=Laurae;132905]\r\n\r\n> Error in train.default(x = data.matrix(data_train[wltst, features, with = FALSE]), : Metric RMSE not applicable for classification models\r\n\r\ncaret should not support RMSE for classification (it might still run), even if xgboost supports it.\r\n\r\n> Warning messages: 1: In data.matrix(data_train[wltst, features, with = FALSE]) : NAs introduced by coercion 2: In data.matrix(data_train[wltst, features, with = FALSE]) : NAs introduced by coercion 3: In data.matrix(data_train[wltst, features, with = FALSE]) : NAs introduced by coercion\"\r\n\r\nYou have character columns in data_train, they are converted as NAs by data.matrix through as.numeric in data.matrix.\r\n\r\n xgb_grid_1 = expand.grid(\r\n nrounds = 2, \r\n max_depth = c(5, 10, 15), \r\n eta = c(0.01, 0.001, 0.0001), \r\n gamma = c(1, 2, 3), \r\n colsample_bytree = c(0.4, 0.7, 1.0), \r\n min_child_weight = c(0.5, 1, 1.5)\r\n )\r\n\r\neta does not to be tuned for the purpose of grid searching. You can keep it constant:\r\n\r\n* The higher the eta the faster the grid search (but less accurate)\r\n* The lower the eta the slower the grid search (but more accurate)\r\n\r\n xgb_trcontrol_1 = trainControl(\r\n method = \"cv\",\r\n number = 3, \r\n allowParallel = TRUE\r\n )\r\n\r\nDon't use parallel because you will spawn core^2 threads if you are using also doParallel, which is not what you want (xgboost in caret is supposed to run multithreaded already). It will slow down the learning.\r\n\r\n xgb_train_1 = train(\r\n x = data.matrix(data_train[wltst,features,with=FALSE]),\r\n y = data.matrix(data_train[wltst,target]),\r\n trControl = xgb_trcontrol_1,\r\n tuneGrid = xgb_grid_1,\r\n method = \"xgbLinear\"\r\n )\r\n\r\nShould be \"xgbTree\" for the method, and you are using data.matrix on a data.frame/data.table with character columns. To coerce a data.frame/data.table in a matrix using data.matrix, data.matrix checks the type of each column:\r\n\r\n* If it is numeric, it is left as is\r\n* If it is not numeric, it is coerced to numeric using as.numeric (factors/logicals/integers can be coerced to numeric, characters cannot)\r\n\r\n[/quote]", "133023": "[quote=Lateral Analytics;133022]\r\n\r\nI'm not trying to do classification, but regression. I'm not sure how\r\nto specify that I want regression. Also, in this code I didn't\r\nspecify that the Metric should be \"RMSE\". Why does Caret make this\r\nassumption?\r\n\r\n[/quote]\r\n\r\ncaret does not support RMSE for classification. You need to use a custom metric that replicates RMSE, or you need to tell caret you are using a regression (by not passing factors as label).", "133100": "Here is also a nice article about how to tune xgboost automatically with help of the mlr (and mlrMBO) R-package: \r\n\r\nhttp://mlr-org.github.io/How-to-win-a-drone-in-20-lines-of-R-code/\r\n\r\nThey won a competition with very few code lines.\r\n\r\nThe mlrMBO package, which currently only is on github can be installed via devtools: \r\n\r\n install.packages(\"devtools\")\r\n library(devtools)\r\n install_github(\"berndbischl/ParamHelpers\")\r\n install_github(\"mlr-org/mlr\")\r\n install_github(\"mlr-org/mlrMBO\")", "133118": "mlrMBO, it`s very useful" }, "source": "meta" }