Hello all,
\n\ni wrote my own function for one hot encoding in R but run always out of memory. R says it cannot allocate vectors of size...\nAdditionally mr. google didn´t tell me any helpfull things.\nAfter one hot encoding i would like to use xgboost.\nTherefore i would like to know, if someone can help me with this issue and provide a little example using one hot encoding or something like that.
\n\nThank you in advance
", "messages": [ { "id": "126645", "postDate": "07/11/2016 12:39:12", "content": "Hello all,
\n\ni wrote my own function for one hot encoding in R but run always out of memory. R says it cannot allocate vectors of size...\nAdditionally mr. google didn´t tell me any helpfull things.\nAfter one hot encoding i would like to use xgboost.\nTherefore i would like to know, if someone can help me with this issue and provide a little example using one hot encoding or something like that.
\n\nThank you in advance
", "rawMarkdown": "Hello all,\r\n\r\ni wrote my own function for one hot encoding in R but run always out of memory. R says it cannot allocate vectors of size...\r\nAdditionally mr. google didn´t tell me any helpfull things.\r\nAfter one hot encoding i would like to use xgboost.\r\nTherefore i would like to know, if someone can help me with this issue and provide a little example using one hot encoding or something like that.\r\n\r\nThank you in advance", "votes": null }, { "id": "126646", "postDate": "07/11/2016 12:50:10", "content": "Just in case, one hot encoding isn't really required for xgboost if you use tree boosters.
", "rawMarkdown": "Just in case, one hot encoding isn't really required for xgboost if you use tree boosters.", "votes": null }, { "id": "126647", "postDate": "07/11/2016 13:24:33", "content": "My best suggestions are caret::dummyVars and FeatureHashing::hashed.model.matrix . The FeatureHashing approach generates a sparse matrix and plays well with xgboost.
", "rawMarkdown": "My best suggestions are caret::dummyVars and FeatureHashing::hashed.model.matrix . The FeatureHashing approach generates a sparse matrix and plays well with xgboost.", "votes": null }, { "id": "126658", "postDate": "07/11/2016 17:17:36", "content": "You can do this by generating a sparse matrix using sparseMatrix from the Matrix package. I wrote an example of this (combined with xgboost) in the *predict_income_xgb2.R* script here.
Has anyone been able to create a sparse matrix with caret's dummyVars? It claims to have the option, and is exactly what you need, but it never worked for me. Creating a non-sparse matrix with 74 million rows and more than million columns is not the best use of one's time.
", "rawMarkdown": "Has anyone been able to create a sparse matrix with caret's dummyVars? It claims to have the option, and is exactly what you need, but it never worked for me. Creating a non-sparse matrix with 74 million rows and more than million columns is not the best use of one's time.", "votes": null }, { "id": "126754", "postDate": "07/12/2016 05:46:17", "content": "Thanx all. I will have a look to your script Ben. Thank you very much.
", "rawMarkdown": "Thanx all. I will have a look to your script Ben. Thank you very much.", "votes": null }, { "id": "126807", "postDate": "07/12/2016 12:05:46", "content": "[quote=CPMP;126646]
\n\nJust in case, one hot encoding isn't really required for xgboost if you use tree boosters.
\n\n[/quote]
\n\nThis is very interesting, can you please elaborate on that? All sources I have read so far indicate completely the opposite- xgboost does not support categorical features and they have to be one-hot-encoded.
", "rawMarkdown": "[quote=CPMP;126646]\r\n\r\nJust in case, one hot encoding isn't really required for xgboost if you use tree boosters. \r\n\r\n[/quote]\r\n\r\nThis is very interesting, can you please elaborate on that? All sources I have read so far indicate completely the opposite- xgboost does not support categorical features and they have to be one-hot-encoded.", "votes": null }, { "id": "126817", "postDate": "07/12/2016 12:43:23", "content": "One hot encoding is not mandatory for boosted trees. More precisely, using categorical variables will yield different results than using one encoded variables. It is not clear to me why one hot encoding would always be better.
\n\nTo understand why, one needs to understand how xgboost tree booster work. If you use linear regression booster then one hot encoding is mandatory, but that's a separate discussion.
\n\nAt each level of the current tree, and for each not at that level , xgboost loops over each feature, to find where splitting values for that feature leads to the better node split.
\n\nAssume for the sake of discussion that we have only one variable, a categorical one, with n possible values.
\n\nIf you do not hot encode, then xgboost will split on the categorical variable. It will find a pivot value, say V1 for the first level. Examples with feature value equal to V1 or lower than V1 will got to the left branch, and the rest of examples will go the right node. And this process is applied recursively.
\n\nIf you use on hot encoding, then xgboost will loop over all features, and evaluates splitting for each. It means that it will select a value of the categorical variable, say V1, and create two nodes: one with all the examples with value V1 for the categorical variable, and one node for the other examples. Then it recurses. Recursing on the node with value V1 yields no further split, while the other one can yield new split.
\n\nWe see that a different tree will be created. I frankly don't know why the latter would necessarily always be better than the former. This said, I think it is good to try both to see which one yields best result on a given data set.
\n\nYou can read the details about how xgboost splits here: arxiv.org/pdf/1603.02754v1.pdf .
", "rawMarkdown": "One hot encoding is not mandatory for boosted trees. More precisely, using categorical variables will yield different results than using one encoded variables. It is not clear to me why one hot encoding would always be better.\r\n\r\n To understand why, one needs to understand how xgboost tree booster work. If you use linear regression booster then one hot encoding *is* mandatory, but that's a separate discussion.\r\n\r\n\r\nAt each level of the current tree, and for each not at that level , xgboost loops over each feature, to find where splitting values for that feature leads to the better node split. \r\n\r\nAssume for the sake of discussion that we have only one variable, a categorical one, with n possible values. \r\n\r\nIf you do not hot encode, then xgboost will split on the categorical variable. It will find a pivot value, say V1 for the first level. Examples with feature value equal to V1 or lower than V1 will got to the left branch, and the rest of examples will go the right node. And this process is applied recursively.\r\n\r\nIf you use on hot encoding, then xgboost will loop over all features, and evaluates splitting for each. It means that it will select a value of the categorical variable, say V1, and create two nodes: one with all the examples with value V1 for the categorical variable, and one node for the other examples. Then it recurses. Recursing on the node with value V1 yields no further split, while the other one can yield new split.\r\n\r\nWe see that a different tree will be created. I frankly don't know why the latter would necessarily always be better than the former. This said, I think it is good to try both to see which one yields best result on a given data set.\r\n\r\nYou can read the details about how xgboost splits here: arxiv.org/pdf/1603.02754v1.pdf .", "votes": null }, { "id": "127101", "postDate": "07/13/2016 11:18:01", "content": "@Ben How do you measure the impact of the features after one hot encoding? xgb.importance treat every column as own feature and that leads to too much features. Isn´t there any trick to "aggregate" the importance back to your original feature?
", "rawMarkdown": "Ben How do you measure the impact of the features after one hot encoding? xgb.importance treat every column as own feature and that leads to too much features. Isn´t there any trick to \"aggregate\" the importance back to your original feature?", "votes": null }, { "id": "135427", "postDate": "09/14/2016 01:49:42", "content": "See this package:
\n\nhttps://cran.r-project.org/web/packages/CatEncoders/index.html
", "rawMarkdown": "See this package:\r\n\r\nhttps://cran.r-project.org/web/packages/CatEncoders/index.html", "votes": null }, { "id": "143788", "postDate": "11/10/2016 17:06:24", "content": "[quote=Ben Gorman;126658]
\n\nYou can do this by generating a sparse matrix using sparseMatrix from the Matrix package. I wrote an example of this (combined with xgboost) in the predict_income_xgb2.R script here.
[/quote]\nHi Ben, the link is dead. Can you repost it? Thanks!
\n\nJianshe
", "rawMarkdown": "[quote=Ben Gorman;126658]\r\n\r\nYou can do this by generating a sparse matrix using `sparseMatrix` from the `Matrix` package. I wrote an example of this (combined with xgboost) in the *predict_income_xgb2.R* script [here][1].\r\n\r\n\r\n [1]: https://github.com/ben519/MLPB/tree/master/Projects/AverageIncome/PredictIncome\r\n\r\n[/quote]\r\nHi Ben, the link is dead. Can you repost it? Thanks!\r\n\r\nJianshe", "votes": null }, { "id": "253699", "postDate": "12/05/2017 14:03:02", "content": "The link shows a 404 error..Appreciate if you could help out..
", "rawMarkdown": "The link shows a 404 error..Appreciate if you could help out..", "votes": null }, { "id": "253729", "postDate": "12/05/2017 15:05:56", "content": "Thanks. Try it now.
", "rawMarkdown": "Thanks. Try it now.", "votes": null } ], "comments": [ { "id": 126646, "author_name": "cpmpml", "author_url": "", "post_date": "07/11/2016 12:50:10", "content": "Just in case, one hot encoding isn't really required for xgboost if you use tree boosters.
", "votes": null, "replies": [] }, { "id": 126647, "author_name": "cartographic", "author_url": "", "post_date": "07/11/2016 13:24:33", "content": "My best suggestions are caret::dummyVars and FeatureHashing::hashed.model.matrix . The FeatureHashing approach generates a sparse matrix and plays well with xgboost.
", "votes": null, "replies": [] }, { "id": 126658, "author_name": "ben519", "author_url": "", "post_date": "07/11/2016 17:17:36", "content": "You can do this by generating a sparse matrix using sparseMatrix from the Matrix package. I wrote an example of this (combined with xgboost) in the *predict_income_xgb2.R* script here.
The link shows a 404 error..Appreciate if you could help out..
", "votes": null, "replies": [] }, { "id": 253729, "author_name": "ben519", "author_url": "", "post_date": "12/05/2017 15:05:56", "content": "Thanks. Try it now.
", "votes": null, "replies": [] } ] }, { "id": 126733, "author_name": "dmitriyguller", "author_url": "", "post_date": "07/12/2016 02:16:08", "content": "Has anyone been able to create a sparse matrix with caret's dummyVars? It claims to have the option, and is exactly what you need, but it never worked for me. Creating a non-sparse matrix with 74 million rows and more than million columns is not the best use of one's time.
", "votes": null, "replies": [] }, { "id": 126754, "author_name": "andreasgruber", "author_url": "", "post_date": "07/12/2016 05:46:17", "content": "Thanx all. I will have a look to your script Ben. Thank you very much.
", "votes": null, "replies": [] }, { "id": 126807, "author_name": "databang", "author_url": "", "post_date": "07/12/2016 12:05:46", "content": "[quote=CPMP;126646]
\n\nJust in case, one hot encoding isn't really required for xgboost if you use tree boosters.
\n\n[/quote]
\n\nThis is very interesting, can you please elaborate on that? All sources I have read so far indicate completely the opposite- xgboost does not support categorical features and they have to be one-hot-encoded.
", "votes": null, "replies": [] }, { "id": 126817, "author_name": "cpmpml", "author_url": "", "post_date": "07/12/2016 12:43:23", "content": "One hot encoding is not mandatory for boosted trees. More precisely, using categorical variables will yield different results than using one encoded variables. It is not clear to me why one hot encoding would always be better.
\n\nTo understand why, one needs to understand how xgboost tree booster work. If you use linear regression booster then one hot encoding is mandatory, but that's a separate discussion.
\n\nAt each level of the current tree, and for each not at that level , xgboost loops over each feature, to find where splitting values for that feature leads to the better node split.
\n\nAssume for the sake of discussion that we have only one variable, a categorical one, with n possible values.
\n\nIf you do not hot encode, then xgboost will split on the categorical variable. It will find a pivot value, say V1 for the first level. Examples with feature value equal to V1 or lower than V1 will got to the left branch, and the rest of examples will go the right node. And this process is applied recursively.
\n\nIf you use on hot encoding, then xgboost will loop over all features, and evaluates splitting for each. It means that it will select a value of the categorical variable, say V1, and create two nodes: one with all the examples with value V1 for the categorical variable, and one node for the other examples. Then it recurses. Recursing on the node with value V1 yields no further split, while the other one can yield new split.
\n\nWe see that a different tree will be created. I frankly don't know why the latter would necessarily always be better than the former. This said, I think it is good to try both to see which one yields best result on a given data set.
\n\nYou can read the details about how xgboost splits here: arxiv.org/pdf/1603.02754v1.pdf .
", "votes": null, "replies": [] }, { "id": 127101, "author_name": "andreasgruber", "author_url": "", "post_date": "07/13/2016 11:18:01", "content": "@Ben How do you measure the impact of the features after one hot encoding? xgb.importance treat every column as own feature and that leads to too much features. Isn´t there any trick to "aggregate" the importance back to your original feature?
", "votes": null, "replies": [] }, { "id": 135427, "author_name": "setseed2016", "author_url": "", "post_date": "09/14/2016 01:49:42", "content": "See this package:
\n\nhttps://cran.r-project.org/web/packages/CatEncoders/index.html
", "votes": null, "replies": [] }, { "id": 143788, "author_name": "jianshefeng", "author_url": "", "post_date": "11/10/2016 17:06:24", "content": "[quote=Ben Gorman;126658]
\n\nYou can do this by generating a sparse matrix using sparseMatrix from the Matrix package. I wrote an example of this (combined with xgboost) in the predict_income_xgb2.R script here.
[/quote]\nHi Ben, the link is dead. Can you repost it? Thanks!
\n\nJianshe
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "126645": "Hello all,\r\n\r\ni wrote my own function for one hot encoding in R but run always out of memory. R says it cannot allocate vectors of size...\r\nAdditionally mr. google didn´t tell me any helpfull things.\r\nAfter one hot encoding i would like to use xgboost.\r\nTherefore i would like to know, if someone can help me with this issue and provide a little example using one hot encoding or something like that.\r\n\r\nThank you in advance", "126646": "Just in case, one hot encoding isn't really required for xgboost if you use tree boosters.", "126647": "My best suggestions are caret::dummyVars and FeatureHashing::hashed.model.matrix . The FeatureHashing approach generates a sparse matrix and plays well with xgboost.", "126658": "You can do this by generating a sparse matrix using `sparseMatrix` from the `Matrix` package. I wrote an example of this (combined with xgboost) in the *predict_income_xgb2.R* script [here][1].\n\n\n [1]: https://github.com/ben519/MLPB/tree/master/Problems/Predict%20Income", "126733": "Has anyone been able to create a sparse matrix with caret's dummyVars? It claims to have the option, and is exactly what you need, but it never worked for me. Creating a non-sparse matrix with 74 million rows and more than million columns is not the best use of one's time.", "126754": "Thanx all. I will have a look to your script Ben. Thank you very much.", "126807": "[quote=CPMP;126646]\r\n\r\nJust in case, one hot encoding isn't really required for xgboost if you use tree boosters. \r\n\r\n[/quote]\r\n\r\nThis is very interesting, can you please elaborate on that? All sources I have read so far indicate completely the opposite- xgboost does not support categorical features and they have to be one-hot-encoded.", "126817": "One hot encoding is not mandatory for boosted trees. More precisely, using categorical variables will yield different results than using one encoded variables. It is not clear to me why one hot encoding would always be better.\r\n\r\n To understand why, one needs to understand how xgboost tree booster work. If you use linear regression booster then one hot encoding *is* mandatory, but that's a separate discussion.\r\n\r\n\r\nAt each level of the current tree, and for each not at that level , xgboost loops over each feature, to find where splitting values for that feature leads to the better node split. \r\n\r\nAssume for the sake of discussion that we have only one variable, a categorical one, with n possible values. \r\n\r\nIf you do not hot encode, then xgboost will split on the categorical variable. It will find a pivot value, say V1 for the first level. Examples with feature value equal to V1 or lower than V1 will got to the left branch, and the rest of examples will go the right node. And this process is applied recursively.\r\n\r\nIf you use on hot encoding, then xgboost will loop over all features, and evaluates splitting for each. It means that it will select a value of the categorical variable, say V1, and create two nodes: one with all the examples with value V1 for the categorical variable, and one node for the other examples. Then it recurses. Recursing on the node with value V1 yields no further split, while the other one can yield new split.\r\n\r\nWe see that a different tree will be created. I frankly don't know why the latter would necessarily always be better than the former. This said, I think it is good to try both to see which one yields best result on a given data set.\r\n\r\nYou can read the details about how xgboost splits here: arxiv.org/pdf/1603.02754v1.pdf .", "127101": "Ben How do you measure the impact of the features after one hot encoding? xgb.importance treat every column as own feature and that leads to too much features. Isn´t there any trick to \"aggregate\" the importance back to your original feature?", "135427": "See this package:\r\n\r\nhttps://cran.r-project.org/web/packages/CatEncoders/index.html", "143788": "[quote=Ben Gorman;126658]\r\n\r\nYou can do this by generating a sparse matrix using `sparseMatrix` from the `Matrix` package. I wrote an example of this (combined with xgboost) in the *predict_income_xgb2.R* script [here][1].\r\n\r\n\r\n [1]: https://github.com/ben519/MLPB/tree/master/Projects/AverageIncome/PredictIncome\r\n\r\n[/quote]\r\nHi Ben, the link is dead. Can you repost it? Thanks!\r\n\r\nJianshe", "253699": "The link shows a 404 error..Appreciate if you could help out..", "253729": "Thanks. Try it now." }, "source": "meta" }