I see various topics on memory issues with xgboost. Some useful suggestions were made, like using the out of core option (in beta) or the spark implementation (xgboost4j). Others suggested decomposing the data in order to do bagging.
\n\nThese are all good suggestions. I'll stress another one: load data using 32 bits integers and floats (I gave the code somewhere in the forum).
\n\nI want to also stress that there is hope. I have a 16GB machine (macbook pro) and I use xgboost via its Python api. My process never goes to more than 10 GB and I could probably make it under 8GB at the expense of longer running times. Yet, I could get 0.44 on public LB that way. Anyone else can achieve the same (or better) if I could...
\n\nEdited to fix typos.
", "messages": [ { "id": "128042", "postDate": "07/17/2016 08:39:01", "content": "I see various topics on memory issues with xgboost. Some useful suggestions were made, like using the out of core option (in beta) or the spark implementation (xgboost4j). Others suggested decomposing the data in order to do bagging.
\n\nThese are all good suggestions. I'll stress another one: load data using 32 bits integers and floats (I gave the code somewhere in the forum).
\n\nI want to also stress that there is hope. I have a 16GB machine (macbook pro) and I use xgboost via its Python api. My process never goes to more than 10 GB and I could probably make it under 8GB at the expense of longer running times. Yet, I could get 0.44 on public LB that way. Anyone else can achieve the same (or better) if I could...
\n\nEdited to fix typos.
", "rawMarkdown": "I see various topics on memory issues with xgboost. Some useful suggestions were made, like using the out of core option (in beta) or the spark implementation (xgboost4j). Others suggested decomposing the data in order to do bagging. \r\n\r\nThese are all good suggestions. I'll stress another one: load data using 32 bits integers and floats (I gave the code somewhere in the forum).\r\n\r\nI want to also stress that there is hope. I have a 16GB machine (macbook pro) and I use xgboost via its Python api. My process never goes to more than 10 GB and I could probably make it under 8GB at the expense of longer running times. Yet, I could get 0.44 on public LB that way. Anyone else can achieve the same (or better) if I could...\r\n\r\n*Edited to fix typos.*", "votes": null }, { "id": "128044", "postDate": "07/17/2016 08:58:57", "content": "Can you give us some insight of how many features you can add at 16gb?
", "rawMarkdown": "Can you give us some insight of how many features you can add at 16gb?", "votes": null }, { "id": "128048", "postDate": "07/17/2016 09:51:28", "content": "I don't apply xgboost to the full data set at once. I use between 100 and 300 features depending on the run.
\n\nI am not using one hot encoding. I know, this is not what people say I should do, but this is what I do anyway.
", "rawMarkdown": "I don't apply xgboost to the full data set at once. I use between 100 and 300 features depending on the run. \r\n\r\nI am not using one hot encoding. I know, this is not what people say I should do, but this is what I do anyway.", "votes": null }, { "id": "128049", "postDate": "07/17/2016 09:53:01", "content": "@CPMP: actually i don't think it's necessary to do OHE - and your rank kind of confirms my intuition here :-) As long as you encode a categorical variable as integer, xgboost can treat it just fine - and probably extract more info than it would've been able to with a really sparse OHE matrix...
", "rawMarkdown": "CPMP: actually i don't think it's necessary to do OHE - and your rank kind of confirms my intuition here :-) As long as you encode a categorical variable as integer, xgboost can treat it just fine - and probably extract more info than it would've been able to with a really sparse OHE matrix...", "votes": null }, { "id": "128054", "postDate": "07/17/2016 10:10:15", "content": "[quote=CPMP;128048]\nI don't apply xgboost to the full data set at once.
\n[/quote]
You mean you are applying xgboost (on smaller train set) many times to generate 1 submission, or just reducing your train set, doing your magic and run your script to generate output ?
", "rawMarkdown": "[quote=CPMP;128048]\r\nI don't apply xgboost to the full data set at once. \r\n[/quote]\r\n\r\nYou mean you are applying xgboost (on smaller train set) many times to generate 1 submission, or just reducing your train set, doing your magic and run your script to generate output ?", "votes": null }, { "id": "128061", "postDate": "07/17/2016 11:39:11", "content": "@salenator I use xgboost many times to generate one solution\n@Konrad I agree. Xgboost will use different branching strategy between OHE and integer encoding of categorical. OHE may be better in some cases, but not always. Regarding my rank, this is prone to change, I am not sure I'll sustain being in the top 10 till the end... And there may be surprises when using the private LB ;)
", "rawMarkdown": "salenator I use xgboost many times to generate one solution\r\n@Konrad I agree. Xgboost will use different branching strategy between OHE and integer encoding of categorical. OHE may be better in some cases, but not always. Regarding my rank, this is prone to change, I am not sure I'll sustain being in the top 10 till the end... And there may be surprises when using the private LB ;)", "votes": null }, { "id": "128159", "postDate": "07/18/2016 08:54:34", "content": "[quote=CPMP;128048]
\n\nI don't apply xgboost to the full data set at once. I use between 100 and 300 features depending on the run.
\n\nI am not using one hot encoding. I know, this is not what people say I should do, but this is what I do anyway.
\n\n[/quote]
\n\nHi , just wanting to clarify, when you say you use 100-300 feature, what does that mean ?
\n\nIf you want to keep it a secret till the end of the competition, it is fine too :)
\n\nXA
", "rawMarkdown": "[quote=CPMP;128048]\r\n\r\nI don't apply xgboost to the full data set at once. I use between 100 and 300 features depending on the run. \r\n\r\nI am not using one hot encoding. I know, this is not what people say I should do, but this is what I do anyway.\r\n\r\n[/quote]\r\n\r\nHi , just wanting to clarify, when you say you use 100-300 feature, what does that mean ?\r\n\r\nIf you want to keep it a secret till the end of the competition, it is fine too :)\r\n\r\nXA", "votes": null }, { "id": "128161", "postDate": "07/18/2016 09:00:22", "content": "It means I use between 100 and 300 features. I construct these features out of the training data. Not sure about what you don't understand.
", "rawMarkdown": "It means I use between 100 and 300 features. I construct these features out of the training data. Not sure about what you don't understand.", "votes": null }, { "id": "128165", "postDate": "07/18/2016 09:32:27", "content": "I was a little confused intitally, but now i get it. I am wondering how did 8 features become 100-300 features though
", "rawMarkdown": "I was a little confused intitally, but now i get it. I am wondering how did 8 features become 100-300 features though", "votes": null }, { "id": "128168", "postDate": "07/18/2016 09:44:50", "content": "I'll explain after competition end.
", "rawMarkdown": "I'll explain after competition end.", "votes": null }, { "id": "128207", "postDate": "07/18/2016 14:16:18", "content": "@CPMP Thank you for your suggestions, really helped me !
", "rawMarkdown": "CPMP Thank you for your suggestions, really helped me !", "votes": null }, { "id": "128243", "postDate": "07/18/2016 17:16:01", "content": "@salenator, glad it helped.
", "rawMarkdown": "salenator, glad it helped.", "votes": null }, { "id": "128477", "postDate": "07/20/2016 12:13:56", "content": "@CPMP & @Konrad..
\n\nHi, total newb here; doing this competition for a class project.
\n\nI was having trouble using OHE for the data:\n1: client and or product ids yield thousands of columns which don't fit in the memory even with sparse matrix.\n2: reading and training data partially was not possible unless we read the full data.
\n\nQUESTION:\nYou've mentioned that if categorical columns are defined as integers, xgboost will be able to use them in training. Is this done by simple defining the dtype as int? or is there something else that should be done?
\n\nmany thanks in advance.
", "rawMarkdown": "CPMP & @Konrad..\r\n\r\nHi, total newb here; doing this competition for a class project.\r\n\r\nI was having trouble using OHE for the data:\r\n1: client and or product ids yield thousands of columns which don't fit in the memory even with sparse matrix.\r\n2: reading and training data partially was not possible unless we read the full data.\r\n\r\nQUESTION:\r\nYou've mentioned that if categorical columns are defined as integers, xgboost will be able to use them in training. Is this done by simple defining the dtype as int? or is there something else that should be done?\r\n\r\nmany thanks in advance.", "votes": null }, { "id": "128480", "postDate": "07/20/2016 12:50:21", "content": "@Cankoklu,
\n\nno need to do anything if you data is integer or float. If you have strings, then you need to encode them as integers one way or another.
\n\nNote that I am not saying OHE is bad. It may well be that OHE works better for some features.
", "rawMarkdown": "Cankoklu,\r\n\r\nno need to do anything if you data is integer or float. If you have strings, then you need to encode them as integers one way or another.\r\n\r\nNote that I am not saying OHE is bad. It may well be that OHE works better for some features.", "votes": null }, { "id": "128485", "postDate": "07/20/2016 13:21:27", "content": "@CPMP If i use xgboost, lets say I train on 2 weeks data, add some features and than predicting on the real test, and I repeat that for all weeks 3-4, 5-6, 7-8 and 9, then i will have lets say 4 results, and merging them is almost impossible, any say about that ?
", "rawMarkdown": "CPMP If i use xgboost, lets say I train on 2 weeks data, add some features and than predicting on the real test, and I repeat that for all weeks 3-4, 5-6, 7-8 and 9, then i will have lets say 4 results, and merging them is almost impossible, any say about that ?", "votes": null }, { "id": "128486", "postDate": "07/20/2016 13:34:46", "content": "@CPMP, thank you for the quick resposne.\nSo if I understand correctly:\nI can just pass the client id column as it is, (without splitting into new columns like OHE), xgboost will then treat it as a categorical variable, am I right?
", "rawMarkdown": "CPMP, thank you for the quick resposne.\r\nSo if I understand correctly:\r\nI can just pass the client id column as it is, (without splitting into new columns like OHE), xgboost will then treat it as a categorical variable, am I right?", "votes": null }, { "id": "128498", "postDate": "07/20/2016 16:13:57", "content": "@CanKoLu, I tried to explain what happens here: https://www.kaggle.com/c/grupo-bimbo-inventory-demand/forums/t/22153/one-hot-encoding-with-r/126817#post126817
", "rawMarkdown": "CanKoLu, I tried to explain what happens here: https://www.kaggle.com/c/grupo-bimbo-inventory-demand/forums/t/22153/one-hot-encoding-with-r/126817#post126817", "votes": null }, { "id": "128593", "postDate": "07/21/2016 15:34:53", "content": "@Salka I tried using xgboost with training multiple times at smaller batches and I try to predict those rows from test that have same Cliente , Producto, Agencia with existing row in a train set, then i merged the results and find mean, i get 0.49 on LB, but there are still 1.4 milion that does not match. And i think that 0.49 is about not predicting them right. \n@CPMP any better suggestions ?
", "rawMarkdown": "Salka I tried using xgboost with training multiple times at smaller batches and I try to predict those rows from test that have same Cliente , Producto, Agencia with existing row in a train set, then i merged the results and find mean, i get 0.49 on LB, but there are still 1.4 milion that does not match. And i think that 0.49 is about not predicting them right. \r\n@CPMP any better suggestions ?", "votes": null }, { "id": "128624", "postDate": "07/22/2016 02:20:23", "content": "@salenator, you are on the right track IMHO. Try more new ideas.
", "rawMarkdown": "salenator, you are on the right track IMHO. Try more new ideas.", "votes": null } ], "comments": [ { "id": 128044, "author_name": "", "author_url": "", "post_date": "07/17/2016 08:58:57", "content": "Can you give us some insight of how many features you can add at 16gb?
", "votes": null, "replies": [] }, { "id": 128048, "author_name": "cpmpml", "author_url": "", "post_date": "07/17/2016 09:51:28", "content": "I don't apply xgboost to the full data set at once. I use between 100 and 300 features depending on the run.
\n\nI am not using one hot encoding. I know, this is not what people say I should do, but this is what I do anyway.
", "votes": null, "replies": [] }, { "id": 128049, "author_name": "konradb", "author_url": "", "post_date": "07/17/2016 09:53:01", "content": "@CPMP: actually i don't think it's necessary to do OHE - and your rank kind of confirms my intuition here :-) As long as you encode a categorical variable as integer, xgboost can treat it just fine - and probably extract more info than it would've been able to with a really sparse OHE matrix...
", "votes": null, "replies": [] }, { "id": 128054, "author_name": "", "author_url": "", "post_date": "07/17/2016 10:10:15", "content": "[quote=CPMP;128048]\nI don't apply xgboost to the full data set at once.
\n[/quote]
You mean you are applying xgboost (on smaller train set) many times to generate 1 submission, or just reducing your train set, doing your magic and run your script to generate output ?
", "votes": null, "replies": [] }, { "id": 128061, "author_name": "cpmpml", "author_url": "", "post_date": "07/17/2016 11:39:11", "content": "@salenator I use xgboost many times to generate one solution\n@Konrad I agree. Xgboost will use different branching strategy between OHE and integer encoding of categorical. OHE may be better in some cases, but not always. Regarding my rank, this is prone to change, I am not sure I'll sustain being in the top 10 till the end... And there may be surprises when using the private LB ;)
", "votes": null, "replies": [] }, { "id": 128159, "author_name": "leexiongan", "author_url": "", "post_date": "07/18/2016 08:54:34", "content": "[quote=CPMP;128048]
\n\nI don't apply xgboost to the full data set at once. I use between 100 and 300 features depending on the run.
\n\nI am not using one hot encoding. I know, this is not what people say I should do, but this is what I do anyway.
\n\n[/quote]
\n\nHi , just wanting to clarify, when you say you use 100-300 feature, what does that mean ?
\n\nIf you want to keep it a secret till the end of the competition, it is fine too :)
\n\nXA
", "votes": null, "replies": [] }, { "id": 128161, "author_name": "cpmpml", "author_url": "", "post_date": "07/18/2016 09:00:22", "content": "It means I use between 100 and 300 features. I construct these features out of the training data. Not sure about what you don't understand.
", "votes": null, "replies": [] }, { "id": 128165, "author_name": "leexiongan", "author_url": "", "post_date": "07/18/2016 09:32:27", "content": "I was a little confused intitally, but now i get it. I am wondering how did 8 features become 100-300 features though
", "votes": null, "replies": [] }, { "id": 128168, "author_name": "cpmpml", "author_url": "", "post_date": "07/18/2016 09:44:50", "content": "I'll explain after competition end.
", "votes": null, "replies": [] }, { "id": 128207, "author_name": "", "author_url": "", "post_date": "07/18/2016 14:16:18", "content": "@CPMP Thank you for your suggestions, really helped me !
", "votes": null, "replies": [] }, { "id": 128243, "author_name": "cpmpml", "author_url": "", "post_date": "07/18/2016 17:16:01", "content": "@salenator, glad it helped.
", "votes": null, "replies": [] }, { "id": 128477, "author_name": "cankoklu", "author_url": "", "post_date": "07/20/2016 12:13:56", "content": "@CPMP & @Konrad..
\n\nHi, total newb here; doing this competition for a class project.
\n\nI was having trouble using OHE for the data:\n1: client and or product ids yield thousands of columns which don't fit in the memory even with sparse matrix.\n2: reading and training data partially was not possible unless we read the full data.
\n\nQUESTION:\nYou've mentioned that if categorical columns are defined as integers, xgboost will be able to use them in training. Is this done by simple defining the dtype as int? or is there something else that should be done?
\n\nmany thanks in advance.
", "votes": null, "replies": [] }, { "id": 128480, "author_name": "cpmpml", "author_url": "", "post_date": "07/20/2016 12:50:21", "content": "@Cankoklu,
\n\nno need to do anything if you data is integer or float. If you have strings, then you need to encode them as integers one way or another.
\n\nNote that I am not saying OHE is bad. It may well be that OHE works better for some features.
", "votes": null, "replies": [] }, { "id": 128485, "author_name": "", "author_url": "", "post_date": "07/20/2016 13:21:27", "content": "@CPMP If i use xgboost, lets say I train on 2 weeks data, add some features and than predicting on the real test, and I repeat that for all weeks 3-4, 5-6, 7-8 and 9, then i will have lets say 4 results, and merging them is almost impossible, any say about that ?
", "votes": null, "replies": [] }, { "id": 128486, "author_name": "cankoklu", "author_url": "", "post_date": "07/20/2016 13:34:46", "content": "@CPMP, thank you for the quick resposne.\nSo if I understand correctly:\nI can just pass the client id column as it is, (without splitting into new columns like OHE), xgboost will then treat it as a categorical variable, am I right?
", "votes": null, "replies": [] }, { "id": 128498, "author_name": "cpmpml", "author_url": "", "post_date": "07/20/2016 16:13:57", "content": "@CanKoLu, I tried to explain what happens here: https://www.kaggle.com/c/grupo-bimbo-inventory-demand/forums/t/22153/one-hot-encoding-with-r/126817#post126817
", "votes": null, "replies": [] }, { "id": 128593, "author_name": "", "author_url": "", "post_date": "07/21/2016 15:34:53", "content": "@Salka I tried using xgboost with training multiple times at smaller batches and I try to predict those rows from test that have same Cliente , Producto, Agencia with existing row in a train set, then i merged the results and find mean, i get 0.49 on LB, but there are still 1.4 milion that does not match. And i think that 0.49 is about not predicting them right. \n@CPMP any better suggestions ?
", "votes": null, "replies": [] }, { "id": 128624, "author_name": "cpmpml", "author_url": "", "post_date": "07/22/2016 02:20:23", "content": "@salenator, you are on the right track IMHO. Try more new ideas.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "128042": "I see various topics on memory issues with xgboost. Some useful suggestions were made, like using the out of core option (in beta) or the spark implementation (xgboost4j). Others suggested decomposing the data in order to do bagging. \r\n\r\nThese are all good suggestions. I'll stress another one: load data using 32 bits integers and floats (I gave the code somewhere in the forum).\r\n\r\nI want to also stress that there is hope. I have a 16GB machine (macbook pro) and I use xgboost via its Python api. My process never goes to more than 10 GB and I could probably make it under 8GB at the expense of longer running times. Yet, I could get 0.44 on public LB that way. Anyone else can achieve the same (or better) if I could...\r\n\r\n*Edited to fix typos.*", "128044": "Can you give us some insight of how many features you can add at 16gb?", "128048": "I don't apply xgboost to the full data set at once. I use between 100 and 300 features depending on the run. \r\n\r\nI am not using one hot encoding. I know, this is not what people say I should do, but this is what I do anyway.", "128049": "CPMP: actually i don't think it's necessary to do OHE - and your rank kind of confirms my intuition here :-) As long as you encode a categorical variable as integer, xgboost can treat it just fine - and probably extract more info than it would've been able to with a really sparse OHE matrix...", "128054": "[quote=CPMP;128048]\r\nI don't apply xgboost to the full data set at once. \r\n[/quote]\r\n\r\nYou mean you are applying xgboost (on smaller train set) many times to generate 1 submission, or just reducing your train set, doing your magic and run your script to generate output ?", "128061": "salenator I use xgboost many times to generate one solution\r\n@Konrad I agree. Xgboost will use different branching strategy between OHE and integer encoding of categorical. OHE may be better in some cases, but not always. Regarding my rank, this is prone to change, I am not sure I'll sustain being in the top 10 till the end... And there may be surprises when using the private LB ;)", "128159": "[quote=CPMP;128048]\r\n\r\nI don't apply xgboost to the full data set at once. I use between 100 and 300 features depending on the run. \r\n\r\nI am not using one hot encoding. I know, this is not what people say I should do, but this is what I do anyway.\r\n\r\n[/quote]\r\n\r\nHi , just wanting to clarify, when you say you use 100-300 feature, what does that mean ?\r\n\r\nIf you want to keep it a secret till the end of the competition, it is fine too :)\r\n\r\nXA", "128161": "It means I use between 100 and 300 features. I construct these features out of the training data. Not sure about what you don't understand.", "128165": "I was a little confused intitally, but now i get it. I am wondering how did 8 features become 100-300 features though", "128168": "I'll explain after competition end.", "128207": "CPMP Thank you for your suggestions, really helped me !", "128243": "salenator, glad it helped.", "128477": "CPMP & @Konrad..\r\n\r\nHi, total newb here; doing this competition for a class project.\r\n\r\nI was having trouble using OHE for the data:\r\n1: client and or product ids yield thousands of columns which don't fit in the memory even with sparse matrix.\r\n2: reading and training data partially was not possible unless we read the full data.\r\n\r\nQUESTION:\r\nYou've mentioned that if categorical columns are defined as integers, xgboost will be able to use them in training. Is this done by simple defining the dtype as int? or is there something else that should be done?\r\n\r\nmany thanks in advance.", "128480": "Cankoklu,\r\n\r\nno need to do anything if you data is integer or float. If you have strings, then you need to encode them as integers one way or another.\r\n\r\nNote that I am not saying OHE is bad. It may well be that OHE works better for some features.", "128485": "CPMP If i use xgboost, lets say I train on 2 weeks data, add some features and than predicting on the real test, and I repeat that for all weeks 3-4, 5-6, 7-8 and 9, then i will have lets say 4 results, and merging them is almost impossible, any say about that ?", "128486": "CPMP, thank you for the quick resposne.\r\nSo if I understand correctly:\r\nI can just pass the client id column as it is, (without splitting into new columns like OHE), xgboost will then treat it as a categorical variable, am I right?", "128498": "CanKoLu, I tried to explain what happens here: https://www.kaggle.com/c/grupo-bimbo-inventory-demand/forums/t/22153/one-hot-encoding-with-r/126817#post126817", "128593": "Salka I tried using xgboost with training multiple times at smaller batches and I try to predict those rows from test that have same Cliente , Producto, Agencia with existing row in a train set, then i merged the results and find mean, i get 0.49 on LB, but there are still 1.4 milion that does not match. And i think that 0.49 is about not predicting them right. \r\n@CPMP any better suggestions ?", "128624": "salenator, you are on the right track IMHO. Try more new ideas." }, "source": "meta" }