Hi All,\nI've been working on this competition for 2 weeks now with R and mlr .\nI'm working on a laptop with 8GB RAM which crashes often due to memory peeks.\nWhat's your hardware configuration?\nIs there any other way to handle this within R?
", "messages": [ { "id": "406975", "postDate": "10/20/2018 05:22:24", "content": "Hi All,\nI've been working on this competition for 2 weeks now with R and mlr .\nI'm working on a laptop with 8GB RAM which crashes often due to memory peeks.\nWhat's your hardware configuration?\nIs there any other way to handle this within R?
", "rawMarkdown": "Hi All,\nI've been working on this competition for 2 weeks now with R and mlr .\nI'm working on a laptop with 8GB RAM which crashes often due to memory peeks.\nWhat's your hardware configuration?\nIs there any other way to handle this within R?", "votes": null }, { "id": "407245", "postDate": "10/20/2018 17:43:54", "content": "Hi Anton,
\n\nI was using laptop with the 8GB RAM for many years - also for Kaggle competitions and I was able to use R without too many issues even when data were quite big. However, sometimes it required batch processing or other tricks to save the memory.
\n\nHowever, you can check if there are any previously created objects in R session which can use the memory. If so, you can remove them and release RAM (you can fore it with gc() function).
\n\nIf it is matter of a data size which you need to train a model, you can consider using data.table instead built-in data.frame (https://cran.r-project.org/web/packages/data.table/vignettes/datatable-intro.html).
\n\nIf the reason is related to mlr package then I'm not sure as I've not used it.
", "rawMarkdown": "Hi Anton,\n\nI was using laptop with the 8GB RAM for many years - also for Kaggle competitions and I was able to use R without too many issues even when data were quite big. However, sometimes it required batch processing or other tricks to save the memory.\n\nHowever, you can check if there are any previously created objects in R session which can use the memory. If so, you can remove them and release RAM (you can fore it with *gc()* function).\n\nIf it is matter of a data size which you need to train a model, you can consider using *data.table* instead built-in *data.frame* (https://cran.r-project.org/web/packages/data.table/vignettes/datatable-intro.html).\n\nIf the reason is related to *mlr* package then I'm not sure as I've not used it.", "votes": null }, { "id": "407279", "postDate": "10/20/2018 19:39:32", "content": "I would suggest trying Kaggle kernels with R.
", "rawMarkdown": "I would suggest trying Kaggle kernels with R.", "votes": null }, { "id": "407479", "postDate": "10/21/2018 09:20:09", "content": "8G is enough for simple models without much feature engineering. I'd suggest to train a model, remove unnecessary variables (rm() and gc()) and only then load the test set.
", "rawMarkdown": "8G is enough for simple models without much feature engineering. I'd suggest to train a model, remove unnecessary variables (**rm()** and **gc()**) and only then load the test set.", "votes": null }, { "id": "407506", "postDate": "10/21/2018 10:57:00", "content": "Thank you for your replies,\nI converted my dataframes to datatables and noticed they occupy the same size , nonetheless data.table looks like a nice structure to use in R given the easy syntax and commands. It also loads faster from disk.\nI loaded a train set in a kernel and it ran OK, thanks Konrad.\nHowever, you loose out on having to upload and download the sets.\nWell, it beats seeing R crash every time due to insufficient memory.\nKxx, the dataframe is totally unmanageble when converting an attribute into a dummyfeature with createdummyfeatures()
\n\ngood look with the competition
\n\nOne more thing, are you guys running on a linux or windows machine?\nI've heard that ubuntu handles memory better
\n\nThanks again\nAnton
", "rawMarkdown": "Thank you for your replies,\nI converted my dataframes to datatables and noticed they occupy the same size , nonetheless data.table looks like a nice structure to use in R given the easy syntax and commands. It also loads faster from disk.\nI loaded a train set in a kernel and it ran OK, thanks Konrad.\nHowever, you loose out on having to upload and download the sets.\nWell, it beats seeing R crash every time due to insufficient memory.\nKxx, the dataframe is totally unmanageble when converting an attribute into a dummyfeature with createdummyfeatures()\n\ngood look with the competition\n\nOne more thing, are you guys running on a linux or windows machine?\nI've heard that ubuntu handles memory better\n\nThanks again\nAnton", "votes": null }, { "id": "407511", "postDate": "10/21/2018 11:19:42", "content": "I'm using R with a 16GB RAM windows machine.\nI've used it in this competition without any encounters of insufficient memory, with extensive feature engineering, and using the data.table package. (I love data.table!)
", "rawMarkdown": "I'm using R with a 16GB RAM windows machine.\nI've used it in this competition without any encounters of insufficient memory, with extensive feature engineering, and using the data.table package. (I love data.table!)", "votes": null }, { "id": "408048", "postDate": "10/22/2018 08:14:22", "content": "Thanks Amir
", "rawMarkdown": "Thanks Amir", "votes": null }, { "id": "410797", "postDate": "10/26/2018 17:24:00", "content": "sapply(df, object.size) showed me the fullVisitorId was a real memory hog. I used bit64 library to turn it into an integer64. Also massively sped up dplyr::group_by which seems to struggle on strings.
", "rawMarkdown": "sapply(df, object.size) showed me the fullVisitorId was a real memory hog. I used bit64 library to turn it into an integer64. Also massively sped up dplyr::group_by which seems to struggle on strings.", "votes": null } ], "comments": [ { "id": 407245, "author_name": "aharasim", "author_url": "", "post_date": "10/20/2018 17:43:54", "content": "Hi Anton,
\n\nI was using laptop with the 8GB RAM for many years - also for Kaggle competitions and I was able to use R without too many issues even when data were quite big. However, sometimes it required batch processing or other tricks to save the memory.
\n\nHowever, you can check if there are any previously created objects in R session which can use the memory. If so, you can remove them and release RAM (you can fore it with gc() function).
\n\nIf it is matter of a data size which you need to train a model, you can consider using data.table instead built-in data.frame (https://cran.r-project.org/web/packages/data.table/vignettes/datatable-intro.html).
\n\nIf the reason is related to mlr package then I'm not sure as I've not used it.
", "votes": null, "replies": [] }, { "id": 407279, "author_name": "konradb", "author_url": "", "post_date": "10/20/2018 19:39:32", "content": "I would suggest trying Kaggle kernels with R.
", "votes": null, "replies": [] }, { "id": 407479, "author_name": "kailex", "author_url": "", "post_date": "10/21/2018 09:20:09", "content": "8G is enough for simple models without much feature engineering. I'd suggest to train a model, remove unnecessary variables (rm() and gc()) and only then load the test set.
", "votes": null, "replies": [] }, { "id": 407506, "author_name": "antonspartan", "author_url": "", "post_date": "10/21/2018 10:57:00", "content": "Thank you for your replies,\nI converted my dataframes to datatables and noticed they occupy the same size , nonetheless data.table looks like a nice structure to use in R given the easy syntax and commands. It also loads faster from disk.\nI loaded a train set in a kernel and it ran OK, thanks Konrad.\nHowever, you loose out on having to upload and download the sets.\nWell, it beats seeing R crash every time due to insufficient memory.\nKxx, the dataframe is totally unmanageble when converting an attribute into a dummyfeature with createdummyfeatures()
\n\ngood look with the competition
\n\nOne more thing, are you guys running on a linux or windows machine?\nI've heard that ubuntu handles memory better
\n\nThanks again\nAnton
", "votes": null, "replies": [] }, { "id": 407511, "author_name": "tpthegreat", "author_url": "", "post_date": "10/21/2018 11:19:42", "content": "I'm using R with a 16GB RAM windows machine.\nI've used it in this competition without any encounters of insufficient memory, with extensive feature engineering, and using the data.table package. (I love data.table!)
", "votes": null, "replies": [] }, { "id": 408048, "author_name": "antonspartan", "author_url": "", "post_date": "10/22/2018 08:14:22", "content": "Thanks Amir
", "votes": null, "replies": [ { "id": 410797, "author_name": "burritodan", "author_url": "", "post_date": "10/26/2018 17:24:00", "content": "sapply(df, object.size) showed me the fullVisitorId was a real memory hog. I used bit64 library to turn it into an integer64. Also massively sped up dplyr::group_by which seems to struggle on strings.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "406975": "Hi All,\nI've been working on this competition for 2 weeks now with R and mlr .\nI'm working on a laptop with 8GB RAM which crashes often due to memory peeks.\nWhat's your hardware configuration?\nIs there any other way to handle this within R?", "407245": "Hi Anton,\n\nI was using laptop with the 8GB RAM for many years - also for Kaggle competitions and I was able to use R without too many issues even when data were quite big. However, sometimes it required batch processing or other tricks to save the memory.\n\nHowever, you can check if there are any previously created objects in R session which can use the memory. If so, you can remove them and release RAM (you can fore it with *gc()* function).\n\nIf it is matter of a data size which you need to train a model, you can consider using *data.table* instead built-in *data.frame* (https://cran.r-project.org/web/packages/data.table/vignettes/datatable-intro.html).\n\nIf the reason is related to *mlr* package then I'm not sure as I've not used it.", "407279": "I would suggest trying Kaggle kernels with R.", "407479": "8G is enough for simple models without much feature engineering. I'd suggest to train a model, remove unnecessary variables (**rm()** and **gc()**) and only then load the test set.", "407506": "Thank you for your replies,\nI converted my dataframes to datatables and noticed they occupy the same size , nonetheless data.table looks like a nice structure to use in R given the easy syntax and commands. It also loads faster from disk.\nI loaded a train set in a kernel and it ran OK, thanks Konrad.\nHowever, you loose out on having to upload and download the sets.\nWell, it beats seeing R crash every time due to insufficient memory.\nKxx, the dataframe is totally unmanageble when converting an attribute into a dummyfeature with createdummyfeatures()\n\ngood look with the competition\n\nOne more thing, are you guys running on a linux or windows machine?\nI've heard that ubuntu handles memory better\n\nThanks again\nAnton", "407511": "I'm using R with a 16GB RAM windows machine.\nI've used it in this competition without any encounters of insufficient memory, with extensive feature engineering, and using the data.table package. (I love data.table!)", "408048": "Thanks Amir", "410797": "sapply(df, object.size) showed me the fullVisitorId was a real memory hog. I used bit64 library to turn it into an integer64. Also massively sped up dplyr::group_by which seems to struggle on strings." }, "source": "meta" }