I am getting memory error when fitting my model.\nHow can i overcome this ?
\n\nPS : using 64 bit system and R version 3.3.0
", "messages": [ { "id": "120236", "postDate": "05/16/2016 16:08:09", "content": "I am getting memory error when fitting my model.\nHow can i overcome this ?
\n\nPS : using 64 bit system and R version 3.3.0
", "rawMarkdown": "I am getting memory error when fitting my model.\r\nHow can i overcome this ?\r\n\r\nPS : using 64 bit system and R version 3.3.0", "votes": null }, { "id": "120241", "postDate": "05/16/2016 17:16:58", "content": "Hi, I had the same issue so I decided to store my data in Hadoop and use sparkR.
\n\nAs you can see, the data is rather big and since R needs to load this into memory it gives problems\n.... see http://analyticstraining.com/2014/memory-management-in-r-and-how-it-handles-big-data/
\n\nI am using the following references:
\n\nI hope this might be helpful.\nRegards
", "rawMarkdown": "Hi, I had the same issue so I decided to store my data in Hadoop and use sparkR. \r\n\r\nAs you can see, the data is rather big and since R needs to load this into memory it gives problems\r\n.... see http://analyticstraining.com/2014/memory-management-in-r-and-how-it-handles-big-data/\r\n\r\nI am using the following references:\r\n\r\n - https://community.hortonworks.com/articles/25558/running-sparkr-in-rstudio-using-hdp-24.html\r\n - https://spark.apache.org/docs/latest/sparkr.html\r\n\r\n\r\nI hope this might be helpful.\r\nRegards", "votes": null }, { "id": "120261", "postDate": "05/16/2016 21:47:20", "content": "What model are you trying to fit?
", "rawMarkdown": "What model are you trying to fit?", "votes": null }, { "id": "120327", "postDate": "05/17/2016 13:32:55", "content": "random forest
", "rawMarkdown": "random forest", "votes": null }, { "id": "120339", "postDate": "05/17/2016 15:07:35", "content": "[quote=tino;120241]
\n\nHi, I had the same issue so I decided to store my data in Hadoop and use sparkR.
\n\nAs you can see, the data is rather big and since R needs to load this into memory it gives problems\n.... see http://analyticstraining.com/2014/memory-management-in-r-and-how-it-handles-big-data/
\n\nI am using the following references:
\n\nI hope this might be helpful.\nRegards
\n\n[/quote]
", "rawMarkdown": "[quote=tino;120241]\r\n\r\nHi, I had the same issue so I decided to store my data in Hadoop and use sparkR. \r\n\r\nAs you can see, the data is rather big and since R needs to load this into memory it gives problems\r\n.... see http://analyticstraining.com/2014/memory-management-in-r-and-how-it-handles-big-data/\r\n\r\nI am using the following references:\r\n\r\n - https://community.hortonworks.com/articles/25558/running-sparkr-in-rstudio-using-hdp-24.html\r\n - https://spark.apache.org/docs/latest/sparkr.html\r\n\r\n\r\nI hope this might be helpful.\r\nRegards\r\n\r\n[/quote]", "votes": null }, { "id": "120340", "postDate": "05/17/2016 15:09:56", "content": "unable to launch spark library :( code to set enviroment variable is executed but on library(sparkR) it throws error "Error in library(SparkR) : there is no package called ‘SparkR’"
", "rawMarkdown": "unable to launch spark library :( code to set enviroment variable is executed but on library(sparkR) it throws error \"Error in library(SparkR) : there is no package called ‘SparkR’\"", "votes": null }, { "id": "120348", "postDate": "05/17/2016 17:04:22", "content": "All of the classification methods I've tested or read about seem like they won't work well with 108k+ classes. They are often limited at 1000, or even 32.
", "rawMarkdown": "All of the classification methods I've tested or read about seem like they won't work well with 108k+ classes. They are often limited at 1000, or even 32.", "votes": null }, { "id": "120870", "postDate": "05/21/2016 06:25:55", "content": "I wrote a script here that outlines how to do random forests on this data set by subdividing it into grids and doing a model on each subdivision - this makes sense in this case since clusers of place_id's are localized.
The only thing I could find that works directly on the entire dataset is k nearest neighbors.
", "rawMarkdown": "I wrote a script [here](https://www.kaggle.com/apapiu/facebook-v-predicting-check-ins/random-forest-on-a-few-blocks) that outlines how to do random forests on this data set by subdividing it into grids and doing a model on each subdivision - this makes sense in this case since clusers of `place_id`'s are localized.\r\n\r\nThe only thing I could find that works directly on the entire dataset is k nearest neighbors.", "votes": null } ], "comments": [ { "id": 120241, "author_name": "tinosibanda", "author_url": "", "post_date": "05/16/2016 17:16:58", "content": "Hi, I had the same issue so I decided to store my data in Hadoop and use sparkR.
\n\nAs you can see, the data is rather big and since R needs to load this into memory it gives problems\n.... see http://analyticstraining.com/2014/memory-management-in-r-and-how-it-handles-big-data/
\n\nI am using the following references:
\n\nI hope this might be helpful.\nRegards
", "votes": null, "replies": [] }, { "id": 120261, "author_name": "apapiu", "author_url": "", "post_date": "05/16/2016 21:47:20", "content": "What model are you trying to fit?
", "votes": null, "replies": [] }, { "id": 120327, "author_name": "snehil786", "author_url": "", "post_date": "05/17/2016 13:32:55", "content": "random forest
", "votes": null, "replies": [] }, { "id": 120339, "author_name": "snehil786", "author_url": "", "post_date": "05/17/2016 15:07:35", "content": "[quote=tino;120241]
\n\nHi, I had the same issue so I decided to store my data in Hadoop and use sparkR.
\n\nAs you can see, the data is rather big and since R needs to load this into memory it gives problems\n.... see http://analyticstraining.com/2014/memory-management-in-r-and-how-it-handles-big-data/
\n\nI am using the following references:
\n\nI hope this might be helpful.\nRegards
\n\n[/quote]
", "votes": null, "replies": [] }, { "id": 120340, "author_name": "snehil786", "author_url": "", "post_date": "05/17/2016 15:09:56", "content": "unable to launch spark library :( code to set enviroment variable is executed but on library(sparkR) it throws error "Error in library(SparkR) : there is no package called ‘SparkR’"
", "votes": null, "replies": [] }, { "id": 120348, "author_name": "zeraes", "author_url": "", "post_date": "05/17/2016 17:04:22", "content": "All of the classification methods I've tested or read about seem like they won't work well with 108k+ classes. They are often limited at 1000, or even 32.
", "votes": null, "replies": [] }, { "id": 120870, "author_name": "apapiu", "author_url": "", "post_date": "05/21/2016 06:25:55", "content": "I wrote a script here that outlines how to do random forests on this data set by subdividing it into grids and doing a model on each subdivision - this makes sense in this case since clusers of place_id's are localized.
The only thing I could find that works directly on the entire dataset is k nearest neighbors.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "120236": "I am getting memory error when fitting my model.\r\nHow can i overcome this ?\r\n\r\nPS : using 64 bit system and R version 3.3.0", "120241": "Hi, I had the same issue so I decided to store my data in Hadoop and use sparkR. \r\n\r\nAs you can see, the data is rather big and since R needs to load this into memory it gives problems\r\n.... see http://analyticstraining.com/2014/memory-management-in-r-and-how-it-handles-big-data/\r\n\r\nI am using the following references:\r\n\r\n - https://community.hortonworks.com/articles/25558/running-sparkr-in-rstudio-using-hdp-24.html\r\n - https://spark.apache.org/docs/latest/sparkr.html\r\n\r\n\r\nI hope this might be helpful.\r\nRegards", "120261": "What model are you trying to fit?", "120327": "random forest", "120339": "[quote=tino;120241]\r\n\r\nHi, I had the same issue so I decided to store my data in Hadoop and use sparkR. \r\n\r\nAs you can see, the data is rather big and since R needs to load this into memory it gives problems\r\n.... see http://analyticstraining.com/2014/memory-management-in-r-and-how-it-handles-big-data/\r\n\r\nI am using the following references:\r\n\r\n - https://community.hortonworks.com/articles/25558/running-sparkr-in-rstudio-using-hdp-24.html\r\n - https://spark.apache.org/docs/latest/sparkr.html\r\n\r\n\r\nI hope this might be helpful.\r\nRegards\r\n\r\n[/quote]", "120340": "unable to launch spark library :( code to set enviroment variable is executed but on library(sparkR) it throws error \"Error in library(SparkR) : there is no package called ‘SparkR’\"", "120348": "All of the classification methods I've tested or read about seem like they won't work well with 108k+ classes. They are often limited at 1000, or even 32.", "120870": "I wrote a script [here](https://www.kaggle.com/apapiu/facebook-v-predicting-check-ins/random-forest-on-a-few-blocks) that outlines how to do random forests on this data set by subdividing it into grids and doing a model on each subdivision - this makes sense in this case since clusers of `place_id`'s are localized.\r\n\r\nThe only thing I could find that works directly on the entire dataset is k nearest neighbors." }, "source": "meta" }