Hello,
\n\nI have machine with 8GB Ram but due to large size of data my machine not working properly.
\n\nAny solution to this ? I am using R.
", "messages": [ { "id": "123877", "postDate": "06/14/2016 08:55:21", "content": "Hello,
\n\nI have machine with 8GB Ram but due to large size of data my machine not working properly.
\n\nAny solution to this ? I am using R.
", "rawMarkdown": "Hello,\r\n\r\nI have machine with 8GB Ram but due to large size of data my machine not working properly.\r\n\r\n Any solution to this ? I am using R.", "votes": null }, { "id": "123881", "postDate": "06/14/2016 09:07:42", "content": "I am working in MATLAB, but I think if you have small RAM (as I do) . It is better to first dump data into a local database and read/write via a database wrapper. You also need to build index to accelerate the random access of data.
\n\nThe actual data (about 500MB, binary) is not that large. More or less, you need to think harder what should be in RAM and what should be in database, so as to minimize the overheads of computations.
", "rawMarkdown": "I am working in MATLAB, but I think if you have small RAM (as I do) . It is better to first dump data into a local database and read/write via a database wrapper. You also need to build index to accelerate the random access of data. \r\n\r\nThe actual data (about 500MB, binary) is not that large. More or less, you need to think harder what should be in RAM and what should be in database, so as to minimize the overheads of computations.", "votes": null }, { "id": "123915", "postDate": "06/14/2016 13:21:57", "content": "A database is a good idea. You could also stream through the file one row at a time in R and accumulate statistics.
", "rawMarkdown": "A database is a good idea. You could also stream through the file one row at a time in R and accumulate statistics.", "votes": null }, { "id": "123939", "postDate": "06/14/2016 15:27:47", "content": "I would suggest trying data.table: https://cran.r-project.org/web/packages/data.table/vignettes/datatable-intro.pdf
", "rawMarkdown": "I would suggest trying data.table: https://cran.r-project.org/web/packages/data.table/vignettes/datatable-intro.pdf", "votes": null }, { "id": "123943", "postDate": "06/14/2016 15:36:25", "content": "data.table is an excellent suggestion— try the fread function.
\n\nNote that as soon as you try to do any kinds of joins or aggregations with this dataset, (even with data.table) you may run over your 8GB of RAM.
", "rawMarkdown": "data.table is an excellent suggestion— try the fread function.\r\n\r\nNote that as soon as you try to do any kinds of joins or aggregations with this dataset, (even with data.table) you may run over your 8GB of RAM.", "votes": null }, { "id": "124009", "postDate": "06/15/2016 00:02:04", "content": "Thanks for the tip but as soon as I start performing any action (even downsampling), memory allocation issues crop up. I, too, have a machine with 8 GB RAM. Any suggestions?
", "rawMarkdown": "Thanks for the tip but as soon as I start performing any action (even downsampling), memory allocation issues crop up. I, too, have a machine with 8 GB RAM. Any suggestions?", "votes": null }, { "id": "124017", "postDate": "06/15/2016 01:02:23", "content": "Read in the data one line at a time. For each line, pick a random number between 1 and 10,000. If that number equals 42, keep the line, otherwise discard it.
", "rawMarkdown": "Read in the data one line at a time. For each line, pick a random number between 1 and 10,000. If that number equals 42, keep the line, otherwise discard it.", "votes": null }, { "id": "124018", "postDate": "06/15/2016 01:03:04", "content": "Actually, here is some useful code from stack overflow:\nhttp://stackoverflow.com/a/18708007/345660
\n\nn <- 1000\ncon <- file("input/train.csv", open = "r")\nhead <- readLines(con, 1)\nsampdat <- readLines(con, n)\nk <- n\nwhile (length(curline <- readLines(con, 1))) {\n k <- k + 1\n if (runif(1) < n/k)\n sampdat[sample(n, 1)] <- curline\n}\nclose(con)\ndelaysamp <- read.csv(textConnection(c(head, sampdat)))\nThis will sample 1000 rows of data. You can increase n for a larger sample.
", "rawMarkdown": "Actually, here is some useful code from stack overflow:\r\nhttp://stackoverflow.com/a/18708007/345660\r\n\r\n\r\n n <- 1000\r\n con <- file(\"input/train.csv\", open = \"r\")\r\n head <- readLines(con, 1)\r\n sampdat <- readLines(con, n)\r\n k <- n\r\n while (length(curline <- readLines(con, 1))) {\r\n k <- k + 1\r\n if (runif(1) < n/k)\r\n sampdat[sample(n, 1)] <- curline\r\n }\r\n close(con)\r\n delaysamp <- read.csv(textConnection(c(head, sampdat)))\r\n\r\nThis will sample 1000 rows of data. You can increase n for a larger sample.", "votes": null }, { "id": "124023", "postDate": "06/15/2016 02:04:14", "content": "Thank you, Zach. That helped.
", "rawMarkdown": "Thank you, Zach. That helped.", "votes": null }, { "id": "124062", "postDate": "06/15/2016 08:43:40", "content": "Thank you Zach. Can you tell me how it samples the data. is it stratified sampling ?\nIf not then how to do stratified sampling
", "rawMarkdown": "Thank you Zach. Can you tell me how it samples the data. is it stratified sampling ?\r\nIf not then how to do stratified sampling", "votes": null }, { "id": "124064", "postDate": "06/15/2016 08:56:10", "content": "[quote=Ajay_Jadhav;124062]
\n\nThank you Zach. Can you tell me how it samples the data. is it stratified sampling ?\nIf not then how to do stratified sampling
\n\n[/quote]
\n\nClearly, that is not stratified sampling. If you do want stratified sampling, you should dump data into a database and build index for the stratified variables. Then stratified sampling becomes much simpler and efficient.
\n\nJianbo
", "rawMarkdown": "[quote=Ajay_Jadhav;124062]\r\n\r\nThank you Zach. Can you tell me how it samples the data. is it stratified sampling ?\r\nIf not then how to do stratified sampling\r\n\r\n[/quote]\r\n\r\nClearly, that is not stratified sampling. If you do want stratified sampling, you should dump data into a database and build index for the stratified variables. Then stratified sampling becomes much simpler and efficient. \r\n\r\nJianbo", "votes": null }, { "id": "124080", "postDate": "06/15/2016 12:17:45", "content": "Yes, this is a random sample. You don't need to do stratified sampling for this competition (and even if you did, I can't think of an easy way to do that in a streaming fashion).
\n\nIf you really want stratified sampling, you're on your own to do it. If you're satisfied with a random sample, use the above code to get 10k or 100k rows of data and go from there.
", "rawMarkdown": "Yes, this is a random sample. You don't need to do stratified sampling for this competition (and even if you did, I can't think of an easy way to do that in a streaming fashion).\r\n\r\nIf you really want stratified sampling, you're on your own to do it. If you're satisfied with a random sample, use the above code to get 10k or 100k rows of data and go from there.", "votes": null }, { "id": "124081", "postDate": "06/15/2016 12:20:53", "content": "Also, there's some useful scripts already in the scripts section:\nhttps://www.kaggle.com/c/grupo-bimbo-inventory-demand/scripts
\n\nIn particular, this one demonstrates how to use fread to only select a few columns:\nhttps://www.kaggle.com/nigelcarpenter/grupo-bimbo-inventory-demand/r-medians/code
", "rawMarkdown": "Also, there's some useful scripts already in the scripts section:\r\nhttps://www.kaggle.com/c/grupo-bimbo-inventory-demand/scripts\r\n\r\nIn particular, this one demonstrates how to use fread to only select a few columns:\r\nhttps://www.kaggle.com/nigelcarpenter/grupo-bimbo-inventory-demand/r-medians/code", "votes": null }, { "id": "124153", "postDate": "06/15/2016 20:32:31", "content": "The readr package is very helpful. As well as allowing you to read in larger data files, the main read_* functions all have a maximum number of lines parameter.
\n\nvignette;
\nhttps://cran.r-project.org/web/packages/readr/readr.pdf
\nrelease blog;
\nhttps://blog.rstudio.org/2015/04/09/readr-0-1-0/
fread and read.csv also have a maximum number of lines parameter.
\n\nBut I agree, readr is excellent and well documented.
", "rawMarkdown": "fread and read.csv also have a maximum number of lines parameter.\r\n\r\nBut I agree, readr is excellent and well documented.", "votes": null }, { "id": "124214", "postDate": "06/16/2016 09:29:28", "content": "I have used readr package and it is working fine for importing data.
\n\nlibrary(readr)\ntrain=read_csv("train.csv")\nIn case you face "Reached total allocation of 8097Mb: see help(memory.size)" error while precessing data then you can increase size by using following command in R.
\n\nmemory.limit(size = 15000)\nFor 64 bit version of R, maximum size is 8 Tb.
", "rawMarkdown": "I have used readr package and it is working fine for importing data.\r\n\r\n library(readr)\r\n train=read_csv(\"train.csv\")\r\n\r\nIn case you face \"**Reached total allocation of 8097Mb: see help(memory.size)**\" error while precessing data then you can increase size by using following command in R.\r\n\r\n memory.limit(size = 15000)\r\n\r\nFor 64 bit version of R, maximum size is 8 Tb.", "votes": null }, { "id": "124233", "postDate": "06/16/2016 11:21:06", "content": "I used fread and it works well and fast too.
\n\nThe other issue you may encounter is that while doing your calculations (apply, select, transpose etc ...) the large dataset may result in your computer running out of disk space.
\n\nWhat I found useful would be to transform / join / merge your data and formulate your algorithim in such a manner that will result in only needing to process the test data set. Test data is approx 6 million, train data is approx 75 million so it makes sense to run the calculations against 6 million.
\n\nHope this helps as I too was struggling to process 75 million records :D
\n\nIan
", "rawMarkdown": "I used fread and it works well and fast too.\r\n\r\nThe other issue you may encounter is that while doing your calculations (apply, select, transpose etc ...) the large dataset may result in your computer running out of disk space.\r\n\r\nWhat I found useful would be to transform / join / merge your data and formulate your algorithim in such a manner that will result in only needing to process the test data set. Test data is approx 6 million, train data is approx 75 million so it makes sense to run the calculations against 6 million.\r\n\r\nHope this helps as I too was struggling to process 75 million records :D\r\n\r\nIan", "votes": null } ], "comments": [ { "id": 123881, "author_name": "amazingbob", "author_url": "", "post_date": "06/14/2016 09:07:42", "content": "I am working in MATLAB, but I think if you have small RAM (as I do) . It is better to first dump data into a local database and read/write via a database wrapper. You also need to build index to accelerate the random access of data.
\n\nThe actual data (about 500MB, binary) is not that large. More or less, you need to think harder what should be in RAM and what should be in database, so as to minimize the overheads of computations.
", "votes": null, "replies": [] }, { "id": 123915, "author_name": "zachmayer", "author_url": "", "post_date": "06/14/2016 13:21:57", "content": "A database is a good idea. You could also stream through the file one row at a time in R and accumulate statistics.
", "votes": null, "replies": [] }, { "id": 123939, "author_name": "konradb", "author_url": "", "post_date": "06/14/2016 15:27:47", "content": "I would suggest trying data.table: https://cran.r-project.org/web/packages/data.table/vignettes/datatable-intro.pdf
", "votes": null, "replies": [] }, { "id": 123943, "author_name": "zachmayer", "author_url": "", "post_date": "06/14/2016 15:36:25", "content": "data.table is an excellent suggestion— try the fread function.
\n\nNote that as soon as you try to do any kinds of joins or aggregations with this dataset, (even with data.table) you may run over your 8GB of RAM.
", "votes": null, "replies": [] }, { "id": 124009, "author_name": "su13odh", "author_url": "", "post_date": "06/15/2016 00:02:04", "content": "Thanks for the tip but as soon as I start performing any action (even downsampling), memory allocation issues crop up. I, too, have a machine with 8 GB RAM. Any suggestions?
", "votes": null, "replies": [] }, { "id": 124017, "author_name": "zachmayer", "author_url": "", "post_date": "06/15/2016 01:02:23", "content": "Read in the data one line at a time. For each line, pick a random number between 1 and 10,000. If that number equals 42, keep the line, otherwise discard it.
", "votes": null, "replies": [] }, { "id": 124018, "author_name": "zachmayer", "author_url": "", "post_date": "06/15/2016 01:03:04", "content": "Actually, here is some useful code from stack overflow:\nhttp://stackoverflow.com/a/18708007/345660
\n\nn <- 1000\ncon <- file("input/train.csv", open = "r")\nhead <- readLines(con, 1)\nsampdat <- readLines(con, n)\nk <- n\nwhile (length(curline <- readLines(con, 1))) {\n k <- k + 1\n if (runif(1) < n/k)\n sampdat[sample(n, 1)] <- curline\n}\nclose(con)\ndelaysamp <- read.csv(textConnection(c(head, sampdat)))\nThis will sample 1000 rows of data. You can increase n for a larger sample.
", "votes": null, "replies": [] }, { "id": 124023, "author_name": "su13odh", "author_url": "", "post_date": "06/15/2016 02:04:14", "content": "Thank you, Zach. That helped.
", "votes": null, "replies": [] }, { "id": 124062, "author_name": "ajayjadhav", "author_url": "", "post_date": "06/15/2016 08:43:40", "content": "Thank you Zach. Can you tell me how it samples the data. is it stratified sampling ?\nIf not then how to do stratified sampling
", "votes": null, "replies": [] }, { "id": 124064, "author_name": "amazingbob", "author_url": "", "post_date": "06/15/2016 08:56:10", "content": "[quote=Ajay_Jadhav;124062]
\n\nThank you Zach. Can you tell me how it samples the data. is it stratified sampling ?\nIf not then how to do stratified sampling
\n\n[/quote]
\n\nClearly, that is not stratified sampling. If you do want stratified sampling, you should dump data into a database and build index for the stratified variables. Then stratified sampling becomes much simpler and efficient.
\n\nJianbo
", "votes": null, "replies": [] }, { "id": 124080, "author_name": "zachmayer", "author_url": "", "post_date": "06/15/2016 12:17:45", "content": "Yes, this is a random sample. You don't need to do stratified sampling for this competition (and even if you did, I can't think of an easy way to do that in a streaming fashion).
\n\nIf you really want stratified sampling, you're on your own to do it. If you're satisfied with a random sample, use the above code to get 10k or 100k rows of data and go from there.
", "votes": null, "replies": [] }, { "id": 124081, "author_name": "zachmayer", "author_url": "", "post_date": "06/15/2016 12:20:53", "content": "Also, there's some useful scripts already in the scripts section:\nhttps://www.kaggle.com/c/grupo-bimbo-inventory-demand/scripts
\n\nIn particular, this one demonstrates how to use fread to only select a few columns:\nhttps://www.kaggle.com/nigelcarpenter/grupo-bimbo-inventory-demand/r-medians/code
", "votes": null, "replies": [] }, { "id": 124153, "author_name": "johnflynn", "author_url": "", "post_date": "06/15/2016 20:32:31", "content": "The readr package is very helpful. As well as allowing you to read in larger data files, the main read_* functions all have a maximum number of lines parameter.
\n\nvignette;
\nhttps://cran.r-project.org/web/packages/readr/readr.pdf
\nrelease blog;
\nhttps://blog.rstudio.org/2015/04/09/readr-0-1-0/
fread and read.csv also have a maximum number of lines parameter.
\n\nBut I agree, readr is excellent and well documented.
", "votes": null, "replies": [] }, { "id": 124214, "author_name": "ksdwor", "author_url": "", "post_date": "06/16/2016 09:29:28", "content": "I have used readr package and it is working fine for importing data.
\n\nlibrary(readr)\ntrain=read_csv("train.csv")\nIn case you face "Reached total allocation of 8097Mb: see help(memory.size)" error while precessing data then you can increase size by using following command in R.
\n\nmemory.limit(size = 15000)\nFor 64 bit version of R, maximum size is 8 Tb.
", "votes": null, "replies": [] }, { "id": 124233, "author_name": "ianlokh", "author_url": "", "post_date": "06/16/2016 11:21:06", "content": "I used fread and it works well and fast too.
\n\nThe other issue you may encounter is that while doing your calculations (apply, select, transpose etc ...) the large dataset may result in your computer running out of disk space.
\n\nWhat I found useful would be to transform / join / merge your data and formulate your algorithim in such a manner that will result in only needing to process the test data set. Test data is approx 6 million, train data is approx 75 million so it makes sense to run the calculations against 6 million.
\n\nHope this helps as I too was struggling to process 75 million records :D
\n\nIan
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "123877": "Hello,\r\n\r\nI have machine with 8GB Ram but due to large size of data my machine not working properly.\r\n\r\n Any solution to this ? I am using R.", "123881": "I am working in MATLAB, but I think if you have small RAM (as I do) . It is better to first dump data into a local database and read/write via a database wrapper. You also need to build index to accelerate the random access of data. \r\n\r\nThe actual data (about 500MB, binary) is not that large. More or less, you need to think harder what should be in RAM and what should be in database, so as to minimize the overheads of computations.", "123915": "A database is a good idea. You could also stream through the file one row at a time in R and accumulate statistics.", "123939": "I would suggest trying data.table: https://cran.r-project.org/web/packages/data.table/vignettes/datatable-intro.pdf", "123943": "data.table is an excellent suggestion— try the fread function.\r\n\r\nNote that as soon as you try to do any kinds of joins or aggregations with this dataset, (even with data.table) you may run over your 8GB of RAM.", "124009": "Thanks for the tip but as soon as I start performing any action (even downsampling), memory allocation issues crop up. I, too, have a machine with 8 GB RAM. Any suggestions?", "124017": "Read in the data one line at a time. For each line, pick a random number between 1 and 10,000. If that number equals 42, keep the line, otherwise discard it.", "124018": "Actually, here is some useful code from stack overflow:\r\nhttp://stackoverflow.com/a/18708007/345660\r\n\r\n\r\n n <- 1000\r\n con <- file(\"input/train.csv\", open = \"r\")\r\n head <- readLines(con, 1)\r\n sampdat <- readLines(con, n)\r\n k <- n\r\n while (length(curline <- readLines(con, 1))) {\r\n k <- k + 1\r\n if (runif(1) < n/k)\r\n sampdat[sample(n, 1)] <- curline\r\n }\r\n close(con)\r\n delaysamp <- read.csv(textConnection(c(head, sampdat)))\r\n\r\nThis will sample 1000 rows of data. You can increase n for a larger sample.", "124023": "Thank you, Zach. That helped.", "124062": "Thank you Zach. Can you tell me how it samples the data. is it stratified sampling ?\r\nIf not then how to do stratified sampling", "124064": "[quote=Ajay_Jadhav;124062]\r\n\r\nThank you Zach. Can you tell me how it samples the data. is it stratified sampling ?\r\nIf not then how to do stratified sampling\r\n\r\n[/quote]\r\n\r\nClearly, that is not stratified sampling. If you do want stratified sampling, you should dump data into a database and build index for the stratified variables. Then stratified sampling becomes much simpler and efficient. \r\n\r\nJianbo", "124080": "Yes, this is a random sample. You don't need to do stratified sampling for this competition (and even if you did, I can't think of an easy way to do that in a streaming fashion).\r\n\r\nIf you really want stratified sampling, you're on your own to do it. If you're satisfied with a random sample, use the above code to get 10k or 100k rows of data and go from there.", "124081": "Also, there's some useful scripts already in the scripts section:\r\nhttps://www.kaggle.com/c/grupo-bimbo-inventory-demand/scripts\r\n\r\nIn particular, this one demonstrates how to use fread to only select a few columns:\r\nhttps://www.kaggle.com/nigelcarpenter/grupo-bimbo-inventory-demand/r-medians/code", "124153": "The readr package is very helpful. As well as allowing you to read in larger data files, the main read_* functions all have a maximum number of lines parameter.\r\n\r\nvignette;