![\"enter](\"https://www.kaggle.com/blobs/download/forum-message-attachment-files/4706/bosch_memory_sparse.png\")
If you read this post, you might be interested in three different things (if you fit in one of these, then it will be of interest obviously):
\n\nRequirements:
\n\nThe following script (3 paragraphs below) should make you able to load the train_numeric set when you have less than 6GB RAM, store it locally (700MB), and then be able to load the whole set under 10 seconds. There are three requirements.
\n\nThe first requirement is to make a pass on the whole dataset by batches, to achieve the creation of a sparse matrix. This takes about 30 minutes with very small batches (10 features per batch) and a 5GB peak. On a AMD E-350 processor (yes, slower than Intel Atom) and a SSD, this took me 25 minutes.
\n\nThe second requirement is to save the sparse matrix to a RDS file, so you don't "pay" the hassle of loading the dataset everytime. Saving as RDS is very quick, and you have a ~700MB file at the end. But the best comes here: it takes only 10 seconds to load the dataset back! (if you are suspicious of whether the initial and loaded datasets are identical or not, use the function identical(matrix1, matrix2) ).
\n\nThe third requirement is to tune appropriately the first requirement (and what you do after the second requirement) so you don't explode the memory usage suddenly. You can also tune the requirements according to the sparse matrix requirements in the picture at the end of this post. This requirement is explained some paragraphs below this code (which does the first+second requirements):
\n\nlibrary(data.table)\nlibrary(recommenderlab)\nfor (i in 1:97) {\n\n cat("Loading ", i, "th part.\\n", sep = "")\n train_data_temp <- fread(input = "train_numeric.csv",\n select = (10*(i-1)+1):(10*i),\n header = TRUE,\n sep = ",",\n stringsAsFactors = FALSE,\n colClasses = rep("numeric", 970),\n data.table = TRUE)\n gc(verbose = FALSE)\n\n if (i > 1) {\n\n cat("Coercing to matrix.\\n", sep = "")\n train_numeric_temp <- as.matrix(train_data_temp)\n rm(train_data_temp)\n gc(verbose = FALSE)\n\n cat("Coercing into dgCMatrix with NA as blank.\\n", sep = "")\n train_numeric_temp <- dropNA(train_numeric_temp)\n gc(verbose = FALSE)\n\n cat("Column binding the full matrix with the newly created matrix.\\n", sep = "")\n train_numeric <- cbind(train_numeric, train_numeric_temp)\n rm(train_numeric_temp)\n gc(verbose = FALSE)\n\n } else {\n\n cat("Coercing to matrix.\\n", sep = "")\n train_numeric_temp <- as.matrix(train_data_temp)\n rm(train_data_temp)\n gc(verbose = FALSE)\n\n cat("Coercing into dgCMatrix with NA as blank.\\n", sep = "")\n train_numeric <- dropNA(train_numeric_temp)\n gc(verbose = FALSE)\n\n }\n\n}\n\ngc()\nsaveRDS(train_numeric, file = "train_numeric.rds")\nExample of verbose content output by the script:
\n\nLoading 94th part.\nRead 1183747 rows and 10 (of 970) columns from 1.993 GB file in 00:00:13\nCoercing to matrix.\nCoercing into dgCMatrix with NA as blank.\nColumn binding the full matrix with the newly created matrix.\nLoading 95th part.\nRead 1183747 rows and 10 (of 970) columns from 1.993 GB file in 00:00:13\nCoercing to matrix.\nCoercing into dgCMatrix with NA as blank.\nColumn binding the full matrix with the newly created matrix.\nLoading 96th part.\nRead 1183747 rows and 10 (of 970) columns from 1.993 GB file in 00:00:13\nCoercing to matrix.\nCoercing into dgCMatrix with NA as blank.\nColumn binding the full matrix with the newly created matrix.\nLoading 97th part.\nRead 1183747 rows and 10 (of 970) columns from 1.993 GB file in 00:00:13\nCoercing to matrix.\nCoercing into dgCMatrix with NA as blank.\nColumn binding the full matrix with the newly created matrix.\nThe last gc() has verbose=TRUE by default and you should get similar to this:
\n\n> gc()\n used (Mb) gc trigger (Mb) max used (Mb)\nNcells 1660022 88.7 2637877 140.9 2164898 115.7\nVcells 330920318 2524.8 668140682 5097.6 659586557 5032.3\nOnly ~5GB memory usage at peak! (and at the end: ~2.5GB only!)
\n\nYou can reduce further the memory usage by splitting the loading into two large chunks (470 first columns in one sparse matrix with small batches, 470 last columns in a second sparse matrix with small batches), saving both sparse matrices, restarting R, loading both sparse matrices, and column binding them. In this case, it should make a 3.6GB + swap peak during the merge. At the end, you have a 2.4GB matrix that fits in memory. This makes it largely feasible to load the entire set in a 4GB laptop, and work with it. You can even do feature selection if done appropriately (and loading back the whole dataset is very quick if what you are doing to select features take lot of RAM).
\n\nIn any case, if you increase the batch size (more features loaded per batch), you will increase the memory usage. But the benefit is a smaller loading speed overall (when taking into account the whole dataset).
\n\nTune i lower for faster processing: you have 970 features in train_numeric, therefore if the max i=97, you should:
\n\nHence, modify the "select" in fread appropriately if you need to load more features in one batch. For instance, using i from 1 to 10 leads up to the following (97 features to load per batch):
\n\n train_data_temp <- fread(input = "train_numeric.csv",\n select = (97*(i-1)+1):(97*i),\n header = TRUE,\n sep = ",",\n stringsAsFactors = FALSE,\n colClasses = rep("numeric", 970),\n data.table = TRUE)\nWhat you get?:
\n\nIf you work with less than 4GB RAM (like 3GB), you will need to work very smartly to achieve the lowest RAM usage possible. I provide the following picture to give you an insight on where your memory will literally explode, as it is clearly visible where it should explode:
\n\nYou can tune aggressively the batches if you want to keep under 3GB usage. However, you will need at least 2.4GB to make it work properly at the end, so you are left with 600MB free (in theory).\n
If you read this post, you might be interested in three different things (if you fit in one of these, then it will be of interest obviously):
\n\nRequirements:
\n\nThe following script (3 paragraphs below) should make you able to load the train_numeric set when you have less than 6GB RAM, store it locally (700MB), and then be able to load the whole set under 10 seconds. There are three requirements.
\n\nThe first requirement is to make a pass on the whole dataset by batches, to achieve the creation of a sparse matrix. This takes about 30 minutes with very small batches (10 features per batch) and a 5GB peak. On a AMD E-350 processor (yes, slower than Intel Atom) and a SSD, this took me 25 minutes.
\n\nThe second requirement is to save the sparse matrix to a RDS file, so you don't "pay" the hassle of loading the dataset everytime. Saving as RDS is very quick, and you have a ~700MB file at the end. But the best comes here: it takes only 10 seconds to load the dataset back! (if you are suspicious of whether the initial and loaded datasets are identical or not, use the function identical(matrix1, matrix2) ).
\n\nThe third requirement is to tune appropriately the first requirement (and what you do after the second requirement) so you don't explode the memory usage suddenly. You can also tune the requirements according to the sparse matrix requirements in the picture at the end of this post. This requirement is explained some paragraphs below this code (which does the first+second requirements):
\n\nlibrary(data.table)\nlibrary(recommenderlab)\nfor (i in 1:97) {\n\n cat("Loading ", i, "th part.\\n", sep = "")\n train_data_temp <- fread(input = "train_numeric.csv",\n select = (10*(i-1)+1):(10*i),\n header = TRUE,\n sep = ",",\n stringsAsFactors = FALSE,\n colClasses = rep("numeric", 970),\n data.table = TRUE)\n gc(verbose = FALSE)\n\n if (i > 1) {\n\n cat("Coercing to matrix.\\n", sep = "")\n train_numeric_temp <- as.matrix(train_data_temp)\n rm(train_data_temp)\n gc(verbose = FALSE)\n\n cat("Coercing into dgCMatrix with NA as blank.\\n", sep = "")\n train_numeric_temp <- dropNA(train_numeric_temp)\n gc(verbose = FALSE)\n\n cat("Column binding the full matrix with the newly created matrix.\\n", sep = "")\n train_numeric <- cbind(train_numeric, train_numeric_temp)\n rm(train_numeric_temp)\n gc(verbose = FALSE)\n\n } else {\n\n cat("Coercing to matrix.\\n", sep = "")\n train_numeric_temp <- as.matrix(train_data_temp)\n rm(train_data_temp)\n gc(verbose = FALSE)\n\n cat("Coercing into dgCMatrix with NA as blank.\\n", sep = "")\n train_numeric <- dropNA(train_numeric_temp)\n gc(verbose = FALSE)\n\n }\n\n}\n\ngc()\nsaveRDS(train_numeric, file = "train_numeric.rds")\nExample of verbose content output by the script:
\n\nLoading 94th part.\nRead 1183747 rows and 10 (of 970) columns from 1.993 GB file in 00:00:13\nCoercing to matrix.\nCoercing into dgCMatrix with NA as blank.\nColumn binding the full matrix with the newly created matrix.\nLoading 95th part.\nRead 1183747 rows and 10 (of 970) columns from 1.993 GB file in 00:00:13\nCoercing to matrix.\nCoercing into dgCMatrix with NA as blank.\nColumn binding the full matrix with the newly created matrix.\nLoading 96th part.\nRead 1183747 rows and 10 (of 970) columns from 1.993 GB file in 00:00:13\nCoercing to matrix.\nCoercing into dgCMatrix with NA as blank.\nColumn binding the full matrix with the newly created matrix.\nLoading 97th part.\nRead 1183747 rows and 10 (of 970) columns from 1.993 GB file in 00:00:13\nCoercing to matrix.\nCoercing into dgCMatrix with NA as blank.\nColumn binding the full matrix with the newly created matrix.\nThe last gc() has verbose=TRUE by default and you should get similar to this:
\n\n> gc()\n used (Mb) gc trigger (Mb) max used (Mb)\nNcells 1660022 88.7 2637877 140.9 2164898 115.7\nVcells 330920318 2524.8 668140682 5097.6 659586557 5032.3\nOnly ~5GB memory usage at peak! (and at the end: ~2.5GB only!)
\n\nYou can reduce further the memory usage by splitting the loading into two large chunks (470 first columns in one sparse matrix with small batches, 470 last columns in a second sparse matrix with small batches), saving both sparse matrices, restarting R, loading both sparse matrices, and column binding them. In this case, it should make a 3.6GB + swap peak during the merge. At the end, you have a 2.4GB matrix that fits in memory. This makes it largely feasible to load the entire set in a 4GB laptop, and work with it. You can even do feature selection if done appropriately (and loading back the whole dataset is very quick if what you are doing to select features take lot of RAM).
\n\nIn any case, if you increase the batch size (more features loaded per batch), you will increase the memory usage. But the benefit is a smaller loading speed overall (when taking into account the whole dataset).
\n\nTune i lower for faster processing: you have 970 features in train_numeric, therefore if the max i=97, you should:
\n\nHence, modify the "select" in fread appropriately if you need to load more features in one batch. For instance, using i from 1 to 10 leads up to the following (97 features to load per batch):
\n\n train_data_temp <- fread(input = "train_numeric.csv",\n select = (97*(i-1)+1):(97*i),\n header = TRUE,\n sep = ",",\n stringsAsFactors = FALSE,\n colClasses = rep("numeric", 970),\n data.table = TRUE)\nWhat you get?:
\n\nIf you work with less than 4GB RAM (like 3GB), you will need to work very smartly to achieve the lowest RAM usage possible. I provide the following picture to give you an insight on where your memory will literally explode, as it is clearly visible where it should explode:
\n\nYou can tune aggressively the batches if you want to keep under 3GB usage. However, you will need at least 2.4GB to make it work properly at the end, so you are left with 600MB free (in theory).\n
Thanks Laurae - nicely written up as always.
\n\nTo complement yours, here's a less polished example of a script to process the categorical data, in this case to an integer representation of each category with tree methods in mind. I will return with an edit to confirm the RAM use, but it's 5GB or thereabouts. The final sparse matrix is 800MB.
\n\nlibrary(data.table)\nlibrary(FeatureHashing)\nlibrary(Matrix)\n\nhash <- function(x) ifelse(is.na(x), 0, hashed.value(x) %% 2^16)\n\nbatches <- 20\ncol_first <- seq(2, 2140, by = 2140 / batches)\ncol_last <- col_first + (2140 / batches) - 1\n\nprocessed_data <- vector("list", length(col_first))\n\nfor(i in seq_along(col_first)) {\n\n dt <- fread("Data/train_categorical.csv",\n colClasses = "character",\n na.strings = "",\n select = c(col_first[i]:col_last[i]))\n\n for(col in names(dt)) set(dt, j = col, value = hash(dt[[col]]))\n\n processed_data[[i]] <- Matrix(as.matrix(dt), sparse = T)\n\n}\n\nall_data <- do.call("cbind", processed_data)\nsaveRDS(all_data, file = "Data/cat_train.rds")\n\nsum(colSums(all_data) == 0)\n#[1] 163\nsum(rowSums(all_data) == 0)\n#[1] 229186\nedit:
\n\nMy mistake, more like 5GB of RAM use with 20 column batches, rather than 3GB. Also, if anyone is more concerned about the speed of saveRDS/readRDS than they are disk usage, then just use compress = FALSE in Laurae's script or mine - makes quite a difference.
", "rawMarkdown": "Thanks Laurae - nicely written up as always.\r\n\r\nTo complement yours, here's a less polished example of a script to process the categorical data, in this case to an integer representation of each category with tree methods in mind. I will return with an edit to confirm the RAM use, but it's 5GB or thereabouts. The final sparse matrix is 800MB.\r\n\r\n library(data.table)\r\n library(FeatureHashing)\r\n library(Matrix)\r\n \r\n hash <- function(x) ifelse(is.na(x), 0, hashed.value(x) %% 2^16)\r\n \r\n batches <- 20\r\n col_first <- seq(2, 2140, by = 2140 / batches)\r\n col_last <- col_first + (2140 / batches) - 1\r\n \r\n processed_data <- vector(\"list\", length(col_first))\r\n \r\n for(i in seq_along(col_first)) {\r\n \r\n dt <- fread(\"Data/train_categorical.csv\",\r\n colClasses = \"character\",\r\n na.strings = \"\",\r\n select = c(col_first[i]:col_last[i]))\r\n \r\n for(col in names(dt)) set(dt, j = col, value = hash(dt[[col]]))\r\n \r\n processed_data[[i]] <- Matrix(as.matrix(dt), sparse = T)\r\n \r\n }\r\n \r\n all_data <- do.call(\"cbind\", processed_data)\r\n saveRDS(all_data, file = \"Data/cat_train.rds\")\r\n \r\n sum(colSums(all_data) == 0)\r\n #[1] 163\r\n sum(rowSums(all_data) == 0)\r\n #[1] 229186\r\n\r\nedit:\r\n\r\nMy mistake, more like 5GB of RAM use with 20 column batches, rather than 3GB. Also, if anyone is more concerned about the speed of saveRDS/readRDS than they are disk usage, then just use compress = FALSE in Laurae's script or mine - makes quite a difference.", "votes": 8 }, { "id": 136482, "postDate": "2016-09-22T10:18:17.183Z", "content": "@Laurae, thank you for spreading RDS knowledge around kaggle. I have been using it for few years now - it indeed is very convenient, and for new guys it is something I recommend to learn :)
", "rawMarkdown": "@Laurae, thank you for spreading RDS knowledge around kaggle. I have been using it for few years now - it indeed is very convenient, and for new guys it is something I recommend to learn :)", "votes": 3 }, { "id": 138174, "postDate": "2016-10-07T14:09:52.987Z", "content": "Yes, only certain packages support it, including glmnet and (I think) extraTrees. If you want to use another algorithm you could always create several balanced, small sample of 0/1 cases and convert them to dense.
", "rawMarkdown": "Yes, only certain packages support it, including glmnet and (I think) extraTrees. If you want to use another algorithm you could always create several balanced, small sample of 0/1 cases and convert them to dense.", "votes": 1 }, { "id": 136435, "postDate": "2016-09-22T00:21:10.807Z", "content": "Laurae,
\n\nI ran the script to get the train_numeric.rds. Then, I read this rds file with readRDS function and stored it into a variable say "x" which gives me an output as dgCmatrix(size is 2.4gb). Assuming complete matrix is loaded into variable x.\nNow, am trying to access the response column from variable x but its not letting me do so.
\n\ncan you help me with this ?
", "rawMarkdown": "Laurae,\r\n\r\nI ran the script to get the train_numeric.rds. Then, I read this rds file with readRDS function and stored it into a variable say \"x\" which gives me an output as dgCmatrix(size is 2.4gb). Assuming complete matrix is loaded into variable x.\r\nNow, am trying to access the response column from variable x but its not letting me do so. \r\n\r\ncan you help me with this ?", "votes": 1 }, { "id": 142934, "postDate": "2016-11-05T15:55:07.707Z", "content": "Is also useful bigmemory package to load train numeric dataset.\nFor example:\nlibrary(bigmemory) \nx<- read.big.matrix("train_numeric.csv", sep=",", header=T)
", "rawMarkdown": "Is also useful bigmemory package to load train numeric dataset.\r\nFor example:\r\nlibrary(bigmemory) \r\nx<- read.big.matrix(\"train_numeric.csv\", sep=\",\", header=T)\r\n" }, { "id": 138657, "postDate": "2016-10-10T06:41:02.617Z", "content": "[quote=Guobin Liu;138537]
\n\n[quote=Icedragon;138156]
\n\n[quote=Guobin Liu;137686]
\n\nIsn't dropNA used in sparkR?
\n\n[/quote]
\n\nIt is called dropna there.
\n\n[/quote]
\n\nbut i cannot use dropNA in my r(rstudio). Did i miss any package?
\n\n[/quote]
\n\nInstall R-package recommenderlab.
", "rawMarkdown": "[quote=Guobin Liu;138537]\r\n\r\n[quote=Icedragon;138156]\r\n\r\n[quote=Guobin Liu;137686]\r\n\r\nIsn't dropNA used in sparkR?\r\n\r\n[/quote]\r\n\r\nIt is called dropna there.\r\n\r\n\r\n[/quote]\r\n\r\nbut i cannot use dropNA in my r(rstudio). Did i miss any package?\r\n\r\n[/quote]\r\n\r\nInstall R-package recommenderlab." }, { "id": 138537, "postDate": "2016-10-09T12:27:18.783Z", "content": "[quote=Icedragon;138156]
\n\n[quote=Guobin Liu;137686]
\n\nIsn't dropNA used in sparkR?
\n\n[/quote]
\n\nIt is called dropna there.
\n\n[/quote]
\n\nbut i cannot use dropNA in my r(rstudio). Did i miss any package?
", "rawMarkdown": "[quote=Icedragon;138156]\r\n\r\n[quote=Guobin Liu;137686]\r\n\r\nIsn't dropNA used in sparkR?\r\n\r\n[/quote]\r\n\r\nIt is called dropna there.\r\n\r\n\r\n[/quote]\r\n\r\nbut i cannot use dropNA in my r(rstudio). Did i miss any package?" }, { "id": 138177, "postDate": "2016-10-07T14:44:29.017Z", "content": "[quote=NewportKing;138035]
\n\nThanks for the script.\nI am just curious about how will the NA blank be treated when running model, say XGB. Will it be treated as zero?
\n\n[/quote]
\n\nFrom this answer on github, the XGboost creator writes:
\n\n\n\n\n"Internally, XGBoost will automatically learn what is the best direction to go when a value is missing. Equivalently, this can be viewed as automatically "learn" what is the best imputation value for missing values based on reduction on training loss."
\n
https://github.com/dmlc/xgboost/issues/21
", "rawMarkdown": "[quote=NewportKing;138035]\r\n\r\nThanks for the script.\r\nI am just curious about how will the NA blank be treated when running model, say XGB. Will it be treated as zero?\r\n\r\n[/quote]\r\n\r\n\r\nFrom this answer on github, the XGboost creator writes:\r\n\r\n>\"Internally, XGBoost will automatically learn what is the best direction to go when a value is missing. Equivalently, this can be viewed as automatically \"learn\" what is the best imputation value for missing values based on reduction on training loss.\"\r\n\r\nhttps://github.com/dmlc/xgboost/issues/21" }, { "id": 138170, "postDate": "2016-10-07T13:54:40.217Z", "content": "This kind of dgCMatrix only can be used with some ML-algorithms like xgboost?
", "rawMarkdown": "This kind of dgCMatrix only can be used with some ML-algorithms like xgboost?" }, { "id": 138156, "postDate": "2016-10-07T13:12:31.137Z", "content": "[quote=Guobin Liu;137686]
\n\nIsn't dropNA used in sparkR?
\n\n[/quote]
\n\nIt is called dropna there.
", "rawMarkdown": "[quote=Guobin Liu;137686]\r\n\r\nIsn't dropNA used in sparkR?\r\n\r\n[/quote]\r\n\r\nIt is called dropna there.\r\n" }, { "id": 138035, "postDate": "2016-10-06T18:29:38.190Z", "content": "Thanks for the script.\nI am just curious about how will the NA blank be treated when running model, say XGB. Will it be treated as zero?
", "rawMarkdown": "Thanks for the script.\r\nI am just curious about how will the NA blank be treated when running model, say XGB. Will it be treated as zero?" }, { "id": 137686, "postDate": "2016-10-04T02:45:04.403Z", "content": "Isn't dropNA used in sparkR?
", "rawMarkdown": "Isn't dropNA used in sparkR?" }, { "id": 136485, "postDate": "2016-09-22T10:28:50.627Z", "content": "@Laurae:
\n\nYep, easier just to take the response from the matrix. Thanks again!
", "rawMarkdown": "@Laurae:\r\n\r\nYep, easier just to take the response from the matrix. Thanks again!" }, { "id": 136484, "postDate": "2016-09-22T10:27:19.457Z", "content": "[quote=Pablo Araya-Melo;136483]
\n\n@Laurae:
\n\nThanks for the script! Regarding the response column, I see there are 970 columns in the train_numeric, last one being response. Shouldn't then the colClasses just go to 969?
\n\n[/quote]
\n\nThen the sparse matrix already has the label column.. (which would mean "label not there" was wrong initially). I didn't check Bosch for weeks until today... (so I assumed the label was not there if someone asked where it was).
\n\nTo access the label, then it should be simply: my_labels <- my_matrix[, 970]
(yes, I confirm there are 969 features and the 970th is the label - checked on my R logs)
", "rawMarkdown": "[quote=Pablo Araya-Melo;136483]\r\n\r\n@Laurae:\r\n\r\nThanks for the script! Regarding the response column, I see there are 970 columns in the train_numeric, last one being response. Shouldn't then the colClasses just go to 969?\r\n\r\n[/quote]\r\n\r\nThen the sparse matrix already has the label column.. (which would mean \"label not there\" was wrong initially). I didn't check Bosch for weeks until today... (so I assumed the label was not there if someone asked where it was).\r\n\r\nTo access the label, then it should be simply: `my_labels <- my_matrix[, 970]`\r\n\r\n(yes, I confirm there are 969 features and the 970th is the label - checked on my R logs)" }, { "id": 136483, "postDate": "2016-09-22T10:22:14.507Z", "content": "@Laurae:
\n\nThanks for the script! Regarding the response column, I see there are 970 columns in the train_numeric, last one being response. Shouldn't then the colClasses just go to 969?
", "rawMarkdown": "@Laurae:\r\n\r\nThanks for the script! Regarding the response column, I see there are 970 columns in the train_numeric, last one being response. Shouldn't then the colClasses just go to 969?" }, { "id": 136481, "postDate": "2016-09-22T10:10:21.823Z", "content": "[quote=NikitRajiv;136435]
\n\nNow, am trying to access the response column from variable x but its not letting me do so.
\n\n[/quote]
\n\nThe response column variable was not loaded in the script so you don't have to reshape the dgCMatrix afterwards.
\n\nYou can load the response column variable using fread and selecting only the last column in train_numeric.csv (which should be the 971th). It's very fast for one column.
\n\nedit: the response column was already there, user error (just access the 970th column).
", "rawMarkdown": "[quote=NikitRajiv;136435]\r\n\r\nNow, am trying to access the response column from variable x but its not letting me do so. \r\n\r\n[/quote]\r\n\r\nThe response column variable was not loaded in the script so you don't have to reshape the dgCMatrix afterwards.\r\n\r\nYou can load the response column variable using fread and selecting only the last column in train_numeric.csv (which should be the 971th). It's very fast for one column.\r\n\r\nedit: the response column was already there, user error (just access the 970th column)." }, { "id": 136459, "postDate": "2016-09-22T05:53:22.003Z", "content": "@NikitRajiv
\n\ntable(x[,"Response",])
", "rawMarkdown": "@NikitRajiv\r\n\r\ntable(x[,\"Response\",])" }, { "id": 138028, "postDate": "2016-10-06T17:42:31.800Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 132015, "author_name": "lewis", "author_url": "", "post_date": "2016-08-21T17:38:19.423000", "content": "Thanks Laurae - nicely written up as always.
\n\nTo complement yours, here's a less polished example of a script to process the categorical data, in this case to an integer representation of each category with tree methods in mind. I will return with an edit to confirm the RAM use, but it's 5GB or thereabouts. The final sparse matrix is 800MB.
\n\nlibrary(data.table)\nlibrary(FeatureHashing)\nlibrary(Matrix)\n\nhash <- function(x) ifelse(is.na(x), 0, hashed.value(x) %% 2^16)\n\nbatches <- 20\ncol_first <- seq(2, 2140, by = 2140 / batches)\ncol_last <- col_first + (2140 / batches) - 1\n\nprocessed_data <- vector("list", length(col_first))\n\nfor(i in seq_along(col_first)) {\n\n dt <- fread("Data/train_categorical.csv",\n colClasses = "character",\n na.strings = "",\n select = c(col_first[i]:col_last[i]))\n\n for(col in names(dt)) set(dt, j = col, value = hash(dt[[col]]))\n\n processed_data[[i]] <- Matrix(as.matrix(dt), sparse = T)\n\n}\n\nall_data <- do.call("cbind", processed_data)\nsaveRDS(all_data, file = "Data/cat_train.rds")\n\nsum(colSums(all_data) == 0)\n#[1] 163\nsum(rowSums(all_data) == 0)\n#[1] 229186\nedit:
\n\nMy mistake, more like 5GB of RAM use with 20 column batches, rather than 3GB. Also, if anyone is more concerned about the speed of saveRDS/readRDS than they are disk usage, then just use compress = FALSE in Laurae's script or mine - makes quite a difference.
", "votes": 8, "replies": [] }, { "id": 136482, "author_name": "raddar", "author_url": "", "post_date": "2016-09-22T10:18:17.183000", "content": "@Laurae, thank you for spreading RDS knowledge around kaggle. I have been using it for few years now - it indeed is very convenient, and for new guys it is something I recommend to learn :)
", "votes": 3, "replies": [] }, { "id": 138174, "author_name": "lewis", "author_url": "", "post_date": "2016-10-07T14:09:52.987000", "content": "Yes, only certain packages support it, including glmnet and (I think) extraTrees. If you want to use another algorithm you could always create several balanced, small sample of 0/1 cases and convert them to dense.
", "votes": 1, "replies": [] }, { "id": 136435, "author_name": "NikitRajiv", "author_url": "", "post_date": "2016-09-22T00:21:10.807000", "content": "Laurae,
\n\nI ran the script to get the train_numeric.rds. Then, I read this rds file with readRDS function and stored it into a variable say "x" which gives me an output as dgCmatrix(size is 2.4gb). Assuming complete matrix is loaded into variable x.\nNow, am trying to access the response column from variable x but its not letting me do so.
\n\ncan you help me with this ?
", "votes": 1, "replies": [] }, { "id": 142934, "author_name": "Kenshiro75", "author_url": "", "post_date": "2016-11-05T15:55:07.707000", "content": "Is also useful bigmemory package to load train numeric dataset.\nFor example:\nlibrary(bigmemory) \nx<- read.big.matrix("train_numeric.csv", sep=",", header=T)
", "votes": 0, "replies": [] }, { "id": 138657, "author_name": "Icedragon", "author_url": "", "post_date": "2016-10-10T06:41:02.617000", "content": "[quote=Guobin Liu;138537]
\n\n[quote=Icedragon;138156]
\n\n[quote=Guobin Liu;137686]
\n\nIsn't dropNA used in sparkR?
\n\n[/quote]
\n\nIt is called dropna there.
\n\n[/quote]
\n\nbut i cannot use dropNA in my r(rstudio). Did i miss any package?
\n\n[/quote]
\n\nInstall R-package recommenderlab.
", "votes": 0, "replies": [] }, { "id": 138537, "author_name": "Guobin Liu", "author_url": "", "post_date": "2016-10-09T12:27:18.783000", "content": "[quote=Icedragon;138156]
\n\n[quote=Guobin Liu;137686]
\n\nIsn't dropNA used in sparkR?
\n\n[/quote]
\n\nIt is called dropna there.
\n\n[/quote]
\n\nbut i cannot use dropNA in my r(rstudio). Did i miss any package?
", "votes": 0, "replies": [] }, { "id": 138177, "author_name": "MaxPower", "author_url": "", "post_date": "2016-10-07T14:44:29.017000", "content": "[quote=NewportKing;138035]
\n\nThanks for the script.\nI am just curious about how will the NA blank be treated when running model, say XGB. Will it be treated as zero?
\n\n[/quote]
\n\nFrom this answer on github, the XGboost creator writes:
\n\n\n\n\n"Internally, XGBoost will automatically learn what is the best direction to go when a value is missing. Equivalently, this can be viewed as automatically "learn" what is the best imputation value for missing values based on reduction on training loss."
\n
https://github.com/dmlc/xgboost/issues/21
", "votes": 0, "replies": [] }, { "id": 138170, "author_name": "Icedragon", "author_url": "", "post_date": "2016-10-07T13:54:40.217000", "content": "This kind of dgCMatrix only can be used with some ML-algorithms like xgboost?
", "votes": 0, "replies": [] }, { "id": 138156, "author_name": "Icedragon", "author_url": "", "post_date": "2016-10-07T13:12:31.137000", "content": "[quote=Guobin Liu;137686]
\n\nIsn't dropNA used in sparkR?
\n\n[/quote]
\n\nIt is called dropna there.
", "votes": 0, "replies": [] }, { "id": 138035, "author_name": "NewportKing", "author_url": "", "post_date": "2016-10-06T18:29:38.190000", "content": "Thanks for the script.\nI am just curious about how will the NA blank be treated when running model, say XGB. Will it be treated as zero?
", "votes": 0, "replies": [] }, { "id": 137686, "author_name": "Guobin Liu", "author_url": "", "post_date": "2016-10-04T02:45:04.403000", "content": "Isn't dropNA used in sparkR?
", "votes": 0, "replies": [] }, { "id": 136485, "author_name": "Pablo Araya-Melo", "author_url": "", "post_date": "2016-09-22T10:28:50.627000", "content": "@Laurae:
\n\nYep, easier just to take the response from the matrix. Thanks again!
", "votes": 0, "replies": [] }, { "id": 136484, "author_name": "Laurae", "author_url": "", "post_date": "2016-09-22T10:27:19.457000", "content": "[quote=Pablo Araya-Melo;136483]
\n\n@Laurae:
\n\nThanks for the script! Regarding the response column, I see there are 970 columns in the train_numeric, last one being response. Shouldn't then the colClasses just go to 969?
\n\n[/quote]
\n\nThen the sparse matrix already has the label column.. (which would mean "label not there" was wrong initially). I didn't check Bosch for weeks until today... (so I assumed the label was not there if someone asked where it was).
\n\nTo access the label, then it should be simply: my_labels <- my_matrix[, 970]
(yes, I confirm there are 969 features and the 970th is the label - checked on my R logs)
", "votes": 0, "replies": [] }, { "id": 136483, "author_name": "Pablo Araya-Melo", "author_url": "", "post_date": "2016-09-22T10:22:14.507000", "content": "@Laurae:
\n\nThanks for the script! Regarding the response column, I see there are 970 columns in the train_numeric, last one being response. Shouldn't then the colClasses just go to 969?
", "votes": 0, "replies": [] }, { "id": 136481, "author_name": "Laurae", "author_url": "", "post_date": "2016-09-22T10:10:21.823000", "content": "[quote=NikitRajiv;136435]
\n\nNow, am trying to access the response column from variable x but its not letting me do so.
\n\n[/quote]
\n\nThe response column variable was not loaded in the script so you don't have to reshape the dgCMatrix afterwards.
\n\nYou can load the response column variable using fread and selecting only the last column in train_numeric.csv (which should be the 971th). It's very fast for one column.
\n\nedit: the response column was already there, user error (just access the 970th column).
", "votes": 0, "replies": [] }, { "id": 136459, "author_name": "vikash", "author_url": "", "post_date": "2016-09-22T05:53:22.003000", "content": "@NikitRajiv
\n\ntable(x[,"Response",])
", "votes": 0, "replies": [] }, { "id": 138028, "author_name": "", "author_url": "", "post_date": "2016-10-06T17:42:31.800000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "132009": "### What are the reasons I might want to read this post?\r\n\r\nIf you read this post, you might be interested in three different things (if you fit in one of these, then it will be of interest obviously):\r\n\r\n1. Loading the data in-memory under 6GB RAM, with a ~10s loading speed after preprocessing\r\n2. Tuning the data loading in-memory for computers under 3GB RAM, with a ~10s loading speed after preprocessing\r\n3. Just for learning (not out-of-memory stuff)\r\n\r\n\r\n----------\r\n\r\n\r\n### Requirements?\r\n\r\nRequirements:\r\n\r\n* Low RAM computer (6GB maximum, 4GB if you want to tune, 3GB for nightmare tuning)\r\n* SSD if possible (might speed up the process by 3x I think, not sure though)\r\n* Any CPU (it should not be slow even on a Intel Atom, as lot of time is spent reading on disk)\r\n* You want to work in-memory, not out-of-memory\r\n\r\n\r\n----------\r\n\r\n\r\n### Other requirements for the script\r\n\r\nThe following script (3 paragraphs below) should make you able to load the train_numeric set when you have less than 6GB RAM, store it locally (700MB), and then be able to load the whole set under 10 seconds. There are three requirements.\r\n\r\nThe first requirement is to make a pass on the whole dataset by batches, to achieve the creation of a sparse matrix. This takes about 30 minutes with very small batches (10 features per batch) and a 5GB peak. On a AMD E-350 processor (yes, slower than Intel Atom) and a SSD, this took me 25 minutes.\r\n\r\nThe second requirement is to save the sparse matrix to a RDS file, so you don't \"pay\" the hassle of loading the dataset everytime. Saving as RDS is very quick, and you have a ~700MB file at the end. But the best comes here: it takes only 10 seconds to load the dataset back! (if you are suspicious of whether the initial and loaded datasets are identical or not, use the function identical(matrix1, matrix2) ).\r\n\r\nThe third requirement is to tune appropriately the first requirement (and what you do after the second requirement) so you don't explode the memory usage suddenly. You can also tune the requirements according to the sparse matrix requirements in the picture at the end of this post. This requirement is explained some paragraphs below this code (which does the first+second requirements):\r\n\r\n library(data.table)\r\n library(recommenderlab)\r\n for (i in 1:97) {\r\n \r\n cat(\"Loading \", i, \"th part.\\n\", sep = \"\")\r\n train_data_temp <- fread(input = \"train_numeric.csv\",\r\n select = (10*(i-1)+1):(10*i),\r\n header = TRUE,\r\n sep = \",\",\r\n stringsAsFactors = FALSE,\r\n colClasses = rep(\"numeric\", 970),\r\n data.table = TRUE)\r\n gc(verbose = FALSE)\r\n \r\n if (i > 1) {\r\n \r\n cat(\"Coercing to matrix.\\n\", sep = \"\")\r\n train_numeric_temp <- as.matrix(train_data_temp)\r\n rm(train_data_temp)\r\n gc(verbose = FALSE)\r\n \r\n cat(\"Coercing into dgCMatrix with NA as blank.\\n\", sep = \"\")\r\n train_numeric_temp <- dropNA(train_numeric_temp)\r\n gc(verbose = FALSE)\r\n \r\n cat(\"Column binding the full matrix with the newly created matrix.\\n\", sep = \"\")\r\n train_numeric <- cbind(train_numeric, train_numeric_temp)\r\n rm(train_numeric_temp)\r\n gc(verbose = FALSE)\r\n \r\n } else {\r\n \r\n cat(\"Coercing to matrix.\\n\", sep = \"\")\r\n train_numeric_temp <- as.matrix(train_data_temp)\r\n rm(train_data_temp)\r\n gc(verbose = FALSE)\r\n \r\n cat(\"Coercing into dgCMatrix with NA as blank.\\n\", sep = \"\")\r\n train_numeric <- dropNA(train_numeric_temp)\r\n gc(verbose = FALSE)\r\n \r\n }\r\n \r\n }\r\n \r\n gc()\r\n saveRDS(train_numeric, file = \"train_numeric.rds\")\r\n\r\nExample of verbose content output by the script:\r\n\r\n Loading 94th part.\r\n Read 1183747 rows and 10 (of 970) columns from 1.993 GB file in 00:00:13\r\n Coercing to matrix.\r\n Coercing into dgCMatrix with NA as blank.\r\n Column binding the full matrix with the newly created matrix.\r\n Loading 95th part.\r\n Read 1183747 rows and 10 (of 970) columns from 1.993 GB file in 00:00:13\r\n Coercing to matrix.\r\n Coercing into dgCMatrix with NA as blank.\r\n Column binding the full matrix with the newly created matrix.\r\n Loading 96th part.\r\n Read 1183747 rows and 10 (of 970) columns from 1.993 GB file in 00:00:13\r\n Coercing to matrix.\r\n Coercing into dgCMatrix with NA as blank.\r\n Column binding the full matrix with the newly created matrix.\r\n Loading 97th part.\r\n Read 1183747 rows and 10 (of 970) columns from 1.993 GB file in 00:00:13\r\n Coercing to matrix.\r\n Coercing into dgCMatrix with NA as blank.\r\n Column binding the full matrix with the newly created matrix.\r\n\r\nThe last gc() has verbose=TRUE by default and you should get similar to this:\r\n\r\n > gc()\r\n used (Mb) gc trigger (Mb) max used (Mb)\r\n Ncells 1660022 88.7 2637877 140.9 2164898 115.7\r\n Vcells 330920318 2524.8 668140682 5097.6 659586557 5032.3\r\n\r\nOnly ~5GB memory usage at peak! (and at the end: ~2.5GB only!)\r\n\r\n----------\r\n\r\n\r\n### Optimizing RAM usage (tuning loading)\r\n\r\nYou can reduce further the memory usage by splitting the loading into two large chunks (470 first columns in one sparse matrix with small batches, 470 last columns in a second sparse matrix with small batches), saving both sparse matrices, restarting R, loading both sparse matrices, and column binding them. In this case, it should make a 3.6GB + swap peak during the merge. At the end, you have a 2.4GB matrix that fits in memory. This makes it largely feasible to load the entire set in a 4GB laptop, and work with it. You can even do feature selection if done appropriately (and loading back the whole dataset is very quick if what you are doing to select features take lot of RAM).\r\n\r\nIn any case, if you increase the batch size (more features loaded per batch), you will increase the memory usage. But the benefit is a smaller loading speed overall (when taking into account the whole dataset).\r\n\r\nTune i lower for faster processing: you have 970 features in train_numeric, therefore if the max i=97, you should:\r\n\r\n* i = 1 => load 1:10 features\r\n* i = 2 => load 11:20 features\r\n* ...\r\n* i = 97 => load 961:970 features\r\n\r\nHence, modify the \"select\" in fread appropriately if you need to load more features in one batch. For instance, using i from 1 to 10 leads up to the following (97 features to load per batch):\r\n\r\n train_data_temp <- fread(input = \"train_numeric.csv\",\r\n select = (97*(i-1)+1):(97*i),\r\n header = TRUE,\r\n sep = \",\",\r\n stringsAsFactors = FALSE,\r\n colClasses = rep(\"numeric\", 970),\r\n data.table = TRUE)\r\n\r\nWhat you get?:\r\n\r\n* i = 1 => load 1:97 features\r\n* i = 2 => load 98:194 features\r\n* ...\r\n* i = 10 => load 874:970 features\r\n\r\n\r\n----------\r\n\r\n\r\n### The 4GB tuning issue (and the 3GB nightmare)\r\n\r\nIf you work with less than 4GB RAM (like 3GB), you will need to work very smartly to achieve the lowest RAM usage possible. I provide the following picture to give you an insight on where your memory will literally explode, as it is clearly visible where it should explode:\r\n\r\nYou can tune aggressively the batches if you want to keep under 3GB usage. However, you will need at least 2.4GB to make it work properly at the end, so you are left with 600MB free (in theory).\r\n![enter image description here][1]\r\n\r\n [1]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4706/bosch_memory_sparse.png", "132015": "Thanks Laurae - nicely written up as always.\r\n\r\nTo complement yours, here's a less polished example of a script to process the categorical data, in this case to an integer representation of each category with tree methods in mind. I will return with an edit to confirm the RAM use, but it's 5GB or thereabouts. The final sparse matrix is 800MB.\r\n\r\n library(data.table)\r\n library(FeatureHashing)\r\n library(Matrix)\r\n \r\n hash <- function(x) ifelse(is.na(x), 0, hashed.value(x) %% 2^16)\r\n \r\n batches <- 20\r\n col_first <- seq(2, 2140, by = 2140 / batches)\r\n col_last <- col_first + (2140 / batches) - 1\r\n \r\n processed_data <- vector(\"list\", length(col_first))\r\n \r\n for(i in seq_along(col_first)) {\r\n \r\n dt <- fread(\"Data/train_categorical.csv\",\r\n colClasses = \"character\",\r\n na.strings = \"\",\r\n select = c(col_first[i]:col_last[i]))\r\n \r\n for(col in names(dt)) set(dt, j = col, value = hash(dt[[col]]))\r\n \r\n processed_data[[i]] <- Matrix(as.matrix(dt), sparse = T)\r\n \r\n }\r\n \r\n all_data <- do.call(\"cbind\", processed_data)\r\n saveRDS(all_data, file = \"Data/cat_train.rds\")\r\n \r\n sum(colSums(all_data) == 0)\r\n #[1] 163\r\n sum(rowSums(all_data) == 0)\r\n #[1] 229186\r\n\r\nedit:\r\n\r\nMy mistake, more like 5GB of RAM use with 20 column batches, rather than 3GB. Also, if anyone is more concerned about the speed of saveRDS/readRDS than they are disk usage, then just use compress = FALSE in Laurae's script or mine - makes quite a difference.", "136482": "@Laurae, thank you for spreading RDS knowledge around kaggle. I have been using it for few years now - it indeed is very convenient, and for new guys it is something I recommend to learn :)", "138174": "Yes, only certain packages support it, including glmnet and (I think) extraTrees. If you want to use another algorithm you could always create several balanced, small sample of 0/1 cases and convert them to dense.", "136435": "Laurae,\r\n\r\nI ran the script to get the train_numeric.rds. Then, I read this rds file with readRDS function and stored it into a variable say \"x\" which gives me an output as dgCmatrix(size is 2.4gb). Assuming complete matrix is loaded into variable x.\r\nNow, am trying to access the response column from variable x but its not letting me do so. \r\n\r\ncan you help me with this ?", "142934": "Is also useful bigmemory package to load train numeric dataset.\r\nFor example:\r\nlibrary(bigmemory) \r\nx<- read.big.matrix(\"train_numeric.csv\", sep=\",\", header=T)\r\n", "138657": "[quote=Guobin Liu;138537]\r\n\r\n[quote=Icedragon;138156]\r\n\r\n[quote=Guobin Liu;137686]\r\n\r\nIsn't dropNA used in sparkR?\r\n\r\n[/quote]\r\n\r\nIt is called dropna there.\r\n\r\n\r\n[/quote]\r\n\r\nbut i cannot use dropNA in my r(rstudio). Did i miss any package?\r\n\r\n[/quote]\r\n\r\nInstall R-package recommenderlab.", "138537": "[quote=Icedragon;138156]\r\n\r\n[quote=Guobin Liu;137686]\r\n\r\nIsn't dropNA used in sparkR?\r\n\r\n[/quote]\r\n\r\nIt is called dropna there.\r\n\r\n\r\n[/quote]\r\n\r\nbut i cannot use dropNA in my r(rstudio). Did i miss any package?", "138177": "[quote=NewportKing;138035]\r\n\r\nThanks for the script.\r\nI am just curious about how will the NA blank be treated when running model, say XGB. Will it be treated as zero?\r\n\r\n[/quote]\r\n\r\n\r\nFrom this answer on github, the XGboost creator writes:\r\n\r\n>\"Internally, XGBoost will automatically learn what is the best direction to go when a value is missing. Equivalently, this can be viewed as automatically \"learn\" what is the best imputation value for missing values based on reduction on training loss.\"\r\n\r\nhttps://github.com/dmlc/xgboost/issues/21", "138170": "This kind of dgCMatrix only can be used with some ML-algorithms like xgboost?", "138156": "[quote=Guobin Liu;137686]\r\n\r\nIsn't dropNA used in sparkR?\r\n\r\n[/quote]\r\n\r\nIt is called dropna there.\r\n", "138035": "Thanks for the script.\r\nI am just curious about how will the NA blank be treated when running model, say XGB. Will it be treated as zero?", "137686": "Isn't dropNA used in sparkR?", "136485": "@Laurae:\r\n\r\nYep, easier just to take the response from the matrix. Thanks again!", "136484": "[quote=Pablo Araya-Melo;136483]\r\n\r\n@Laurae:\r\n\r\nThanks for the script! Regarding the response column, I see there are 970 columns in the train_numeric, last one being response. Shouldn't then the colClasses just go to 969?\r\n\r\n[/quote]\r\n\r\nThen the sparse matrix already has the label column.. (which would mean \"label not there\" was wrong initially). I didn't check Bosch for weeks until today... (so I assumed the label was not there if someone asked where it was).\r\n\r\nTo access the label, then it should be simply: `my_labels <- my_matrix[, 970]`\r\n\r\n(yes, I confirm there are 969 features and the 970th is the label - checked on my R logs)", "136483": "@Laurae:\r\n\r\nThanks for the script! Regarding the response column, I see there are 970 columns in the train_numeric, last one being response. Shouldn't then the colClasses just go to 969?", "136481": "[quote=NikitRajiv;136435]\r\n\r\nNow, am trying to access the response column from variable x but its not letting me do so. \r\n\r\n[/quote]\r\n\r\nThe response column variable was not loaded in the script so you don't have to reshape the dgCMatrix afterwards.\r\n\r\nYou can load the response column variable using fread and selecting only the last column in train_numeric.csv (which should be the 971th). It's very fast for one column.\r\n\r\nedit: the response column was already there, user error (just access the 970th column).", "136459": "@NikitRajiv\r\n\r\ntable(x[,\"Response\",])", "138028": "" } }