![\"enter](\"https://www.kaggle.com/blobs/download/forum-message-attachment-files/4677/Bosch_Size.JPG\")
Sneak peak on the dataset size for people interested at looking at the file sizes uncompressed before downloading the compressed files:
\n\nRaw: 14.3GB
NTFS Compressed: 2.83GB
Files to download: 697MB
![\"enter](\"https://www.kaggle.com/blobs/download/forum-message-attachment-files/4678/Bosch_Size_Disagg.JPG\")
Sneak peak on the dataset size for people interested at looking at the file sizes uncompressed before downloading the compressed files:
\n\nRaw: 14.3GB
NTFS Compressed: 2.83GB
Files to download: 697MB
Hmm, we can try to convert categorical data and dates to numeric and generate new fles. It can reduce size of a files a little. \nBut... Need to look at categorical and date data first :)
", "rawMarkdown": "Hmm, we can try to convert categorical data and dates to numeric and generate new fles. It can reduce size of a files a little. \r\nBut... Need to look at categorical and date data first :)", "votes": null }, { "id": "131476", "postDate": "08/17/2016 14:50:36", "content": "[quote=Arty.Erokhin;131474]
\n\nHmm, we can try to convert categorical data and dates to numeric and generate new fles. It can reduce size of a files a little. \nBut... Need to look at categorical and date data first :)
\n\n[/quote]
\n\nI quickly looked at categorical and date data.
\n\nSee my "expeditive" exploration: https://www.kaggle.com/c/bosch-production-line-performance/forums/t/22909/expeditive-exploration-models-on-data
", "rawMarkdown": "[quote=Arty.Erokhin;131474]\r\n\r\nHmm, we can try to convert categorical data and dates to numeric and generate new fles. It can reduce size of a files a little. \r\nBut... Need to look at categorical and date data first :)\r\n\r\n[/quote]\r\n\r\nI quickly looked at categorical and date data.\r\n\r\n* For categorical: it seems to be (extremely) sparse data with numeric inputs.\r\n* For date: dates are normalized already, it is also (very) sparse with numeric inputs.\r\n\r\nSee my \"expeditive\" exploration: https://www.kaggle.com/c/bosch-production-line-performance/forums/t/22909/expeditive-exploration-models-on-data", "votes": null }, { "id": "131530", "postDate": "08/17/2016 23:00:22", "content": "128GB of memory isn't enough for me to read and pickle the train_categorical csv. Looks like this competition might be "Who has the most RAM?", or "Who has the most swap space + patience?"
\n\nEDIT: Each of the categorical/date files takes around 100GB of RAM for me to load with pandas, meaning that the full dataset is 450GB+ when read as a full csv...
", "rawMarkdown": "128GB of memory isn't enough for me to read and pickle the train_categorical csv. Looks like this competition might be \"Who has the most RAM?\", or \"Who has the most swap space + patience?\"\r\n\r\nEDIT: Each of the categorical/date files takes around 100GB of RAM for me to load with pandas, meaning that the full dataset is 450GB+ when read as a full csv...", "votes": null }, { "id": "131531", "postDate": "08/17/2016 23:03:43", "content": "@anokas: good that you mentioned it - i was beginning to wonder if i can improve things on a 16gb laptop on that front :-) As an intermediate solution, I suggest (since you're clearly using Python) experimenting with the reading by chunks via pandas::read_csv - I know that unless I get more RAM, this is my only option atm.
", "rawMarkdown": "anokas: good that you mentioned it - i was beginning to wonder if i can improve things on a 16gb laptop on that front :-) As an intermediate solution, I suggest (since you're clearly using Python) experimenting with the reading by chunks via pandas::read_csv - I know that unless I get more RAM, this is my only option atm.", "votes": null }, { "id": "131548", "postDate": "08/18/2016 02:36:31", "content": "Can someone recommend what tool[s] to use in order to process this data (working with Python)? Also, can you point out to any particular tutorials?\nReally appreciate any help on that.
", "rawMarkdown": "Can someone recommend what tool[s] to use in order to process this data (working with Python)? Also, can you point out to any particular tutorials?\r\nReally appreciate any help on that.", "votes": null }, { "id": "131549", "postDate": "08/18/2016 02:46:31", "content": "Hmmm, I've managed to read both categorical and numeric training sets in memory in R using data.table::fread. The categorical train took 6 mins to read in memory and takes up 20GB in R - big difference from pandas, I wonder why. Maybe some encodings or sparsity considerations? Things are running pretty smoothly even though I am on an Macbook Pro with 8GB of RAM (?!). What is this sorcery?
You can often reduce Panda's load by typing variables: https://www.kaggle.com/ericcouto/grupo-bimbo-inventory-demand/using-82-less-memory/discussion
\n\nAnd look at ZFTurbo's scripts for direct .csv parsing.
", "rawMarkdown": "You can often reduce Panda's load by typing variables: https://www.kaggle.com/ericcouto/grupo-bimbo-inventory-demand/using-82-less-memory/discussion\r\n\r\nAnd look at ZFTurbo's scripts for direct .csv parsing.", "votes": null }, { "id": "131589", "postDate": "08/18/2016 12:38:00", "content": "It seems that pandas does not support loading sparse matrices with read_csv (please correct me if I'm wrong). Same goes for numpy.loadtxt. Right now I am trying to roll my own loader to convert the data to scipy.sparse.csr_matrix. I will report back if it works...
", "rawMarkdown": "It seems that pandas does not support loading sparse matrices with read_csv (please correct me if I'm wrong). Same goes for numpy.loadtxt. Right now I am trying to roll my own loader to convert the data to scipy.sparse.csr_matrix. I will report back if it works...", "votes": null }, { "id": "131599", "postDate": "08/18/2016 13:57:08", "content": "[quote=anokas;131530]
\n\n128GB of memory isn't enough for me to read and pickle the train_categorical csv. Looks like this competition might be "Who has the most RAM?", or "Who has the most swap space + patience?"
\n\nEDIT: Each of the categorical/date files takes around 100GB of RAM for me to load with pandas, meaning that the full dataset is 450GB+ when read as a full csv...
\n\n[/quote]
\n\nAgreed, too much RAM cost even when dataframes are converted into sparse matrix
", "rawMarkdown": "[quote=anokas;131530]\r\n\r\n128GB of memory isn't enough for me to read and pickle the train_categorical csv. Looks like this competition might be \"Who has the most RAM?\", or \"Who has the most swap space + patience?\"\r\n\r\nEDIT: Each of the categorical/date files takes around 100GB of RAM for me to load with pandas, meaning that the full dataset is 450GB+ when read as a full csv...\r\n\r\n[/quote]\r\n\r\nAgreed, too much RAM cost even when dataframes are converted into sparse matrix", "votes": null }, { "id": "131624", "postDate": "08/18/2016 18:10:03", "content": "", "rawMarkdown": "", "votes": null }, { "id": "131625", "postDate": "08/18/2016 18:14:33", "content": "Downsample and / or use an online learning technique such as FTLR
", "rawMarkdown": "Downsample and / or use an online learning technique such as FTLR", "votes": null }, { "id": "131639", "postDate": "08/18/2016 20:11:18", "content": "[quote=Alexandru Papiu;131549]
\n\nHmmm, I've managed to read both categorical and numeric training sets in memory in R using data.table::fread. The categorical train took 6 mins to read in memory and takes up 20GB in R - big difference from pandas, I wonder why. Maybe some encodings or sparsity considerations? Things are running pretty smoothly even though I am on an Macbook Pro with 8GB of RAM (?!). What is this sorcery?
[/quote]
\n\nOne reason might because characters items are a pointer to the single cached string. Data table are extremely efficient for large data sets by allowing you to update the table by reference .
", "rawMarkdown": "[quote=Alexandru Papiu;131549]\r\n\r\nHmmm, I've managed to read both categorical and numeric training sets in memory in R using `data.table::fread`. The categorical train took 6 mins to read in memory and takes up 20GB in R - big difference from pandas, I wonder why. Maybe some encodings or sparsity considerations? Things are running pretty smoothly even though I am on an Macbook Pro with 8GB of RAM (?!). What is this sorcery?\r\n\r\n[/quote]\r\n\r\nOne reason might because characters items are a pointer to the single cached string. Data table are extremely efficient for large data sets by allowing you to update the table by reference .", "votes": null }, { "id": "131642", "postDate": "08/18/2016 20:23:26", "content": "one way of analyzing this large data set could be loop trough it row by row. In the script below I do this and calculates some simple statistics for the training data. The script took about 5 minutes to run per file.
\n\nfrom collections import Counter\nimport csv\nfilename="train_numeric.csv"\nfilename="train_categorical.csv"\nfilename="train_date.csv"\nrowCount=0\nnumberEmptyValues=0\n\n\nwith open(filename, "rt") as csvfile:\n filereader = csv.reader(csvfile)\n for curRow in filereader:\n if rowCount == 0 :\n headerRow=curRow\n numberColumns=len(headerRow)\n emptyList=['']*numberColumns # create list of empty emtries to compare against\n emptyCounter=Counter(emptyList)\n\n else:\n curCounter=Counter(curRow)\n diff = curCounter-emptyCounter # create list of the the non empty values\n\n numberNotEmpty=len(diff)# calc the length of this list\n numEmpty=numberColumns-numberNotEmpty \n numberEmptyValues=numberEmptyValues+numEmpty\n\n\n rowCount=rowCount+1\n\n\ntotalnumber=rowCount*numberColumns\npctEmpty=100*numberEmptyValues/totalnumber\n\nprint("nuber of rows :",rowCount)\nprint("number of columns :",numberColumns)\nprint("number of empty values:",numberEmptyValues)\nprint("% of empty values :",pctEmpty)\nI got the following results from this script:
\n\ntrain_numeric.csv\nnuber of rows : 1183748\nnumber of columns : 970\nnumber of empty values: 1035542147\n% of empty values : 90.18551446011652\n\ntrain_categorical.csv\nnuber of rows : 1183748\nnumber of columns : 2141\nnumber of empty values: 2531522206\n% of empty values : 99.886274584961\n\ntrain_date.csv\nnuber of rows : 1183748\nnumber of columns : 1157\nnumber of empty values: 1360147063\n% of empty values : 99.31006150778272\nThanks!
\n\nIs Kaggle going Big Data?
", "rawMarkdown": "Thanks! \r\n\r\nIs Kaggle going Big Data?", "votes": null }, { "id": "131682", "postDate": "08/19/2016 00:23:24", "content": "Using R, my windows machine couldn't handle fread so I used the ff package, which reads data in chunks and writes to disk. It took over 15 minutes, but used <8GB. I then took a sample to mess around with for now.
\n\n library(data.table)\n library(ff)\n\n datecols <- fread("train_date.csv", nrows = 0L) # just getting the columns here\n classvector <- c("integer", rep.int("numeric", ncol(datecols)-1)) # avoids type error\n\n dates <- read.csv.ffdf(file = "train_date.csv", header = TRUE, colClasses = classvector)\n\n sampleidx <- sample(nrow(dates), nrow(dates)/10)\n dates2 <- dates[sampleidx, ]\nI want to join this, but my 4G machine is saying 'DONT'.
", "rawMarkdown": "I want to join this, but my 4G machine is saying 'DONT'.", "votes": null }, { "id": "131724", "postDate": "08/19/2016 07:54:44", "content": "Those of us with modest hardware may not need to give up so quickly - I've only glanced at the categorical and numeric data so far but there's lots of scope for reducing the memory requirements:
\n\nI read the train_categorical file in two parts and managed to hash it down from 20GB dense to 800MB sparse (using the same SSD overflow sorcery as Alexandru, this used about 10GB of memory in R for each half but was doable on my old 4GB machine in about 15mins total)
the numeric data has lots of NAs and has been scaled so each feature has a mean of zero. But if we have tree methods in mind I guess we could convert the NAs to zero and then a sparse format may help (still playing with this one but I suspect it may be possible to shrink it from 9GB to 2GB).
Also for those of us with humble hardware--You could also entirely ignore certain columns(/features). My hunch is that Lines 1, 2, and the first two stations of Line 3 is where the action is at, so to speak. There's a paper by Andrew Ng of Stanford (forgot the title but it's on his website) that it's actually okay to NOT use all of the variables that matter but a set of variables that get you close if not exactly at the right as answer. More features isn't always the answer anyway; sometimes it's the opposite, depending on how things look. This may help reduce memory and computing power requirements. For that matter you can test if increasing the sample size from 10,000 to one million will help much. If it doesn't look like that extra million rows will add much, then well... sayonara. :)
", "rawMarkdown": "Also for those of us with humble hardware--You could also entirely ignore certain columns(/features). My hunch is that Lines 1, 2, and the first two stations of Line 3 is where the action is at, so to speak. There's a paper by Andrew Ng of Stanford (forgot the title but it's on his website) that it's actually okay to NOT use all of the variables that matter but a set of variables that get you close if not exactly at the right as answer. More features isn't always the answer anyway; sometimes it's the opposite, depending on how things look. This may help reduce memory and computing power requirements. For that matter you can test if increasing the sample size from 10,000 to one million will help much. If it doesn't look like that extra million rows will add much, then well... sayonara. :)", "votes": null }, { "id": "131947", "postDate": "08/20/2016 20:56:27", "content": "Start Looking for Amazon VM!
", "rawMarkdown": "Start Looking for Amazon VM!", "votes": null }, { "id": "132073", "postDate": "08/22/2016 10:56:48", "content": "[quote=Alpha;131947]
\n\nStart Looking for Amazon VM!
\n\n[/quote]
\n\nOr better yet, start looking at online learning algorithms... (as Scirpus already mentioned)
\n\nI feel that vw is making a comeback in this one, after having been dominated a bit by xgboost for the last year or so (at least in Kaggle).
\n\n[Edit: I haven't looked at the data, so please do not blindly trust my statement about vw being good for this one]
", "rawMarkdown": "[quote=Alpha;131947]\r\n\r\nStart Looking for Amazon VM!\r\n\r\n[/quote]\r\n\r\nOr better yet, start looking at online learning algorithms... (as Scirpus already [mentioned][1])\r\n\r\nI feel that [vw][2] is making a comeback in this one, after having been dominated a bit by xgboost for the last year or so (at least in Kaggle).\r\n\r\n\r\n[Edit: I haven't looked at the data, so please do not blindly trust my statement about vw being good for this one]\r\n\r\n\r\n [1]: https://www.kaggle.com/c/bosch-production-line-performance/forums/t/22908/datasets-size-uncompressed-14-3gb/131625#post131625\r\n [2]: http://hunch.net/~vw/", "votes": null }, { "id": "132438", "postDate": "08/25/2016 14:26:00", "content": "I too had same problem, using a 4GB laptop is not helping anyway. Then I used H2O, with option -Xmx4g, I was able to load train_categorical in under 10min. Atleast now I can say I have seen the data. I'm happy that now I can explore further.
", "rawMarkdown": "I too had same problem, using a 4GB laptop is not helping anyway. Then I used H2O, with option -Xmx4g, I was able to load train_categorical in under 10min. Atleast now I can say I have seen the data. I'm happy that now I can explore further.", "votes": null } ], "comments": [ { "id": 131474, "author_name": "ggofat", "author_url": "", "post_date": "08/17/2016 14:30:22", "content": "Hmm, we can try to convert categorical data and dates to numeric and generate new fles. It can reduce size of a files a little. \nBut... Need to look at categorical and date data first :)
", "votes": null, "replies": [] }, { "id": 131476, "author_name": "laurae2", "author_url": "", "post_date": "08/17/2016 14:50:36", "content": "[quote=Arty.Erokhin;131474]
\n\nHmm, we can try to convert categorical data and dates to numeric and generate new fles. It can reduce size of a files a little. \nBut... Need to look at categorical and date data first :)
\n\n[/quote]
\n\nI quickly looked at categorical and date data.
\n\nSee my "expeditive" exploration: https://www.kaggle.com/c/bosch-production-line-performance/forums/t/22909/expeditive-exploration-models-on-data
", "votes": null, "replies": [] }, { "id": 131530, "author_name": "anokas", "author_url": "", "post_date": "08/17/2016 23:00:22", "content": "128GB of memory isn't enough for me to read and pickle the train_categorical csv. Looks like this competition might be "Who has the most RAM?", or "Who has the most swap space + patience?"
\n\nEDIT: Each of the categorical/date files takes around 100GB of RAM for me to load with pandas, meaning that the full dataset is 450GB+ when read as a full csv...
", "votes": null, "replies": [] }, { "id": 131531, "author_name": "konradb", "author_url": "", "post_date": "08/17/2016 23:03:43", "content": "@anokas: good that you mentioned it - i was beginning to wonder if i can improve things on a 16gb laptop on that front :-) As an intermediate solution, I suggest (since you're clearly using Python) experimenting with the reading by chunks via pandas::read_csv - I know that unless I get more RAM, this is my only option atm.
", "votes": null, "replies": [] }, { "id": 131548, "author_name": "sinilochka", "author_url": "", "post_date": "08/18/2016 02:36:31", "content": "Can someone recommend what tool[s] to use in order to process this data (working with Python)? Also, can you point out to any particular tutorials?\nReally appreciate any help on that.
", "votes": null, "replies": [] }, { "id": 131549, "author_name": "apapiu", "author_url": "", "post_date": "08/18/2016 02:46:31", "content": "Hmmm, I've managed to read both categorical and numeric training sets in memory in R using data.table::fread. The categorical train took 6 mins to read in memory and takes up 20GB in R - big difference from pandas, I wonder why. Maybe some encodings or sparsity considerations? Things are running pretty smoothly even though I am on an Macbook Pro with 8GB of RAM (?!). What is this sorcery?
You can often reduce Panda's load by typing variables: https://www.kaggle.com/ericcouto/grupo-bimbo-inventory-demand/using-82-less-memory/discussion
\n\nAnd look at ZFTurbo's scripts for direct .csv parsing.
", "votes": null, "replies": [] }, { "id": 131589, "author_name": "michfeld", "author_url": "", "post_date": "08/18/2016 12:38:00", "content": "It seems that pandas does not support loading sparse matrices with read_csv (please correct me if I'm wrong). Same goes for numpy.loadtxt. Right now I am trying to roll my own loader to convert the data to scipy.sparse.csr_matrix. I will report back if it works...
", "votes": null, "replies": [] }, { "id": 131599, "author_name": "icyblade", "author_url": "", "post_date": "08/18/2016 13:57:08", "content": "[quote=anokas;131530]
\n\n128GB of memory isn't enough for me to read and pickle the train_categorical csv. Looks like this competition might be "Who has the most RAM?", or "Who has the most swap space + patience?"
\n\nEDIT: Each of the categorical/date files takes around 100GB of RAM for me to load with pandas, meaning that the full dataset is 450GB+ when read as a full csv...
\n\n[/quote]
\n\nAgreed, too much RAM cost even when dataframes are converted into sparse matrix
", "votes": null, "replies": [] }, { "id": 131624, "author_name": "igauty", "author_url": "", "post_date": "08/18/2016 18:10:03", "content": "", "votes": null, "replies": [] }, { "id": 131625, "author_name": "scirpus", "author_url": "", "post_date": "08/18/2016 18:14:33", "content": "Downsample and / or use an online learning technique such as FTLR
", "votes": null, "replies": [] }, { "id": 131639, "author_name": "notaapple", "author_url": "", "post_date": "08/18/2016 20:11:18", "content": "[quote=Alexandru Papiu;131549]
\n\nHmmm, I've managed to read both categorical and numeric training sets in memory in R using data.table::fread. The categorical train took 6 mins to read in memory and takes up 20GB in R - big difference from pandas, I wonder why. Maybe some encodings or sparsity considerations? Things are running pretty smoothly even though I am on an Macbook Pro with 8GB of RAM (?!). What is this sorcery?
[/quote]
\n\nOne reason might because characters items are a pointer to the single cached string. Data table are extremely efficient for large data sets by allowing you to update the table by reference .
", "votes": null, "replies": [] }, { "id": 131642, "author_name": "playfulsynapse", "author_url": "", "post_date": "08/18/2016 20:23:26", "content": "one way of analyzing this large data set could be loop trough it row by row. In the script below I do this and calculates some simple statistics for the training data. The script took about 5 minutes to run per file.
\n\nfrom collections import Counter\nimport csv\nfilename="train_numeric.csv"\nfilename="train_categorical.csv"\nfilename="train_date.csv"\nrowCount=0\nnumberEmptyValues=0\n\n\nwith open(filename, "rt") as csvfile:\n filereader = csv.reader(csvfile)\n for curRow in filereader:\n if rowCount == 0 :\n headerRow=curRow\n numberColumns=len(headerRow)\n emptyList=['']*numberColumns # create list of empty emtries to compare against\n emptyCounter=Counter(emptyList)\n\n else:\n curCounter=Counter(curRow)\n diff = curCounter-emptyCounter # create list of the the non empty values\n\n numberNotEmpty=len(diff)# calc the length of this list\n numEmpty=numberColumns-numberNotEmpty \n numberEmptyValues=numberEmptyValues+numEmpty\n\n\n rowCount=rowCount+1\n\n\ntotalnumber=rowCount*numberColumns\npctEmpty=100*numberEmptyValues/totalnumber\n\nprint("nuber of rows :",rowCount)\nprint("number of columns :",numberColumns)\nprint("number of empty values:",numberEmptyValues)\nprint("% of empty values :",pctEmpty)\nI got the following results from this script:
\n\ntrain_numeric.csv\nnuber of rows : 1183748\nnumber of columns : 970\nnumber of empty values: 1035542147\n% of empty values : 90.18551446011652\n\ntrain_categorical.csv\nnuber of rows : 1183748\nnumber of columns : 2141\nnumber of empty values: 2531522206\n% of empty values : 99.886274584961\n\ntrain_date.csv\nnuber of rows : 1183748\nnumber of columns : 1157\nnumber of empty values: 1360147063\n% of empty values : 99.31006150778272\nThanks!
\n\nIs Kaggle going Big Data?
", "votes": null, "replies": [] }, { "id": 131682, "author_name": "jpmiller", "author_url": "", "post_date": "08/19/2016 00:23:24", "content": "Using R, my windows machine couldn't handle fread so I used the ff package, which reads data in chunks and writes to disk. It took over 15 minutes, but used <8GB. I then took a sample to mess around with for now.
\n\n library(data.table)\n library(ff)\n\n datecols <- fread("train_date.csv", nrows = 0L) # just getting the columns here\n classvector <- c("integer", rep.int("numeric", ncol(datecols)-1)) # avoids type error\n\n dates <- read.csv.ffdf(file = "train_date.csv", header = TRUE, colClasses = classvector)\n\n sampleidx <- sample(nrow(dates), nrow(dates)/10)\n dates2 <- dates[sampleidx, ]\nI want to join this, but my 4G machine is saying 'DONT'.
", "votes": null, "replies": [] }, { "id": 131724, "author_name": "cartographic", "author_url": "", "post_date": "08/19/2016 07:54:44", "content": "Those of us with modest hardware may not need to give up so quickly - I've only glanced at the categorical and numeric data so far but there's lots of scope for reducing the memory requirements:
\n\nI read the train_categorical file in two parts and managed to hash it down from 20GB dense to 800MB sparse (using the same SSD overflow sorcery as Alexandru, this used about 10GB of memory in R for each half but was doable on my old 4GB machine in about 15mins total)
the numeric data has lots of NAs and has been scaled so each feature has a mean of zero. But if we have tree methods in mind I guess we could convert the NAs to zero and then a sparse format may help (still playing with this one but I suspect it may be possible to shrink it from 9GB to 2GB).
Also for those of us with humble hardware--You could also entirely ignore certain columns(/features). My hunch is that Lines 1, 2, and the first two stations of Line 3 is where the action is at, so to speak. There's a paper by Andrew Ng of Stanford (forgot the title but it's on his website) that it's actually okay to NOT use all of the variables that matter but a set of variables that get you close if not exactly at the right as answer. More features isn't always the answer anyway; sometimes it's the opposite, depending on how things look. This may help reduce memory and computing power requirements. For that matter you can test if increasing the sample size from 10,000 to one million will help much. If it doesn't look like that extra million rows will add much, then well... sayonara. :)
", "votes": null, "replies": [] }, { "id": 131947, "author_name": "arpangupta87", "author_url": "", "post_date": "08/20/2016 20:56:27", "content": "Start Looking for Amazon VM!
", "votes": null, "replies": [] }, { "id": 132073, "author_name": "tobycheese", "author_url": "", "post_date": "08/22/2016 10:56:48", "content": "[quote=Alpha;131947]
\n\nStart Looking for Amazon VM!
\n\n[/quote]
\n\nOr better yet, start looking at online learning algorithms... (as Scirpus already mentioned)
\n\nI feel that vw is making a comeback in this one, after having been dominated a bit by xgboost for the last year or so (at least in Kaggle).
\n\n[Edit: I haven't looked at the data, so please do not blindly trust my statement about vw being good for this one]
", "votes": null, "replies": [] }, { "id": 132438, "author_name": "sivakumard", "author_url": "", "post_date": "08/25/2016 14:26:00", "content": "I too had same problem, using a 4GB laptop is not helping anyway. Then I used H2O, with option -Xmx4g, I was able to load train_categorical in under 10min. Atleast now I can say I have seen the data. I'm happy that now I can explore further.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "131473": "Sneak peak on the dataset size for people interested at looking at the file sizes **uncompressed** before downloading the compressed files:\r\n\r\n* Raw: 14.3GB\r\n\r\n* NTFS Compressed: 2.83GB\r\n\r\n* Files to download: 697MB\r\n\r\n![enter image description here][1]\r\n\r\n![enter image description here][2]\r\n\r\n\r\n [1]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4677/Bosch_Size.JPG\r\n [2]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4678/Bosch_Size_Disagg.JPG", "131474": "Hmm, we can try to convert categorical data and dates to numeric and generate new fles. It can reduce size of a files a little. \r\nBut... Need to look at categorical and date data first :)", "131476": "[quote=Arty.Erokhin;131474]\r\n\r\nHmm, we can try to convert categorical data and dates to numeric and generate new fles. It can reduce size of a files a little. \r\nBut... Need to look at categorical and date data first :)\r\n\r\n[/quote]\r\n\r\nI quickly looked at categorical and date data.\r\n\r\n* For categorical: it seems to be (extremely) sparse data with numeric inputs.\r\n* For date: dates are normalized already, it is also (very) sparse with numeric inputs.\r\n\r\nSee my \"expeditive\" exploration: https://www.kaggle.com/c/bosch-production-line-performance/forums/t/22909/expeditive-exploration-models-on-data", "131530": "128GB of memory isn't enough for me to read and pickle the train_categorical csv. Looks like this competition might be \"Who has the most RAM?\", or \"Who has the most swap space + patience?\"\r\n\r\nEDIT: Each of the categorical/date files takes around 100GB of RAM for me to load with pandas, meaning that the full dataset is 450GB+ when read as a full csv...", "131531": "anokas: good that you mentioned it - i was beginning to wonder if i can improve things on a 16gb laptop on that front :-) As an intermediate solution, I suggest (since you're clearly using Python) experimenting with the reading by chunks via pandas::read_csv - I know that unless I get more RAM, this is my only option atm.", "131548": "Can someone recommend what tool[s] to use in order to process this data (working with Python)? Also, can you point out to any particular tutorials?\r\nReally appreciate any help on that.", "131549": "Hmmm, I've managed to read both categorical and numeric training sets in memory in R using `data.table::fread`. The categorical train took 6 mins to read in memory and takes up 20GB in R - big difference from pandas, I wonder why. Maybe some encodings or sparsity considerations? Things are running pretty smoothly even though I am on an Macbook Pro with 8GB of RAM (?!). What is this sorcery?", "131555": "You can often reduce Panda's load by typing variables: https://www.kaggle.com/ericcouto/grupo-bimbo-inventory-demand/using-82-less-memory/discussion\r\n\r\nAnd look at ZFTurbo's scripts for direct .csv parsing.", "131589": "It seems that pandas does not support loading sparse matrices with read_csv (please correct me if I'm wrong). Same goes for numpy.loadtxt. Right now I am trying to roll my own loader to convert the data to scipy.sparse.csr_matrix. I will report back if it works...", "131599": "[quote=anokas;131530]\r\n\r\n128GB of memory isn't enough for me to read and pickle the train_categorical csv. Looks like this competition might be \"Who has the most RAM?\", or \"Who has the most swap space + patience?\"\r\n\r\nEDIT: Each of the categorical/date files takes around 100GB of RAM for me to load with pandas, meaning that the full dataset is 450GB+ when read as a full csv...\r\n\r\n[/quote]\r\n\r\nAgreed, too much RAM cost even when dataframes are converted into sparse matrix", "131624": "", "131625": "Downsample and / or use an online learning technique such as FTLR", "131639": "[quote=Alexandru Papiu;131549]\r\n\r\nHmmm, I've managed to read both categorical and numeric training sets in memory in R using `data.table::fread`. The categorical train took 6 mins to read in memory and takes up 20GB in R - big difference from pandas, I wonder why. Maybe some encodings or sparsity considerations? Things are running pretty smoothly even though I am on an Macbook Pro with 8GB of RAM (?!). What is this sorcery?\r\n\r\n[/quote]\r\n\r\nOne reason might because characters items are a pointer to the single cached string. Data table are extremely efficient for large data sets by allowing you to update the table by reference .", "131642": "one way of analyzing this large data set could be loop trough it row by row. In the script below I do this and calculates some simple statistics for the training data. The script took about 5 minutes to run per file.\r\n\r\n-------------------------------------------------------------------------------\r\n\r\n from collections import Counter\r\n import csv\r\n filename=\"train_numeric.csv\"\r\n filename=\"train_categorical.csv\"\r\n filename=\"train_date.csv\"\r\n rowCount=0\r\n numberEmptyValues=0\r\n \r\n \r\n with open(filename, \"rt\") as csvfile:\r\n filereader = csv.reader(csvfile)\r\n for curRow in filereader:\r\n if rowCount == 0 :\r\n headerRow=curRow\r\n numberColumns=len(headerRow)\r\n emptyList=['']*numberColumns # create list of empty emtries to compare against\r\n emptyCounter=Counter(emptyList)\r\n \r\n else:\r\n curCounter=Counter(curRow)\r\n diff = curCounter-emptyCounter # create list of the the non empty values\r\n \r\n numberNotEmpty=len(diff)# calc the length of this list\r\n numEmpty=numberColumns-numberNotEmpty \r\n numberEmptyValues=numberEmptyValues+numEmpty\r\n \r\n \r\n rowCount=rowCount+1\r\n \r\n \r\n totalnumber=rowCount*numberColumns\r\n pctEmpty=100*numberEmptyValues/totalnumber\r\n\r\n print(\"nuber of rows :\",rowCount)\r\n print(\"number of columns :\",numberColumns)\r\n print(\"number of empty values:\",numberEmptyValues)\r\n print(\"% of empty values :\",pctEmpty)\r\n\r\n-----------------------------------------------------\r\n\r\nI got the following results from this script:\r\n\r\n train_numeric.csv\r\n nuber of rows : 1183748\r\n number of columns : 970\r\n number of empty values: 1035542147\r\n % of empty values : 90.18551446011652\r\n\r\n train_categorical.csv\r\n nuber of rows : 1183748\r\n number of columns : 2141\r\n number of empty values: 2531522206\r\n % of empty values : 99.886274584961\r\n\r\n train_date.csv\r\n nuber of rows : 1183748\r\n number of columns : 1157\r\n number of empty values: 1360147063\r\n % of empty values : 99.31006150778272", "131655": "Thanks! \r\n\r\nIs Kaggle going Big Data?", "131682": "Using R, my windows machine couldn't handle fread so I used the ff package, which reads data in chunks and writes to disk. It took over 15 minutes, but used <8GB. I then took a sample to mess around with for now.\r\n\r\n library(data.table)\r\n library(ff)\r\n\r\n datecols <- fread(\"train_date.csv\", nrows = 0L) # just getting the columns here\r\n classvector <- c(\"integer\", rep.int(\"numeric\", ncol(datecols)-1)) # avoids type error\r\n\r\n dates <- read.csv.ffdf(file = \"train_date.csv\", header = TRUE, colClasses = classvector)\r\n\r\n sampleidx <- sample(nrow(dates), nrow(dates)/10)\r\n dates2 <- dates[sampleidx, ]", "131685": "I want to join this, but my 4G machine is saying 'DONT'.", "131724": "Those of us with modest hardware may not need to give up so quickly - I've only glanced at the categorical and numeric data so far but there's lots of scope for reducing the memory requirements:\r\n\r\n- I read the train_categorical file in two parts and managed to hash it down from 20GB dense to 800MB sparse (using the same SSD overflow sorcery as Alexandru, this used about 10GB of memory in R for each half but was doable on my old 4GB machine in about 15mins total)\r\n\r\n- the numeric data has lots of NAs and has been scaled so each feature has a mean of zero. But if we have tree methods in mind I guess we could convert the NAs to zero and then a sparse format may help (still playing with this one but I suspect it may be possible to shrink it from 9GB to 2GB).", "131895": "Also for those of us with humble hardware--You could also entirely ignore certain columns(/features). My hunch is that Lines 1, 2, and the first two stations of Line 3 is where the action is at, so to speak. There's a paper by Andrew Ng of Stanford (forgot the title but it's on his website) that it's actually okay to NOT use all of the variables that matter but a set of variables that get you close if not exactly at the right as answer. More features isn't always the answer anyway; sometimes it's the opposite, depending on how things look. This may help reduce memory and computing power requirements. For that matter you can test if increasing the sample size from 10,000 to one million will help much. If it doesn't look like that extra million rows will add much, then well... sayonara. :)", "131947": "Start Looking for Amazon VM!", "132073": "[quote=Alpha;131947]\r\n\r\nStart Looking for Amazon VM!\r\n\r\n[/quote]\r\n\r\nOr better yet, start looking at online learning algorithms... (as Scirpus already [mentioned][1])\r\n\r\nI feel that [vw][2] is making a comeback in this one, after having been dominated a bit by xgboost for the last year or so (at least in Kaggle).\r\n\r\n\r\n[Edit: I haven't looked at the data, so please do not blindly trust my statement about vw being good for this one]\r\n\r\n\r\n [1]: https://www.kaggle.com/c/bosch-production-line-performance/forums/t/22908/datasets-size-uncompressed-14-3gb/131625#post131625\r\n [2]: http://hunch.net/~vw/", "132438": "I too had same problem, using a 4GB laptop is not helping anyway. Then I used H2O, with option -Xmx4g, I was able to load train_categorical in under 10min. Atleast now I can say I have seen the data. I'm happy that now I can explore further." }, "source": "meta" }