{ "id": 22048, "title": "Train.csv - File not loaded completely", "url": "/competitions/grupo-bimbo-inventory-demand/discussion/22048", "author_name": "ClaireS", "post_date": "2016-07-05T09:39:22.777000", "votes": 0, "comment_count": 8, "views": 1299, "content": "

Hi\nI've downloaded all datasets but can't manage to open the train.csv.\nI've tried open it in Excel and R; the other datasets open fine.\nI get the message "FIle not loaded completely", "Windows cannot complete the extraction. The destination file could not be created". \nI've tried on three different machines.\nDoes anyone else had the same issue? How did you fix it?\nThanks

", "messages": [ { "id": 126064, "postDate": "2016-07-05T23:36:03.633Z", "content": "

@ClaireS: Take a look at similar thread here: https://www.kaggle.com/c/grupo-bimbo-inventory-demand/forums/t/21930/anaconda-python-2-7-x-64bit-overcoming-out-of-momory-error/

", "rawMarkdown": "@ClaireS: Take a look at similar thread here: https://www.kaggle.com/c/grupo-bimbo-inventory-demand/forums/t/21930/anaconda-python-2-7-x-64bit-overcoming-out-of-momory-error/\r\n", "votes": 1 }, { "id": 130864, "postDate": "2016-08-11T00:23:35.617Z", "content": "

[quote=Fish;128792]

\n\n

Can you be more specific on how to do this? I'm new to R/Kaggle and stuck on trying to load the data in

\n\n

[/quote]

\n\n
install.packages("readr")\n
\n\n

then after install, run

\n\n
library(readr)\ntrain <- read_csv("train.csv")\n
", "rawMarkdown": "[quote=Fish;128792]\r\n\r\nCan you be more specific on how to do this? I'm new to R/Kaggle and stuck on trying to load the data in\r\n\r\n[/quote]\r\n\r\n install.packages(\"readr\")\r\n\r\nthen after install, run\r\n\r\n library(readr)\r\n train <- read_csv(\"train.csv\")" }, { "id": 128792, "postDate": "2016-07-24T00:38:01.147Z", "content": "

Can you be more specific on how to do this? I'm new to R/Kaggle and stuck on trying to load the data in

", "rawMarkdown": "Can you be more specific on how to do this? I'm new to R/Kaggle and stuck on trying to load the data in" }, { "id": 128791, "postDate": "2016-07-24T00:37:31.693Z", "content": "

[quote=sumedh;126562]

\n\n

You can use fread from data.table and read_csv from readr packages. On a 8GB RAM, I can read the entire train data is just under 2 mins.

\n\n

[/quote]

", "rawMarkdown": "[quote=sumedh;126562]\r\n\r\nYou can use `fread` from `data.table` and `read_csv` from `readr` packages. On a 8GB RAM, I can read the entire train data is just under 2 mins. \r\n\r\n[/quote]\r\n" }, { "id": 126809, "postDate": "2016-07-12T12:19:17.053Z", "content": "

[quote=sumedh;126562]

\n\n

You can use fread from data.table and read_csv from readr packages. On a 8GB RAM, I can read the entire train data is just under 2 mins.

\n\n

[/quote]\n+1 to the above comment. fread is the only way I could read the dataset into my system.\nread.csv for the entire train file of 3GB failed on a machine of 12GB RAM as well!

", "rawMarkdown": "[quote=sumedh;126562]\r\n\r\nYou can use `fread` from `data.table` and `read_csv` from `readr` packages. On a 8GB RAM, I can read the entire train data is just under 2 mins. \r\n\r\n[/quote]\r\n+1 to the above comment. fread is the only way I could read the dataset into my system.\r\nread.csv for the entire train file of 3GB failed on a machine of 12GB RAM as well!" }, { "id": 126558, "postDate": "2016-07-09T19:31:22.543Z", "content": "

In R, you may use:

\n\n
\n

train1<-read.table("train.csv",header=TRUE,sep=",",colClasses=c("integer","integer","integer","integer","integer","integer","integer","character","integer","character","integer"),nrows=1000000)\n train2<-read.table("train.csv",sep=",",colClasses=c("integer","integer","integer","integer","integer","integer","integer","character","integer","character","integer"),skip=1000001,nrows=1000000)\n train3<-read.table("train.csv",sep=",",colClasses=c("integer","integer","integer","integer","integer","integer","integer","character","integer","character","integer"),skip=2000001,nrows=1000000)

\n
\n\n

and so on...\nI think near 75 repetitions should be sufficient.\nFor details, see R help:

\n\n
\n

?read.table

\n
", "rawMarkdown": "In R, you may use:\r\n\r\n> train1<-read.table(\"train.csv\",header=TRUE,sep=\",\",colClasses=c(\"integer\",\"integer\",\"integer\",\"integer\",\"integer\",\"integer\",\"integer\",\"character\",\"integer\",\"character\",\"integer\"),nrows=1000000)\r\n> train2<-read.table(\"train.csv\",sep=\",\",colClasses=c(\"integer\",\"integer\",\"integer\",\"integer\",\"integer\",\"integer\",\"integer\",\"character\",\"integer\",\"character\",\"integer\"),skip=1000001,nrows=1000000)\r\n> train3<-read.table(\"train.csv\",sep=\",\",colClasses=c(\"integer\",\"integer\",\"integer\",\"integer\",\"integer\",\"integer\",\"integer\",\"character\",\"integer\",\"character\",\"integer\"),skip=2000001,nrows=1000000)\r\n\r\nand so on...\r\nI think near 75 repetitions should be sufficient.\r\nFor details, see R help:\r\n> ?read.table\r\n\r\n" }, { "id": 126001, "postDate": "2016-07-05T12:45:17.237Z", "content": "

The message is caused simply by the size of the file. For example, Excel file limits the number of rows to 1,048,576 rows and any row exceeding this limit will be truncated. Since this is the limitation, not an error, you cannot fix it.

", "rawMarkdown": "The message is caused simply by the size of the file. For example, Excel file limits the number of rows to 1,048,576 rows and any row exceeding this limit will be truncated. Since this is the limitation, not an error, you cannot fix it. " }, { "id": 125986, "postDate": "2016-07-05T09:39:22.777Z", "content": "

Hi\nI've downloaded all datasets but can't manage to open the train.csv.\nI've tried open it in Excel and R; the other datasets open fine.\nI get the message "FIle not loaded completely", "Windows cannot complete the extraction. The destination file could not be created". \nI've tried on three different machines.\nDoes anyone else had the same issue? How did you fix it?\nThanks

", "rawMarkdown": "Hi\r\nI've downloaded all datasets but can't manage to open the train.csv.\r\nI've tried open it in Excel and R; the other datasets open fine.\r\nI get the message \"FIle not loaded completely\", \"Windows cannot complete the extraction. The destination file could not be created\". \r\nI've tried on three different machines.\r\nDoes anyone else had the same issue? How did you fix it?\r\nThanks\r\n" }, { "id": 126562, "postDate": "2016-07-09T19:53:32.840Z", "rawMarkdown": "", "votes": 3, "isDeleted": true } ], "comments": [ { "id": 126064, "author_name": "RB", "author_url": "", "post_date": "2016-07-05T23:36:03.633000", "content": "

@ClaireS: Take a look at similar thread here: https://www.kaggle.com/c/grupo-bimbo-inventory-demand/forums/t/21930/anaconda-python-2-7-x-64bit-overcoming-out-of-momory-error/

", "votes": 1, "replies": [] }, { "id": 130864, "author_name": "Denis Zaff", "author_url": "", "post_date": "2016-08-11T00:23:35.617000", "content": "

[quote=Fish;128792]

\n\n

Can you be more specific on how to do this? I'm new to R/Kaggle and stuck on trying to load the data in

\n\n

[/quote]

\n\n
install.packages("readr")\n
\n\n

then after install, run

\n\n
library(readr)\ntrain <- read_csv("train.csv")\n
", "votes": 0, "replies": [] }, { "id": 128792, "author_name": "Fish", "author_url": "", "post_date": "2016-07-24T00:38:01.147000", "content": "

Can you be more specific on how to do this? I'm new to R/Kaggle and stuck on trying to load the data in

", "votes": 0, "replies": [] }, { "id": 128791, "author_name": "Fish", "author_url": "", "post_date": "2016-07-24T00:37:31.693000", "content": "

[quote=sumedh;126562]

\n\n

You can use fread from data.table and read_csv from readr packages. On a 8GB RAM, I can read the entire train data is just under 2 mins.

\n\n

[/quote]

", "votes": 0, "replies": [] }, { "id": 126809, "author_name": "Naval Singh", "author_url": "", "post_date": "2016-07-12T12:19:17.053000", "content": "

[quote=sumedh;126562]

\n\n

You can use fread from data.table and read_csv from readr packages. On a 8GB RAM, I can read the entire train data is just under 2 mins.

\n\n

[/quote]\n+1 to the above comment. fread is the only way I could read the dataset into my system.\nread.csv for the entire train file of 3GB failed on a machine of 12GB RAM as well!

", "votes": 0, "replies": [] }, { "id": 126558, "author_name": "patsyuck", "author_url": "", "post_date": "2016-07-09T19:31:22.543000", "content": "

In R, you may use:

\n\n
\n

train1<-read.table("train.csv",header=TRUE,sep=",",colClasses=c("integer","integer","integer","integer","integer","integer","integer","character","integer","character","integer"),nrows=1000000)\n train2<-read.table("train.csv",sep=",",colClasses=c("integer","integer","integer","integer","integer","integer","integer","character","integer","character","integer"),skip=1000001,nrows=1000000)\n train3<-read.table("train.csv",sep=",",colClasses=c("integer","integer","integer","integer","integer","integer","integer","character","integer","character","integer"),skip=2000001,nrows=1000000)

\n
\n\n

and so on...\nI think near 75 repetitions should be sufficient.\nFor details, see R help:

\n\n
\n

?read.table

\n
", "votes": 0, "replies": [] }, { "id": 126001, "author_name": "HenryVu", "author_url": "", "post_date": "2016-07-05T12:45:17.237000", "content": "

The message is caused simply by the size of the file. For example, Excel file limits the number of rows to 1,048,576 rows and any row exceeding this limit will be truncated. Since this is the limitation, not an error, you cannot fix it.

", "votes": 0, "replies": [] }, { "id": 126562, "author_name": "", "author_url": "", "post_date": "2016-07-09T19:53:32.840000", "content": "", "votes": 3, "replies": [] } ], "raw_markdown_by_id": { "126064": "@ClaireS: Take a look at similar thread here: https://www.kaggle.com/c/grupo-bimbo-inventory-demand/forums/t/21930/anaconda-python-2-7-x-64bit-overcoming-out-of-momory-error/\r\n", "130864": "[quote=Fish;128792]\r\n\r\nCan you be more specific on how to do this? I'm new to R/Kaggle and stuck on trying to load the data in\r\n\r\n[/quote]\r\n\r\n install.packages(\"readr\")\r\n\r\nthen after install, run\r\n\r\n library(readr)\r\n train <- read_csv(\"train.csv\")", "128792": "Can you be more specific on how to do this? I'm new to R/Kaggle and stuck on trying to load the data in", "128791": "[quote=sumedh;126562]\r\n\r\nYou can use `fread` from `data.table` and `read_csv` from `readr` packages. On a 8GB RAM, I can read the entire train data is just under 2 mins. \r\n\r\n[/quote]\r\n", "126809": "[quote=sumedh;126562]\r\n\r\nYou can use `fread` from `data.table` and `read_csv` from `readr` packages. On a 8GB RAM, I can read the entire train data is just under 2 mins. \r\n\r\n[/quote]\r\n+1 to the above comment. fread is the only way I could read the dataset into my system.\r\nread.csv for the entire train file of 3GB failed on a machine of 12GB RAM as well!", "126558": "In R, you may use:\r\n\r\n> train1<-read.table(\"train.csv\",header=TRUE,sep=\",\",colClasses=c(\"integer\",\"integer\",\"integer\",\"integer\",\"integer\",\"integer\",\"integer\",\"character\",\"integer\",\"character\",\"integer\"),nrows=1000000)\r\n> train2<-read.table(\"train.csv\",sep=\",\",colClasses=c(\"integer\",\"integer\",\"integer\",\"integer\",\"integer\",\"integer\",\"integer\",\"character\",\"integer\",\"character\",\"integer\"),skip=1000001,nrows=1000000)\r\n> train3<-read.table(\"train.csv\",sep=\",\",colClasses=c(\"integer\",\"integer\",\"integer\",\"integer\",\"integer\",\"integer\",\"integer\",\"character\",\"integer\",\"character\",\"integer\"),skip=2000001,nrows=1000000)\r\n\r\nand so on...\r\nI think near 75 repetitions should be sufficient.\r\nFor details, see R help:\r\n> ?read.table\r\n\r\n", "126001": "The message is caused simply by the size of the file. For example, Excel file limits the number of rows to 1,048,576 rows and any row exceeding this limit will be truncated. Since this is the limitation, not an error, you cannot fix it. ", "125986": "Hi\r\nI've downloaded all datasets but can't manage to open the train.csv.\r\nI've tried open it in Excel and R; the other datasets open fine.\r\nI get the message \"FIle not loaded completely\", \"Windows cannot complete the extraction. The destination file could not be created\". \r\nI've tried on three different machines.\r\nDoes anyone else had the same issue? How did you fix it?\r\nThanks\r\n", "126562": "" } }