I found this post on stackoverflow useful to read training data in R. The function read.csv was constantly running out of memory while reading the big training data set.
\nlibrary(RSQLite)
# Create/Connect to a database
con <- dbConnect(RSQLite::SQLite(), dbname = "sample_db.sqlite")
# read csv file into sql database
# Warning: this is going to take some time and disk space,
# as your complete CSV file is transferred into an SQLite database.
dbWriteTable(con, name="sample_table", value="Your_Big_CSV_File.csv",
row.names=FALSE, header=TRUE, sep = ",")
# Query your data as you like
yourData <- dbGetQuery(con, "SELECT * FROM sample_table LIMIT 10")
dbDisconnect(con)
", "messages": [ { "id": "56969", "postDate": "10/29/2014 09:43:23", "content": "I found this post on stackoverflow useful to read training data in R. The function read.csv was constantly running out of memory while reading the big training data set.
\nlibrary(RSQLite)
# Create/Connect to a database
con <- dbConnect(RSQLite::SQLite(), dbname = "sample_db.sqlite")
# read csv file into sql database
# Warning: this is going to take some time and disk space,
# as your complete CSV file is transferred into an SQLite database.
dbWriteTable(con, name="sample_table", value="Your_Big_CSV_File.csv",
row.names=FALSE, header=TRUE, sep = ",")
# Query your data as you like
yourData <- dbGetQuery(con, "SELECT * FROM sample_table LIMIT 10")
dbDisconnect(con)
", "rawMarkdown": "", "votes": null }, { "id": "56972", "postDate": "10/29/2014 10:08:42", "content": "Thanks. Might give that a try. So far, I've been relying on a perl one liner - also via stackoverflow - to create a sample of the training set. To create a 1% sample from the terminal you just type:
\nperl -ne 'print if (rand() < 0.01)' train.csv > trainSample.csv
It only takes a few seconds to run through the entire training set, even on my basic laptop.
", "rawMarkdown": "", "votes": null }, { "id": "57004", "postDate": "10/29/2014 17:54:50", "content": "The ff and ffbase packages are great for reading in huge files and storing them on disk. They are transparently accessed using standard data.frame notation:
\nlibrary(ff)
library(ffbase)
df <- read.csv.ffdf(file=gzfile('./train_rev2.gz'), VERBOSE=TRUE)
You can then sample very easily like so:
\ns <- sample(nrow(df), nrow(df)*0.01)
df.in.mem <- df[s, ]
I find this workflow to be great for having very quick access to the data I need using the same semantics I use for every day EDA.
", "rawMarkdown": "", "votes": null }, { "id": "57026", "postDate": "10/29/2014 22:36:27", "content": "My strategy is to build out an online version of gradient descent, and reading in the files via stdin. I'm using hashing to simulate how R handles factor variables.
\n\nObviously feel free to tell me if I'm going off the deep end. My biggest concern is that the growth of the object will result in long training times (I saw 1.3 seconds for the first 1,000 lines of data, 16 seconds for the first 10K, so these results are definitely less than encouraging to handle 40M.
", "rawMarkdown": "", "votes": null }, { "id": "63174", "postDate": "01/26/2015 04:43:51", "content": "I am using a same method. it seems just reading the file takes a while. My machine has been running for several minutes now and still reading it.
\nhow long did it take R to read all the data, Zelazny7?
\nThanks
", "rawMarkdown": "", "votes": null }, { "id": "63175", "postDate": "01/26/2015 04:46:13", "content": "It just finished reading. here is total time:
\ncsv-read=1091.917sec ffdf-write=947.458sec
\nTOTAL=2039.375sec
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 56972, "author_name": "cartographic", "author_url": "", "post_date": "10/29/2014 10:08:42", "content": "Thanks. Might give that a try. So far, I've been relying on a perl one liner - also via stackoverflow - to create a sample of the training set. To create a 1% sample from the terminal you just type:
\nperl -ne 'print if (rand() < 0.01)' train.csv > trainSample.csv
It only takes a few seconds to run through the entire training set, even on my basic laptop.
", "votes": null, "replies": [] }, { "id": 57004, "author_name": "zelazny7", "author_url": "", "post_date": "10/29/2014 17:54:50", "content": "The ff and ffbase packages are great for reading in huge files and storing them on disk. They are transparently accessed using standard data.frame notation:
\nlibrary(ff)
library(ffbase)
df <- read.csv.ffdf(file=gzfile('./train_rev2.gz'), VERBOSE=TRUE)
You can then sample very easily like so:
\ns <- sample(nrow(df), nrow(df)*0.01)
df.in.mem <- df[s, ]
I find this workflow to be great for having very quick access to the data I need using the same semantics I use for every day EDA.
", "votes": null, "replies": [] }, { "id": 57026, "author_name": "jdr0317", "author_url": "", "post_date": "10/29/2014 22:36:27", "content": "My strategy is to build out an online version of gradient descent, and reading in the files via stdin. I'm using hashing to simulate how R handles factor variables.
\n\nObviously feel free to tell me if I'm going off the deep end. My biggest concern is that the growth of the object will result in long training times (I saw 1.3 seconds for the first 1,000 lines of data, 16 seconds for the first 10K, so these results are definitely less than encouraging to handle 40M.
", "votes": null, "replies": [] }, { "id": 63174, "author_name": "nubeee", "author_url": "", "post_date": "01/26/2015 04:43:51", "content": "I am using a same method. it seems just reading the file takes a while. My machine has been running for several minutes now and still reading it.
\nhow long did it take R to read all the data, Zelazny7?
\nThanks
", "votes": null, "replies": [] }, { "id": 63175, "author_name": "nubeee", "author_url": "", "post_date": "01/26/2015 04:46:13", "content": "It just finished reading. here is total time:
\ncsv-read=1091.917sec ffdf-write=947.458sec
\nTOTAL=2039.375sec
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "56969": "", "56972": "", "57004": "", "57026": "", "63174": "", "63175": "" }, "source": "meta" }