{ "id": 9724, "title": "The ids after the 202513283rd row in transactions data are showing as 'NA'", "url": "/competitions/acquire-valued-shoppers-challenge/discussion/9724", "author_name": "", "post_date": "2014-07-12T10:02:13.007Z", "votes": null, "comment_count": 4, "views": 1898, "content": "

I am reading the transactions data set in R using :

\n

x<-read.csv.ffdf(file="transactions.csv", header=TRUE, VERBOSE=TRUE, nrows=350000000, next.rows=10000000, colClasses=NA)

\n

The values of the ids after 202513283rd row are showing as NA. Is there some argument that should be passed in read.csv.ffdf to make them valid(is it due to some limit on values accepted) or are they really NA....? 

", "messages": [ { "id": "50389", "postDate": "07/12/2014 10:02:13", "content": "

I am reading the transactions data set in R using :

\n

x<-read.csv.ffdf(file="transactions.csv", header=TRUE, VERBOSE=TRUE, nrows=350000000, next.rows=10000000, colClasses=NA)

\n

The values of the ids after 202513283rd row are showing as NA. Is there some argument that should be passed in read.csv.ffdf to make them valid(is it due to some limit on values accepted) or are they really NA....? 

", "rawMarkdown": "", "votes": null }, { "id": "50414", "postDate": "07/12/2014 19:39:19", "content": "

Someone please reply to this 

", "rawMarkdown": "", "votes": null }, { "id": "50451", "postDate": "07/14/2014 00:09:03", "content": "

I am very new to kaggle and just trying to learn about applying data analysis on large data sets in R. I am really stuck on this and want to know out of curiosity that after some value around 202 Millionth row the values of customer ids are shown as NA when read by using read.csv.ffdf, are they really NA or  is it due to some memory limit of this command...? The command that I am using:

\n

read.csv.ffdf(file="transactions.csv", header=TRUE, VERBOSE=TRUE, nrows=350000000, next.rows=10000000, colClasses=NA)

\n\n\n

                read.table(file = "transactions.csv", header = TRUE, sep = ",", skip =100000000,                              nrows = 50000000)

\n\n\n

Someone please suggest the mistake  that I making in read.csv.ffdf or suggest some other command that can do the job in lesser time. 

", "rawMarkdown": "", "votes": null }, { "id": "50452", "postDate": "07/14/2014 00:35:22", "content": "

See:

\n

https://www.kaggle.com/c/acquire-valued-shoppers-challenge/forums/t/8258/fread-function-in-r

\n

http://www.kaggle.com/c/acquire-valued-shoppers-challenge/forums/t/8249/python-script-to-read-the-transactions-csv

\n

http://www.kaggle.com/c/acquire-valued-shoppers-challenge/forums/t/8298/trouble-loading-transactions-csv-into-r

\n

http://www.kaggle.com/c/acquire-valued-shoppers-challenge/forums/t/9394/my-r-code-is-too-slow

\n

http://www.kaggle.com/c/acquire-valued-shoppers-challenge/forums/t/9507/read-specific-range-of-rows-from-a-transaction-csv-in-r

", "rawMarkdown": "", "votes": null }, { "id": "50481", "postDate": "07/14/2014 16:54:41", "content": "

Do you have a 64 bit version of R >3.0?  It may be that your machine cannot allocate enough RAM to the R process.

\n\n

There could also be issues if you are using windows... it's been a while since I've used R on windows, but I think there are per-process RAM limits.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 50414, "author_name": "shreedharpawar", "author_url": "", "post_date": "07/12/2014 19:39:19", "content": "

Someone please reply to this 

", "votes": null, "replies": [] }, { "id": 50451, "author_name": "shreedharpawar", "author_url": "", "post_date": "07/14/2014 00:09:03", "content": "

I am very new to kaggle and just trying to learn about applying data analysis on large data sets in R. I am really stuck on this and want to know out of curiosity that after some value around 202 Millionth row the values of customer ids are shown as NA when read by using read.csv.ffdf, are they really NA or  is it due to some memory limit of this command...? The command that I am using:

\n

read.csv.ffdf(file="transactions.csv", header=TRUE, VERBOSE=TRUE, nrows=350000000, next.rows=10000000, colClasses=NA)

\n\n\n

                read.table(file = "transactions.csv", header = TRUE, sep = ",", skip =100000000,                              nrows = 50000000)

\n\n\n

Someone please suggest the mistake  that I making in read.csv.ffdf or suggest some other command that can do the job in lesser time. 

", "votes": null, "replies": [] }, { "id": 50452, "author_name": "triskelion", "author_url": "", "post_date": "07/14/2014 00:35:22", "content": "

See:

\n

https://www.kaggle.com/c/acquire-valued-shoppers-challenge/forums/t/8258/fread-function-in-r

\n

http://www.kaggle.com/c/acquire-valued-shoppers-challenge/forums/t/8249/python-script-to-read-the-transactions-csv

\n

http://www.kaggle.com/c/acquire-valued-shoppers-challenge/forums/t/8298/trouble-loading-transactions-csv-into-r

\n

http://www.kaggle.com/c/acquire-valued-shoppers-challenge/forums/t/9394/my-r-code-is-too-slow

\n

http://www.kaggle.com/c/acquire-valued-shoppers-challenge/forums/t/9507/read-specific-range-of-rows-from-a-transaction-csv-in-r

", "votes": null, "replies": [] }, { "id": 50481, "author_name": "zachmayer", "author_url": "", "post_date": "07/14/2014 16:54:41", "content": "

Do you have a 64 bit version of R >3.0?  It may be that your machine cannot allocate enough RAM to the R process.

\n\n

There could also be issues if you are using windows... it's been a while since I've used R on windows, but I think there are per-process RAM limits.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "50389": "", "50414": "", "50451": "", "50452": "", "50481": "" }, "source": "meta" }