{ "id": 11073, "title": "ff -package", "url": "/competitions/avazu-ctr-prediction/discussion/11073", "author_name": "", "post_date": "2014-11-27T19:47:11.517Z", "votes": 1, "comment_count": 6, "views": 2698, "content": "

Hello.

\n

Im new to analyzing so big files of data and have been searching around the web and the forums here for a solution to my RAM - problems. I figured out that the ff-package was the way to go. This seemed fine until I was going to take a look at my new and exciting ffdf file.... and then figured out it was only one column ...

\n

Is it supposed to do this? Or does anyone have a solution for me ?

Thanks
Magnus

\n

....it just occured to me that there is such a thing as seperators... which I guess will solve the problem.

", "messages": [ { "id": "59116", "postDate": "11/27/2014 19:47:11", "content": "

Hello.

\n

Im new to analyzing so big files of data and have been searching around the web and the forums here for a solution to my RAM - problems. I figured out that the ff-package was the way to go. This seemed fine until I was going to take a look at my new and exciting ffdf file.... and then figured out it was only one column ...

\n

Is it supposed to do this? Or does anyone have a solution for me ?

Thanks
Magnus

\n

....it just occured to me that there is such a thing as seperators... which I guess will solve the problem.

", "rawMarkdown": "", "votes": null }, { "id": "59137", "postDate": "11/28/2014 04:52:07", "content": "

Hi Magnus,  A package SOAR may also be of interest to you as it helps to save memory while storing the records on disk in R.

\n

Cheers

", "rawMarkdown": "", "votes": null }, { "id": "59165", "postDate": "11/28/2014 15:37:09", "content": "

[quote=Magnus R. Aunevik-Berntsen;59116]

\n

and then figured out it was only one column ...

\n

Is it supposed to do this? Or does anyone have a solution for me ?

\n

[/quote]

\n

I've been using the ff package for data manipulation and transformations in this competition. If you haven't solved the problem yet here is a good way to load data into ffdf objects.

\n

http://pastebin.com/4745tB13

\n

The reason why I read the first 1000 lines using the standard read.csv function is that the parameter x in the read.table.ffdf /read.csv.ffdf function, "...defines crucial features that are otherwise determined during the 'first' chunk of reading: vmodes, colnames, colClasses, sequence of predefined levels." So basically that's how I let  read.table.ffdf know what colClasses I want. Just an FYI read.table.ffdf cannot read class "character", you have to change its class to factor.

\n

In the end it should look something like this:

\n

https://imgur.com/QeefEi2

\n

I hope this helps.

", "rawMarkdown": "", "votes": null }, { "id": "59166", "postDate": "11/28/2014 15:38:13", "content": "

DUPLICATE POST,

", "rawMarkdown": "", "votes": null }, { "id": "59213", "postDate": "11/29/2014 10:24:16", "content": "

Thanks for the help! Just got to see if Im able to get anything sensible out of the data then.. =)

", "rawMarkdown": "", "votes": null }, { "id": "59217", "postDate": "11/29/2014 10:55:58", "content": "

Yep, I didn't have any problem with using ff. Make sure you are using read.csv.ffdf, since it is comma delimited. Also always load ffbase since it has a lot of good tools. But mine worked fine as is:

\n\n


require(ffbase)
require(ff)

\n\n

#load data into ffdf

\n

train.data = read.csv.ffdf(file = './data/train.data.csv', header = T)

\n

#save data
save.ffdf(train.data, dir='./train.data')

\n

#get dimensions of data
dim(train.data)

", "rawMarkdown": "", "votes": null }, { "id": "59465", "postDate": "12/03/2014 15:16:11", "content": "

pi_informatics what  method  to  you    use  for  training  with  so  much  data?Or  are you just sampling   on the   train  data set?

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 59137, "author_name": "drpatrickchan", "author_url": "", "post_date": "11/28/2014 04:52:07", "content": "

Hi Magnus,  A package SOAR may also be of interest to you as it helps to save memory while storing the records on disk in R.

\n

Cheers

", "votes": null, "replies": [] }, { "id": 59165, "author_name": "wacaxx", "author_url": "", "post_date": "11/28/2014 15:37:09", "content": "

[quote=Magnus R. Aunevik-Berntsen;59116]

\n

and then figured out it was only one column ...

\n

Is it supposed to do this? Or does anyone have a solution for me ?

\n

[/quote]

\n

I've been using the ff package for data manipulation and transformations in this competition. If you haven't solved the problem yet here is a good way to load data into ffdf objects.

\n

http://pastebin.com/4745tB13

\n

The reason why I read the first 1000 lines using the standard read.csv function is that the parameter x in the read.table.ffdf /read.csv.ffdf function, "...defines crucial features that are otherwise determined during the 'first' chunk of reading: vmodes, colnames, colClasses, sequence of predefined levels." So basically that's how I let  read.table.ffdf know what colClasses I want. Just an FYI read.table.ffdf cannot read class "character", you have to change its class to factor.

\n

In the end it should look something like this:

\n

https://imgur.com/QeefEi2

\n

I hope this helps.

", "votes": null, "replies": [] }, { "id": 59166, "author_name": "wacaxx", "author_url": "", "post_date": "11/28/2014 15:38:13", "content": "

DUPLICATE POST,

", "votes": null, "replies": [] }, { "id": 59213, "author_name": "berntsen", "author_url": "", "post_date": "11/29/2014 10:24:16", "content": "

Thanks for the help! Just got to see if Im able to get anything sensible out of the data then.. =)

", "votes": null, "replies": [] }, { "id": 59217, "author_name": "piinformatics", "author_url": "", "post_date": "11/29/2014 10:55:58", "content": "

Yep, I didn't have any problem with using ff. Make sure you are using read.csv.ffdf, since it is comma delimited. Also always load ffbase since it has a lot of good tools. But mine worked fine as is:

\n\n


require(ffbase)
require(ff)

\n\n

#load data into ffdf

\n

train.data = read.csv.ffdf(file = './data/train.data.csv', header = T)

\n

#save data
save.ffdf(train.data, dir='./train.data')

\n

#get dimensions of data
dim(train.data)

", "votes": null, "replies": [] }, { "id": 59465, "author_name": "", "author_url": "", "post_date": "12/03/2014 15:16:11", "content": "

pi_informatics what  method  to  you    use  for  training  with  so  much  data?Or  are you just sampling   on the   train  data set?

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "59116": "", "59137": "", "59165": "", "59166": "", "59213": "", "59217": "", "59465": "" }, "source": "meta" }