{ "id": 718, "title": "Memory problems with reading the train set file with the R language.", "url": "/competitions/ClaimPredictionChallenge/discussion/718", "author_name": "", "post_date": "2011-07-18T14:05:02.477Z", "votes": null, "comment_count": 13, "views": 15871, "content": "

I tried to read the train_set by the commands

\r\n

memory.limit(4095)

\r\n

aaa<-read.csv(file="train_set.csv")

\r\n

Unfortunately I get the error cannot allocate a vector of size 62.5 M

\r\n

"If 32-bit R is run on most 64-bit versions\r\n of Windows the maximum value of obtainable memory is just under 4Gb. For a 64-bit versions of R under\r\n 64-bit Windows the limit is currently 8Tb."

\r\n

4 Gb and 8 Tb is a big difference and I wonder if people who use R for this competition use only 64 bit versions of R under 64 bit Windows. 

", "messages": [ { "id": "4642", "postDate": "07/18/2011 14:05:02", "content": "

I tried to read the train_set by the commands

\r\n

memory.limit(4095)

\r\n

aaa<-read.csv(file="train_set.csv")

\r\n

Unfortunately I get the error cannot allocate a vector of size 62.5 M

\r\n

"If 32-bit R is run on most 64-bit versions\r\n of Windows the maximum value of obtainable memory is just under 4Gb. For a 64-bit versions of R under\r\n 64-bit Windows the limit is currently 8Tb."

\r\n

4 Gb and 8 Tb is a big difference and I wonder if people who use R for this competition use only 64 bit versions of R under 64 bit Windows. 

", "rawMarkdown": "", "votes": null }, { "id": "4643", "postDate": "07/18/2011 18:10:18", "content": "

Yes Uri, you would need 64bit operating system and and a large amount of RAM.

\r\n

Other options (in order of my preference) you have ( & which I found practical in handling large datasets)

\r\n

1. Find a 64 bit OS with RAM 2-3 times the size of the dataset.

\r\n

2. Make use of SQL Server Express/MySQL database and use R 32bit on 32bit OS (if not #1) to explore data by brining in only the columns you need to do your analysis.

\r\n

3. Use cloud serves such as Amazon EC2, where you can get access to upto 68GB RAM 64bit machines for ~$2.5/hours. (if you want more details about how to setup your cloud machine let me know)

\r\n

4. Use some R packages for out of memory analysis - but I could not get them to work the way I wanted.

\r\n

 

", "rawMarkdown": "", "votes": null }, { "id": "4651", "postDate": "07/19/2011 16:57:38", "content": "

so far the immense size of the training and test sets have been the biggest obstacle.

\r\n

i am using an Amazon EC2 with 17GB, and i am still hitting memory limits and find myself having to be careful how i handle the data and process it in batches as opposed to all at once.

", "rawMarkdown": "", "votes": null }, { "id": "4652", "postDate": "07/19/2011 17:15:47", "content": "

A couple of other options: use the big data features of Revolution R, which is available free to Kaggle competitors (see the link on Kaggle's home page) - this software has various libraries and features that allow really big data sets to be handled, even\r\n if they don't fit in memory. Another approach is to filter out some of the data using a simple DB (e.g. sqlite is a good choice) - there's a\r\nlot of rows where the dependent variable is zero; I bet randomly removing a bunch of them won't really impact your model at all, but will make it much easier to analyse.

", "rawMarkdown": "", "votes": null }, { "id": "4658", "postDate": "07/19/2011 22:22:04", "content": "

I am trying to set up a SQL 2008 R2 instance using Amazon EC2, but I can't figure out what to put in for the server name? I just want a local environment where I can run the data through analysis services. Any tips?

", "rawMarkdown": "", "votes": null }, { "id": "4671", "postDate": "07/21/2011 00:49:43", "content": "

[quote=Tom Seward;4658]

\r\n

I am trying to set up a SQL 2008 R2 instance using Amazon EC2, but I can't figure out what to put in for the server name?

\r\n

[/quote]

\r\n

When you select the EC2 instance the lower panel will have a value titled "Public DNS". That is the name of the server. Also, you have to ensure that\r\n the security group has the SQL Server port open, which by default is 1433.

", "rawMarkdown": "", "votes": null }, { "id": "4680", "postDate": "07/21/2011 19:30:54", "content": "

R has prohibitive memory consumption. You should use a tool that is more memory efficient: for example: the "TIMi Suite". See you! Frank

", "rawMarkdown": "", "votes": null }, { "id": "4729", "postDate": "07/25/2011 15:54:25", "content": "

On a 64bit Windows system, with Revolution R 64 bit, this works for me (I have 8GB of RAM, not sure if this might become a bottleneck for you) It takes a few minutes to read in from CSV, but the read in from xdf is couple of seconds at most... once you have\r\n the dataframe, you can export it to whatever you wanna use this data in - note that all car charactersitics will be the same for the same submodel, so you can effectively break the table in 2, a look up and the main - making the data transfer to other programs\r\n smaller.

\r\n

rxTextToXdf( inFile = "C:\\Users\\InsuranceContest\\trainset.csv",
\r\noutFile = "C:\\Users\\InsuranceContest\\trainset.xdf", rowSelection = NULL,
\r\ntransforms = NULL, append = "none", overwrite = FALSE,
\r\nstringsAsFactors = TRUE, columnDelimiters = NULL,
\r\nrowsPerRead = 500000)

\r\n

data <- rxXdfToDataFrame(file = "C:\\Users\\InsuranceContest\\train_set.xdf", varsToKeep = NULL, varsToDrop = NULL, rowVarName = NULL,\r\n
\r\nrowSelection = NULL, transforms = NULL, blocksPerRead = 1, reportProgress = 2, maxRowsByCols = 9e+09)

", "rawMarkdown": "", "votes": null }, { "id": "4730", "postDate": "07/25/2011 16:04:01", "content": "

"note that all car charactersitics will be the same for the same submodel"

\r\n

Maybe this observation will help with modeling (model on a smaller, summarized dataset), and not just data transfer?

", "rawMarkdown": "", "votes": null }, { "id": "4837", "postDate": "08/01/2011 10:43:51", "content": "

[quote=Jeremy Howard (Kaggle);4652]

\r\n

A couple of other options: use the big data features of Revolution R, which is available free to Kaggle competitors (see the link on Kaggle's home page) - this software has various libraries and features that allow really big data sets to be handled, even\r\n if they don't fit in memory. Another approach is to filter out some of the data using a simple DB (e.g. sqlite is a good choice) - there's a\r\nlot of rows where the dependent variable is zero; I bet randomly removing a bunch of them won't really impact your model at all, but will make it much easier to analyse.

\r\n

[/quote]

\r\n

I do not understand how Revolution R helps to read big data.

\r\n

page 20 of RevoScaleRGetStart.pdf gives the following as an example

\r\n

testDF <- rxReadXdf(file=dataName, varsToKeep = c("ArrDelay", "DepDelay", "DayOfWeek"), startRow = 100000, numRows = 1000)

\r\n

summary(testDF)

\r\n

I read that I should be able to analyze also data that is too big to fit into memory by using RevoScaleR functions but I do not understand how I can do it

\r\n

I am not interested in regression but only in the same details that I get with summary(testDF) and in the ability to have big vectors like testDF$Row_ID that I can do calculations on them.

\r\n

I wonder if there is a good way to do it without installing 64 bits window or maybe if I install 64 bit windows I can simply write something like memory.limit(100000) that is going to solve all the problems

\r\n

with 32 bits I can write at most memory.limit(4095) without getting an error.

\r\n

I guess that more RAM can also help but the computer has no limit of 4095 mbytes even if it has no more RAM than it and I think that practically there should be no problem with arrays of 100,000 mbytes considering the fact that the computer clearly has enough\r\n memory not in RAM for it.

\r\n

 

\r\n

 

", "rawMarkdown": "", "votes": null }, { "id": "4871", "postDate": "08/02/2011 04:02:05", "content": "

[quote=Uri Blass;4837]

\r\n

I do not understand how Revolution R helps to read big data.

\r\n

[/quote]

\r\n

The RevoScaleR getting started guide introduces functions such as rxSummary, rxHistogram, rxLinMod, rxLinePlot, rxCrossTabs etc. For example using rxSummary(~., data="traindata") will produce univariate summaries for each variable if you have the file traindata.xdf\r\n in your working directory. You first need to convert the csv file to a xdf file using rxTextToXdf.

\r\n

I must admit I've thrown the towel in with my 32bit windows on an old machine, but Revolution R seems like a great tool for out-of-memory operations. Incidentally, my one submission (team smidsy 0.038) can be seen as a benchmark: it's what you get if you\r\n take the Blind_Model averages as the predictions (which overfits).

", "rawMarkdown": "", "votes": null }, { "id": "4916", "postDate": "08/04/2011 02:21:12", "content": "

Hi,

\r\n

Using RevoScaleR from Revolution Analytics, I was able to read the training set into an XDF file, which is their file format. I barely have 1 GB RAM in my trusty Acer Netbook runing Windows XP Home Edition. 

\r\n

So far, so good.

\r\n

Jan

", "rawMarkdown": "", "votes": null }, { "id": "5989", "postDate": "10/10/2011 14:06:29", "content": "

Thanks for this tip, I've been looking for the Revolution R link on the URL \r\nwww.kaggle.com.

\r\n

It's probably just me, but I can't seem to spot it, can you help ? 

\r\n

 

\r\n

Many thanks.

\r\n


\r\n

\r\n

 

\r\n


\r\n

\r\n

 

\r\n

[quote=Jeremy Howard (Kaggle);4652]

\r\n

A couple of other options: use the big data features of Revolution R, which is available free to Kaggle competitors (see the link on Kaggle's home page) - this software has various libraries and features that allow really big data sets to be handled, even\r\n if they don't fit in memory. Another approach is to filter out some of the data using a simple DB (e.g. sqlite is a good choice) - there's a\r\nlot of rows where the dependent variable is zero; I bet randomly removing a bunch of them won't really impact your model at all, but will make it much easier to analyse.

\r\n

[/quote]

", "rawMarkdown": "", "votes": null }, { "id": "5990", "postDate": "10/10/2011 14:36:26", "content": "

I know I saw it before, but don't see the link now, but this one works:

\r\n

http://info.revolutionanalytics.com/Kaggle.html

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 4643, "author_name": "sashikanthdareddy", "author_url": "", "post_date": "07/18/2011 18:10:18", "content": "

Yes Uri, you would need 64bit operating system and and a large amount of RAM.

\r\n

Other options (in order of my preference) you have ( & which I found practical in handling large datasets)

\r\n

1. Find a 64 bit OS with RAM 2-3 times the size of the dataset.

\r\n

2. Make use of SQL Server Express/MySQL database and use R 32bit on 32bit OS (if not #1) to explore data by brining in only the columns you need to do your analysis.

\r\n

3. Use cloud serves such as Amazon EC2, where you can get access to upto 68GB RAM 64bit machines for ~$2.5/hours. (if you want more details about how to setup your cloud machine let me know)

\r\n

4. Use some R packages for out of memory analysis - but I could not get them to work the way I wanted.

\r\n

 

", "votes": null, "replies": [] }, { "id": 4651, "author_name": "markwaddle", "author_url": "", "post_date": "07/19/2011 16:57:38", "content": "

so far the immense size of the training and test sets have been the biggest obstacle.

\r\n

i am using an Amazon EC2 with 17GB, and i am still hitting memory limits and find myself having to be careful how i handle the data and process it in batches as opposed to all at once.

", "votes": null, "replies": [] }, { "id": 4652, "author_name": "jhoward", "author_url": "", "post_date": "07/19/2011 17:15:47", "content": "

A couple of other options: use the big data features of Revolution R, which is available free to Kaggle competitors (see the link on Kaggle's home page) - this software has various libraries and features that allow really big data sets to be handled, even\r\n if they don't fit in memory. Another approach is to filter out some of the data using a simple DB (e.g. sqlite is a good choice) - there's a\r\nlot of rows where the dependent variable is zero; I bet randomly removing a bunch of them won't really impact your model at all, but will make it much easier to analyse.

", "votes": null, "replies": [] }, { "id": 4658, "author_name": "tomseward", "author_url": "", "post_date": "07/19/2011 22:22:04", "content": "

I am trying to set up a SQL 2008 R2 instance using Amazon EC2, but I can't figure out what to put in for the server name? I just want a local environment where I can run the data through analysis services. Any tips?

", "votes": null, "replies": [] }, { "id": 4671, "author_name": "markwaddle", "author_url": "", "post_date": "07/21/2011 00:49:43", "content": "

[quote=Tom Seward;4658]

\r\n

I am trying to set up a SQL 2008 R2 instance using Amazon EC2, but I can't figure out what to put in for the server name?

\r\n

[/quote]

\r\n

When you select the EC2 instance the lower panel will have a value titled "Public DNS". That is the name of the server. Also, you have to ensure that\r\n the security group has the SQL Server port open, which by default is 1433.

", "votes": null, "replies": [] }, { "id": 4680, "author_name": "frank913", "author_url": "", "post_date": "07/21/2011 19:30:54", "content": "

R has prohibitive memory consumption. You should use a tool that is more memory efficient: for example: the "TIMi Suite". See you! Frank

", "votes": null, "replies": [] }, { "id": 4729, "author_name": "", "author_url": "", "post_date": "07/25/2011 15:54:25", "content": "

On a 64bit Windows system, with Revolution R 64 bit, this works for me (I have 8GB of RAM, not sure if this might become a bottleneck for you) It takes a few minutes to read in from CSV, but the read in from xdf is couple of seconds at most... once you have\r\n the dataframe, you can export it to whatever you wanna use this data in - note that all car charactersitics will be the same for the same submodel, so you can effectively break the table in 2, a look up and the main - making the data transfer to other programs\r\n smaller.

\r\n

rxTextToXdf( inFile = "C:\\Users\\InsuranceContest\\trainset.csv",
\r\noutFile = "C:\\Users\\InsuranceContest\\trainset.xdf", rowSelection = NULL,
\r\ntransforms = NULL, append = "none", overwrite = FALSE,
\r\nstringsAsFactors = TRUE, columnDelimiters = NULL,
\r\nrowsPerRead = 500000)

\r\n

data <- rxXdfToDataFrame(file = "C:\\Users\\InsuranceContest\\train_set.xdf", varsToKeep = NULL, varsToDrop = NULL, rowVarName = NULL,\r\n
\r\nrowSelection = NULL, transforms = NULL, blocksPerRead = 1, reportProgress = 2, maxRowsByCols = 9e+09)

", "votes": null, "replies": [] }, { "id": 4730, "author_name": "allstateinsurancecompany", "author_url": "", "post_date": "07/25/2011 16:04:01", "content": "

"note that all car charactersitics will be the same for the same submodel"

\r\n

Maybe this observation will help with modeling (model on a smaller, summarized dataset), and not just data transfer?

", "votes": null, "replies": [] }, { "id": 4837, "author_name": "uriblass", "author_url": "", "post_date": "08/01/2011 10:43:51", "content": "

[quote=Jeremy Howard (Kaggle);4652]

\r\n

A couple of other options: use the big data features of Revolution R, which is available free to Kaggle competitors (see the link on Kaggle's home page) - this software has various libraries and features that allow really big data sets to be handled, even\r\n if they don't fit in memory. Another approach is to filter out some of the data using a simple DB (e.g. sqlite is a good choice) - there's a\r\nlot of rows where the dependent variable is zero; I bet randomly removing a bunch of them won't really impact your model at all, but will make it much easier to analyse.

\r\n

[/quote]

\r\n

I do not understand how Revolution R helps to read big data.

\r\n

page 20 of RevoScaleRGetStart.pdf gives the following as an example

\r\n

testDF <- rxReadXdf(file=dataName, varsToKeep = c("ArrDelay", "DepDelay", "DayOfWeek"), startRow = 100000, numRows = 1000)

\r\n

summary(testDF)

\r\n

I read that I should be able to analyze also data that is too big to fit into memory by using RevoScaleR functions but I do not understand how I can do it

\r\n

I am not interested in regression but only in the same details that I get with summary(testDF) and in the ability to have big vectors like testDF$Row_ID that I can do calculations on them.

\r\n

I wonder if there is a good way to do it without installing 64 bits window or maybe if I install 64 bit windows I can simply write something like memory.limit(100000) that is going to solve all the problems

\r\n

with 32 bits I can write at most memory.limit(4095) without getting an error.

\r\n

I guess that more RAM can also help but the computer has no limit of 4095 mbytes even if it has no more RAM than it and I think that practically there should be no problem with arrays of 100,000 mbytes considering the fact that the computer clearly has enough\r\n memory not in RAM for it.

\r\n

 

\r\n

 

", "votes": null, "replies": [] }, { "id": 4871, "author_name": "alecstephenson", "author_url": "", "post_date": "08/02/2011 04:02:05", "content": "

[quote=Uri Blass;4837]

\r\n

I do not understand how Revolution R helps to read big data.

\r\n

[/quote]

\r\n

The RevoScaleR getting started guide introduces functions such as rxSummary, rxHistogram, rxLinMod, rxLinePlot, rxCrossTabs etc. For example using rxSummary(~., data="traindata") will produce univariate summaries for each variable if you have the file traindata.xdf\r\n in your working directory. You first need to convert the csv file to a xdf file using rxTextToXdf.

\r\n

I must admit I've thrown the towel in with my 32bit windows on an old machine, but Revolution R seems like a great tool for out-of-memory operations. Incidentally, my one submission (team smidsy 0.038) can be seen as a benchmark: it's what you get if you\r\n take the Blind_Model averages as the predictions (which overfits).

", "votes": null, "replies": [] }, { "id": 4916, "author_name": "janko6362", "author_url": "", "post_date": "08/04/2011 02:21:12", "content": "

Hi,

\r\n

Using RevoScaleR from Revolution Analytics, I was able to read the training set into an XDF file, which is their file format. I barely have 1 GB RAM in my trusty Acer Netbook runing Windows XP Home Edition. 

\r\n

So far, so good.

\r\n

Jan

", "votes": null, "replies": [] }, { "id": 5989, "author_name": "imagedoctor", "author_url": "", "post_date": "10/10/2011 14:06:29", "content": "

Thanks for this tip, I've been looking for the Revolution R link on the URL \r\nwww.kaggle.com.

\r\n

It's probably just me, but I can't seem to spot it, can you help ? 

\r\n

 

\r\n

Many thanks.

\r\n


\r\n

\r\n

 

\r\n


\r\n

\r\n

 

\r\n

[quote=Jeremy Howard (Kaggle);4652]

\r\n

A couple of other options: use the big data features of Revolution R, which is available free to Kaggle competitors (see the link on Kaggle's home page) - this software has various libraries and features that allow really big data sets to be handled, even\r\n if they don't fit in memory. Another approach is to filter out some of the data using a simple DB (e.g. sqlite is a good choice) - there's a\r\nlot of rows where the dependent variable is zero; I bet randomly removing a bunch of them won't really impact your model at all, but will make it much easier to analyse.

\r\n

[/quote]

", "votes": null, "replies": [] }, { "id": 5990, "author_name": "chrisraimondi", "author_url": "", "post_date": "10/10/2011 14:36:26", "content": "

I know I saw it before, but don't see the link now, but this one works:

\r\n

http://info.revolutionanalytics.com/Kaggle.html

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "4642": "", "4643": "", "4651": "", "4652": "", "4658": "", "4671": "", "4680": "", "4729": "", "4730": "", "4837": "", "4871": "", "4916": "", "5989": "", "5990": "" }, "source": "meta" }