{ "id": 9394, "title": "My R code is too slow!", "url": "/competitions/acquire-valued-shoppers-challenge/discussion/9394", "author_name": "", "post_date": "2014-06-05T06:56:33.737Z", "votes": null, "comment_count": 2, "views": 1521, "content": "

Hi, everyone,

\n

I would like to make my data set like Triskelion's in another thread, but  it is not successful because my code is too slow in R.

\n

Here is my code sample:

\n

 

\n

 

\n

for(i in 1:nrow(dtrain))
{
    dsub<-reduced2[reduced2$id==dtrain$id[i],]
    dtrain$has_bought_company[i]<-nrow(dsub[dsub$company==dtrain$company[i],])

\n

    ........
}

\n

 

\n

 

\n

I made dtrain by combining "offers" with "trainHistory".

\n

Then I count up how many times the shopper has bought the company.

\n

Is there any faster code or is it better to use another tools such as python / SQL?

", "messages": [ { "id": "48689", "postDate": "06/05/2014 06:56:33", "content": "

Hi, everyone,

\n

I would like to make my data set like Triskelion's in another thread, but  it is not successful because my code is too slow in R.

\n

Here is my code sample:

\n

 

\n

 

\n

for(i in 1:nrow(dtrain))
{
    dsub<-reduced2[reduced2$id==dtrain$id[i],]
    dtrain$has_bought_company[i]<-nrow(dsub[dsub$company==dtrain$company[i],])

\n

    ........
}

\n

 

\n

 

\n

I made dtrain by combining "offers" with "trainHistory".

\n

Then I count up how many times the shopper has bought the company.

\n

Is there any faster code or is it better to use another tools such as python / SQL?

", "rawMarkdown": "", "votes": null }, { "id": "48789", "postDate": "06/06/2014 18:16:19", "content": "

Subsetting reduced2 iteratively is very expensive:

\n

system.time(length(reduced[trans$id==1,1]))
user system elapsed
9.022 4.089 17.168 <br/><br/>

\n

So say ~10sec * ~311000 users = 3110000 secs ~= 35 days +/-

\n

Consider package data.table with which the following takes a total of ~12 secs. <br/><br/>

\n

> system.time(t1 <- trans[,sum(purchaseamount),by=list(id,category)])
user system elapsed
12.004 14.401 38.805
> head(t1)
   id  category  V1
1: 86246  707  535.96
2: 86246 6319  165.62
3: 86246 9753 9286.86
4: 86246 2509  146.33
5: 86246 5555  134.73
6: 86246 9909  699.81<br/><br/>

\n

Data tables are a subclass of data.frames. They can be confusing at first, but the performance is worth the cognitive dissonance. Also this:

\n

trans=fread("../data/reduced2.csv",sep=",",header=T)

\n

is up to five times faster than

\n

read.csv()

", "rawMarkdown": "", "votes": null }, { "id": "48828", "postDate": "06/07/2014 13:53:41", "content": "

Thank you for your help , mariobluedog.

\n

It is really good information for me.

\n

I will try this way!

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 48789, "author_name": "numinary", "author_url": "", "post_date": "06/06/2014 18:16:19", "content": "

Subsetting reduced2 iteratively is very expensive:

\n

system.time(length(reduced[trans$id==1,1]))
user system elapsed
9.022 4.089 17.168 <br/><br/>

\n

So say ~10sec * ~311000 users = 3110000 secs ~= 35 days +/-

\n

Consider package data.table with which the following takes a total of ~12 secs. <br/><br/>

\n

> system.time(t1 <- trans[,sum(purchaseamount),by=list(id,category)])
user system elapsed
12.004 14.401 38.805
> head(t1)
   id  category  V1
1: 86246  707  535.96
2: 86246 6319  165.62
3: 86246 9753 9286.86
4: 86246 2509  146.33
5: 86246 5555  134.73
6: 86246 9909  699.81<br/><br/>

\n

Data tables are a subclass of data.frames. They can be confusing at first, but the performance is worth the cognitive dissonance. Also this:

\n

trans=fread("../data/reduced2.csv",sep=",",header=T)

\n

is up to five times faster than

\n

read.csv()

", "votes": null, "replies": [] }, { "id": 48828, "author_name": "chanfuku18", "author_url": "", "post_date": "06/07/2014 13:53:41", "content": "

Thank you for your help , mariobluedog.

\n

It is really good information for me.

\n

I will try this way!

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "48689": "", "48789": "", "48828": "" }, "source": "meta" }