{ "id": 13847, "title": "Help with R", "url": "/competitions/facebook-recruiting-iv-human-or-bot/discussion/13847", "author_name": "", "post_date": "2015-04-29T06:22:09.060Z", "votes": null, "comment_count": 12, "views": 3011, "content": "

Hi all!

\n

I am trying to learn R for this competition. Can someone tell me how can I merge bids and train files by bidder_id ? I have tried merge command so far. is there anything else for this?

\n\n

Here is the command I use.

\n

bids <- data.frame(read.csv(file="data/bids.csv", head=TRUE, sep=","))
train <- data.frame(read.csv(file="data/train.csv",head=TRUE, sep=","))

\n

m <- merge(bids, train, by.x="bidder_id", by.y="bidder_id")

\n\n

Thanks.

", "messages": [ { "id": "75366", "postDate": "04/29/2015 06:22:09", "content": "

Hi all!

\n

I am trying to learn R for this competition. Can someone tell me how can I merge bids and train files by bidder_id ? I have tried merge command so far. is there anything else for this?

\n\n

Here is the command I use.

\n

bids <- data.frame(read.csv(file="data/bids.csv", head=TRUE, sep=","))
train <- data.frame(read.csv(file="data/train.csv",head=TRUE, sep=","))

\n

m <- merge(bids, train, by.x="bidder_id", by.y="bidder_id")

\n\n

Thanks.

", "rawMarkdown": "", "votes": null }, { "id": "75399", "postDate": "04/29/2015 12:05:32", "content": "

@K0stla: I would expect that to work. Are you getting an error message?

\n

using merge() can be a little memory-heavy at times. If you are short on RAM (<= 2GB, perhaps) then a workaround might start with

\n

bidTrainIndex <- match(bids$bidder_id,train$bidder_id)

\n

## note, don't assign this into bids$trainIndex as that uses a lot more memory

\n

Other alternatives on a limited-memory system are only keeping those rows of bids that correspond to entries in the training data:

\n

bids <- bids[bids$bidder_id %in% train$bidder_id,]

\n

gc() ## return the freed memory to the system

\n

But perhaps memory is not your problem?

", "rawMarkdown": "", "votes": null }, { "id": "75409", "postDate": "04/29/2015 12:57:47", "content": "

it works for me, it as you pointed out consumes a lot of memory sometimes. However, particulary in this case it works, I was wondering if there is something else, more efficient for merging two data frames.

\n

Thanks for your reply.

", "rawMarkdown": "", "votes": null }, { "id": "75416", "postDate": "04/29/2015 14:17:59", "content": "

This is what I use most of the time, hope it helps;

\n

library('dplyr')

\n

left_join(train, bids) # detects similar named field names and joins on them

\n

# OR

\n

left_join(train, bids, by = "bidder_id")  # to select a specific field to join on

\n

However, I do believe that left_join uses MERGE under the hood. could be wrong about that. 

\n\n

If you are very used to SQL, you might want to look at the 'sqldf' package. Then you can write something like "SELECT * FROM df.a INNER JOIN df.b ON id=id "

", "rawMarkdown": "", "votes": null }, { "id": "75417", "postDate": "04/29/2015 14:29:18", "content": "

I think you'll find that the data.table package is easier on the memory and much faster for this sort of thing.

\n

install.packages("data.table")

\n

library(data.table)

\n

bids<-fread("data/bids.csv")

\n

train<-fread("data/train.csv")

\n

m<-merge(bids,train,by="bidder_id")

\n

#cast to data.frame if needed

\n

m<-as.data.frame(m)

\n

Good luck

", "rawMarkdown": "", "votes": null }, { "id": "75466", "postDate": "04/29/2015 18:17:24", "content": "

One more question. Loading of bids csv file takes me some time. Is there a way to speed up this?

", "rawMarkdown": "", "votes": null }, { "id": "75522", "postDate": "04/29/2015 22:39:23", "content": "

You can use the package data.table and the function fread as described above. It speeds up a lot.

", "rawMarkdown": "", "votes": null }, { "id": "75830", "postDate": "05/01/2015 01:44:17", "content": "

The bids table is a transactional table (potentially multiple rows per bidder) I suspect that a good amount of aggregation and date preprocessing is necessary before it can be meaningfully joined to the training and test sets.

", "rawMarkdown": "", "votes": null }, { "id": "76009", "postDate": "05/02/2015 01:18:21", "content": "

I am also trying to learn R  and I am doing the MITx class. I also decided to enter this competition just to see if I can solve some real world problems. However beyond merging my files and trying a failed  CART model I am totally clueless as to what  tools I might consider. Can someone give me a small hint? I was thinking text mining but that will not work because I believe that I cannot stem the IDs

", "rawMarkdown": "", "votes": null }, { "id": "76032", "postDate": "05/02/2015 05:16:46", "content": "

Take a look at this list: http://www.infoworld.com/article/2853707/machine-learning/11-open-source-tools-machine-learning.html . In particular, scikit-learn is a great place to start: http://scikit-learn.org/stable/ . This is a more generic answer than you are looking for, but scanning through the features offered by scikit-learn might provide some inspiration (that's what I'm going to do :).

", "rawMarkdown": "", "votes": null }, { "id": "79884", "postDate": "05/24/2015 03:09:25", "content": "

I just start learning R and I need some help.

\n

I have a data frame

\n

column1 column2
1 89
2 45
3 82

and I want to add a new column with its value is the difference between column2 value from present row and previous row.

\n

column1 column2 column3

\n

1 89 0
2 45 -44
3 82 37

Could anyone give me some clue how to archive it? I try to search from search engine but I couldn't find it.

Thanks.

", "rawMarkdown": "", "votes": null }, { "id": "79885", "postDate": "05/24/2015 03:15:57", "content": "

With basic functions you could do:

column3 = t(c(0,diff(t(column2),1)))

where,
t(): transpose,
diff(): lagged difference.

PS.: Probably there is a direct way to do it - as usually in R :)

", "rawMarkdown": "", "votes": null }, { "id": "79934", "postDate": "05/24/2015 23:01:25", "content": "

you don't need to transpose the data. just use something like this

\n\n

df$column3 <- diff(df$column2)

\n\n

You may have to add or remove a value at the beginning depending on how you want to correlate the results. I rarely get it right the first time.

\n\n

df$column3 <- c(0,diff(df$column2))  #Add a 0 in the first row, shift down

\n

df$column3 <- diff(df$column2)[-1]   #Remove the first row, shift up

\n\n

Good luck,

\n

Jeff

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 75399, "author_name": "jamesdlawrence", "author_url": "", "post_date": "04/29/2015 12:05:32", "content": "

@K0stla: I would expect that to work. Are you getting an error message?

\n

using merge() can be a little memory-heavy at times. If you are short on RAM (<= 2GB, perhaps) then a workaround might start with

\n

bidTrainIndex <- match(bids$bidder_id,train$bidder_id)

\n

## note, don't assign this into bids$trainIndex as that uses a lot more memory

\n

Other alternatives on a limited-memory system are only keeping those rows of bids that correspond to entries in the training data:

\n

bids <- bids[bids$bidder_id %in% train$bidder_id,]

\n

gc() ## return the freed memory to the system

\n

But perhaps memory is not your problem?

", "votes": null, "replies": [] }, { "id": 75409, "author_name": "", "author_url": "", "post_date": "04/29/2015 12:57:47", "content": "

it works for me, it as you pointed out consumes a lot of memory sometimes. However, particulary in this case it works, I was wondering if there is something else, more efficient for merging two data frames.

\n

Thanks for your reply.

", "votes": null, "replies": [] }, { "id": 75416, "author_name": "michelblaauw", "author_url": "", "post_date": "04/29/2015 14:17:59", "content": "

This is what I use most of the time, hope it helps;

\n

library('dplyr')

\n

left_join(train, bids) # detects similar named field names and joins on them

\n

# OR

\n

left_join(train, bids, by = "bidder_id")  # to select a specific field to join on

\n

However, I do believe that left_join uses MERGE under the hood. could be wrong about that. 

\n\n

If you are very used to SQL, you might want to look at the 'sqldf' package. Then you can write something like "SELECT * FROM df.a INNER JOIN df.b ON id=id "

", "votes": null, "replies": [] }, { "id": 75417, "author_name": "speculation", "author_url": "", "post_date": "04/29/2015 14:29:18", "content": "

I think you'll find that the data.table package is easier on the memory and much faster for this sort of thing.

\n

install.packages("data.table")

\n

library(data.table)

\n

bids<-fread("data/bids.csv")

\n

train<-fread("data/train.csv")

\n

m<-merge(bids,train,by="bidder_id")

\n

#cast to data.frame if needed

\n

m<-as.data.frame(m)

\n

Good luck

", "votes": null, "replies": [] }, { "id": 75466, "author_name": "", "author_url": "", "post_date": "04/29/2015 18:17:24", "content": "

One more question. Loading of bids csv file takes me some time. Is there a way to speed up this?

", "votes": null, "replies": [] }, { "id": 75522, "author_name": "asahin", "author_url": "", "post_date": "04/29/2015 22:39:23", "content": "

You can use the package data.table and the function fread as described above. It speeds up a lot.

", "votes": null, "replies": [] }, { "id": 75830, "author_name": "rocelot", "author_url": "", "post_date": "05/01/2015 01:44:17", "content": "

The bids table is a transactional table (potentially multiple rows per bidder) I suspect that a good amount of aggregation and date preprocessing is necessary before it can be meaningfully joined to the training and test sets.

", "votes": null, "replies": [] }, { "id": 76009, "author_name": "awitong", "author_url": "", "post_date": "05/02/2015 01:18:21", "content": "

I am also trying to learn R  and I am doing the MITx class. I also decided to enter this competition just to see if I can solve some real world problems. However beyond merging my files and trying a failed  CART model I am totally clueless as to what  tools I might consider. Can someone give me a small hint? I was thinking text mining but that will not work because I believe that I cannot stem the IDs

", "votes": null, "replies": [] }, { "id": 76032, "author_name": "wgillett", "author_url": "", "post_date": "05/02/2015 05:16:46", "content": "

Take a look at this list: http://www.infoworld.com/article/2853707/machine-learning/11-open-source-tools-machine-learning.html . In particular, scikit-learn is a great place to start: http://scikit-learn.org/stable/ . This is a more generic answer than you are looking for, but scanning through the features offered by scikit-learn might provide some inspiration (that's what I'm going to do :).

", "votes": null, "replies": [] }, { "id": 79884, "author_name": "winsxx", "author_url": "", "post_date": "05/24/2015 03:09:25", "content": "

I just start learning R and I need some help.

\n

I have a data frame

\n

column1 column2
1 89
2 45
3 82

and I want to add a new column with its value is the difference between column2 value from present row and previous row.

\n

column1 column2 column3

\n

1 89 0
2 45 -44
3 82 37

Could anyone give me some clue how to archive it? I try to search from search engine but I couldn't find it.

Thanks.

", "votes": null, "replies": [] }, { "id": 79885, "author_name": "", "author_url": "", "post_date": "05/24/2015 03:15:57", "content": "

With basic functions you could do:

column3 = t(c(0,diff(t(column2),1)))

where,
t(): transpose,
diff(): lagged difference.

PS.: Probably there is a direct way to do it - as usually in R :)

", "votes": null, "replies": [] }, { "id": 79934, "author_name": "jeffhebert", "author_url": "", "post_date": "05/24/2015 23:01:25", "content": "

you don't need to transpose the data. just use something like this

\n\n

df$column3 <- diff(df$column2)

\n\n

You may have to add or remove a value at the beginning depending on how you want to correlate the results. I rarely get it right the first time.

\n\n

df$column3 <- c(0,diff(df$column2))  #Add a 0 in the first row, shift down

\n

df$column3 <- diff(df$column2)[-1]   #Remove the first row, shift up

\n\n

Good luck,

\n

Jeff

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "75366": "", "75399": "", "75409": "", "75416": "", "75417": "", "75466": "", "75522": "", "75830": "", "76009": "", "76032": "", "79884": "", "79885": "", "79934": "" }, "source": "meta" }