{ "id": 11270, "title": "Is the FeatureHasher function available in R?", "url": "/competitions/avazu-ctr-prediction/discussion/11270", "author_name": "", "post_date": "2014-12-16T17:57:14.657Z", "votes": 1, "comment_count": 19, "views": 7631, "content": "

Hi folks,

\n

Is any of you know if there exist something similar (package) to the FeatureHasher function from sci-kit learn in R? I implemented one following the algo in Wikipedia, but I'm not pleased with the result. 

\n

Thanks,

", "messages": [ { "id": "60380", "postDate": "12/16/2014 17:57:14", "content": "

Hi folks,

\n

Is any of you know if there exist something similar (package) to the FeatureHasher function from sci-kit learn in R? I implemented one following the algo in Wikipedia, but I'm not pleased with the result. 

\n

Thanks,

", "rawMarkdown": "", "votes": null }, { "id": "60384", "postDate": "12/16/2014 19:17:43", "content": "

I guess No. The closest to hash that I could get is the hash library that gives us a dictionary data structure. But, feature hashing is a No in R. If there is one, please let us know. Happy to learn.

\n

EDIT: There's a hash generator in the digest package. But I'm afraid this doesn't answer your question though.

", "rawMarkdown": "", "votes": null }, { "id": "60447", "postDate": "12/17/2014 08:57:49", "content": "

you can use digest package.... for example

\n\n

library(digest)

\n

D <- 2^20

\n

#feature hashing
hash <- function(s) {

\n

    hstr <- digest(paste0(s[1], '_', s[2]), algo='xxhash32')
    as.numeric(paste0('0x', hstr)) %% D
}

", "rawMarkdown": "", "votes": null }, { "id": "60510", "postDate": "12/17/2014 19:19:39", "content": "

Hi Xin,

\n

In your formula, what is the variable s? How do you make sure there's a match in the number of rows in the hashed test data set and the ID's number? I actually implemented something more elegant than that, but the rows in my test set ended being way more than the original test data. And the problem was how do I match the test ID variables to the new hashed test data set.

\n

Thank you,

", "rawMarkdown": "", "votes": null }, { "id": "61754", "postDate": "12/23/2014 14:23:18", "content": "

Have you considered encoding without the hash trick in R?  You can one-hot encode all of the features into a sparse matrix automatically using the Matrix package or model.matrix.

", "rawMarkdown": "", "votes": null }, { "id": "61759", "postDate": "12/23/2014 16:54:42", "content": "

http://cran.at.r-project.org/web/packages/FeatureHashing/index.html

", "rawMarkdown": "", "votes": null }, { "id": "61770", "postDate": "12/23/2014 19:19:23", "content": "

Thanks a lot piotrek. That looks like what I was looking for. I'll mess around with it and see how it compares to the python implementation.

", "rawMarkdown": "", "votes": null }, { "id": "61881", "postDate": "12/25/2014 23:18:01", "content": "

for what is feature hashing good for - just compression or is calc also faster?

", "rawMarkdown": "", "votes": null }, { "id": "62664", "postDate": "01/12/2015 18:37:12", "content": "

@Piotrek - Did you figure out how to use FeatureHashing package on test data for submission? I mean after you hashed the training data and you've built the model 

", "rawMarkdown": "", "votes": null }, { "id": "62934", "postDate": "01/20/2015 19:12:03", "content": "

[quote=Filip Floegel;61881]

\n

for what is feature hashing good for - just compression or is calc also faster?

\n

[/quote]

\n

This post is a good introduction to some of the reasons for feature hashing: http://webcache.googleusercontent.com/search?q=cache:PRrGZpsAUJAJ:metaoptimize.com/qa/questions/6943/what-is-the-hashing-trick+&cd=2&hl=en&ct=clnk&gl=us

", "rawMarkdown": "", "votes": null }, { "id": "63049", "postDate": "01/23/2015 09:18:33", "content": "

Dear all,

\n

I am the author of FeatureHashing .

\n

Here is a demo code to train the logistic regression with glmnet and gradient boosted decision tree with gxboost on the ipinyou dataset

\n

Hope that helps.These example will be included in the documentation of the next patch.

", "rawMarkdown": "", "votes": null }, { "id": "63118", "postDate": "01/24/2015 22:18:07", "content": "

[quote=Wush Wu;63049]

\n

Dear all,

\n

I am the author of FeatureHashing .

\n

Here is a demo code to train the logistic regression with glmnet and gradient boosted decision tree with gxboost on the ipinyou dataset

\n

Hope that helps.These example will be included in the documentation of the next patch.

\n

[/quote]

\n\n

There seems to be some formatting issue in the sample code. I'm getting following errors when trying it out. Can you please into it? Let me know if I'm missing something obvious here.

\n\n

Thanks.

\n

> suppressPackageStartupMessages({
+ library(FeatureHashing)
+ library(dplyr)
+ })
>
> f <- ~="" ip="" +="" region="" +="" city="" +="" adexchange="" +="" domain="">
Error: unexpected '=' in "f <- ~="
> URL + AdSlotId + AdSlotWidth + AdSlotHeight +
+ AdSlotVisibility + AdSlotFormat + CreativeID +
+ weekday + hour + tag(usertag, split = ",")
Error: object 'URL' not found
> m.train <- hashed.model.matrix(f,="" imp.train,="" 2^20,="" false)="" %="">%
Error: unexpected '=' in "m.train <- hashed.model.matrix(f,="
> as("dgCMatrix")
Error in .identC(Class, "double") :
argument "Class" is missing, with no default
> m.test <- hashed.model.matrix(f,="" imp.test,="" 2^20,="" false)="" %="">%
Error: unexpected '=' in "m.test <- hashed.model.matrix(f,="
> as("dgCMatrix")
Error in .identC(Class, "double") :
argument "Class" is missing, with no default

", "rawMarkdown": "", "votes": null }, { "id": "63119", "postDate": "01/24/2015 23:48:02", "content": "

A few additional questions. Please pardon my ignorance as I'm a novice.

\n

1) It seems that the feature hasher by default returns a transposed matrix. Shouldn't it be transposed again before training so that the rows represent the samples and the columns the new features.

\n

2) From the sample I'm trying to understand how the features are hashed and mapped.

\n

> m <- hashed.model.matrix(~ ., CO2, 2^6, keep.hashing_mapping = TRUE)
> mapping <- as.list(attr(m, "mapping"))
> mapping <- unlist(as.list(attr(m, "mapping")))
>
> mapping %% 2^6
PlantQn1 PlantQn2 PlantQn3 uptake TypeMississippi Treatmentchilled PlantMn1 PlantMn2 PlantMn3
33 37 6 48 16 18 22 1 20
PlantQc1 PlantQc2 PlantQc3 Treatmentnonchilled PlantMc1 PlantMc2 PlantMc3 conc TypeQuebec
5 26 45 31 54 41 41 30 6

\n

From the above my understanding is that value of Qn1 for Plant will be 33, Qn2 be 37, etc. This implies that in the one hot encoded vector of 64 features, the 33rd and 37th values will be set to 1.

\n

Let's look at the CO2 data.> head(CO2)
Grouped Data: uptake ~ conc | Plant
Plant Type Treatment conc uptake
1 Qn1 Quebec nonchilled 95 16.0
2 Qn1 Quebec nonchilled 175 30.4
3 Qn1 Quebec nonchilled 250 34.8
4 Qn1 Quebec nonchilled 350 37.2
5 Qn1 Quebec nonchilled 500 35.3
6 Qn1 Quebec nonchilled 675 39.2

\n

Comparing the above with the first column of the feature hash, I don't see the 33rd element set as 1 (this is what QN1 above was mapped to).

\n

> dgm = as(m, "dgCMatrix")

\n

> dgm[,1]
[1] 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 95 -1 0 -1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
[49] -16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

\n

And lastly why are there elements like -16 and 95.

\n

It's possible that I'm not just understanding how feature hashing works. So please pardon my ignorance and the questions asked.

\n

Thanks.

", "rawMarkdown": "", "votes": null }, { "id": "63127", "postDate": "01/25/2015 05:23:29", "content": "

Thanks for your asking.

\n\n

I correct the link above. But please do not use them directly. Replace the formula and the variable name of the data.frame to adapt to your dataset.

\n\n

Your question indicates some incomplete explanation of the document.

\n

[quote=Hitesh Sharma;63119]

\n

A few additional questions. Please pardon my ignorance as I'm a novice.

\n

1) It seems that the feature hasher by default returns a transposed matrix. Shouldn't it be transposed again before training so that the rows represent the samples and the columns the new features.

\n\n

There is a parameter `transpose` to decide whether to transpose the returned matrix or not. The default value is `TRUE` and I am considering whether to change it in the next patch or not.

\n\n

2) From the sample I'm trying to understand how the features are hashed and mapped.

\n

> m <- hashed.model.matrix(~ ., CO2, 2^6, keep.hashing_mapping = TRUE)
> mapping <- as.list(attr(m, "mapping"))
> mapping <- unlist(as.list(attr(m, "mapping")))
>
> mapping %% 2^6
PlantQn1 PlantQn2 PlantQn3 uptake TypeMississippi Treatmentchilled PlantMn1 PlantMn2 PlantMn3
33 37 6 48 16 18 22 1 20
PlantQc1 PlantQc2 PlantQc3 Treatmentnonchilled PlantMc1 PlantMc2 PlantMc3 conc TypeQuebec
5 26 45 31 54 41 41 30 6

\n

From the above my understanding is that value of Qn1 for Plant will be 33, Qn2 be 37, etc. This implies that in the one hot encoded vector of 64 features, the 33rd and 37th values will be set to 1.

\n

Let's look at the CO2 data.> head(CO2)
Grouped Data: uptake ~ conc | Plant
Plant Type Treatment conc uptake
1 Qn1 Quebec nonchilled 95 16.0
2 Qn1 Quebec nonchilled 175 30.4
3 Qn1 Quebec nonchilled 250 34.8
4 Qn1 Quebec nonchilled 350 37.2
5 Qn1 Quebec nonchilled 500 35.3
6 Qn1 Quebec nonchilled 675 39.2

\n

Comparing the above with the first column of the feature hash, I don't see the 33rd element set as 1 (this is what QN1 above was mapped to).

\n

> dgm = as(m, "dgCMatrix")

\n

> dgm[,1]
[1] 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 95 -1 0 -1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
[49] -16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

\n

And lastly why are there elements like -16 and 95.

\n

It's possible that I'm not just understanding how feature hashing works. So please pardon my ignorance and the questions asked.

\n

Thanks.

\n

[/quote]

\n\n

At first, I need to address that the implementation uses a binary hashed function to correct the sign of the result. Therefore, there are 1 and -1 in the returned matrix. You can read the second formula in here: feature vectorization using the hashing trick

\n

Second, the range of `%% 2^6` is 0 ~ 2^6 - 1, so we need to shift the index by 1 Because the starting index in R is 1

\n

The first row of CO2 is:

\n

Plant Type Treatment conc uptake
1 Qn1 Quebec nonchilled 95 16

\n

The mapping of them are:

\n

> mapping["Treatmentnonchilled"] %% 2^6 + 1
Treatmentnonchilled
32
> mapping["TypeQuebec"] %% 2^6 + 1
TypeQuebec
7
> mapping["PlantQn1"] %% 2^6 + 1
PlantQn1
34
> mapping["conc"] %% 2^6 + 1
conc
31
> mapping["uptake"] %% 2^6 + 1
uptake
49

\n\n

Therefore, 

\n

> dgm[c(7,31,32,34,49),1]
<NA> <NA> <NA> <NA> <NA>
1 95 -1 -1 -16

\n

Note that the variable conc and uptake are numeric, so the value is 95 and -16.

\n

Hope that helps.

", "rawMarkdown": "", "votes": null }, { "id": "63169", "postDate": "01/26/2015 02:53:33", "content": "

Hi Wush,

\n

Thanks for the explanation and response. I tried to use the FeatureHasher with glmnet on the CO2 data but getting this error:

\n

>
> co = CO2
>
> co$Treatment = NULL
> co = hashed.model.matrix(~ ., co, 2^6, keep.hashing_mapping = TRUE)
> co = as(co, "dgCMatrix")
> co = t(co)
>
>
> glmnetmod <- cv.glmnet(y = as.factor(CO2$Treatment), x = co, family="binomial", nfolds=3, type.measure="auc")
> result = predict(glmnetmod, co, s = "lambda.min")
Error in `dimnames<-`(`*tmp*`, value = list("(Intercept)", "1")) :
  invalid dimnames given for “dgCMatrix” object
>

\n

I'm not sure why it's happening but have a hunch that it might be something to do with the transpose that I'm taking. I noted that in your sample glmnet snippet you are not taking a transpose. Is the input data in that sample already transposed?

\n

Appreciate the pointers in getting past this error. I have tried to search a lot on the web but had no luck.

\n

Thanks.

", "rawMarkdown": "", "votes": null }, { "id": "63173", "postDate": "01/26/2015 04:42:50", "content": "

[quote=Hitesh Sharma;63169]

\n

Hi Wush,

\n

Thanks for the explanation and response. I tried to use the FeatureHasher with glmnet on the CO2 data but getting this error:

\n

>
> co = CO2
>
> co$Treatment = NULL
> co = hashed.model.matrix(~ ., co, 2^6, keep.hashing_mapping = TRUE)
> co = as(co, "dgCMatrix")
> co = t(co)
>
>
> glmnetmod <- cv.glmnet(y = as.factor(CO2$Treatment), x = co, family="binomial", nfolds=3, type.measure="auc")
> result = predict(glmnetmod, co, s = "lambda.min")
Error in `dimnames<-`(`*tmp*`, value = list("(Intercept)", "1")) :
  invalid dimnames given for “dgCMatrix” object
>

\n

I'm not sure why it's happening but have a hunch that it might be something to do with the transpose that I'm taking. I noted that in your sample glmnet snippet you are not taking a transpose. Is the input data in that sample already transposed?

\n

Appreciate the pointers in getting past this error. I have tried to search a lot on the web but had no luck.

\n

Thanks.

\n

[/quote]

\n

As suspected something was going wrong when a transpose was being done. I set transpose = FALSE in FeatureHasher and everything worked. Thanks for creating this package!

", "rawMarkdown": "", "votes": null }, { "id": "63216", "postDate": "01/26/2015 22:05:06", "content": "

[quote=jwjohnson314;62934]

\n

[quote=Filip Floegel;61881]

\n

for what is feature hashing good for - just compression or is calc also faster?

\n

[/quote]

\n

This post is a good introduction to some of the reasons for feature hashing: http://webcache.googleusercontent.com/search?q=cache:PRrGZpsAUJAJ:metaoptimize.com/qa/questions/6943/what-is-the-hashing-trick+&cd=2&hl=en&ct=clnk&gl=us

\n

[/quote]

\n\n

oh :-) thank you very much for the web link

\n\n

multitask + interaction are very good reasonsa apart from online learning + feature space

", "rawMarkdown": "", "votes": null }, { "id": "63385", "postDate": "01/31/2015 11:03:20", "content": "

[quote=Hitesh Sharma;63118]

\n

[quote=Wush Wu;63049]

\n

Dear all,

\n

I am the author of FeatureHashing .

\n

Here is a demo code to train the logistic regression with glmnet and gradient boosted decision tree with gxboost on the ipinyou dataset

\n

Hope that helps.These example will be included in the documentation of the next patch.

\n

[/quote]

\n

There seems to be some formatting issue in the sample code. I'm getting following errors when trying it out. Can you please into it? Let me know if I'm missing something obvious here.

\n

Thanks.

\n

> suppressPackageStartupMessages({
+ library(FeatureHashing)
+ library(dplyr)
+ })
>
> f <- ~="" ip="" +="" region="" +="" city="" +="" adexchange="" +="" domain="">
Error: unexpected '=' in "f <- ~="
> URL + AdSlotId + AdSlotWidth + AdSlotHeight +
+ AdSlotVisibility + AdSlotFormat + CreativeID +
+ weekday + hour + tag(usertag, split = ",")
Error: object 'URL' not found
> m.train <- hashed.model.matrix(f,="" imp.train,="" 2^20,="" false)="" %="">%
Error: unexpected '=' in "m.train <- hashed.model.matrix(f,="
> as("dgCMatrix")
Error in .identC(Class, "double") :
argument "Class" is missing, with no default
> m.test <- hashed.model.matrix(f,="" imp.test,="" 2^20,="" false)="" %="">%
Error: unexpected '=' in "m.test <- hashed.model.matrix(f,="
> as("dgCMatrix")
Error in .identC(Class, "double") :
argument "Class" is missing, with no default

\n

[/quote]

", "rawMarkdown": "", "votes": null }, { "id": "63387", "postDate": "01/31/2015 11:04:28", "content": "

in my case it doesnt work out too with those formating problems

", "rawMarkdown": "", "votes": null }, { "id": "63543", "postDate": "02/04/2015 03:01:57", "content": "

You could use sparse.model.matrix() in R. I'm not sure if it's worse/beter than the hash trick, but it works. Here is an example script.

\n

rm(list =ls(all=T)); gc()

\n

library(FeatureHashing)
library(dplyr)
library(data.table)
library(Matrix)
library(glmnet)
library(xgboost)

\n

test <- fread("test.txt", colClasses='character', header=T)
id <- test$id
test <- sparse.model.matrix(object= ~ ., data=test)
test$id <- NULL

\n\n

LogLoss<-function(actual, predicted){
predicted<-(pmax(predicted, 0.00001))
predicted<-(pmin(predicted, 0.99999))
result<- -1/length(actual)*(sum((actual*log(predicted)+(1-actual)*log(1-predicted))))
return(result)

\n

}

\n

set.seed(123)

\n\n

train = fread("train.txt", nrows=1e6)
y = train[['click']]gc()x_all <- sparse.model.matrix(object=click ~ ., data=train[, -1, with=F])
inx <- sample(1e6, 1e6 * 0.7)
x_train <- x_all[inx, ]
x_test <- x_all[-inx, ]

\n

rm(train); gc()

\n\n

# Parameters

\n

max.depth = 10
gamma = 1.2
eta = 0.8
nround = 10
min_child_weight = 3

\n\n

bst <- xgboost(data = x_train, label = y[inx], max.depth=max.depth, gamma = gamma,
eta = eta, nround = nround, min_child_weight = min_child_weight,
objective = "binary:logistic", booster = 'gbtree')
pred <- predict(bst, x_test)
print(LogLoss(y[-inx], pred))

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 60384, "author_name": "phanisrikanth", "author_url": "", "post_date": "12/16/2014 19:17:43", "content": "

I guess No. The closest to hash that I could get is the hash library that gives us a dictionary data structure. But, feature hashing is a No in R. If there is one, please let us know. Happy to learn.

\n

EDIT: There's a hash generator in the digest package. But I'm afraid this doesn't answer your question though.

", "votes": null, "replies": [] }, { "id": 60447, "author_name": "xinliu", "author_url": "", "post_date": "12/17/2014 08:57:49", "content": "

you can use digest package.... for example

\n\n

library(digest)

\n

D <- 2^20

\n

#feature hashing
hash <- function(s) {

\n

    hstr <- digest(paste0(s[1], '_', s[2]), algo='xxhash32')
    as.numeric(paste0('0x', hstr)) %% D
}

", "votes": null, "replies": [] }, { "id": 60510, "author_name": "maxsop", "author_url": "", "post_date": "12/17/2014 19:19:39", "content": "

Hi Xin,

\n

In your formula, what is the variable s? How do you make sure there's a match in the number of rows in the hashed test data set and the ID's number? I actually implemented something more elegant than that, but the rows in my test set ended being way more than the original test data. And the problem was how do I match the test ID variables to the new hashed test data set.

\n

Thank you,

", "votes": null, "replies": [] }, { "id": 61754, "author_name": "cvarrichio", "author_url": "", "post_date": "12/23/2014 14:23:18", "content": "

Have you considered encoding without the hash trick in R?  You can one-hot encode all of the features into a sparse matrix automatically using the Matrix package or model.matrix.

", "votes": null, "replies": [] }, { "id": 61759, "author_name": "piotrw", "author_url": "", "post_date": "12/23/2014 16:54:42", "content": "

http://cran.at.r-project.org/web/packages/FeatureHashing/index.html

", "votes": null, "replies": [] }, { "id": 61770, "author_name": "maxsop", "author_url": "", "post_date": "12/23/2014 19:19:23", "content": "

Thanks a lot piotrek. That looks like what I was looking for. I'll mess around with it and see how it compares to the python implementation.

", "votes": null, "replies": [] }, { "id": 61881, "author_name": "filipus", "author_url": "", "post_date": "12/25/2014 23:18:01", "content": "

for what is feature hashing good for - just compression or is calc also faster?

", "votes": null, "replies": [] }, { "id": 62664, "author_name": "maxsop", "author_url": "", "post_date": "01/12/2015 18:37:12", "content": "

@Piotrek - Did you figure out how to use FeatureHashing package on test data for submission? I mean after you hashed the training data and you've built the model 

", "votes": null, "replies": [] }, { "id": 62934, "author_name": "jwjohnson314", "author_url": "", "post_date": "01/20/2015 19:12:03", "content": "

[quote=Filip Floegel;61881]

\n

for what is feature hashing good for - just compression or is calc also faster?

\n

[/quote]

\n

This post is a good introduction to some of the reasons for feature hashing: http://webcache.googleusercontent.com/search?q=cache:PRrGZpsAUJAJ:metaoptimize.com/qa/questions/6943/what-is-the-hashing-trick+&cd=2&hl=en&ct=clnk&gl=us

", "votes": null, "replies": [] }, { "id": 63049, "author_name": "wush978", "author_url": "", "post_date": "01/23/2015 09:18:33", "content": "

Dear all,

\n

I am the author of FeatureHashing .

\n

Here is a demo code to train the logistic regression with glmnet and gradient boosted decision tree with gxboost on the ipinyou dataset

\n

Hope that helps.These example will be included in the documentation of the next patch.

", "votes": null, "replies": [] }, { "id": 63118, "author_name": "hitesh", "author_url": "", "post_date": "01/24/2015 22:18:07", "content": "

[quote=Wush Wu;63049]

\n

Dear all,

\n

I am the author of FeatureHashing .

\n

Here is a demo code to train the logistic regression with glmnet and gradient boosted decision tree with gxboost on the ipinyou dataset

\n

Hope that helps.These example will be included in the documentation of the next patch.

\n

[/quote]

\n\n

There seems to be some formatting issue in the sample code. I'm getting following errors when trying it out. Can you please into it? Let me know if I'm missing something obvious here.

\n\n

Thanks.

\n

> suppressPackageStartupMessages({
+ library(FeatureHashing)
+ library(dplyr)
+ })
>
> f <- ~="" ip="" +="" region="" +="" city="" +="" adexchange="" +="" domain="">
Error: unexpected '=' in "f <- ~="
> URL + AdSlotId + AdSlotWidth + AdSlotHeight +
+ AdSlotVisibility + AdSlotFormat + CreativeID +
+ weekday + hour + tag(usertag, split = ",")
Error: object 'URL' not found
> m.train <- hashed.model.matrix(f,="" imp.train,="" 2^20,="" false)="" %="">%
Error: unexpected '=' in "m.train <- hashed.model.matrix(f,="
> as("dgCMatrix")
Error in .identC(Class, "double") :
argument "Class" is missing, with no default
> m.test <- hashed.model.matrix(f,="" imp.test,="" 2^20,="" false)="" %="">%
Error: unexpected '=' in "m.test <- hashed.model.matrix(f,="
> as("dgCMatrix")
Error in .identC(Class, "double") :
argument "Class" is missing, with no default

", "votes": null, "replies": [] }, { "id": 63119, "author_name": "hitesh", "author_url": "", "post_date": "01/24/2015 23:48:02", "content": "

A few additional questions. Please pardon my ignorance as I'm a novice.

\n

1) It seems that the feature hasher by default returns a transposed matrix. Shouldn't it be transposed again before training so that the rows represent the samples and the columns the new features.

\n

2) From the sample I'm trying to understand how the features are hashed and mapped.

\n

> m <- hashed.model.matrix(~ ., CO2, 2^6, keep.hashing_mapping = TRUE)
> mapping <- as.list(attr(m, "mapping"))
> mapping <- unlist(as.list(attr(m, "mapping")))
>
> mapping %% 2^6
PlantQn1 PlantQn2 PlantQn3 uptake TypeMississippi Treatmentchilled PlantMn1 PlantMn2 PlantMn3
33 37 6 48 16 18 22 1 20
PlantQc1 PlantQc2 PlantQc3 Treatmentnonchilled PlantMc1 PlantMc2 PlantMc3 conc TypeQuebec
5 26 45 31 54 41 41 30 6

\n

From the above my understanding is that value of Qn1 for Plant will be 33, Qn2 be 37, etc. This implies that in the one hot encoded vector of 64 features, the 33rd and 37th values will be set to 1.

\n

Let's look at the CO2 data.> head(CO2)
Grouped Data: uptake ~ conc | Plant
Plant Type Treatment conc uptake
1 Qn1 Quebec nonchilled 95 16.0
2 Qn1 Quebec nonchilled 175 30.4
3 Qn1 Quebec nonchilled 250 34.8
4 Qn1 Quebec nonchilled 350 37.2
5 Qn1 Quebec nonchilled 500 35.3
6 Qn1 Quebec nonchilled 675 39.2

\n

Comparing the above with the first column of the feature hash, I don't see the 33rd element set as 1 (this is what QN1 above was mapped to).

\n

> dgm = as(m, "dgCMatrix")

\n

> dgm[,1]
[1] 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 95 -1 0 -1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
[49] -16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

\n

And lastly why are there elements like -16 and 95.

\n

It's possible that I'm not just understanding how feature hashing works. So please pardon my ignorance and the questions asked.

\n

Thanks.

", "votes": null, "replies": [] }, { "id": 63127, "author_name": "wush978", "author_url": "", "post_date": "01/25/2015 05:23:29", "content": "

Thanks for your asking.

\n\n

I correct the link above. But please do not use them directly. Replace the formula and the variable name of the data.frame to adapt to your dataset.

\n\n

Your question indicates some incomplete explanation of the document.

\n

[quote=Hitesh Sharma;63119]

\n

A few additional questions. Please pardon my ignorance as I'm a novice.

\n

1) It seems that the feature hasher by default returns a transposed matrix. Shouldn't it be transposed again before training so that the rows represent the samples and the columns the new features.

\n\n

There is a parameter `transpose` to decide whether to transpose the returned matrix or not. The default value is `TRUE` and I am considering whether to change it in the next patch or not.

\n\n

2) From the sample I'm trying to understand how the features are hashed and mapped.

\n

> m <- hashed.model.matrix(~ ., CO2, 2^6, keep.hashing_mapping = TRUE)
> mapping <- as.list(attr(m, "mapping"))
> mapping <- unlist(as.list(attr(m, "mapping")))
>
> mapping %% 2^6
PlantQn1 PlantQn2 PlantQn3 uptake TypeMississippi Treatmentchilled PlantMn1 PlantMn2 PlantMn3
33 37 6 48 16 18 22 1 20
PlantQc1 PlantQc2 PlantQc3 Treatmentnonchilled PlantMc1 PlantMc2 PlantMc3 conc TypeQuebec
5 26 45 31 54 41 41 30 6

\n

From the above my understanding is that value of Qn1 for Plant will be 33, Qn2 be 37, etc. This implies that in the one hot encoded vector of 64 features, the 33rd and 37th values will be set to 1.

\n

Let's look at the CO2 data.> head(CO2)
Grouped Data: uptake ~ conc | Plant
Plant Type Treatment conc uptake
1 Qn1 Quebec nonchilled 95 16.0
2 Qn1 Quebec nonchilled 175 30.4
3 Qn1 Quebec nonchilled 250 34.8
4 Qn1 Quebec nonchilled 350 37.2
5 Qn1 Quebec nonchilled 500 35.3
6 Qn1 Quebec nonchilled 675 39.2

\n

Comparing the above with the first column of the feature hash, I don't see the 33rd element set as 1 (this is what QN1 above was mapped to).

\n

> dgm = as(m, "dgCMatrix")

\n

> dgm[,1]
[1] 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 95 -1 0 -1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
[49] -16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

\n

And lastly why are there elements like -16 and 95.

\n

It's possible that I'm not just understanding how feature hashing works. So please pardon my ignorance and the questions asked.

\n

Thanks.

\n

[/quote]

\n\n

At first, I need to address that the implementation uses a binary hashed function to correct the sign of the result. Therefore, there are 1 and -1 in the returned matrix. You can read the second formula in here: feature vectorization using the hashing trick

\n

Second, the range of `%% 2^6` is 0 ~ 2^6 - 1, so we need to shift the index by 1 Because the starting index in R is 1

\n

The first row of CO2 is:

\n

Plant Type Treatment conc uptake
1 Qn1 Quebec nonchilled 95 16

\n

The mapping of them are:

\n

> mapping["Treatmentnonchilled"] %% 2^6 + 1
Treatmentnonchilled
32
> mapping["TypeQuebec"] %% 2^6 + 1
TypeQuebec
7
> mapping["PlantQn1"] %% 2^6 + 1
PlantQn1
34
> mapping["conc"] %% 2^6 + 1
conc
31
> mapping["uptake"] %% 2^6 + 1
uptake
49

\n\n

Therefore, 

\n

> dgm[c(7,31,32,34,49),1]
<NA> <NA> <NA> <NA> <NA>
1 95 -1 -1 -16

\n

Note that the variable conc and uptake are numeric, so the value is 95 and -16.

\n

Hope that helps.

", "votes": null, "replies": [] }, { "id": 63169, "author_name": "hitesh", "author_url": "", "post_date": "01/26/2015 02:53:33", "content": "

Hi Wush,

\n

Thanks for the explanation and response. I tried to use the FeatureHasher with glmnet on the CO2 data but getting this error:

\n

>
> co = CO2
>
> co$Treatment = NULL
> co = hashed.model.matrix(~ ., co, 2^6, keep.hashing_mapping = TRUE)
> co = as(co, "dgCMatrix")
> co = t(co)
>
>
> glmnetmod <- cv.glmnet(y = as.factor(CO2$Treatment), x = co, family="binomial", nfolds=3, type.measure="auc")
> result = predict(glmnetmod, co, s = "lambda.min")
Error in `dimnames<-`(`*tmp*`, value = list("(Intercept)", "1")) :
  invalid dimnames given for “dgCMatrix” object
>

\n

I'm not sure why it's happening but have a hunch that it might be something to do with the transpose that I'm taking. I noted that in your sample glmnet snippet you are not taking a transpose. Is the input data in that sample already transposed?

\n

Appreciate the pointers in getting past this error. I have tried to search a lot on the web but had no luck.

\n

Thanks.

", "votes": null, "replies": [] }, { "id": 63173, "author_name": "hitesh", "author_url": "", "post_date": "01/26/2015 04:42:50", "content": "

[quote=Hitesh Sharma;63169]

\n

Hi Wush,

\n

Thanks for the explanation and response. I tried to use the FeatureHasher with glmnet on the CO2 data but getting this error:

\n

>
> co = CO2
>
> co$Treatment = NULL
> co = hashed.model.matrix(~ ., co, 2^6, keep.hashing_mapping = TRUE)
> co = as(co, "dgCMatrix")
> co = t(co)
>
>
> glmnetmod <- cv.glmnet(y = as.factor(CO2$Treatment), x = co, family="binomial", nfolds=3, type.measure="auc")
> result = predict(glmnetmod, co, s = "lambda.min")
Error in `dimnames<-`(`*tmp*`, value = list("(Intercept)", "1")) :
  invalid dimnames given for “dgCMatrix” object
>

\n

I'm not sure why it's happening but have a hunch that it might be something to do with the transpose that I'm taking. I noted that in your sample glmnet snippet you are not taking a transpose. Is the input data in that sample already transposed?

\n

Appreciate the pointers in getting past this error. I have tried to search a lot on the web but had no luck.

\n

Thanks.

\n

[/quote]

\n

As suspected something was going wrong when a transpose was being done. I set transpose = FALSE in FeatureHasher and everything worked. Thanks for creating this package!

", "votes": null, "replies": [] }, { "id": 63216, "author_name": "filipus", "author_url": "", "post_date": "01/26/2015 22:05:06", "content": "

[quote=jwjohnson314;62934]

\n

[quote=Filip Floegel;61881]

\n

for what is feature hashing good for - just compression or is calc also faster?

\n

[/quote]

\n

This post is a good introduction to some of the reasons for feature hashing: http://webcache.googleusercontent.com/search?q=cache:PRrGZpsAUJAJ:metaoptimize.com/qa/questions/6943/what-is-the-hashing-trick+&cd=2&hl=en&ct=clnk&gl=us

\n

[/quote]

\n\n

oh :-) thank you very much for the web link

\n\n

multitask + interaction are very good reasonsa apart from online learning + feature space

", "votes": null, "replies": [] }, { "id": 63385, "author_name": "filipus", "author_url": "", "post_date": "01/31/2015 11:03:20", "content": "

[quote=Hitesh Sharma;63118]

\n

[quote=Wush Wu;63049]

\n

Dear all,

\n

I am the author of FeatureHashing .

\n

Here is a demo code to train the logistic regression with glmnet and gradient boosted decision tree with gxboost on the ipinyou dataset

\n

Hope that helps.These example will be included in the documentation of the next patch.

\n

[/quote]

\n

There seems to be some formatting issue in the sample code. I'm getting following errors when trying it out. Can you please into it? Let me know if I'm missing something obvious here.

\n

Thanks.

\n

> suppressPackageStartupMessages({
+ library(FeatureHashing)
+ library(dplyr)
+ })
>
> f <- ~="" ip="" +="" region="" +="" city="" +="" adexchange="" +="" domain="">
Error: unexpected '=' in "f <- ~="
> URL + AdSlotId + AdSlotWidth + AdSlotHeight +
+ AdSlotVisibility + AdSlotFormat + CreativeID +
+ weekday + hour + tag(usertag, split = ",")
Error: object 'URL' not found
> m.train <- hashed.model.matrix(f,="" imp.train,="" 2^20,="" false)="" %="">%
Error: unexpected '=' in "m.train <- hashed.model.matrix(f,="
> as("dgCMatrix")
Error in .identC(Class, "double") :
argument "Class" is missing, with no default
> m.test <- hashed.model.matrix(f,="" imp.test,="" 2^20,="" false)="" %="">%
Error: unexpected '=' in "m.test <- hashed.model.matrix(f,="
> as("dgCMatrix")
Error in .identC(Class, "double") :
argument "Class" is missing, with no default

\n

[/quote]

", "votes": null, "replies": [] }, { "id": 63387, "author_name": "filipus", "author_url": "", "post_date": "01/31/2015 11:04:28", "content": "

in my case it doesnt work out too with those formating problems

", "votes": null, "replies": [] }, { "id": 63543, "author_name": "marbel", "author_url": "", "post_date": "02/04/2015 03:01:57", "content": "

You could use sparse.model.matrix() in R. I'm not sure if it's worse/beter than the hash trick, but it works. Here is an example script.

\n

rm(list =ls(all=T)); gc()

\n

library(FeatureHashing)
library(dplyr)
library(data.table)
library(Matrix)
library(glmnet)
library(xgboost)

\n

test <- fread("test.txt", colClasses='character', header=T)
id <- test$id
test <- sparse.model.matrix(object= ~ ., data=test)
test$id <- NULL

\n\n

LogLoss<-function(actual, predicted){
predicted<-(pmax(predicted, 0.00001))
predicted<-(pmin(predicted, 0.99999))
result<- -1/length(actual)*(sum((actual*log(predicted)+(1-actual)*log(1-predicted))))
return(result)

\n

}

\n

set.seed(123)

\n\n

train = fread("train.txt", nrows=1e6)
y = train[['click']]gc()x_all <- sparse.model.matrix(object=click ~ ., data=train[, -1, with=F])
inx <- sample(1e6, 1e6 * 0.7)
x_train <- x_all[inx, ]
x_test <- x_all[-inx, ]

\n

rm(train); gc()

\n\n

# Parameters

\n

max.depth = 10
gamma = 1.2
eta = 0.8
nround = 10
min_child_weight = 3

\n\n

bst <- xgboost(data = x_train, label = y[inx], max.depth=max.depth, gamma = gamma,
eta = eta, nround = nround, min_child_weight = min_child_weight,
objective = "binary:logistic", booster = 'gbtree')
pred <- predict(bst, x_test)
print(LogLoss(y[-inx], pred))

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "60380": "", "60384": "", "60447": "", "60510": "", "61754": "", "61759": "", "61770": "", "61881": "", "62664": "", "62934": "", "63049": "", "63118": "", "63119": "", "63127": "", "63169": "", "63173": "", "63216": "", "63385": "", "63387": "", "63543": "" }, "source": "meta" }