{ "id": 7917, "title": "Performance of Logistic Regression on Features Described by Triskelion ", "url": "/competitions/acquire-valued-shoppers-challenge/discussion/7917", "author_name": "", "post_date": "2014-04-29T04:25:59.650Z", "votes": 2, "comment_count": 45, "views": 20400, "content": "

Hi All,

\n

I trained a Logistic Regression model, using the features described in following thread.

\n

Feature engineering and beat the benchmark (~0.59347)

\n

Unfortunately, its performance is much less than the performance of Vowpal Wabbit model. So I just want to know, whether a similar behavior is observed by others.

\n

Regards,
Upul

", "messages": [ { "id": "43265", "postDate": "04/29/2014 04:25:59", "content": "

Hi All,

\n

I trained a Logistic Regression model, using the features described in following thread.

\n

Feature engineering and beat the benchmark (~0.59347)

\n

Unfortunately, its performance is much less than the performance of Vowpal Wabbit model. So I just want to know, whether a similar behavior is observed by others.

\n

Regards,
Upul

", "rawMarkdown": "", "votes": null }, { "id": "43307", "postDate": "04/29/2014 15:45:55", "content": "

Yes, VW does some magic.  It uses stochastic gradient descent, which is a little less sensitive to outliers.  The code you linked to also uses quantile regression, rather than logistic regression, which seems to give better results on this dataset.  I haven't figured out why they use a quantile_tau of 0.60 rather than the median, but that magic number also seems to give better results on this dataset.

\n

Finally, it's possible that the featureset you built yourself is different from the one built in the linked post.  Check your subsets and aggregations carefully!

", "rawMarkdown": "", "votes": null }, { "id": "44415", "postDate": "05/12/2014 16:43:49", "content": "

If you want to use another tool, bear in mind that vw has a couple of features on by default that may or may not be implemented (e.g., adaptive, normalized, invariant update rules / using holdout data to determine how many passes you need / ...)

\n

Here is a very brief description of what it does: https://github.com/JohnLangford/vowpal_wabbit/wiki/Command-line-arguments

", "rawMarkdown": "", "votes": null }, { "id": "47261", "postDate": "05/29/2014 14:35:21", "content": "

What percentage of repeaters classifies the model right?

", "rawMarkdown": "", "votes": null }, { "id": "48814", "postDate": "06/07/2014 00:07:57", "content": "

How can quantile regression be used on a binomial problem?

", "rawMarkdown": "", "votes": null }, { "id": "48824", "postDate": "06/07/2014 09:30:16", "content": "

[quote=Upul Bandara;43265]

\n

Hi All,

\n

I trained a Logistic Regression model, using the features described in following thread.

\n

Feature engineering and beat the benchmark (~0.59347)

\n

Unfortunately, its performance is much less than the performance of Vowpal Wabbit model. So I just want to know, whether a similar behavior is observed by others.

\n

Regards,
Upul

\n

[/quote]

\n

I tried R package glmnet(alpha = 0, lambda = 2^17, family = "binomial") on the Triskelion's features except offer_quantity and got 0.59565. The reason for the difference between Upul's score and mine may be the parameter estimation method employed by glmnet. See Regularization Paths for Generalized Linear Models via Coordinate Descent in detail.
Glmnet model can be improved by Bagging. The LB score of bagged glmnet using the same features and paramters was 0.60427. 

", "rawMarkdown": "", "votes": null }, { "id": "48833", "postDate": "06/07/2014 14:58:15", "content": "

Thanks!  I'm not too familiar with glmnet, though - would you mind posting a bit of sample code / pointers, just for running the un-bagged glmnet and predict.glmnet?  I think I'm using it properly, but I don't seem to be getting useful predictions out of it.

", "rawMarkdown": "", "votes": null }, { "id": "48837", "postDate": "06/07/2014 17:25:37", "content": "

Ok

\n

Here are my codes vw2csv.py and glmnet01.r.

\n

vw2csv.py converts train.vw and test.vw generated by Triskelion's code to csv format.

\n

glmnet01.r outputs a submission file.

", "rawMarkdown": "", "votes": null }, { "id": "48838", "postDate": "06/07/2014 17:38:21", "content": "

Thank you very much!

", "rawMarkdown": "", "votes": null }, { "id": "48839", "postDate": "06/07/2014 17:49:14", "content": "

Oops! Sorry for the duplicate files. The last 2 files are correct version.

\n

Does anyone know how to remove those attachments? I don't see any menu item for removing attachment.

", "rawMarkdown": "", "votes": null }, { "id": "48860", "postDate": "06/08/2014 12:41:47", "content": "

[quote=tks;48837]

\n

Ok

\n

Here are my codes vw2csv.py and glmnet01.r.

\n

vw2csv.py converts train.vw and test.vw generated by Triskelion's code to csv format.

\n

glmnet01.r outputs a submission file.

\n

[/quote]

\n\n

Thanks for sharing :D 

", "rawMarkdown": "", "votes": null }, { "id": "49083", "postDate": "06/13/2014 03:59:56", "content": "

Hi Thomas,

\n\n

the submissions are being evaluated based on AUC/ROC.

\n\n

What this does is try every possible cutoff threshold for what constitutes True/False

\n\n

For instance if we have 4 regression score values [0.0, .25, .5, 1.0] and we have [0, 0, 1, 1] as the real scores.

\n\n

For threshold of

\n

1.0, 1 true, 3 false - 1 error

\n

.5, 2 true, 2 false - 0 error

\n

.25, 3 true, 1 false - 1 error

\n

.0 , 4 true, 0 false - 2 errors

\n\n

So the best cutoff is .5 which gives a perfect AUC score of 1.0

", "rawMarkdown": "", "votes": null }, { "id": "49100", "postDate": "06/13/2014 13:58:05", "content": "

Hi tks,

\n\n

It's interesting to me that such a large value of lambda wouldn't cause the model to have high bias. Could your logic in setting it so high ? Is the bias less of a problem for logistic regression? 

", "rawMarkdown": "", "votes": null }, { "id": "49130", "postDate": "06/13/2014 23:09:17", "content": "

No logic. I simply tried several values and chose the best one.

", "rawMarkdown": "", "votes": null }, { "id": "49170", "postDate": "06/16/2014 00:56:06", "content": "

Hi TKS - what methods are you using to bag your results? The below code only showed a small improvement (0.00155) over running the regression alone:

\n\n

I ran the following code for 5000 iterations and averaged the predicted results on each 'fit'. I've included the code with the pseudo code in comments:

\n\n

# Sample 1-1/e (63%) percent of dataset

\n

pos <- sample(nrow(X),size=floor(nrow(X)*(1 - 1/exp(1)) ))

\n\n

# resample (m - 0.63*m) duplicates from this subset and add to the original sample
pos <- c(pos,sample(pos,size=nrow(X)-length(pos)))

\n\n

# run a regression on this dataset
fit <- glmnet(X[pos,], Y[pos,], family = "binomial",
alpha = 0, lambda=2^17)

\n\n

# Predict on Test set

\n\n

# average the results of the 5000 predictions

\n\n

Thanks

", "rawMarkdown": "", "votes": null }, { "id": "49173", "postDate": "06/16/2014 04:17:06", "content": "

Hi,

\n

Is there an R package that can wrap bagging around glmnet (or any other trainer)? 

\n

Thanks,

\n

C

", "rawMarkdown": "", "votes": null }, { "id": "49200", "postDate": "06/16/2014 14:10:12", "content": "

My current bagging implementation was based on this article:

\n\n\n

http://www.r-bloggers.com/improve-predictive-performance-in-r-with-bagging/

\n\n\n

I significantly improved my glmnet-based leaderboard score by bagging along those lines.

\n\n\n

Also, in general, if you're new to R, I've found the caret package (and accompanying book, Applied Predictive Modeling) to be incredibly useful.

", "rawMarkdown": "", "votes": null }, { "id": "49284", "postDate": "06/18/2014 21:43:22", "content": "

[quote=ddunder;49170]

\n

# Sample 1-1/e (63%) percent of dataset

\n

[/quote]

\n

The sample size is a parameter to tune. I used much smaller.

", "rawMarkdown": "", "votes": null }, { "id": "49291", "postDate": "06/19/2014 07:43:37", "content": "

Hi,

\n

I can't get the tks's solution. I was running the code posted by tks using glmnet package, but the result is much worse. Does anybody have the same problem?

\n\n

Thanks.

", "rawMarkdown": "", "votes": null }, { "id": "49294", "postDate": "06/19/2014 08:17:06", "content": "

For me I get the exact LB score tks got for the un-bagged solution, but when I try to use bagging the way suggested by the article Phil Culliton suggested I get lower LB score (0.60186) compared to the  0.60427 tks got.

\n\n

Perroquet, maybe something in your feature extraction phase was different.

", "rawMarkdown": "", "votes": null }, { "id": "49303", "postDate": "06/19/2014 14:48:01", "content": "

[quote=culstifier;49294]
For me I get the exact LB score tks got for the un-bagged solution, but when I try to use bagging the way suggested by the article Phil Culliton suggested I get lower LB score (0.60186) compared to the 0.60427 tks got.
[/quote]

\n\n

The article I posted uses a pretty large sample size.  As tks mentions, that's something that needs to be tuned for the data - a different sample size or sampling method may improve your score.

", "rawMarkdown": "", "votes": null }, { "id": "49329", "postDate": "06/20/2014 09:35:51", "content": "

Thanks Phil!

\n

It worked much better with 5% sample.

", "rawMarkdown": "", "votes": null }, { "id": "49474", "postDate": "06/24/2014 10:49:59", "content": "

Hi, I tried running glmnet for my model and got the exact same score for unbagged, but I'm not able to reach 0.60427. I'm using about 2000 iterations and I've varied sample size from 5% to about 30%. Is there anything apart from sample size that needs tweaking? Probably a different bagging method, I guess?

\n\n

Also, the most important question what i wanted to ask was about the output of the glmnet script. How do you interpret this repeatProbability output? Almost all of the repeat probabilities seem to be starting this way; with "0.2714....". Does anyone know how to interpret this? Or how it is used for ranking the customers?

\n\n

id repeatProbability

\n

12262064 0.271465247

\n

12277270 0.271466099

\n

12332190 0.27146523

\n

12524696 0.27146534 1

\n

3074629 0.271465299

\n\n\n

Thank you and have a great day! :)

", "rawMarkdown": "", "votes": null }, { "id": "49499", "postDate": "06/24/2014 18:47:25", "content": "

meet thakkar,

\n

Not sure what might be the issue with your bagged glmnet implementation. 2000 iterations and 5% sampling should give you close to the 0.60427 LB score. If you would like, provide the relevant code here for review. 

\n\n

Regarding the probabilities question, in ROC curve the order of the probabilities is important the value itself isn't.

", "rawMarkdown": "", "votes": null }, { "id": "49547", "postDate": "06/25/2014 10:47:10", "content": "

[quote=culstifier;49499]

\n

meet thakkar,

\n

Not sure what might be the issue with your bagged glmnet implementation. 2000 iterations and 5% sampling should give you close to the 0.60427 LB score. If you would like, provide the relevant code here for review. 

\n

Regarding the probabilities question, in ROC curve the order of the probabilities is important the value itself isn't.

\n

[/quote]

\n\n

clustifier,

\n

here's my code for bagging

\n\n

#get 5% samples from the training set

\n

training_positions <- sample(nrow(training1), size=floor((nrow(training1)*0.05)))

\n

#sample original again to get n-training_positions number of samples
pos<-sample(nrow(training1),size=nrow(training1)-length(training_positions))
training_positions <- c(training_positions,pos)

\n

xTrain<-training1[training_positions,]
target<-xTrain$repeter

\n

#remove first 3 columns(repeter, offerqty, id)
xTrain <- as.matrix(training1[training_positions,-(1:3)])

\n

#build glmnet model
model <- glmnet(xTrain, target, family = "binomial", alpha = 0, lambda = 2^17)

\n

#predict 
predict(model, xtesting, type="response")

\n\n

I run this code for about 2000 iterations to get 2000 predictions and then average it out

\n

id's not in testset but in testHistory will get 0 probability

\n\n\n

So, kindly help me out by pointing where I'm making a mistake here.

\n\n

Thanks in advance!

", "rawMarkdown": "", "votes": null }, { "id": "49548", "postDate": "06/25/2014 10:51:08", "content": "

how is the performance of LR

", "rawMarkdown": "", "votes": null }, { "id": "49579", "postDate": "06/25/2014 16:26:35", "content": "

meet thakkar,

\n

just delete these lines:

\n

#sample original again to get n-training_positions number of samples
pos<-sample(nrow(training1),size=nrow(training1)-length(training_positions))
training_positions <- c(training_positions,pos)

\n\n

You want to train only on 5% of the data, these three lines add back almost all the data.

\n\n

Just delete those lines and I think it will be good.

\n\n

(please, don't forget to click the Thank link in the bottom of the posts if you would like to thank someone :-)

", "rawMarkdown": "", "votes": null }, { "id": "49604", "postDate": "06/26/2014 05:53:32", "content": "

[quote=culstifier;49579]

\n

meet thakkar,

\n

just delete these lines:

\n

#sample original again to get n-training_positions number of samples
pos<-sample(nrow(training1),size=nrow(training1)-length(training_positions))
training_positions <- c(training_positions,pos)

\n

You want to train only on 5% of the data, these three lines add back almost all the data.

\n

Just delete those lines and I think it will be good.

\n

(please, don't forget to click the Thank link in the bottom of the posts if you would like to thank someone :-)

\n

[/quote]

\n\n\n\n

Worked like a charm. Thanks! :)

", "rawMarkdown": "", "votes": null }, { "id": "49605", "postDate": "06/26/2014 06:08:10", "content": "

meet thakkar,

\n

have you used more or different features than these mentioned in: Feature engineering and beat the benchmark (~0.59347)?

", "rawMarkdown": "", "votes": null }, { "id": "49612", "postDate": "06/26/2014 10:31:31", "content": "

culstifier,

\n

You mentioned that in ROC curve the order of the probabilities is important rather than the value itself , Can you explain more or provide some pointers to refer about that .

\n\n

Thanks!!

", "rawMarkdown": "", "votes": null }, { "id": "49614", "postDate": "06/26/2014 11:01:05", "content": "

Look at the Evaluation page and also the link to Wikipedia there.

\n

To calculate the area under the ROC curve what actually determine the score is the order. The reason is that peeking the point to put the threshold between the positive and negative estimates (see the Wikipedia page for the full explanation about the threshold) doesn't affect by the score (the AUC, area under the curve) itself. Only the order affects that.

\n\n\n

Think of it as ordering your result by probability, moving the threshold probability from 0 to 1 and assign all those smaller than the threshold to one class and all the other to the other class. For each threshold probability count the number of false positives and the number of true positives. These numbers will be affected only by the order not by the values.

\n\n\n

Hope it helps.

", "rawMarkdown": "", "votes": null }, { "id": "49748", "postDate": "06/28/2014 01:18:59", "content": "

Hi All,

\n\n

    I am also using the sparse logistic regression but not the R package 'glmnet'. I have several questions about the experimental settings in the R code.

\n\n

    (1) In previous codes, the training data was inputted to glmnet() and the default setting for standardization is TRUE (from the manual for glmnet). Is it z-score? And I was wondering whether the function predict() standardizes the test data or not. Theoretically, they should be standardized separately, right? Z-score indeed does not perform well in my implementations. Any suggestions for standardization?

\n\n

    (2) It seems that 2^17 is a quite large penalty. Could anyone tell me how sparse the coefficient vector is in this case? For example, how many non-zero entries when using 2^17?

\n\n

Thanks a lot~

", "rawMarkdown": "", "votes": null }, { "id": "49749", "postDate": "06/28/2014 02:09:34", "content": "

Hey hunterluffy. In glmnet:

\n\n

1) Predict will standardize the test data using the means and standard deviations of the training data.

\n\n

2) In the settings of the code posted, alpha=0 so the regularization penalty is the ridge penalty (or in other words ||B||^2 for coefficient vector B). With this squared penalty, the coefficients will be shrumk to a very small size but none will go to zero. An alpha of 1 would penalize the absolute value ||B|| of the coefficients, and a value in-between will penalize a mix of the two. (For me, alpha=0 seems to work the best for this data but maybe you can get something reasonable with another alpha)

\n\n

Hope this helps!

", "rawMarkdown": "", "votes": null }, { "id": "49750", "postDate": "06/28/2014 02:12:48", "content": "

Got it~ Thanks a lot, Thomas !

", "rawMarkdown": "", "votes": null }, { "id": "49819", "postDate": "06/30/2014 05:53:28", "content": "

[quote=clustifier;49605]

\n

meet thakkar,

\n

have you used more or different features than these mentioned in: Feature engineering and beat the benchmark (~0.59347)?

\n

[/quote]

\n

clustifier,

\n

I am using few features from the link as well as few features of my own. How about you?

", "rawMarkdown": "", "votes": null }, { "id": "49820", "postDate": "06/30/2014 06:04:17", "content": "

No feature that I've tried to add significantly changed my LB score.

\n

Playing little with the bins of days features somewhat helped.

", "rawMarkdown": "", "votes": null }, { "id": "49822", "postDate": "06/30/2014 06:13:19", "content": "

Oh, a few new features helped my LB score a little(I tried many. But only a few helped!). I haven't tried binning though. You're using the same glmnet bagging for your model, right?

", "rawMarkdown": "", "votes": null }, { "id": "49878", "postDate": "06/30/2014 18:59:39", "content": "

Yes. Same glmnet & bagging.

\n\n

I'm not sure what to try next.

\n

Adding more new features or trying new algorithm.

\n\n

I it been mentioned that using features from the full data help significantly but I couldn't think of good feature from the full transactions file that are not already in the reduced data.

", "rawMarkdown": "", "votes": null }, { "id": "49889", "postDate": "07/01/2014 05:47:26", "content": "

[quote=clustifier;49878]

\n

Yes. Same glmnet & bagging.

\n

I'm not sure what to try next.

\n

Adding more new features or trying new algorithm.

\n

I it been mentioned that using features from the full data help significantly but I couldn't think of good feature from the full transactions file that are not already in the reduced data.

\n

[/quote]

\n\n

I think adding new features might be a better option than trying new algorithms.

\n

For me, I have a few features, but i need to focus on better preprocessing techniques like binning as you suggested, I reckon.

\n\n

And trying the entire 22GB data will be my last option I guess.

", "rawMarkdown": "", "votes": null }, { "id": "49915", "postDate": "07/01/2014 17:10:58", "content": "

Interesting - I've taken a different path to my current score.  All of my advances have come via Vowpal Wabbit and adding new features.  I've come up with a fair number of new classes of features that have improved my VW performance significantly.  I also switched to the full data set weeks ago.  Apparently, given our respective positions on the leaderboard (I'm a few dozen places behind both of you at the moment) all that may not have been the optimal approach.  It was fun, though!  :-)

\n

When I switch to glmnet and bagging, however, it doesn't perform as well.  So I may have a bug somewhere in my glmnet code... or, also likely, my VW features need to be tuned for the glmnet context.  Hmm.  I've tried cutting out features that seemed like they'd be less useful there, but so far no luck.

", "rawMarkdown": "", "votes": null }, { "id": "49935", "postDate": "07/02/2014 05:44:43", "content": "

[quote=Phil Culliton;49915]

\n

Interesting - I've taken a different path to my current score.  All of my advances have come via Vowpal Wabbit and adding new features.  I've come up with a fair number of new classes of features that have improved my VW performance significantly.  I also switched to the full data set weeks ago.  Apparently, given our respective positions on the leaderboard (I'm a few dozen places behind both of you at the moment) all that may not have been the optimal approach.  It was fun, though!  :-)

\n

When I switch to glmnet and bagging, however, it doesn't perform as well.  So I may have a bug somewhere in my glmnet code... or, also likely, my VW features need to be tuned for the glmnet context.  Hmm.  I've tried cutting out features that seemed like they'd be less useful there, but so far no luck.

\n

[/quote]

\n\n

Phil,

\n

I tried VW in the beginning, but glmnet(without bagging) improved my score. Then I tried it with bagging and it improved my score a little more. I only added a very few new features(Had created a lot. But only a very few helped!). I think I should think of some new and better features to add.

\n\n

Did the full dataset help in improving the score? I'm thinking of using it, but I'm not sure how much of a help it will be?

", "rawMarkdown": "", "votes": null }, { "id": "49956", "postDate": "07/02/2014 12:40:10", "content": "

Thanks!  I went back to the original data set, with just a few features added which had performed well across methods, and glmnet still underperforms significantly for me, so I'm pretty sure I've got some bugs in my glmnet code.  Debugging time!  :-)

\n\n

For glmnet, I'm not sure how much value is added by the full set - I've noted similar CV / LB scores for both.  It seems to help VW pretty significantly - my LB scores jumped up when I started using it.  I think it helps to fill out the weight vectors and make them cover more corner cases.

", "rawMarkdown": "", "votes": null }, { "id": "49965", "postDate": "07/02/2014 18:41:02", "content": "

[quote=Phil Culliton;49956]

\n

I think it helps to fill out the weight vectors and make them cover more corner cases.

\n

[/quote]

\n\n

Can you please explain that more?

\n\n

I can't think why the data not in the reduced will help. It looks to me like completely irrelevant data.

", "rawMarkdown": "", "votes": null }, { "id": "49967", "postDate": "07/02/2014 19:04:01", "content": "

Sure!  You're basically capturing potential indirect relationships.  For instance, if you're tracking specific brands bought, purchasing related brands Y-Z multiple times - even if they didn't have a coupon - may indicate a propensity to purchase brand X more than once, or vice versa.

\n\n

In VW terms, this translates to providing more members of the weight vector which help distinguish similarity, which is probably why it performs so well there.  I definitely had a large jump when I switched to using the full transaction set with VW.

", "rawMarkdown": "", "votes": null }, { "id": "58714", "postDate": "11/23/2014 06:03:45", "content": "

Hi Everyone,

\n

I am new to data analysis and working on this project for knowledge and learning. I have applied the glmnet model on features created, which are quite similar to one mentioned by Treskelion. But when I am trying to plot it I cannot see anything.

\n

For learning purpose I am working just on train dataset and have not included test one.

\n

I have created the model using command in R:

\n

model = glmnet(xmerge,target,family="binomial", alpha = 0, lambda=2^17)

\n

where xmerge contains all my numeric measures. I want to see in all the features which one has highest impact. How can I see that with the help of plots/graphs?

\n

Any help will be appreciated.

\n

Thanks!!

", "rawMarkdown": "", "votes": null }, { "id": "69960", "postDate": "04/08/2015 00:30:11", "content": "

[quote=tks;48824]

\n

[quote=Upul Bandara;43265]

\n

Hi All,

\n

I trained a Logistic Regression model, using the features described in following thread.

\n

Feature engineering and beat the benchmark (~0.59347)

\n

Unfortunately, its performance is much less than the performance of Vowpal Wabbit model. So I just want to know, whether a similar behavior is observed by others.

\n

Regards,
Upul

\n

[/quote]

\n

I tried R package glmnet(alpha = 0, lambda = 2^17, family = "binomial") on the Triskelion's features except offer_quantity and got 0.59565. The reason for the difference between Upul's score and mine may be the parameter estimation method employed by glmnet. See Regularization Paths for Generalized Linear Models via Coordinate Descent in detail.
Glmnet model can be improved by Bagging. The LB score of bagged glmnet using the same features and paramters was 0.60427. 

\n

[/quote]

\n\n

TKS - you were kind enough to post your glmnet R code, so I thought I'd ask if you might be willing to share how you approached translating Triskelion's features from VW/Python into R. I'm working on an academic team that is going through this old competition to practice our R skills. Thanks for considering this request!

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 43307, "author_name": "zachmayer", "author_url": "", "post_date": "04/29/2014 15:45:55", "content": "

Yes, VW does some magic.  It uses stochastic gradient descent, which is a little less sensitive to outliers.  The code you linked to also uses quantile regression, rather than logistic regression, which seems to give better results on this dataset.  I haven't figured out why they use a quantile_tau of 0.60 rather than the median, but that magic number also seems to give better results on this dataset.

\n

Finally, it's possible that the featureset you built yourself is different from the one built in the linked post.  Check your subsets and aggregations carefully!

", "votes": null, "replies": [] }, { "id": 44415, "author_name": "breuker", "author_url": "", "post_date": "05/12/2014 16:43:49", "content": "

If you want to use another tool, bear in mind that vw has a couple of features on by default that may or may not be implemented (e.g., adaptive, normalized, invariant update rules / using holdout data to determine how many passes you need / ...)

\n

Here is a very brief description of what it does: https://github.com/JohnLangford/vowpal_wabbit/wiki/Command-line-arguments

", "votes": null, "replies": [] }, { "id": 47261, "author_name": "mike64", "author_url": "", "post_date": "05/29/2014 14:35:21", "content": "

What percentage of repeaters classifies the model right?

", "votes": null, "replies": [] }, { "id": 48814, "author_name": "omalleyt", "author_url": "", "post_date": "06/07/2014 00:07:57", "content": "

How can quantile regression be used on a binomial problem?

", "votes": null, "replies": [] }, { "id": 48824, "author_name": "tks0123456789", "author_url": "", "post_date": "06/07/2014 09:30:16", "content": "

[quote=Upul Bandara;43265]

\n

Hi All,

\n

I trained a Logistic Regression model, using the features described in following thread.

\n

Feature engineering and beat the benchmark (~0.59347)

\n

Unfortunately, its performance is much less than the performance of Vowpal Wabbit model. So I just want to know, whether a similar behavior is observed by others.

\n

Regards,
Upul

\n

[/quote]

\n

I tried R package glmnet(alpha = 0, lambda = 2^17, family = "binomial") on the Triskelion's features except offer_quantity and got 0.59565. The reason for the difference between Upul's score and mine may be the parameter estimation method employed by glmnet. See Regularization Paths for Generalized Linear Models via Coordinate Descent in detail.
Glmnet model can be improved by Bagging. The LB score of bagged glmnet using the same features and paramters was 0.60427. 

", "votes": null, "replies": [] }, { "id": 48833, "author_name": "philculliton", "author_url": "", "post_date": "06/07/2014 14:58:15", "content": "

Thanks!  I'm not too familiar with glmnet, though - would you mind posting a bit of sample code / pointers, just for running the un-bagged glmnet and predict.glmnet?  I think I'm using it properly, but I don't seem to be getting useful predictions out of it.

", "votes": null, "replies": [] }, { "id": 48837, "author_name": "tks0123456789", "author_url": "", "post_date": "06/07/2014 17:25:37", "content": "

Ok

\n

Here are my codes vw2csv.py and glmnet01.r.

\n

vw2csv.py converts train.vw and test.vw generated by Triskelion's code to csv format.

\n

glmnet01.r outputs a submission file.

", "votes": null, "replies": [] }, { "id": 48838, "author_name": "philculliton", "author_url": "", "post_date": "06/07/2014 17:38:21", "content": "

Thank you very much!

", "votes": null, "replies": [] }, { "id": 48839, "author_name": "tks0123456789", "author_url": "", "post_date": "06/07/2014 17:49:14", "content": "

Oops! Sorry for the duplicate files. The last 2 files are correct version.

\n

Does anyone know how to remove those attachments? I don't see any menu item for removing attachment.

", "votes": null, "replies": [] }, { "id": 48860, "author_name": "tayshin", "author_url": "", "post_date": "06/08/2014 12:41:47", "content": "

[quote=tks;48837]

\n

Ok

\n

Here are my codes vw2csv.py and glmnet01.r.

\n

vw2csv.py converts train.vw and test.vw generated by Triskelion's code to csv format.

\n

glmnet01.r outputs a submission file.

\n

[/quote]

\n\n

Thanks for sharing :D 

", "votes": null, "replies": [] }, { "id": 49083, "author_name": "letterrip", "author_url": "", "post_date": "06/13/2014 03:59:56", "content": "

Hi Thomas,

\n\n

the submissions are being evaluated based on AUC/ROC.

\n\n

What this does is try every possible cutoff threshold for what constitutes True/False

\n\n

For instance if we have 4 regression score values [0.0, .25, .5, 1.0] and we have [0, 0, 1, 1] as the real scores.

\n\n

For threshold of

\n

1.0, 1 true, 3 false - 1 error

\n

.5, 2 true, 2 false - 0 error

\n

.25, 3 true, 1 false - 1 error

\n

.0 , 4 true, 0 false - 2 errors

\n\n

So the best cutoff is .5 which gives a perfect AUC score of 1.0

", "votes": null, "replies": [] }, { "id": 49100, "author_name": "dansterba", "author_url": "", "post_date": "06/13/2014 13:58:05", "content": "

Hi tks,

\n\n

It's interesting to me that such a large value of lambda wouldn't cause the model to have high bias. Could your logic in setting it so high ? Is the bias less of a problem for logistic regression? 

", "votes": null, "replies": [] }, { "id": 49130, "author_name": "tks0123456789", "author_url": "", "post_date": "06/13/2014 23:09:17", "content": "

No logic. I simply tried several values and chose the best one.

", "votes": null, "replies": [] }, { "id": 49170, "author_name": "dansterba", "author_url": "", "post_date": "06/16/2014 00:56:06", "content": "

Hi TKS - what methods are you using to bag your results? The below code only showed a small improvement (0.00155) over running the regression alone:

\n\n

I ran the following code for 5000 iterations and averaged the predicted results on each 'fit'. I've included the code with the pseudo code in comments:

\n\n

# Sample 1-1/e (63%) percent of dataset

\n

pos <- sample(nrow(X),size=floor(nrow(X)*(1 - 1/exp(1)) ))

\n\n

# resample (m - 0.63*m) duplicates from this subset and add to the original sample
pos <- c(pos,sample(pos,size=nrow(X)-length(pos)))

\n\n

# run a regression on this dataset
fit <- glmnet(X[pos,], Y[pos,], family = "binomial",
alpha = 0, lambda=2^17)

\n\n

# Predict on Test set

\n\n

# average the results of the 5000 predictions

\n\n

Thanks

", "votes": null, "replies": [] }, { "id": 49173, "author_name": "clustifier", "author_url": "", "post_date": "06/16/2014 04:17:06", "content": "

Hi,

\n

Is there an R package that can wrap bagging around glmnet (or any other trainer)? 

\n

Thanks,

\n

C

", "votes": null, "replies": [] }, { "id": 49200, "author_name": "philculliton", "author_url": "", "post_date": "06/16/2014 14:10:12", "content": "

My current bagging implementation was based on this article:

\n\n\n

http://www.r-bloggers.com/improve-predictive-performance-in-r-with-bagging/

\n\n\n

I significantly improved my glmnet-based leaderboard score by bagging along those lines.

\n\n\n

Also, in general, if you're new to R, I've found the caret package (and accompanying book, Applied Predictive Modeling) to be incredibly useful.

", "votes": null, "replies": [] }, { "id": 49284, "author_name": "tks0123456789", "author_url": "", "post_date": "06/18/2014 21:43:22", "content": "

[quote=ddunder;49170]

\n

# Sample 1-1/e (63%) percent of dataset

\n

[/quote]

\n

The sample size is a parameter to tune. I used much smaller.

", "votes": null, "replies": [] }, { "id": 49291, "author_name": "quickjanca", "author_url": "", "post_date": "06/19/2014 07:43:37", "content": "

Hi,

\n

I can't get the tks's solution. I was running the code posted by tks using glmnet package, but the result is much worse. Does anybody have the same problem?

\n\n

Thanks.

", "votes": null, "replies": [] }, { "id": 49294, "author_name": "clustifier", "author_url": "", "post_date": "06/19/2014 08:17:06", "content": "

For me I get the exact LB score tks got for the un-bagged solution, but when I try to use bagging the way suggested by the article Phil Culliton suggested I get lower LB score (0.60186) compared to the  0.60427 tks got.

\n\n

Perroquet, maybe something in your feature extraction phase was different.

", "votes": null, "replies": [] }, { "id": 49303, "author_name": "philculliton", "author_url": "", "post_date": "06/19/2014 14:48:01", "content": "

[quote=culstifier;49294]
For me I get the exact LB score tks got for the un-bagged solution, but when I try to use bagging the way suggested by the article Phil Culliton suggested I get lower LB score (0.60186) compared to the 0.60427 tks got.
[/quote]

\n\n

The article I posted uses a pretty large sample size.  As tks mentions, that's something that needs to be tuned for the data - a different sample size or sampling method may improve your score.

", "votes": null, "replies": [] }, { "id": 49329, "author_name": "clustifier", "author_url": "", "post_date": "06/20/2014 09:35:51", "content": "

Thanks Phil!

\n

It worked much better with 5% sample.

", "votes": null, "replies": [] }, { "id": 49474, "author_name": "meetthakkar1", "author_url": "", "post_date": "06/24/2014 10:49:59", "content": "

Hi, I tried running glmnet for my model and got the exact same score for unbagged, but I'm not able to reach 0.60427. I'm using about 2000 iterations and I've varied sample size from 5% to about 30%. Is there anything apart from sample size that needs tweaking? Probably a different bagging method, I guess?

\n\n

Also, the most important question what i wanted to ask was about the output of the glmnet script. How do you interpret this repeatProbability output? Almost all of the repeat probabilities seem to be starting this way; with "0.2714....". Does anyone know how to interpret this? Or how it is used for ranking the customers?

\n\n

id repeatProbability

\n

12262064 0.271465247

\n

12277270 0.271466099

\n

12332190 0.27146523

\n

12524696 0.27146534 1

\n

3074629 0.271465299

\n\n\n

Thank you and have a great day! :)

", "votes": null, "replies": [] }, { "id": 49499, "author_name": "clustifier", "author_url": "", "post_date": "06/24/2014 18:47:25", "content": "

meet thakkar,

\n

Not sure what might be the issue with your bagged glmnet implementation. 2000 iterations and 5% sampling should give you close to the 0.60427 LB score. If you would like, provide the relevant code here for review. 

\n\n

Regarding the probabilities question, in ROC curve the order of the probabilities is important the value itself isn't.

", "votes": null, "replies": [] }, { "id": 49547, "author_name": "meetthakkar1", "author_url": "", "post_date": "06/25/2014 10:47:10", "content": "

[quote=culstifier;49499]

\n

meet thakkar,

\n

Not sure what might be the issue with your bagged glmnet implementation. 2000 iterations and 5% sampling should give you close to the 0.60427 LB score. If you would like, provide the relevant code here for review. 

\n

Regarding the probabilities question, in ROC curve the order of the probabilities is important the value itself isn't.

\n

[/quote]

\n\n

clustifier,

\n

here's my code for bagging

\n\n

#get 5% samples from the training set

\n

training_positions <- sample(nrow(training1), size=floor((nrow(training1)*0.05)))

\n

#sample original again to get n-training_positions number of samples
pos<-sample(nrow(training1),size=nrow(training1)-length(training_positions))
training_positions <- c(training_positions,pos)

\n

xTrain<-training1[training_positions,]
target<-xTrain$repeter

\n

#remove first 3 columns(repeter, offerqty, id)
xTrain <- as.matrix(training1[training_positions,-(1:3)])

\n

#build glmnet model
model <- glmnet(xTrain, target, family = "binomial", alpha = 0, lambda = 2^17)

\n

#predict 
predict(model, xtesting, type="response")

\n\n

I run this code for about 2000 iterations to get 2000 predictions and then average it out

\n

id's not in testset but in testHistory will get 0 probability

\n\n\n

So, kindly help me out by pointing where I'm making a mistake here.

\n\n

Thanks in advance!

", "votes": null, "replies": [] }, { "id": 49548, "author_name": "sistzyhndcw", "author_url": "", "post_date": "06/25/2014 10:51:08", "content": "

how is the performance of LR

", "votes": null, "replies": [] }, { "id": 49579, "author_name": "clustifier", "author_url": "", "post_date": "06/25/2014 16:26:35", "content": "

meet thakkar,

\n

just delete these lines:

\n

#sample original again to get n-training_positions number of samples
pos<-sample(nrow(training1),size=nrow(training1)-length(training_positions))
training_positions <- c(training_positions,pos)

\n\n

You want to train only on 5% of the data, these three lines add back almost all the data.

\n\n

Just delete those lines and I think it will be good.

\n\n

(please, don't forget to click the Thank link in the bottom of the posts if you would like to thank someone :-)

", "votes": null, "replies": [] }, { "id": 49604, "author_name": "meetthakkar1", "author_url": "", "post_date": "06/26/2014 05:53:32", "content": "

[quote=culstifier;49579]

\n

meet thakkar,

\n

just delete these lines:

\n

#sample original again to get n-training_positions number of samples
pos<-sample(nrow(training1),size=nrow(training1)-length(training_positions))
training_positions <- c(training_positions,pos)

\n

You want to train only on 5% of the data, these three lines add back almost all the data.

\n

Just delete those lines and I think it will be good.

\n

(please, don't forget to click the Thank link in the bottom of the posts if you would like to thank someone :-)

\n

[/quote]

\n\n\n\n

Worked like a charm. Thanks! :)

", "votes": null, "replies": [] }, { "id": 49605, "author_name": "clustifier", "author_url": "", "post_date": "06/26/2014 06:08:10", "content": "

meet thakkar,

\n

have you used more or different features than these mentioned in: Feature engineering and beat the benchmark (~0.59347)?

", "votes": null, "replies": [] }, { "id": 49612, "author_name": "vivekaditya", "author_url": "", "post_date": "06/26/2014 10:31:31", "content": "

culstifier,

\n

You mentioned that in ROC curve the order of the probabilities is important rather than the value itself , Can you explain more or provide some pointers to refer about that .

\n\n

Thanks!!

", "votes": null, "replies": [] }, { "id": 49614, "author_name": "clustifier", "author_url": "", "post_date": "06/26/2014 11:01:05", "content": "

Look at the Evaluation page and also the link to Wikipedia there.

\n

To calculate the area under the ROC curve what actually determine the score is the order. The reason is that peeking the point to put the threshold between the positive and negative estimates (see the Wikipedia page for the full explanation about the threshold) doesn't affect by the score (the AUC, area under the curve) itself. Only the order affects that.

\n\n\n

Think of it as ordering your result by probability, moving the threshold probability from 0 to 1 and assign all those smaller than the threshold to one class and all the other to the other class. For each threshold probability count the number of false positives and the number of true positives. These numbers will be affected only by the order not by the values.

\n\n\n

Hope it helps.

", "votes": null, "replies": [] }, { "id": 49748, "author_name": "sundevil", "author_url": "", "post_date": "06/28/2014 01:18:59", "content": "

Hi All,

\n\n

    I am also using the sparse logistic regression but not the R package 'glmnet'. I have several questions about the experimental settings in the R code.

\n\n

    (1) In previous codes, the training data was inputted to glmnet() and the default setting for standardization is TRUE (from the manual for glmnet). Is it z-score? And I was wondering whether the function predict() standardizes the test data or not. Theoretically, they should be standardized separately, right? Z-score indeed does not perform well in my implementations. Any suggestions for standardization?

\n\n

    (2) It seems that 2^17 is a quite large penalty. Could anyone tell me how sparse the coefficient vector is in this case? For example, how many non-zero entries when using 2^17?

\n\n

Thanks a lot~

", "votes": null, "replies": [] }, { "id": 49749, "author_name": "omalleyt", "author_url": "", "post_date": "06/28/2014 02:09:34", "content": "

Hey hunterluffy. In glmnet:

\n\n

1) Predict will standardize the test data using the means and standard deviations of the training data.

\n\n

2) In the settings of the code posted, alpha=0 so the regularization penalty is the ridge penalty (or in other words ||B||^2 for coefficient vector B). With this squared penalty, the coefficients will be shrumk to a very small size but none will go to zero. An alpha of 1 would penalize the absolute value ||B|| of the coefficients, and a value in-between will penalize a mix of the two. (For me, alpha=0 seems to work the best for this data but maybe you can get something reasonable with another alpha)

\n\n

Hope this helps!

", "votes": null, "replies": [] }, { "id": 49750, "author_name": "sundevil", "author_url": "", "post_date": "06/28/2014 02:12:48", "content": "

Got it~ Thanks a lot, Thomas !

", "votes": null, "replies": [] }, { "id": 49819, "author_name": "meetthakkar1", "author_url": "", "post_date": "06/30/2014 05:53:28", "content": "

[quote=clustifier;49605]

\n

meet thakkar,

\n

have you used more or different features than these mentioned in: Feature engineering and beat the benchmark (~0.59347)?

\n

[/quote]

\n

clustifier,

\n

I am using few features from the link as well as few features of my own. How about you?

", "votes": null, "replies": [] }, { "id": 49820, "author_name": "clustifier", "author_url": "", "post_date": "06/30/2014 06:04:17", "content": "

No feature that I've tried to add significantly changed my LB score.

\n

Playing little with the bins of days features somewhat helped.

", "votes": null, "replies": [] }, { "id": 49822, "author_name": "meetthakkar1", "author_url": "", "post_date": "06/30/2014 06:13:19", "content": "

Oh, a few new features helped my LB score a little(I tried many. But only a few helped!). I haven't tried binning though. You're using the same glmnet bagging for your model, right?

", "votes": null, "replies": [] }, { "id": 49878, "author_name": "clustifier", "author_url": "", "post_date": "06/30/2014 18:59:39", "content": "

Yes. Same glmnet & bagging.

\n\n

I'm not sure what to try next.

\n

Adding more new features or trying new algorithm.

\n\n

I it been mentioned that using features from the full data help significantly but I couldn't think of good feature from the full transactions file that are not already in the reduced data.

", "votes": null, "replies": [] }, { "id": 49889, "author_name": "meetthakkar1", "author_url": "", "post_date": "07/01/2014 05:47:26", "content": "

[quote=clustifier;49878]

\n

Yes. Same glmnet & bagging.

\n

I'm not sure what to try next.

\n

Adding more new features or trying new algorithm.

\n

I it been mentioned that using features from the full data help significantly but I couldn't think of good feature from the full transactions file that are not already in the reduced data.

\n

[/quote]

\n\n

I think adding new features might be a better option than trying new algorithms.

\n

For me, I have a few features, but i need to focus on better preprocessing techniques like binning as you suggested, I reckon.

\n\n

And trying the entire 22GB data will be my last option I guess.

", "votes": null, "replies": [] }, { "id": 49915, "author_name": "philculliton", "author_url": "", "post_date": "07/01/2014 17:10:58", "content": "

Interesting - I've taken a different path to my current score.  All of my advances have come via Vowpal Wabbit and adding new features.  I've come up with a fair number of new classes of features that have improved my VW performance significantly.  I also switched to the full data set weeks ago.  Apparently, given our respective positions on the leaderboard (I'm a few dozen places behind both of you at the moment) all that may not have been the optimal approach.  It was fun, though!  :-)

\n

When I switch to glmnet and bagging, however, it doesn't perform as well.  So I may have a bug somewhere in my glmnet code... or, also likely, my VW features need to be tuned for the glmnet context.  Hmm.  I've tried cutting out features that seemed like they'd be less useful there, but so far no luck.

", "votes": null, "replies": [] }, { "id": 49935, "author_name": "meetthakkar1", "author_url": "", "post_date": "07/02/2014 05:44:43", "content": "

[quote=Phil Culliton;49915]

\n

Interesting - I've taken a different path to my current score.  All of my advances have come via Vowpal Wabbit and adding new features.  I've come up with a fair number of new classes of features that have improved my VW performance significantly.  I also switched to the full data set weeks ago.  Apparently, given our respective positions on the leaderboard (I'm a few dozen places behind both of you at the moment) all that may not have been the optimal approach.  It was fun, though!  :-)

\n

When I switch to glmnet and bagging, however, it doesn't perform as well.  So I may have a bug somewhere in my glmnet code... or, also likely, my VW features need to be tuned for the glmnet context.  Hmm.  I've tried cutting out features that seemed like they'd be less useful there, but so far no luck.

\n

[/quote]

\n\n

Phil,

\n

I tried VW in the beginning, but glmnet(without bagging) improved my score. Then I tried it with bagging and it improved my score a little more. I only added a very few new features(Had created a lot. But only a very few helped!). I think I should think of some new and better features to add.

\n\n

Did the full dataset help in improving the score? I'm thinking of using it, but I'm not sure how much of a help it will be?

", "votes": null, "replies": [] }, { "id": 49956, "author_name": "philculliton", "author_url": "", "post_date": "07/02/2014 12:40:10", "content": "

Thanks!  I went back to the original data set, with just a few features added which had performed well across methods, and glmnet still underperforms significantly for me, so I'm pretty sure I've got some bugs in my glmnet code.  Debugging time!  :-)

\n\n

For glmnet, I'm not sure how much value is added by the full set - I've noted similar CV / LB scores for both.  It seems to help VW pretty significantly - my LB scores jumped up when I started using it.  I think it helps to fill out the weight vectors and make them cover more corner cases.

", "votes": null, "replies": [] }, { "id": 49965, "author_name": "clustifier", "author_url": "", "post_date": "07/02/2014 18:41:02", "content": "

[quote=Phil Culliton;49956]

\n

I think it helps to fill out the weight vectors and make them cover more corner cases.

\n

[/quote]

\n\n

Can you please explain that more?

\n\n

I can't think why the data not in the reduced will help. It looks to me like completely irrelevant data.

", "votes": null, "replies": [] }, { "id": 49967, "author_name": "philculliton", "author_url": "", "post_date": "07/02/2014 19:04:01", "content": "

Sure!  You're basically capturing potential indirect relationships.  For instance, if you're tracking specific brands bought, purchasing related brands Y-Z multiple times - even if they didn't have a coupon - may indicate a propensity to purchase brand X more than once, or vice versa.

\n\n

In VW terms, this translates to providing more members of the weight vector which help distinguish similarity, which is probably why it performs so well there.  I definitely had a large jump when I switched to using the full transaction set with VW.

", "votes": null, "replies": [] }, { "id": 58714, "author_name": "shainanhemrajani", "author_url": "", "post_date": "11/23/2014 06:03:45", "content": "

Hi Everyone,

\n

I am new to data analysis and working on this project for knowledge and learning. I have applied the glmnet model on features created, which are quite similar to one mentioned by Treskelion. But when I am trying to plot it I cannot see anything.

\n

For learning purpose I am working just on train dataset and have not included test one.

\n

I have created the model using command in R:

\n

model = glmnet(xmerge,target,family="binomial", alpha = 0, lambda=2^17)

\n

where xmerge contains all my numeric measures. I want to see in all the features which one has highest impact. How can I see that with the help of plots/graphs?

\n

Any help will be appreciated.

\n

Thanks!!

", "votes": null, "replies": [] }, { "id": 69960, "author_name": "dluzansky", "author_url": "", "post_date": "04/08/2015 00:30:11", "content": "

[quote=tks;48824]

\n

[quote=Upul Bandara;43265]

\n

Hi All,

\n

I trained a Logistic Regression model, using the features described in following thread.

\n

Feature engineering and beat the benchmark (~0.59347)

\n

Unfortunately, its performance is much less than the performance of Vowpal Wabbit model. So I just want to know, whether a similar behavior is observed by others.

\n

Regards,
Upul

\n

[/quote]

\n

I tried R package glmnet(alpha = 0, lambda = 2^17, family = "binomial") on the Triskelion's features except offer_quantity and got 0.59565. The reason for the difference between Upul's score and mine may be the parameter estimation method employed by glmnet. See Regularization Paths for Generalized Linear Models via Coordinate Descent in detail.
Glmnet model can be improved by Bagging. The LB score of bagged glmnet using the same features and paramters was 0.60427. 

\n

[/quote]

\n\n

TKS - you were kind enough to post your glmnet R code, so I thought I'd ask if you might be willing to share how you approached translating Triskelion's features from VW/Python into R. I'm working on an academic team that is going through this old competition to practice our R skills. Thanks for considering this request!

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "43265": "", "43307": "", "44415": "", "47261": "", "48814": "", "48824": "", "48833": "", "48837": "", "48838": "", "48839": "", "48860": "", "49083": "", "49100": "", "49130": "", "49170": "", "49173": "", "49200": "", "49284": "", "49291": "", "49294": "", "49303": "", "49329": "", "49474": "", "49499": "", "49547": "", "49548": "", "49579": "", "49604": "", "49605": "", "49612": "", "49614": "", "49748": "", "49749": "", "49750": "", "49819": "", "49820": "", "49822": "", "49878": "", "49889": "", "49915": "", "49935": "", "49956": "", "49965": "", "49967": "", "58714": "", "69960": "" }, "source": "meta" }