{ "id": 7688, "title": "Feature engineering and beat the benchmark (~0.59347)", "url": "/competitions/acquire-valued-shoppers-challenge/discussion/7688", "author_name": "", "post_date": "2014-04-15T05:37:47.573Z", "votes": 77, "comment_count": 132, "views": 59898, "content": "

Here is some code and ideas to get a score of around 0.59347. It uses Python and Vowpal Wabbit, but the idea should work with other languages and other algorithms.

\n

Data reduction

\n

Check out the data reduction thread to understand what this is about. For this benchmark we create the category and company union.

\n

Feature engineering

\n

Feature engineering will be important in this competition, no matter the language or algorithms used. We can check out the benchmarks on the leaderboard to get a feel for features to use.

\n

We will generate the following features:

\n\n

These same features for:

\n\n

Combinations of these:

\n\n

Offer-related:

\n\n

Total shopper spend:

\n\n

Can you name some other possibly interesting features to generate?

\n

Vowpal Wabbit

\n

We train Vowpal Wabbit using quantile regression, 40 passes and a learning rate of 0.85. We turn the predictions into Kaggle's submission format.

\n

Feature visualisation

\n

Using the output from Vowpal Wabbit's wrapper vw-varinfo we generate the feature relevance plot below (code to generate this included):

\n

\"feature

\n

Improvements

\n

If you use this code or feature ideas to improve the score of 0.59347 I'd really appreciate it if you let me know these improvements. It's very possible that another algorithm may do better with these exact features.

\n

If you do improve the score I'd love to team up. I am still working on a few new features and we could combine our approaches for an ever better score.

\n

Code & Tutorial

\n

For latest code see the Github repo: https://github.com/MLWave/kaggle_acquire-valued-shoppers-challenge, for the tutorial/description read: http://mlwave.com/predicting-repeat-buyers-vowpal-wabbit/

\n

All of this runs in about 15 minutes for me, from raw data to submission. Takes less than 1GB of memory.

\n

Todo

\n

Generate more features. Set up local validation. Tweak algorithms. Try other algorithms than quantile regression (sklearn has plenty).

", "messages": [ { "id": "41913", "postDate": "04/15/2014 05:37:47", "content": "

Here is some code and ideas to get a score of around 0.59347. It uses Python and Vowpal Wabbit, but the idea should work with other languages and other algorithms.

\n

Data reduction

\n

Check out the data reduction thread to understand what this is about. For this benchmark we create the category and company union.

\n

Feature engineering

\n

Feature engineering will be important in this competition, no matter the language or algorithms used. We can check out the benchmarks on the leaderboard to get a feel for features to use.

\n

We will generate the following features:

\n\n

These same features for:

\n\n

Combinations of these:

\n\n

Offer-related:

\n\n

Total shopper spend:

\n\n

Can you name some other possibly interesting features to generate?

\n

Vowpal Wabbit

\n

We train Vowpal Wabbit using quantile regression, 40 passes and a learning rate of 0.85. We turn the predictions into Kaggle's submission format.

\n

Feature visualisation

\n

Using the output from Vowpal Wabbit's wrapper vw-varinfo we generate the feature relevance plot below (code to generate this included):

\n

\"feature

\n

Improvements

\n

If you use this code or feature ideas to improve the score of 0.59347 I'd really appreciate it if you let me know these improvements. It's very possible that another algorithm may do better with these exact features.

\n

If you do improve the score I'd love to team up. I am still working on a few new features and we could combine our approaches for an ever better score.

\n

Code & Tutorial

\n

For latest code see the Github repo: https://github.com/MLWave/kaggle_acquire-valued-shoppers-challenge, for the tutorial/description read: http://mlwave.com/predicting-repeat-buyers-vowpal-wabbit/

\n

All of this runs in about 15 minutes for me, from raw data to submission. Takes less than 1GB of memory.

\n

Todo

\n

Generate more features. Set up local validation. Tweak algorithms. Try other algorithms than quantile regression (sklearn has plenty).

", "rawMarkdown": "", "votes": null }, { "id": "41962", "postDate": "04/15/2014 19:54:40", "content": "

So I noticed that the offer_quantity is equal to 1 for all offers of trainHistory, while it is sometimes equal to 2 for some of the offers in testHistory.  This explain why it is not a good feature as it is.  

\n

However, I can imagine someone getting a "good" feature out of this by probing the test data a few times with the kaggle score, though I doubt this is a good use of one's time.  

", "rawMarkdown": "", "votes": null }, { "id": "41978", "postDate": "04/15/2014 22:58:15", "content": "

[quote=Mathieu Cliche;41962]

\n

So I noticed that the offer_quantity is equal to 1 for all offers of trainHistory, while it is sometimes equal to 2 for some of the offers in testHistory.

\n

[/quote]

\n

Astute observation! I probably should have checked that myself. Seemed such a nice feature. I think it is useless now.

", "rawMarkdown": "", "votes": null }, { "id": "42021", "postDate": "04/16/2014 10:41:50", "content": "

Great job!

\n

good start!

", "rawMarkdown": "", "votes": null }, { "id": "42047", "postDate": "04/16/2014 14:49:38", "content": "

Triskelion

\n

Your thoughts of creating new features from the transaction data are absolutely right. One note though. Brand is associated with category and company, So, if a shopper buys a brand, he/she buys the category, and the company as well. So, you can combine these three into one feature. To be more accurate, I think category and company could be left out, because the offers are made on brands, not categories or companies. I think a new binary feature, that indicates if a shopper has purchased the brand prior to the offer, would be sufficient. In other words, a shopper who has prior exposure to the brand, is more likely to respond to the offer, than the shopper who has not had the exposure. 

", "rawMarkdown": "", "votes": null }, { "id": "42056", "postDate": "04/16/2014 15:47:52", "content": "

[quote=Art_Ghazaryan;42047]

\n

Triskelion

\n

Your thoughts of creating new features from the transaction data are absolutely right. One note though. Brand is associated with category and company, So, if a shopper buys a brand, he/she buys the category, and the company as well. So, you can combine these three into one feature. To be more accurate, I think category and company could be left out, because the offers are made on brands, not categories or companies. I think a new binary feature, that indicates if a shopper has purchased the brand prior to the offer, would be sufficient. In other words, a shopper who has prior exposure to the brand, is more likely to respond to the offer, than the shopper who has not had the exposure. 

\n

[/quote]

\n

No.

\n

Brand is not associated with category (with company?). A sparkling water bottle is a category. A company can make many brands but all brands do not belong to one company. Brand is a trademark of a given company. If you buy kinley still mineral water, you are buying from department = water, category = still water, brand = kinley, company = coca cola

", "rawMarkdown": "", "votes": null }, { "id": "42058", "postDate": "04/16/2014 15:54:47", "content": "

[quote=Triskelion;41913]

\n

Here is some code and ideas to get a score of around 0.59347. It uses Python and Vowpal Wabbit, but the idea should work with other languages and other algorithms.

\n

[/quote]

\n\n

Do you get this same (or similar) score when doing crossvalidation on your end, or do you get a much higher one?

\n

It seems to me that an important factor in this competition is generating a validation set that actually works. I had tried using similar features to the ones you described and got a score of ~0.67 in CV, but only 0.55 in the public leaderboard. For my best submission (currently 0.588), I get ~0.60 in CV, which makes this really confusing.

", "rawMarkdown": "", "votes": null }, { "id": "42059", "postDate": "04/16/2014 16:12:32", "content": "

Abhishek,

\n\n

When you get a coupon, it is for a specific brand. You cannot redeem that coupon to get another brand of the same company, or category, or department.

\n

There is also another issue here. What type of association we have between each shopper and coupons? Is it one to one? Were the coupons provided as a collection(coupon book, newspaper insert, etc), or they were provided in some other way?

", "rawMarkdown": "", "votes": null }, { "id": "42069", "postDate": "04/16/2014 17:20:10", "content": "

[quote=Martin Martin;42058]

\n

[quote=Triskelion;41913]

\n

Here is some code and ideas to get a score of around 0.59347. It uses Python and Vowpal Wabbit, but the idea should work with other languages and other algorithms.

\n

[/quote]

\n

Do you get this same (or similar) score when doing crossvalidation on your end, or do you get a much higher one?

\n

[/quote]

\n

I get a higher one. (~3-5 AUC higher) I think this may be due to outliers in the train set (and test set). I suspect that if you denoise the train and test set, your model will be more robust, and your validation scores are then closer to the leaderboard.

\n

I am familiar with removing noisy features (PCA, topkfeatures etc.), not too familiar with removing noisy feature values (more like fraud detection). EDIT: This thread seems to suggest it is not due to noise/outliers, but due to temporal issues. I think that means in 5-year-old speak that the train set and test set are fairly different from each other. CV on only the train set will be higher than public leader board score, as the train set itself is more internally consistent. This may be a hard problem to solve, though an increase in local CV may very well mean an increase in public leaderboard score.

\n

I'll take a better look soon, for now I have a few other competitions that demand my attention.

", "rawMarkdown": "", "votes": null }, { "id": "42072", "postDate": "04/16/2014 17:25:33", "content": "

[quote=Art_Ghazaryan;42059]

\n

Abhishek,

\n

When you get a coupon, it is for a specific brand. You cannot redeem that coupon to get another brand of the same company, or category, or department.

\n

There is also another issue here. What type of association we have between each shopper and coupons? Is it one to one? Were the coupons provided as a collection(coupon book, newspaper insert, etc), or they were provided in some other way?

\n

[/quote]

\n

Hey Art Ghazaryan. I think you may be right. Some features are superfluous. Has_never_bought_brand_company would probably be the same as has_never_bought_brand. As it is unlikely that another company creates the same brand.

\n

I was not too sure of this beforehand. I just extracted some features and ran it through Vowpal Wabbit. I don't like relying on domain expertise too much (as I have very little in all domains). Basically I hope I use tools that are good enough to discard useless features and focus only on the indicative features. Vowpal Wabbit is pretty good at that, sklearn has some algorithms that are better. Most of them need feature values to be scaled between 0-1 and demand all data to be loaded into memory. While a bit more work, this may reduce the hard-to-normalize noise in the data sets.

", "rawMarkdown": "", "votes": null }, { "id": "42125", "postDate": "04/17/2014 02:09:35", "content": "

6,089 out of 35689 brands belong to more than one company ... I am very confused ...

", "rawMarkdown": "", "votes": null }, { "id": "42142", "postDate": "04/17/2014 05:51:48", "content": "

[quote=Y Zhao;42125]

\n

6,089 out of 35689 brands belong to more than one company ... I am very confused ...

\n

[/quote]

\n

My guessing is because in the data description they say:

\n

'The transactions file can be joined to the offers file by (category, brand, company).'

\n

We should consider category, brand, company as a key, so different brands for different companies can have the same code but they are actually different brands.

", "rawMarkdown": "", "votes": null }, { "id": "42326", "postDate": "04/18/2014 15:46:32", "content": "

[quote=Mike Demurtas;42142]

\n

My guessing is because in the data description they say:

\n

'The transactions file can be joined to the offers file by (category, brand, company).'

\n

We should consider category, brand, company as a key, so different brands for different companies can have the same code but they are actually different brands.

\n

[/quote]

\n

I'd really love to get an answer from the competition admin on this-- do we need to combine company and category to get a unique brand id?

", "rawMarkdown": "", "votes": null }, { "id": "42328", "postDate": "04/18/2014 16:38:07", "content": "

My guess is this is due to corporate actions. I know there are quite a few food companies got acquired and spin-off-ed during the past two years, (Kraft, white plain??, etc). I probably will take a look at the time series of brand/company instances to verify my guess.

", "rawMarkdown": "", "votes": null }, { "id": "42346", "postDate": "04/18/2014 19:46:58", "content": "

This is the answer I received from the sponsor admin (who is not at their computer today):

\n\n

The combination of “category, brand, company” could be consider a unique key.

\n

Brands are unique to companies, but brands-company combinations can cross categories.

", "rawMarkdown": "", "votes": null }, { "id": "42356", "postDate": "04/18/2014 21:43:30", "content": "

[quote=Ramzi R;42346]

\n

Brands are unique to companies, ...

\n

[/quote]

\n

I think we know this statement is false, right? Is there a problem with the data?

", "rawMarkdown": "", "votes": null }, { "id": "42557", "postDate": "04/21/2014 00:07:06", "content": "

The sponsor admin's remark is in line with what Mike wrote before:

\n

[quote=Mike Demurtas;42142]

\n

We should consider category, brand, company as a key, so different brands for different companies can have the same code but they are actually different brands.

\n

[/quote]

\n\n

This is also supported by the observation that the brand occuring for the largest number of companies is brand '0', which occurs for 9150 companies (making other explanations like corporate actions very unlikely).

", "rawMarkdown": "", "votes": null }, { "id": "42563", "postDate": "04/21/2014 01:48:15", "content": "

Are there similar duplicates of companies between categories?  Or since companies can have brands in different categories, are all company IDs unique?

", "rawMarkdown": "", "votes": null }, { "id": "42622", "postDate": "04/21/2014 17:32:45", "content": "

Point of clarification:  it is possible for brands to have more than 1 (one) company. 

\n

The combination of "category, brand, company" can be considered a unique key.

", "rawMarkdown": "", "votes": null }, { "id": "42626", "postDate": "04/21/2014 18:18:27", "content": "

First of all, thank you for submitting your code.  This submissions is better than anything I've been able to put together on my own, after many hours of work.

\n\n

Some questions:

\n

1. Why did you choose to use quantile regression?  Isn't this problem a perfect fit for the logistic regression function in vowpal wabbit?

\n

2. Why 0.6 for the quantile tau?  Did you try any other values, and if so, how did you decide if they were good or bad?

\n

3. Why 40 passes?  Did you try any other values, and if so, how did you decide 40 was optimal?

\n

4. Why 0.85 for the learning rate?

\n

5. Are you worried at all about over-fitting?  I noticed that the self-reported loss for VW was 0 for a very long time before the model stopped.  Should you have stopped the model after some number of passes where the self-reported loss was still about zero?

\n

Questions 2-4 are basically the same question: how did you tune the model prior to submission, and how confident are you in your tuning?

\n\n

Thanks again for the code, you've helped me out immensely.

", "rawMarkdown": "", "votes": null }, { "id": "43523", "postDate": "05/02/2014 18:32:43", "content": "

[quote=Zach;42626]

\n

Some questions:

\n

1. Why did you choose to use quantile regression?  Isn't this problem a perfect fit for the logistic regression function in vowpal wabbit?

\n

[/quote]

\n

It probably is a good fit (if it is better, it very well could be: try it!). Langford says that for binary classification problems Vowpal Wabbit usually performs well using the --binary --loss_function logistic operator. This demands labels to be [-1,1] instead of [0,1]

\n\n

But I am not actually doing classification. I am doing a regression between 0 and 1. This will skip the need to post-process the predictions (running through a sigmoid etc.). They already score good for AUC/ROC.

\n\n

[quote=Zach;42626]

\n

2. Why 0.6 for the quantile tau?  Did you try any other values, and if so, how did you decide if they were good or bad?

\n

[/quote]

\n

I always try the standard 0.5 first. There are ways to find out which parameter is the best (its an obvious tweak when using quantile regression), but the easiest for me was just generating a submission with quantile tau 0.4 and 0.6 and see if it improves or decreases the score. You obviously would use local validation and performance evaluation for this if you were to do this properly.

\n

[quote=Zach;42626]

\n

3. Why 40 passes?  Did you try any other values, and if so, how did you decide 40 was optimal?

\n

[/quote]

\n

This used to be based on a hunch more than anything (or I just did not know how to find out, I just tried 20 passes as the standard that just seems to work alright, fastml confirms). The newer version of Vowpal Wabbit has holdout features build-in. This means one in n samples is used to calculate the loss now (standard 1 in 10). Quantile regression seems to need fewer passes to converge to a good solution in general. With new holdout functionality Vowpal Wabbit will stop doing passes if it does not increase performance over the last n passes. So that is why Vowpal Wabbit 7.6 will increase the score on leaderboard. Maybe it only does 9 passes and finds that is optimal over 40 passes. This, you could say, is Vowpal Wabbit's build-in overfitting mechanism.

\n

[quote]

\n

4. Why 0.85 for the learning rate?

\n

[/quote]

\n

This caused a lower average loss. You can manually check parameters like these (--decay_learning_rate, --power_t, l). Usually you don't even need to fiddle with it (it can also mess up your results), but it can improve the score a bit. BTW: From https://raw.githubusercontent.com/wiki/gdfm/vowpal_wabbit/v5.1_tutorial.pdf "for multiple passes --decay_learning_rate between [0.5-1] is sensible. Values smaller than 1 protect against overfitting."

\n

[quote]

\n

5. Are you worried at all about over-fitting?  I noticed that the self-reported loss for VW was 0 for a very long time before the model stopped.  Should you have stopped the model after some number of passes where the self-reported loss was still about zero?

\n

[/quote]

\n

Not with Vowpal Wabbit and with this dataset. Especially not with the new holdout functionality. If you are really worried about over-fitting you could add l1 and l2 regularization with --l1 and --l2. And try ensemble learning to reduce fitting.

\n

[quote]

\n

Questions 2-4 are basically the same question: how did you tune the model prior to submission, and how confident are you in your tuning?

\n

[/quote]

\n

I check average loss and if the learning process looks ok: (average loss and since last decreasing). This is the quick way, because you don't need to set up a holdout set or do k-fold validation, average loss is a pretty decent indicator of leaderboard performance. 

\n

Debug process from the 5.1 tutorial:

\n\n

If you want more precise way: Create a holdout set or two from your train set. Evaluate AUC on the holdout set(s). Parameter tweak (i try bruteforce gridsearch or random search) according to this fitness factor.

\n

You can also employ PERF software to find good cutt-off and min_prediction and max_prediction I think, but I have not figured that out yet. 

\n

I may even output in libsvm format so you can do most of this with sklearn or try other solvers like Sofia-ML. 

\n

[quote]

\n

Thanks again for the code, you've helped me out immensely.

\n

[/quote]

\n

No problem! Happy competition!

", "rawMarkdown": "", "votes": null }, { "id": "43657", "postDate": "05/04/2014 09:03:23", "content": "

As for the choice of the loss function, I tried Triskelions features with all 4 loss functions implemented in Vowpal Wabbit. At first, I also thought logistic regression might be a better fit. Quantile regression scored highest on the leaderboard though.
What kind of problem we are solving is actually not that important afaik, at least in terms of the numbers you want to submit. You can create scores in any range, not just [0,1]. AUC can handle that.

", "rawMarkdown": "", "votes": null }, { "id": "44253", "postDate": "05/08/2014 10:59:41", "content": "

Hi Triskelion,

\n

Many thanks for the detailed explanation. Really impressed with your knowledge.

\n

Being novice, i am participating in this contest and I am trying to create new features in SAS/R so just wanted to check if what I understood from these posts is correct or not. I think one should try to create new features like How many times person bought from a company/category on offer, what is the total amount amount transacted on company/category etc. Now my question is, are you guys suggesting that we should have this count of transactions/amount of transaction variables for each company present in the offers.csv i.e we should have no_of_trxn_comp1, no_of_trxn_comp2, no_of_trxn_comp3, no_of_trxn_comp4, no_of_trxn_comp5 and so on for each company until comp18 that we have in offers.csv (18 companies) and similarly for 20 categories  in offers.csv file. This way we will so many variables for company and categories. In total my question is that we need to create the variables for each company for whatever parameter (i.e amount, count etc.) we are creating. 

\n

Anyone's help on this would be really appreciated.  Thanks in advance.

", "rawMarkdown": "", "votes": null }, { "id": "44259", "postDate": "05/08/2014 14:24:32", "content": "

Hi Decipher

\n\n

Your thinking process is absolutely correct. That is a correct approach to extract intelligence and create variables. I have created 114 variables so far.

", "rawMarkdown": "", "votes": null }, { "id": "44273", "postDate": "05/08/2014 18:22:38", "content": "

@Decipher

\n

Well you do need to calculate all those, but in terms of features to train on you only use the variables for the specific offer.

\n

So if an offer is on company XX you would use the no_of_trxn_compXX variable as a feature. Hope this helps.

", "rawMarkdown": "", "votes": null }, { "id": "44396", "postDate": "05/12/2014 04:24:38", "content": "

Thanks for your sharing, it's a good start

", "rawMarkdown": "", "votes": null }, { "id": "44410", "postDate": "05/12/2014 16:07:34", "content": "

Did anyone use exactly the same features with logistic regression?

\n

My auc using logistic regression is only 0.53, I'm checking whether there is something wrong with my feature extraction process, or it's due to logistic regression.

", "rawMarkdown": "", "votes": null }, { "id": "44411", "postDate": "05/12/2014 16:10:52", "content": "

[quote=身经百战长者;44410]

\n

Did anyone use exact the same features with logistic regression?

\n

My auc using logistic regression is only 0.53, I'm checking whether there is something wrong with my feature extraction process, or is due to logistic regression.

\n

[/quote]

\n\n

Yes.  Logistic regression using vowpal wabbit is a fair bit worse than quantile regression using vowpal wabbit, and logistic regression using another tool seems to be even worse.

\n

Try quantile regression with your features and see what happens.

\n

There's already a discussion on this topic here.

", "rawMarkdown": "", "votes": null }, { "id": "44412", "postDate": "05/12/2014 16:16:20", "content": "

Thanks for reply

\n

Are there any tools in python or R for quantile regression? It seems that sklearn does not implement quantile regression

", "rawMarkdown": "", "votes": null }, { "id": "44413", "postDate": "05/12/2014 16:17:21", "content": "

statsmodels

", "rawMarkdown": "", "votes": null }, { "id": "44414", "postDate": "05/12/2014 16:18:47", "content": "

I'm actually using logistic regression with scikit-learn and roughly the same features, and for me it's working well, but you need to use some sort of regularization to avoid over-fitting.

", "rawMarkdown": "", "votes": null }, { "id": "44429", "postDate": "05/13/2014 03:23:56", "content": "

But I'm still a little confused about applying a regression algorithm(like quantile regression) to a classification problem...

", "rawMarkdown": "", "votes": null }, { "id": "44441", "postDate": "05/13/2014 08:22:00", "content": "

Can you please explain your feature extraction process?

", "rawMarkdown": "", "votes": null }, { "id": "44451", "postDate": "05/13/2014 12:55:08", "content": "

It generate the features provided by Triskelion

", "rawMarkdown": "", "votes": null }, { "id": "44479", "postDate": "05/13/2014 18:50:06", "content": "

@身经百战长者

\n

> It seems that sklearn does not implement quantile regression.

\n

> ... Confused about using regression for a classification problem.

\n

See: sklearn.ensemble.GradientBoostingRegressor which has quantile loss.

\n

You can treat a classification problem as a regression problem, if you regress between 0 (the negative class) and 1 (the positive class). I don't know if this is unorthodox, probably it is -- It looks like standard sklearn in-memory logistic regression with these features (and some regularization) already outperforms this benchmark.

", "rawMarkdown": "", "votes": null }, { "id": "44513", "postDate": "05/14/2014 10:14:32", "content": "

Hmm, but in a machine learning MOOC course   . When introducing logistic regression (the first video lecture of logistic regression), the professor showed how bad it is to apply OLS linear regression to a classification problem.

\n

So, I'm amazed that quantile regression can produce good result.

", "rawMarkdown": "", "votes": null }, { "id": "44562", "postDate": "05/14/2014 22:58:37", "content": "

@身经百战长者

\n

>So, I'm amazed that quantile regression can produce good result.

\n

Try all, keep the best. Sometimes theoretical knowledge on what is possible or accepted practice can be a burden. Most of the time it is beneficial though. Don't be afraid to experiment or break some rules and you got the best of both worlds!

\n

See: Forum thread and FastML experiment for an opposite example: someone using classification for a regression problem, and ending up in 3rd position.

", "rawMarkdown": "", "votes": null }, { "id": "44796", "postDate": "05/18/2014 23:30:35", "content": "

@身经百战长者

\n

I am following the MOOC you are referring to. Here is how I see the problem.

\n

If I remember correctly the professor explained that linear regression is not good for classification because it is very sensitive to outliers. What linear regression is doing is to estimate the conditional mean so if you add a data point far away from the others this will greatly affect the value of the mean.

\n

According to Wikipedia quantile regression aims at estimating either the conditional median or other quantiles of the response variable.

\n

Now if your features are quite dense around e.g. the median then adding features far away will not affect a lot the value of the median.

\n

If anyone has a better understanding of this please share!

", "rawMarkdown": "", "votes": null }, { "id": "47421", "postDate": "06/01/2014 01:05:24", "content": "

Dear Triskelion and all friends,

\n

First, thanks for the provided code. Great job!

\n

I am new to Python and I just start this competition.  Previously, I mostly work with R, Matlab and C. Today, I run your posted python code on https://github.com/MLWave/kaggle_acquire-valued-shoppers-challenge, but run into some errors.

\n

My environment is Python 3.4.1 IDLE ( p.s., what's your python programming environment for this posted Python code?). When I run your posted code, I run into two following errors:

\n

1) invalid syntax at line "print e, reduced, datetime.now() - start". I then changed this line to "print (e, reduced, datetime.now() - star)", and then it doesn't report error, but I am not quite sure whether this indeed solve the problem.

\n

2)  after fixing the invalid syntax error, I run the code again, and it showed following error:

\n

Traceback (most recent call last):
File "D:\\ML_DM Competition\\acquire_shopper_challenge\\gen_vw_features.py", line 231, in

\n

<module> reduce_data(loc_offers, loc_transactions, loc_reduced)

\n

File "D:\\ML_DM Competition\\acquire_shopper_challenge\\gen_vw_features.py", line 41, in

\n

reduce_data  outfile.write( line ) #print header

\n

TypeError: 'str' does not support the buffer interface

\n

I have no idea about how to fix this problem. Would you be so kind to give me some hint to solve this problem? Thanks so much!

\n

Besides, I did following things to run the python code. Correct me if I make some stupid mistakes.

\n

1) I install Python 3.4.1 IDLE in C disk.

\n

2) the python code and data files are located in D disk.

\n

3) I start the Python 3.4.1 IDLE, and use open option to open your python code, and use Run Modules option in the IDLE to run the python code, and I run into these two errors.

\n

Any help are pretty appreciated. Thanks!

\n

Best wishes,

\n

Shize

", "rawMarkdown": "", "votes": null }, { "id": "47422", "postDate": "06/01/2014 01:21:38", "content": "

Shize, there are two major versions of python, 3.x and 2.x. Try python 2.7. The different versions have quite different syntax.

", "rawMarkdown": "", "votes": null }, { "id": "47423", "postDate": "06/01/2014 01:29:08", "content": "

Hi Scott,

\n

Thanks for your reply. Besides, does Python 2.7 also have GUI IDLE programming environment? You see, I prefer the user-friendly GUI IDLE environment rather than a DOS-like terminal environment. Thanks, and have a great night!

\n

Best wishes,

\n

Shize

", "rawMarkdown": "", "votes": null }, { "id": "47424", "postDate": "06/01/2014 02:08:22", "content": "

Dear Scott,

\n

I download Python 2.7.6 and install it on my laptop. Now the Python code works. Thanks so much, and have  a great night!

\n

Best wishes,

\n

Shize

", "rawMarkdown": "", "votes": null }, { "id": "48413", "postDate": "06/01/2014 03:19:55", "content": "

Dear friends,

\n

I run into another problem when run the posted Python code of Triskelion in Python 2.7.

\n

Actually, when I run the code, it generate the reduce.csv correctly, but when it run the generat_features function, it throw out an error about date format. The error is as follows:

\n

Traceback (most recent call last):
File "D:\\ML_DM Competition\\acquire_shopper_challenge\\gen_features_updated.py", line 232, in <module>

\n


generate_features(loc_train, loc_test, loc_reduced, loc_out_train, loc_out_test)
File "D:\\ML_DM Competition\\acquire_shopper_challenge\\gen_features_updated.py", line 162, in generate_features

\n


date_diff_days = diff_days(row[6],history[-1])
File "D:\\ML_DM Competition\\acquire_shopper_challenge\\gen_features_updated.py", line 57, in diff_days

\n


a = datetime.strptime(s1, date_format)
File "C:\\Python27\\lib\\_strptime.py", line 325, in _strptime
(data_string, format))

\n


ValueError: time data ''6/27/2013'' does not match format '%Y-%m-%d'

\n

Trying to fix this error, I changed the line of Python code "date_format = "%Y-%m-%d"  to "date_format = "%m/%d/%Y", and run the Python code again.

\n

But this time I run into another date format error as follows:

\n

ValueError: time data '2012-03-17' does not match format '%m/%d/%Y'

\n

So it seems that there are more than one kind of date format in the data. I don't have any ideas to solve this problem. Would anyone be so kind to provide some hints? Thanks so much!

\n\n

Best wishes,

\n

Shize

", "rawMarkdown": "", "votes": null }, { "id": "48416", "postDate": "06/01/2014 04:03:04", "content": "

try:

\n

    first date format

\n

except ValueError:

\n

    second date format

", "rawMarkdown": "", "votes": null }, { "id": "48417", "postDate": "06/01/2014 04:07:48", "content": "

Dear Neil,

\n

Thanks!

\n\n

I am new to Python, and I am not quite sure how to code in Python about your suggestion of 

\n

try:

\n

first date format

\n

except ValueError:

\n

second date format

\n\n

To be specific, how should I change the Python code? Thanks so much, and good night!

\n\n

Best wishes,

\n

Shize

", "rawMarkdown": "", "votes": null }, { "id": "48418", "postDate": "06/01/2014 04:23:47", "content": "

I suggest you read the python documentation on python handling of errors.

\n

https://docs.python.org/2/tutorial/errors.html

\n

What you can do is catch the exception and execute a different piece of code instead of throwing the exception.

\n

I have not looked at the code you are having trouble with but it appears you are getting the exception at the line

\n

a = datetime.strptime(s1, date_format)

\n

so you want to put the try/except clause around this, but you want to change the date_format when you catch the exception. date_format appears to be set in another part of the script, so you need to reset this after you catch the exception. I would set date_format within the try/except clause for clarity, so the code would look something like this (put the most common date format first).

\n

try:

\n

    date_format = "%Y-%m-%d

\n

    a = datetime.strptime(s1, date_format)

\n

except ValueError:

\n

    date_format = "%m/%d/%Y"

\n

    a = datetime.strptime(s1, date_format)

\n\n

Edit: I now see that "a" is set within the one of the python modules. I cannot see all of your code, but you need to traceback to where in your code the exception is thrown and the date_format is set, and wrap the try/except clause around it there.

", "rawMarkdown": "", "votes": null }, { "id": "48419", "postDate": "06/01/2014 04:33:18", "content": "

Dear Neil,

\n\n

Thanks for your suggestion. It works now. Good night!

\n\n

Best wishes,

\n

Shize

", "rawMarkdown": "", "votes": null }, { "id": "48420", "postDate": "06/01/2014 04:55:11", "content": "

Dear Friends,

\n

I use Python 2.7.6 and VM 7.6 (Latest version) to run Triskelion's Python code, but I got a leader board score of 0.59104 rather than 0.59410. And I found that there are a couple of other competitors also have  such same score 0.59104. 

\n\n

Does anyone has ideas about what might cause such difference? Thanks in advance.

\n\n

Good night!

\n\n

Best wishes,

\n

Shize

", "rawMarkdown": "", "votes": null }, { "id": "48437", "postDate": "06/01/2014 16:36:16", "content": "

[quote=Shize Su;48420]

\n

VM 7.6 (Latest version) 

\n

[/quote]

\n

Use VW version 7.1

", "rawMarkdown": "", "votes": null }, { "id": "48439", "postDate": "06/01/2014 16:56:22", "content": "

Really?

\n

But Zach said that he used VW version 7.6 and got score of 0.59410. See the post at 

\n

http://www.kaggle.com/c/acquire-valued-shoppers-challenge/forums/t/7908/0-59410

\n

Thanks!

\n

Best wishes,

\n

Shize

", "rawMarkdown": "", "votes": null }, { "id": "48448", "postDate": "06/01/2014 21:17:08", "content": "

Valid point!  I tested my code on version 7.6 and only got 0.59104.

", "rawMarkdown": "", "votes": null }, { "id": "48450", "postDate": "06/01/2014 21:43:29", "content": "

Hi friends,

\n

I download VW 7.1 and test the code again, but still got the same score 0.59104. 

\n

Dear Odessa, are you sure that you run the code with VW 7.1 and got a score of 0.59410, while you run the same code with VW 7.6 and got a score of 0.59104? I thought there should be some other contributing factors for such difference.

\n\n

Thanks, and have a great day!

\n\n

Best wishes,

\n

Shize

", "rawMarkdown": "", "votes": null }, { "id": "48451", "postDate": "06/01/2014 22:00:53", "content": "

No, with VW 7.1 I got only 0.59347, not 0.59410. I use 64 bit version of Cygwin for both.

", "rawMarkdown": "", "votes": null }, { "id": "48453", "postDate": "06/01/2014 22:06:46", "content": "

Hi Odessa,

\n

I also use 64 bit version Cygwin for both, and I got score 0.59104 for both VW 7.1 and VW 7.6.    I have no idea what might cause such difference....

\n

Anyway, thanks!

\n

Best wishes,

\n

Shize

", "rawMarkdown": "", "votes": null }, { "id": "48454", "postDate": "06/01/2014 22:28:14", "content": "

[quote=Shize Su;48420]

\n

Does anyone has ideas about what might cause such difference? Thanks in advance.

\n

[/quote]

\n

The difference between Python 2.7 and 3.x should be in changing a few print statements. 0.59410 is caused by newer version of Vowpal Wabbit (equal to using no passes in 7.1 version I think). Since 7.4 VW has holdout functionality and stops after doing 4 passes or so.

\n

Try running any version without doing any passes at all.

", "rawMarkdown": "", "votes": null }, { "id": "48457", "postDate": "06/01/2014 23:34:27", "content": "

Hi Triskelion,

\n

I tried VW 7.1 with no pass just now but got a score of 0.58717.

\n

As I said in my previous post, I actually once tried the latest version of VW, namely, VW 7.6, but got a score of 0.59104 rather than 0.59410. This is what make me confused, since Zach said that he used VW 7.6 and got a score of 0.59410.

\n

Any other thoughts?

\n\n

Thanks, and have a great day!

\n\n

Best wishes,

\n

Shize

", "rawMarkdown": "", "votes": null }, { "id": "48473", "postDate": "06/02/2014 13:21:57", "content": "

[quote=Shize Su;48439]

\n

Really?

\n

But Zach said that he used VW version 7.6 and got score of 0.59410. See the post at 

\n

http://www.kaggle.com/c/acquire-valued-shoppers-challenge/forums/t/7908/0-59410

\n

Thanks!

\n

Best wishes,

\n

Shize

\n

[/quote]

\n\n

I'm pretty certain I was using 7.6.  Did you run the code exactly as written, or did you make any changes?

\n\n

Try with both 7.1 and 7.6 and see what happens.

", "rawMarkdown": "", "votes": null }, { "id": "48476", "postDate": "06/02/2014 13:49:05", "content": "

I'm finding I need to change the learning rate significantly with recent VW builds, in order to get similar results results to a VW version pulled a couple weeks ago.  My guess is folks following the tutorial with very recent builds are going to get very different results from those originally posted.

\n\n

More specifically, with recent builds I seem to get similar results with 0.5 as a learning rate, vice the original recommendation of 0.85.  Unfortunately the loss function seems to be less of a reliable indicator with VW now exiting / converging much earlier than in previous versions.

", "rawMarkdown": "", "votes": null }, { "id": "48482", "postDate": "06/02/2014 14:39:54", "content": "

Hi Zach, 

\n

I run the code exactly as written, except that I change the paths for the data files (e.g., I change loc_offers = "data/offers.csv" to  loc_offers = "offers.csv".)

\n\n

I don't think such change about the data file paths would cause any difference about the accuracy, so I am a bit confused about what might cause the difference.

\n\n

I do have tried both VW 7.1 and VW 7.6, and they gave me exactly same score 0.59104.

\n\n

Thanks, and have a great day!

\n\n

Best wishes,

\n

Shize

", "rawMarkdown": "", "votes": null }, { "id": "48485", "postDate": "06/02/2014 14:46:50", "content": "

Hi Phil,

\n\n

I tried learning rate 0.5 with VW 7.6 just now, but it gave me a score of  0.58984. What's the version of your VW?

\n

Thanks!

\n\n

Best wishes,

\n

Shize

", "rawMarkdown": "", "votes": null }, { "id": "48486", "postDate": "06/02/2014 14:49:05", "content": "

0.59104 is also good. You might wanna improve on that.

", "rawMarkdown": "", "votes": null }, { "id": "48487", "postDate": "06/02/2014 14:53:09", "content": "

[quote=Shize Su;48482]

\n

Hi Zach, 

\n

I run the code exactly as written, except that I change the paths for the data files (e.g., I change loc_offers = "data/offers.csv" to  loc_offers = "offers.csv".)

\n

I don't think such change about the data file paths would cause any difference about the accuracy, so I am a bit confused about what might cause the difference.

\n

I do have tried both VW 7.1 and VW 7.6, and they gave me exactly same score 0.59104.

\n

Thanks, and have a great day!

\n

Best wishes,

\n

Shize

\n

[/quote]

\n\n

Interesting.  For what it's worth, I'm on a mac.  0.59104 is the score a lot of other people got with the earlier version of VW, and they may have been on windows.

", "rawMarkdown": "", "votes": null }, { "id": "48488", "postDate": "06/02/2014 14:55:30", "content": "

Hi Abhishek,

\n

I do agree that 0.59104 is good. And currently I don't care about the performance at all since I just start this competition. In fact,  I am just curious about what might cause such difference, since everything seems to be pretty clear.  It seems interesting for me!

\n

Thanks,

\n

Shize

", "rawMarkdown": "", "votes": null }, { "id": "48489", "postDate": "06/02/2014 14:57:52", "content": "

Hi Zach,

\n

You're right, I do run the code on Windows. Both VW 7.1 and VW 7.6 gave me same score 0.59104. So based on this, maybe the difference between 0.59410 and 0.59104 is simply caused by Mac and Windows? Seems pretty interesting!

\n

Best wishes,

\n

Shize

", "rawMarkdown": "", "votes": null }, { "id": "48490", "postDate": "06/02/2014 15:05:54", "content": "

I'm on a mac too and my score with VW in the begining was also 0.59410

", "rawMarkdown": "", "votes": null }, { "id": "48494", "postDate": "06/02/2014 15:26:24", "content": "

[quote=Shize Su;48485]

\n

Hi Phil,

\n

I tried learning rate 0.5 with VW 7.6 just now, but it gave me a score of 0.58984. What's the version of your VW?

\n

Thanks!

\n

[/quote]

\n\n

I'm using 7.6.1.  I'm also (mostly) on a Mac, although I'm using Windows and Linux builds as well.

\n

WRT to the score you got - yeah, I haven't quite matched my 7.6 results with 7.6.1.  For now I'm using a 7.6 build for most of my VW work.  0.85 doesn't give me good results in 7.6.1, so anyone using 7.6.1 may want to play with the learning rate a bit.

\n

I suppose my main point is that VW changes often.  People following the (excellent) MLWave install tutorial for Windows are going to end up with brand new code which may or may not behave very differently.

", "rawMarkdown": "", "votes": null }, { "id": "48495", "postDate": "06/02/2014 15:27:55", "content": "

AFAIK, Triskelion is using Windows.

", "rawMarkdown": "", "votes": null }, { "id": "48499", "postDate": "06/02/2014 16:19:24", "content": "

Hi Abhishek,

\n\n

Thanks for your provided information. But maybe Triskelion run the code to get score of 0.59347 a couple of weeks ago, and the reason posted by Phil in this posting might cause the difference between my result of 0.59104 and his result of 0.59347.

\n\n

And Phil, could you let me know your operation system (windows or mac?) and the version of VW you use? Thanks!

\n\n

Anyway, can anyone who uses Windows and VW 7.6 (or VW 7.1) verify that they run the same python code recently and get a score of 0.59410 (or 0.59347)?

\n\n

Thanks in advance.

\n\n

Best wishes,

\n

Shize

", "rawMarkdown": "", "votes": null }, { "id": "48514", "postDate": "06/02/2014 19:49:17", "content": "

It is possible that different builds of Vowpal Wabbit give slightly different results. As far as I see this is mostly irrelevant: focus on getting the best score for your build and features. The difference is small enough to be variance in public leaderboard. You can't say for sure which is better.

\n

The settings I gave were for a quick start, they are not the best settings possible.

\n

Vowpal Wabbit's hashing algorithm then also gives different results for little endian and big endian systems: https://github.com/JohnLangford/vowpal_wabbit/blob/master/vowpalwabbit/hash.cc

\n

For the record:

\n

7.1 64-bit Windows = ~0.59347

\n

7.6.1 64-bit Cygwin on Windows = ~0.59410

\n

@Shize Su, I don't know why you are getting 0.59347 with 7.6.1, most are getting 0.59410 with 7.6.1.

\n

I got my current score with 7.1 BTW using just small adjustments (mostly removing things I thought were fancy). I may share these when I optimize my current score some more (if I ever get time for this).

", "rawMarkdown": "", "votes": null }, { "id": "48515", "postDate": "06/02/2014 20:37:54", "content": "

Hi Triskelion,

\n

Thanks for your reply. Actually, I do agree with you that there is no significant difference between 0.59410 and 0.59104 at all. I was not caring about the score since I just start this competition; instead, I am just quite curious about what subtle reasons might cause such difference since everything seems to be that straight forward and clear for me. 

\n

Anyway, if no one can give an precise answer for this issue, I will just forget about it and start to work on my progress.

\n

P.S.: Triskelion, to clarify, I got score of 0.59104 on both VW 7.6 and VW 7.1, not 0.59347. I use windows operation system.

\n

Thanks, and have a great day!

\n

Best wishes,

\n

Shize

", "rawMarkdown": "", "votes": null }, { "id": "48516", "postDate": "06/02/2014 20:45:05", "content": "

Hmm, thanks.  I got wildly different leaderboard scores between 7.6 and 7.6.1 (0.59990 vs 0.59033) on the same data / VW settings / machine, and had assumed it was due to changes made to the learning algorithm, hence my warning.  However, if everybody else is getting very similar scores between versions I may need to dig deeper at some point and find out what's really going on.  :-)

\n\n

Edit: I was able to reproduce it just now - my current high score 0.60393 (IIRC) set, when run against the latest VW, gets ~0.594 or so.  Must be something I'm doing... but this one took place under sterile conditions, everything matched up correctly, so definitely not a one-off mistake.  Odd.

", "rawMarkdown": "", "votes": null }, { "id": "48527", "postDate": "06/02/2014 22:43:21", "content": "

Can somebody give me access to the reduced 1.6 GB file? I tried the Python script, but I always get a syntax error

\n\n

Kind regards,

\n

Alex

", "rawMarkdown": "", "votes": null }, { "id": "48535", "postDate": "06/03/2014 00:07:20", "content": "

I may be able to provide a link to a reduced file if necessary, but it'd probably be best to get you up and running - the script is *really* useful!  :-)  What kind of syntax error are you getting?

", "rawMarkdown": "", "votes": null }, { "id": "48541", "postDate": "06/03/2014 01:18:42", "content": "

Hi Alexander,

\n

Are you using Python version 3.* to run the code?

\n

Try to use Python version 2.*, for example, python 2.7 to run the code as written (except changing the paths for the data files), and there should not be any syntax errors.

\n

I once use python 3.4 to run the code and came across the same syntax error problems, and then as suggested by Scott, I turned to python 2.7, and then there were no syntax errors.

\n

Hope this would be helpful.

\n

Have a great night!

\n

Best wishes,

\n

Shize

", "rawMarkdown": "", "votes": null }, { "id": "48547", "postDate": "06/03/2014 05:25:42", "content": "

While VW is used widely for this completion , I  tried with my favorite scikit-learn. With few more features along with what is mentioned by Triskelion's benchmark code, I could fetch 0.59449 (little better than VW benchmark 0.59410). I used RF without any tuning. I am hoping for better scores with tuning the params.

\n\n

My point is, one shouldn't stuck with VW version issues while there are alternate options to try out.

", "rawMarkdown": "", "votes": null }, { "id": "48549", "postDate": "06/03/2014 07:03:23", "content": "

I also tried these both, so I can say that it has close results, so we hope to use them both

", "rawMarkdown": "", "votes": null }, { "id": "48555", "postDate": "06/03/2014 12:37:53", "content": "

[quote=Chitrasen;48547]

\n

While VW is used widely for this completion , I  tried with my favorite scikit-learn. With few more features along with what is mentioned by Triskelion's benchmark code, I could fetch 0.59449 (little better than VW benchmark 0.59410). I used RF without any tuning. I am hoping for better scores with tuning the params.

\n

My point is, one shouldn't stuck with VW version issues while there are alternate options to try out.

\n

[/quote]

\n

Agreed.  Would you mind posting your early-benchmark sklearn code somewhere to help folks get started down that path?  If it's really simple all the better.

\n

Edit: without the extra features, etc., of course.  Not asking you to give anything away, just the sklearn boilerplate.  :-)

", "rawMarkdown": "", "votes": null }, { "id": "48558", "postDate": "06/03/2014 13:20:46", "content": "

Hi Phil, Hi Shize

\n\n

I use Python 2.7 but I am a beginner. I changed the code to my home directory, where I saved my  csv.files:

\n

loc_offers = "offers.csv"
loc_transactions = "transactions.csv"
loc_reduced = "reduced2.csv" # will be created

\n

But after copy and paste the first line of code I get a syntax error:

\n

def reduce_data(loc_offers, loc_transactions, loc_reduced):  start = datetime.now()
  #get all categories on offer in a dict
  offers = {}
  for e, line in enumerate( open(loc_offers) ):
    offers[ line.split(",")[1] ] = 1

\n

Actually I need the 1.6GB csv for a test if I can read 1.6 GB file from S3 to Graphlabb Create:

\n

http://forum.graphlab.com/discussion/251/s3-sframe-read-problems

\n

Kind regards,

\n

Alex

", "rawMarkdown": "", "votes": null }, { "id": "48561", "postDate": "06/03/2014 14:07:14", "content": "

If you're doing this in the python console you might be better served by running the whole file all at once:

\n

run "gen_vw_features.py"

\n

Also, if you're not going to provide specific paths, make sure you're running python / ipython from the directory where the data files reside, or set the working directory in some other way.

\n\n

Lastly, here's a link to the reduced csv:

\n

https://dl.dropboxusercontent.com/u/12189505/reduced.csv.zip

\n\n

If anyone decides to put it somewhere more performant, let me know and I'll take the link down.  :-)

", "rawMarkdown": "", "votes": null }, { "id": "48595", "postDate": "06/04/2014 05:22:07", "content": "

[quote=Phil Culliton;48555]

\n\n

Agreed.  Would you mind posting your early-benchmark sklearn code somewhere to help folks get started down that path?  If it's really simple all the better.

\n

Edit: without the extra features, etc., of course.  Not asking you to give anything away, just the sklearn boilerplate.  :-)

\n

[/quote]

\n\n

Here is my skeleton code using Pandas and scikit-learn.

\n

Note: My original features have been removed from the code. Code is extracted from my ipython notebook, hence may not be in proper python coding standard. I am sorry for that.

\n\n\n\n

I also added the code to my blog.

", "rawMarkdown": "", "votes": null }, { "id": "48615", "postDate": "06/04/2014 11:39:05", "content": "

Chitrasen, thank you for code sharing. Did you do a cross validation via sklearn?

", "rawMarkdown": "", "votes": null }, { "id": "48616", "postDate": "06/04/2014 11:51:40", "content": "

[quote=rzykov;48615]

\n

Chitrasen, thank you for code sharing. Did you do a cross validation via sklearn?

\n

[/quote]

\n\n

Yes, I did some CV testing, but seems test and train data distributions are either different, or there seems some pattern I am missing to account into. I am getting similar results as mentioned in this thread.

\n\n

I was getting 4 fold CV score of 0.68XXX with the sample code, I provided. (my current CV score improved to 0.715).

", "rawMarkdown": "", "votes": null }, { "id": "48617", "postDate": "06/04/2014 12:05:09", "content": "

I do a cross validation over all offers. It is like K-fold CV where each fold is a an offer.

\n

Then calculate mean of CV's errors. It's very close to my test results.

", "rawMarkdown": "", "votes": null }, { "id": "48623", "postDate": "06/04/2014 12:49:27", "content": "

I just wanted to leave my contribution to this great, collaborative, thread.

\n

I used the whole 22Gb transaction data and ran svm + logistic regression with the features as suggested by Triskelion. The result was a 0.59735 score. Running the same set up with the reduced data set produced a 0.58627 score. Therefore, in this case, using more data improved the score by 0.01 . 

", "rawMarkdown": "", "votes": null }, { "id": "48630", "postDate": "06/04/2014 14:11:30", "content": "

[quote=Chitrasen;48595]

\n

Here is my skeleton code using Pandas and scikit-learn.

\n

[/quote]

\n

What score does this give on public leaderboard?

", "rawMarkdown": "", "votes": null }, { "id": "48632", "postDate": "06/04/2014 15:02:25", "content": "

[quote=Paulo;48623]

\n

I just wanted to leave my contribution to this great, collaborative, thread.

\n

I used the whole 22Gb transaction data and ran svm + logistic regression with the features as suggested by Triskelion. The result was a 0.59735 score. Running the same set up with the reduced data set produced a 0.58627 score. Therefore, in this case, using more data improved the score by 0.01 . 

\n

[/quote]

\n

Cool, this is a useful tip. I wonder why this gives so much of a difference though. Maybe the features that hold "total amount spend" become more indicative?

\n\n

I'll try these with my new settings and report back if it improves. I finally got a little time to optimize my current near-leaderboard score and try out some new features.

", "rawMarkdown": "", "votes": null }, { "id": "48669", "postDate": "06/04/2014 22:41:06", "content": "

[quote=Triskelion;48630]

\n

[quote=Chitrasen;48595]

\n

Here is my skeleton code using Pandas and scikit-learn.

\n

[/quote]

\n

What score does this give on public leaderboard?

\n

[/quote]

\n\n

The additional features like 'median price' and 'median amount' and some others worked better. My initial score with those features and the skeleton code, was 0.59449 on leaderboard.

\n

My current leaderboard score(0.60343) uses some more features and tuning of RF params.

", "rawMarkdown": "", "votes": null }, { "id": "48696", "postDate": "06/05/2014 09:20:46", "content": "

Only one question,

\n\n

how well the model classifies among repeaters (t) ?

", "rawMarkdown": "", "votes": null }, { "id": "48710", "postDate": "06/05/2014 12:21:09", "content": "

Hmm.  I tried your code (exactly as-is, with no changes), Chitrasen, and got a submission error regarding the ids - they were float64 and the submission code wanted them not to be.  I switched them to int64, and got a leaderboard score of 0.54190.  Wonder what's going on there?

", "rawMarkdown": "", "votes": null }, { "id": "48716", "postDate": "06/05/2014 13:24:45", "content": "

I think 0.54190 is OK for this code, it only has features that include only category and company information.

", "rawMarkdown": "", "votes": null }, { "id": "48721", "postDate": "06/05/2014 14:08:29", "content": "

[quote=Selfish Gene;48716]

\n

I think 0.54190 is OK for this code, it only has features that include only category and company information.

\n

[/quote]

\n

True, but I figured I'd ask to make sure!  Since newcomers might be using this code, I thought it would be useful to know what the approximate leaderboard score might be, and wanted to double-check my result.  Also, to report the id bug.  :-)

", "rawMarkdown": "", "votes": null }, { "id": "48747", "postDate": "06/06/2014 02:34:43", "content": "

I face this errors...

\n

Can anyone help me "how to solve this problem?"

\n

ValueError: time data '1' does not match format '%Y-%m-%d'

", "rawMarkdown": "", "votes": null }, { "id": "48755", "postDate": "06/06/2014 06:26:36", "content": "

[quote=Chitrasen;48669]
The additional features like 'median price' and 'median amount' and some others worked better. My initial score with those features and the skeleton code, was 0.59449 on leaderboard.
My current leaderboard score(0.60343) uses some more features and tuning of RF params.
[/quote]

\n

Hi Chitrasen, I used RF model with 5-fold CV on my training set and got score 0.675xx. But when I applied the same model on my test set, my prediction was just like random, score 0.49476. Did you see any similar thing? Do you have any suggestions? Thanks!

", "rawMarkdown": "", "votes": null }, { "id": "48758", "postDate": "06/06/2014 07:01:27", "content": "

[quote=Kien Trinh;48755]

\n

Hi CHi Chitrasen, I used RF model with 5-fold CV on my training set and got score 0.675xx. But when I applied the same model on my test set, my prediction was just like random, score 0.49476. Did you see any similar thing? Do you have any suggestions? Thanks!

\n

[/quote]

\n

I am also struggling to get a reliable CV testing technique :(

\n

Yes,I see the similar effect as you are getting. To put time effect on CV, I sort the training data with offerdate and then apply K-Fold CV. The CV scores vary from 0.58 to 0.68 and on the leader board it scored 0.572 ( I have used some of my features, not the features mentioned in my previous post on skeleton code).

", "rawMarkdown": "", "votes": null }, { "id": "48763", "postDate": "06/06/2014 08:59:34", "content": "

rzykov mentioned very good idea: to train on some offers and to test on others, for those who struggling to get a reliable CV technique it's worth trying. It should be better for sure then any standard CV.

", "rawMarkdown": "", "votes": null }, { "id": "48773", "postDate": "06/06/2014 12:45:17", "content": "

Christansen,

\n\n

only a very specific question that nobody answers me and it's very easy:

\n\n

there are 43.438 customers who repeated. Well, of these 43.438 customers, how you classify correctly?

\n\n

Kind regards

", "rawMarkdown": "", "votes": null }, { "id": "48807", "postDate": "06/06/2014 21:00:40", "content": "

[quote=mike;48773]

\n

Christansen,

\n

only a very specific question that nobody answers me and it's very easy:

\n

there are 43.438 customers who repeated. Well, of these 43.438 customers, how you classify correctly?

\n

Kind regards

\n

[/quote]

\n

Mike - you are doing binary classification {0 or 1}. There are 43438 outcomes { 1 } on training set. You are predicting the probability a shopper will be come a loyal customer { 1 } on test set. You submission file is a list of probability in a range [0,1]. I hope I answer your question. Best!

", "rawMarkdown": "", "votes": null }, { "id": "48853", "postDate": "06/08/2014 02:29:11", "content": "

[quote=nhc;48747]

\n

I face this errors...

\n

Can anyone help me "how to solve this problem?"

\n

ValueError: time data '1' does not match format '%Y-%m-%d'

\n

[/quote]

\n\n

Can you elaborate abit more about the error? I'm going to take a stab here and say that its because the test and train history data has different date format. You'll have to treat them both seperately

", "rawMarkdown": "", "votes": null }, { "id": "48857", "postDate": "06/08/2014 10:49:40", "content": "

thank you Eu Jin Lok!!! I find the problem on my trainHistory Data!!! I make a mistake to check only transaction data...

", "rawMarkdown": "", "votes": null }, { "id": "48864", "postDate": "06/08/2014 16:47:25", "content": "

thanks for you answer.

\n

I clearly explain bad, my question is: if we consider a probability above 0.5 as repeater, then, how well your model classifies ?

\n\n

bye 

", "rawMarkdown": "", "votes": null }, { "id": "48872", "postDate": "06/08/2014 20:28:47", "content": "

Just a note on this program, as I tried it out the other day (and it's a great and I've learned a lot. Thank you to the submitter!) There are some customers in trainhistory and testhistory who have no data, and from what I can tell they don't end up in the training set or more importantly the prediction set (there are less total rows than in the original trainhistory+testhistory). I don't know how kaggle handles this.

", "rawMarkdown": "", "votes": null }, { "id": "48875", "postDate": "06/08/2014 21:20:38", "content": "

[quote=Thomas O'Malley;48872]

\n

Just a note on this program, as I tried it out the other day (and it's a great and I've learned a lot. Thank you to the submitter!) There are some customers in trainhistory and testhistory who have no data, and from what I can tell they don't end up in the training set or more importantly the prediction set (there are less total rows than in the original trainhistory+testhistory). I don't know how kaggle handles this.

\n

[/quote]

\n

If you use transactions.csv file, the number of shoppers is exactly equal to the number of shoppers in two files: trainHistory.csv and testHistory.csv. 

", "rawMarkdown": "", "votes": null }, { "id": "48883", "postDate": "06/09/2014 06:20:46", "content": "

You can beat the "prior category benchmark" without even using the transaction file.

\n

Just a logistic fit on trainhistory  based on  variables suggested by varImpPlot()  on randomForest  model.   :-)

", "rawMarkdown": "", "votes": null }, { "id": "48886", "postDate": "06/09/2014 07:19:05", "content": "

for offerdate  , Changing factor to 'as.Date' worked for me   .

", "rawMarkdown": "", "votes": null }, { "id": "48895", "postDate": "06/09/2014 12:07:51", "content": "

[quote=Thomas O'Malley;48872]

\n

Just a note on this program, as I tried it out the other day (and it's a great and I've learned a lot. Thank you to the submitter!) There are some customers in trainhistory and testhistory who have no data, and from what I can tell they don't end up in the training set or more importantly the prediction set (there are less total rows than in the original trainhistory+testhistory). I don't know how kaggle handles this.

\n

[/quote]

\n

I handled this in the benchmark by predicting a 0 for these customers. Kaggle submissions need all the entries from the original test set, else it will throw a warning.

", "rawMarkdown": "", "votes": null }, { "id": "48896", "postDate": "06/09/2014 12:10:48", "content": "

I think Kien is right.

\n

There are exactly same users in raw transactions as in train + test. So, there are no users that have no data unless you didn't filter transactions first.

", "rawMarkdown": "", "votes": null }, { "id": "48996", "postDate": "06/11/2014 13:05:39", "content": "

[quote=Shize Su;48489]

\n

Hi Zach,

\n

You're right, I do run the code on Windows. Both VW 7.1 and VW 7.6 gave me same score 0.59104. So based on this, maybe the difference between 0.59410 and 0.59104 is simply caused by Mac and Windows? Seems pretty interesting!

\n

Best wishes,

\n

Shize

\n

[/quote]

\n\n

Hi Shize Su, I ran Triskelion's method a couple nights ago with VW 7.6 on Windows. I got the same score (0.59104 ) as you.

\n\n

Triskelion, thank you very much for sharing the details of your method.

", "rawMarkdown": "", "votes": null }, { "id": "49006", "postDate": "06/11/2014 16:09:37", "content": "

Hi guys

\n

Do you think its possible to run Trischlikons model on a GraphLab Create EC2 instance: 

\n

http://graphlab.com/products/create/docs/generated/graphlab.vowpal_wabbit.create.html#graphlab.vowpal_wabbit.create

\n

I am not quite sure how to transmit the the features from Python to this EC2 instance, but it could be an advantage for the full 20 GB file.

", "rawMarkdown": "", "votes": null }, { "id": "49053", "postDate": "06/12/2014 13:43:11", "content": "

Hi All -

\n\n

I've been using R to try and replicate the results that you all have received with Python and VW. Logistic regression using the Lasso method only got me up to .57 with the engineered features, so I wanted to try quantile regression, but am having a hard time figuring out how to run the same kind of quantile regression in R.

\n\n

The most promising package seemed to be 'quantreg', but I've gotten errors with it every time

\n\n

When running: rq(data=tempTR,as.formula(x),tau=.6)   where tempTR is my data.frame with engineered values and 'x' is the text version of my formula, I get the following error

\n

* Error in rq.fit.br(x, y, tau = tau, ...) : Singular design matrix

\n\n

I've also attempted to try other models that looked promising in the 'quantreg' package, such as 'rqProcess': which in the R-help documentation, says: "Computes a standardize quantile regression process for the model specified by the formula, on the partition of [0,1] specified by the taus argument, and standardized according to the argument nullH. Intended for use in KhmaladzeTest.

\n\n

rqProcess(as.formula(x),data=tempTR,taus=c(.5))

\n\n

The problem with this one is it requires I specify a range of 'equally spaced taus' , when I do this:

\n\n

rqProcess(as.formula(x),data=tempTR,taus=c(.2,.4,.6,.8))

\n\n

I get an error saying 'singular design matrix' again.

\n\n

Can anyone advise?

\n\n

My formula is below, and through troubleshooting, I've just narrowed it down to the metrics posted earlier on this thread:

\n\n

"repeater ~ I(last_600_count_brand == 0) + I(last_600_count_brand > 0 & last_600_count_company > 0 & last_600_count_category >0) + I(last_60_count_brand > 0) + I(last_600_count_brand > 0 & last_600_count_company > 0) + I(last_30_count_brand > 0) + I(last_600_count_brand > 0 & last_600_count_category > 0) + I(last_600_count_brand > 0) + last_30_monVal_brand + I(last_90_count_brand > 0) + last_60_monVal_brand + I(last_60_count_company > 0) + last_90_monVal_brand + last_60_q_company + I(last_600_count_category == 0) + last_600_q_brand + last_180_monVal_brand + I(last_180_count_brand > 0) + offervalue + last_30_q_category"

", "rawMarkdown": "", "votes": null }, { "id": "49082", "postDate": "06/13/2014 03:44:08", "content": "

ddunder: Having never worked with the quantreg package I can only hazard some guesses.

\n\n

Based on the message returned by R, I'd guess that quantreg has a matrix inversion in it somewhere and the programmers have set the code to detect linear dependence in the design matrix (nice to see some defensive programming). I would reproduce the design matrix (in R: design.matrix) and then check for linear dependence. Given the formula indicated, I'd look for equal columns in the matrix.

", "rawMarkdown": "", "votes": null }, { "id": "49126", "postDate": "06/13/2014 20:34:05", "content": "

quantreg works for me... not great;I got only 0.58364:

\n\n

cols <- c("V167", ...)
dt <- data.frame(train[,cols])
dt$y <- idy$repeattrips
model <- rq(y~., data=dt, method='fn', tau=0.5)
submit$repeatProbability <- predict(model, test[,cols])

\n\n

Yes, the feature selection (10 of 350) was done first; the CV (10 folds) was 6.02...

", "rawMarkdown": "", "votes": null }, { "id": "49257", "postDate": "06/17/2014 16:49:30", "content": "

Thank you, but when I try to run the gen_vw_features.py file I get:

\n

Alexanders-iMac:~ arigge$ python gen_vw_features.py
File "gen_vw_features.py", line 6
<!DOCTYPE html>
^
SyntaxError: invalid syntax

\n

I placed all my csv files in a folder called data in my home directory arigge. Can somebody tell me what I am missing ?

", "rawMarkdown": "", "votes": null }, { "id": "49259", "postDate": "06/17/2014 18:20:43", "content": "

Hi Alexander,

\n\n

Are you using Python 2.7 or Python 3.*?

\n

~

\n

To run the posted code by Triskelion, you need to use Python 2.7 version rather than 3.* version. They have quite different syntax.

\n

~

\n\n

Best wishes,

\n

Shize

", "rawMarkdown": "", "votes": null }, { "id": "49260", "postDate": "06/17/2014 19:18:51", "content": "

Thank you Shize.

\n\n

I use Python 2.7 which runs on Mac by default.

\n\n

Best,

\n

Alex 

", "rawMarkdown": "", "votes": null }, { "id": "49261", "postDate": "06/17/2014 19:50:08", "content": "

Hi Alexander,

\n\n

Uh, interesting.

\n

~

\n

For me, I just download the code, change the location for data files, and run it with Python 2.7 on windows, and then everything works.

\n

~

\n

Besides, are you sure that the only change you made to the Python code is just the data file locations? If yes, then I don't know why it doesn't work on your computer. Maybe you could  try to run the same code on another computer and see what happens. It seems to me that there is no reason why it will not work.

\n

~

\n

Have a great day!

\n\n

Best wishes,

\n

Shize

", "rawMarkdown": "", "votes": null }, { "id": "49268", "postDate": "06/18/2014 03:55:51", "content": "

Hi Alex,

\n

I am also using Mac and Python 2.7 without any problem.

\n\n

Best,

\n

Tam

", "rawMarkdown": "", "votes": null }, { "id": "49269", "postDate": "06/18/2014 04:33:46", "content": "

Is it important to normalize/standardize the data, since vw seems to be some sort of gradient-based algorithm (plz correct me if I'm wrong)??

\n\n

Triskelion, thanks a million for sharing your solution and introducing us to the powers of vw.

", "rawMarkdown": "", "votes": null }, { "id": "49286", "postDate": "06/19/2014 03:12:17", "content": "

Check for whitespace. Make sure you don't have any stray spaces on empty lines. Especially line 6.

\n\n

[quote=Alexander Riggers;49257]

\n

Thank you, but when I try to run the gen_vw_features.py file I get:

\n

Alexanders-iMac:~ arigge$ python gen_vw_features.py
File "gen_vw_features.py", line 6
<!DOCTYPE html>
^
SyntaxError: invalid syntax

\n

I placed all my csv files in a folder called data in my home directory arigge. Can somebody tell me what I am missing ?

\n

[/quote]

", "rawMarkdown": "", "votes": null }, { "id": "49292", "postDate": "06/19/2014 07:57:07", "content": "

May I ask what the "average loss" and "since last" columns converged to towards the end? I've had cases where the predictions are quite different but those 2 columns gave similar results during training. Then how does one tell which one is better?

\n\n

Many thanks in advance!

", "rawMarkdown": "", "votes": null }, { "id": "49293", "postDate": "06/19/2014 08:02:58", "content": "

Sorry another question: are the Y's used here repeater (i.e. 0s and 1s) or number of repeats?

", "rawMarkdown": "", "votes": null }, { "id": "49304", "postDate": "06/19/2014 15:06:04", "content": "

Wanted to note two things quickly:

\n

1) Namespaces!  They're a great feature of Vowpal Wabbit.  I use them to quickly test features - just put each new feature / feature class into its own namespace, then turn them on and off at the vw command line (using --keep or --ignore) rather than rebuilding the train/test files for each possible combination of features.  Other folks are probably already doing this, but I wanted to throw it out there.

\n\n\n

2) If using namespaces, be aware that unless specified, testing (with -t) will use ALL namespaces, and could change your results slightly.

", "rawMarkdown": "", "votes": null }, { "id": "49999", "postDate": "07/03/2014 16:01:40", "content": "

By the way - Triskelion, I tried to get in touch with you via your website a few weeks ago about VW.  I'm guessing it didn't go through, and I'm still too much of a Kaggle newbie to be able to email other users, so if you'd like to, please feel free to email me.  :-)

", "rawMarkdown": "", "votes": null }, { "id": "50002", "postDate": "07/03/2014 17:51:21", "content": "

[quote=Phil Culliton;49999]

\n

By the way - Triskelion, I tried to get in touch with you via your website a few weeks ago about VW.  I'm guessing it didn't go through, and I'm still too much of a Kaggle newbie to be able to email other users, so if you'd like to, please feel free to email me.  :-)

\n

[/quote]

\n

Whoops, yeah that manual checking of spam vs. ham comments took more time then I expected. Sorry for that, I'll go through the existing queue and install Akismet or something so this won't happen again.

\n\n

I think you can email other users by clicking their profile and then "contact", but maybe you have to be Kaggler for that first. I send you an email over the Kaggle comment system.

\n\n

Thank you for all your comments, here and elsewhere, and helping everyone out!

", "rawMarkdown": "", "votes": null }, { "id": "50004", "postDate": "07/03/2014 18:27:16", "content": "

[quote=Triskelion;50002]

\n

Whoops, yeah that manual checking of spam vs. ham comments took more time then I expected. Sorry for that, I'll go through the existing queue and install Akismet or something so this won't happen again.

\n

[/quote]

\n\n

No worries at all, that sort of stuff can creep up on you.

\n\n

[quote=Triskelion;50002]

\n

I think you can email other users by clicking their profile and then "contact", but maybe you have to be Kaggler for that first. I send you an email over the Kaggle comment system.

\n

[/quote]

\n\n

I'm a Kaggler, but I apparently need more points to be able to contact others.  Thanks for the email!

\n\n

[quote=Triskelion;50002]

\n

Thank you for all your comments, here and elsewhere, and helping everyone out!

\n

[/quote]

\n\n

Just taking an example from you, really!  And from many of the other generous folks here.  I've been really impressed by all of the people stepping forward to help folks figure out the basics.  I'd love to help even more myself, if possible.

", "rawMarkdown": "", "votes": null }, { "id": "50011", "postDate": "07/03/2014 23:00:58", "content": "

[quote=Peemblis;48996]

\n

[quote=Shize Su;48489]

\n

Hi Zach,

\n

You're right, I do run the code on Windows. Both VW 7.1 and VW 7.6 gave me same score 0.59104. So based on this, maybe the difference between 0.59410 and 0.59104 is simply caused by Mac and Windows? Seems pretty interesting!

\n

Best wishes,

\n

Shize

\n

[/quote]

\n

Hi Shize Su, I ran Triskelion's method a couple nights ago with VW 7.6 on Windows. I got the same score (0.59104 ) as you.

\n

Triskelion, thank you very much for sharing the details of your method.

\n

[/quote]

\n

Out of curiosity, do you find you have probability scores that have a truncated decimal string (such as random forest in R)? I have not looked at VW so I am not familiar with the output but when using randomforest in R, I believe it truncates to 3 or 4 decimal places giving many records the same score (especially when using many binary features). A simple change in order upon submission can throw off the final score, even when running the identical model but with a different ordering of input records on each iteration.

", "rawMarkdown": "", "votes": null }, { "id": "50056", "postDate": "07/04/2014 23:10:20", "content": "

[quote=Ben Post;50011]

\n

Out of curiosity, do you find you have probability scores that have a truncated decimal string (such as random forest in R)?

\n

[/quote]

\n

It's likely that they use the same script to convert a VW predictions file to a Kaggle submission.

\n

.

\n

It seems that VW can produce different results depending on your CPU type. VW is compiled with ffast-math flag, which is the least conforming but fastest math mode.

", "rawMarkdown": "", "votes": null }, { "id": "50058", "postDate": "07/05/2014 02:37:42", "content": "

[quote=Triskelion;50056]

\n

It seems that VW can produce different results depending on your CPU type. VW is compiled with ffast-math flag, which is the least conforming but fastest math mode.

\n

[/quote]

\n

And compiler type!  Every compiler plays with math-heavy code differently, and interprets things like fast-math differently.  I wouldn't be at all surprised to find that VW compiled with GCC would behave slightly differently than with, say, Intel or MSVCPP.

", "rawMarkdown": "", "votes": null }, { "id": "50090", "postDate": "07/07/2014 04:05:53", "content": "

i got 0.59474 by Triskelion's feature and vw command line. when i tried to add some features , i got worse scores and curiously found many predict results were zero. after tuned vw parameter lambda，nothing changed.  

", "rawMarkdown": "", "votes": null }, { "id": "50210", "postDate": "07/09/2014 15:55:19", "content": "

What features did you try to add?  And what changes did you make to VW parameters?  I've found that Triskelion's initial settings are actually a pretty darned good fit.

\n\n

Also - when did you grab your Vowpal Wabbit build?  Did you pull straight from github?  There are some recent bugs, I think - some nasty issues with not properly updating average loss, etc.

", "rawMarkdown": "", "votes": null }, { "id": "50218", "postDate": "07/09/2014 17:21:53", "content": "

I have a question regarding the gen_feature_vw.py output. Why do all entries in the test.vw file have a label of one? 

\n\n

VW input format is:

\n

label [weight] |Namespace Feature ... Feature |Namespace ...

\n\n

In our case 1`lastid ....

\n

While int the train file we also have some 0 labels.

", "rawMarkdown": "", "votes": null }, { "id": "50219", "postDate": "07/09/2014 17:36:30", "content": "

VW ignores labels when in test mode - so it doesn't matter what's there.  However, something does need to be there, so we put in some dummy information.  When using the train file we're (most likely) actually training the VW model, so you put real information in the labels.

\n\n

One nice side effect of using all 1s as the dummy labels is that you can use average loss on the test run to gauge how "high" or "low" your current set of features might be pushing the overall results, simply by comparing it to the average testing loss from previous runs.  Since VW doesn't have good support for CV, any kind of meta-data about your current results can be helpful.  :-)

", "rawMarkdown": "", "votes": null }, { "id": "116220", "postDate": "04/22/2016 15:25:03", "content": "

Thanks for sharing!

", "rawMarkdown": "Thanks for sharing!", "votes": null }, { "id": "3229372", "postDate": "06/21/2025 11:53:12", "content": "

nicely explained

", "rawMarkdown": "nicely explained", "votes": null } ], "comments": [ { "id": 3229372, "author_name": "vickey13", "author_url": "", "post_date": "06/21/2025 11:53:12", "content": "

nicely explained

", "votes": null, "replies": [] }, { "id": 41962, "author_name": "mathieucliche", "author_url": "", "post_date": "04/15/2014 19:54:40", "content": "

So I noticed that the offer_quantity is equal to 1 for all offers of trainHistory, while it is sometimes equal to 2 for some of the offers in testHistory.  This explain why it is not a good feature as it is.  

\n

However, I can imagine someone getting a "good" feature out of this by probing the test data a few times with the kaggle score, though I doubt this is a good use of one's time.  

", "votes": null, "replies": [] }, { "id": 41978, "author_name": "triskelion", "author_url": "", "post_date": "04/15/2014 22:58:15", "content": "

[quote=Mathieu Cliche;41962]

\n

So I noticed that the offer_quantity is equal to 1 for all offers of trainHistory, while it is sometimes equal to 2 for some of the offers in testHistory.

\n

[/quote]

\n

Astute observation! I probably should have checked that myself. Seemed such a nice feature. I think it is useless now.

", "votes": null, "replies": [] }, { "id": 42021, "author_name": "zhaogang", "author_url": "", "post_date": "04/16/2014 10:41:50", "content": "

Great job!

\n

good start!

", "votes": null, "replies": [] }, { "id": 42047, "author_name": "artghazaryan", "author_url": "", "post_date": "04/16/2014 14:49:38", "content": "

Triskelion

\n

Your thoughts of creating new features from the transaction data are absolutely right. One note though. Brand is associated with category and company, So, if a shopper buys a brand, he/she buys the category, and the company as well. So, you can combine these three into one feature. To be more accurate, I think category and company could be left out, because the offers are made on brands, not categories or companies. I think a new binary feature, that indicates if a shopper has purchased the brand prior to the offer, would be sufficient. In other words, a shopper who has prior exposure to the brand, is more likely to respond to the offer, than the shopper who has not had the exposure. 

", "votes": null, "replies": [] }, { "id": 42056, "author_name": "abhishek", "author_url": "", "post_date": "04/16/2014 15:47:52", "content": "

[quote=Art_Ghazaryan;42047]

\n

Triskelion

\n

Your thoughts of creating new features from the transaction data are absolutely right. One note though. Brand is associated with category and company, So, if a shopper buys a brand, he/she buys the category, and the company as well. So, you can combine these three into one feature. To be more accurate, I think category and company could be left out, because the offers are made on brands, not categories or companies. I think a new binary feature, that indicates if a shopper has purchased the brand prior to the offer, would be sufficient. In other words, a shopper who has prior exposure to the brand, is more likely to respond to the offer, than the shopper who has not had the exposure. 

\n

[/quote]

\n

No.

\n

Brand is not associated with category (with company?). A sparkling water bottle is a category. A company can make many brands but all brands do not belong to one company. Brand is a trademark of a given company. If you buy kinley still mineral water, you are buying from department = water, category = still water, brand = kinley, company = coca cola

", "votes": null, "replies": [] }, { "id": 42058, "author_name": "martin22", "author_url": "", "post_date": "04/16/2014 15:54:47", "content": "

[quote=Triskelion;41913]

\n

Here is some code and ideas to get a score of around 0.59347. It uses Python and Vowpal Wabbit, but the idea should work with other languages and other algorithms.

\n

[/quote]

\n\n

Do you get this same (or similar) score when doing crossvalidation on your end, or do you get a much higher one?

\n

It seems to me that an important factor in this competition is generating a validation set that actually works. I had tried using similar features to the ones you described and got a score of ~0.67 in CV, but only 0.55 in the public leaderboard. For my best submission (currently 0.588), I get ~0.60 in CV, which makes this really confusing.

", "votes": null, "replies": [] }, { "id": 42059, "author_name": "artghazaryan", "author_url": "", "post_date": "04/16/2014 16:12:32", "content": "

Abhishek,

\n\n

When you get a coupon, it is for a specific brand. You cannot redeem that coupon to get another brand of the same company, or category, or department.

\n

There is also another issue here. What type of association we have between each shopper and coupons? Is it one to one? Were the coupons provided as a collection(coupon book, newspaper insert, etc), or they were provided in some other way?

", "votes": null, "replies": [] }, { "id": 42069, "author_name": "triskelion", "author_url": "", "post_date": "04/16/2014 17:20:10", "content": "

[quote=Martin Martin;42058]

\n

[quote=Triskelion;41913]

\n

Here is some code and ideas to get a score of around 0.59347. It uses Python and Vowpal Wabbit, but the idea should work with other languages and other algorithms.

\n

[/quote]

\n

Do you get this same (or similar) score when doing crossvalidation on your end, or do you get a much higher one?

\n

[/quote]

\n

I get a higher one. (~3-5 AUC higher) I think this may be due to outliers in the train set (and test set). I suspect that if you denoise the train and test set, your model will be more robust, and your validation scores are then closer to the leaderboard.

\n

I am familiar with removing noisy features (PCA, topkfeatures etc.), not too familiar with removing noisy feature values (more like fraud detection). EDIT: This thread seems to suggest it is not due to noise/outliers, but due to temporal issues. I think that means in 5-year-old speak that the train set and test set are fairly different from each other. CV on only the train set will be higher than public leader board score, as the train set itself is more internally consistent. This may be a hard problem to solve, though an increase in local CV may very well mean an increase in public leaderboard score.

\n

I'll take a better look soon, for now I have a few other competitions that demand my attention.

", "votes": null, "replies": [] }, { "id": 42072, "author_name": "triskelion", "author_url": "", "post_date": "04/16/2014 17:25:33", "content": "

[quote=Art_Ghazaryan;42059]

\n

Abhishek,

\n

When you get a coupon, it is for a specific brand. You cannot redeem that coupon to get another brand of the same company, or category, or department.

\n

There is also another issue here. What type of association we have between each shopper and coupons? Is it one to one? Were the coupons provided as a collection(coupon book, newspaper insert, etc), or they were provided in some other way?

\n

[/quote]

\n

Hey Art Ghazaryan. I think you may be right. Some features are superfluous. Has_never_bought_brand_company would probably be the same as has_never_bought_brand. As it is unlikely that another company creates the same brand.

\n

I was not too sure of this beforehand. I just extracted some features and ran it through Vowpal Wabbit. I don't like relying on domain expertise too much (as I have very little in all domains). Basically I hope I use tools that are good enough to discard useless features and focus only on the indicative features. Vowpal Wabbit is pretty good at that, sklearn has some algorithms that are better. Most of them need feature values to be scaled between 0-1 and demand all data to be loaded into memory. While a bit more work, this may reduce the hard-to-normalize noise in the data sets.

", "votes": null, "replies": [] }, { "id": 42125, "author_name": "yzhao0527", "author_url": "", "post_date": "04/17/2014 02:09:35", "content": "

6,089 out of 35689 brands belong to more than one company ... I am very confused ...

", "votes": null, "replies": [] }, { "id": 42142, "author_name": "mikedemurtas", "author_url": "", "post_date": "04/17/2014 05:51:48", "content": "

[quote=Y Zhao;42125]

\n

6,089 out of 35689 brands belong to more than one company ... I am very confused ...

\n

[/quote]

\n

My guessing is because in the data description they say:

\n

'The transactions file can be joined to the offers file by (category, brand, company).'

\n

We should consider category, brand, company as a key, so different brands for different companies can have the same code but they are actually different brands.

", "votes": null, "replies": [] }, { "id": 42326, "author_name": "zachmayer", "author_url": "", "post_date": "04/18/2014 15:46:32", "content": "

[quote=Mike Demurtas;42142]

\n

My guessing is because in the data description they say:

\n

'The transactions file can be joined to the offers file by (category, brand, company).'

\n

We should consider category, brand, company as a key, so different brands for different companies can have the same code but they are actually different brands.

\n

[/quote]

\n

I'd really love to get an answer from the competition admin on this-- do we need to combine company and category to get a unique brand id?

", "votes": null, "replies": [] }, { "id": 42328, "author_name": "yzhao0527", "author_url": "", "post_date": "04/18/2014 16:38:07", "content": "

My guess is this is due to corporate actions. I know there are quite a few food companies got acquired and spin-off-ed during the past two years, (Kraft, white plain??, etc). I probably will take a look at the time series of brand/company instances to verify my guess.

", "votes": null, "replies": [] }, { "id": 42346, "author_name": "rjrest", "author_url": "", "post_date": "04/18/2014 19:46:58", "content": "

This is the answer I received from the sponsor admin (who is not at their computer today):

\n\n

The combination of “category, brand, company” could be consider a unique key.

\n

Brands are unique to companies, but brands-company combinations can cross categories.

", "votes": null, "replies": [] }, { "id": 42356, "author_name": "martin22", "author_url": "", "post_date": "04/18/2014 21:43:30", "content": "

[quote=Ramzi R;42346]

\n

Brands are unique to companies, ...

\n

[/quote]

\n

I think we know this statement is false, right? Is there a problem with the data?

", "votes": null, "replies": [] }, { "id": 42557, "author_name": "jansan", "author_url": "", "post_date": "04/21/2014 00:07:06", "content": "

The sponsor admin's remark is in line with what Mike wrote before:

\n

[quote=Mike Demurtas;42142]

\n

We should consider category, brand, company as a key, so different brands for different companies can have the same code but they are actually different brands.

\n

[/quote]

\n\n

This is also supported by the observation that the brand occuring for the largest number of companies is brand '0', which occurs for 9150 companies (making other explanations like corporate actions very unlikely).

", "votes": null, "replies": [] }, { "id": 42563, "author_name": "zachmayer", "author_url": "", "post_date": "04/21/2014 01:48:15", "content": "

Are there similar duplicates of companies between categories?  Or since companies can have brands in different categories, are all company IDs unique?

", "votes": null, "replies": [] }, { "id": 42622, "author_name": "dmdave", "author_url": "", "post_date": "04/21/2014 17:32:45", "content": "

Point of clarification:  it is possible for brands to have more than 1 (one) company. 

\n

The combination of "category, brand, company" can be considered a unique key.

", "votes": null, "replies": [] }, { "id": 42626, "author_name": "zachmayer", "author_url": "", "post_date": "04/21/2014 18:18:27", "content": "

First of all, thank you for submitting your code.  This submissions is better than anything I've been able to put together on my own, after many hours of work.

\n\n

Some questions:

\n

1. Why did you choose to use quantile regression?  Isn't this problem a perfect fit for the logistic regression function in vowpal wabbit?

\n

2. Why 0.6 for the quantile tau?  Did you try any other values, and if so, how did you decide if they were good or bad?

\n

3. Why 40 passes?  Did you try any other values, and if so, how did you decide 40 was optimal?

\n

4. Why 0.85 for the learning rate?

\n

5. Are you worried at all about over-fitting?  I noticed that the self-reported loss for VW was 0 for a very long time before the model stopped.  Should you have stopped the model after some number of passes where the self-reported loss was still about zero?

\n

Questions 2-4 are basically the same question: how did you tune the model prior to submission, and how confident are you in your tuning?

\n\n

Thanks again for the code, you've helped me out immensely.

", "votes": null, "replies": [] }, { "id": 43523, "author_name": "triskelion", "author_url": "", "post_date": "05/02/2014 18:32:43", "content": "

[quote=Zach;42626]

\n

Some questions:

\n

1. Why did you choose to use quantile regression?  Isn't this problem a perfect fit for the logistic regression function in vowpal wabbit?

\n

[/quote]

\n

It probably is a good fit (if it is better, it very well could be: try it!). Langford says that for binary classification problems Vowpal Wabbit usually performs well using the --binary --loss_function logistic operator. This demands labels to be [-1,1] instead of [0,1]

\n\n

But I am not actually doing classification. I am doing a regression between 0 and 1. This will skip the need to post-process the predictions (running through a sigmoid etc.). They already score good for AUC/ROC.

\n\n

[quote=Zach;42626]

\n

2. Why 0.6 for the quantile tau?  Did you try any other values, and if so, how did you decide if they were good or bad?

\n

[/quote]

\n

I always try the standard 0.5 first. There are ways to find out which parameter is the best (its an obvious tweak when using quantile regression), but the easiest for me was just generating a submission with quantile tau 0.4 and 0.6 and see if it improves or decreases the score. You obviously would use local validation and performance evaluation for this if you were to do this properly.

\n

[quote=Zach;42626]

\n

3. Why 40 passes?  Did you try any other values, and if so, how did you decide 40 was optimal?

\n

[/quote]

\n

This used to be based on a hunch more than anything (or I just did not know how to find out, I just tried 20 passes as the standard that just seems to work alright, fastml confirms). The newer version of Vowpal Wabbit has holdout features build-in. This means one in n samples is used to calculate the loss now (standard 1 in 10). Quantile regression seems to need fewer passes to converge to a good solution in general. With new holdout functionality Vowpal Wabbit will stop doing passes if it does not increase performance over the last n passes. So that is why Vowpal Wabbit 7.6 will increase the score on leaderboard. Maybe it only does 9 passes and finds that is optimal over 40 passes. This, you could say, is Vowpal Wabbit's build-in overfitting mechanism.

\n

[quote]

\n

4. Why 0.85 for the learning rate?

\n

[/quote]

\n

This caused a lower average loss. You can manually check parameters like these (--decay_learning_rate, --power_t, l). Usually you don't even need to fiddle with it (it can also mess up your results), but it can improve the score a bit. BTW: From https://raw.githubusercontent.com/wiki/gdfm/vowpal_wabbit/v5.1_tutorial.pdf "for multiple passes --decay_learning_rate between [0.5-1] is sensible. Values smaller than 1 protect against overfitting."

\n

[quote]

\n

5. Are you worried at all about over-fitting?  I noticed that the self-reported loss for VW was 0 for a very long time before the model stopped.  Should you have stopped the model after some number of passes where the self-reported loss was still about zero?

\n

[/quote]

\n

Not with Vowpal Wabbit and with this dataset. Especially not with the new holdout functionality. If you are really worried about over-fitting you could add l1 and l2 regularization with --l1 and --l2. And try ensemble learning to reduce fitting.

\n

[quote]

\n

Questions 2-4 are basically the same question: how did you tune the model prior to submission, and how confident are you in your tuning?

\n

[/quote]

\n

I check average loss and if the learning process looks ok: (average loss and since last decreasing). This is the quick way, because you don't need to set up a holdout set or do k-fold validation, average loss is a pretty decent indicator of leaderboard performance. 

\n

Debug process from the 5.1 tutorial:

\n\n

If you want more precise way: Create a holdout set or two from your train set. Evaluate AUC on the holdout set(s). Parameter tweak (i try bruteforce gridsearch or random search) according to this fitness factor.

\n

You can also employ PERF software to find good cutt-off and min_prediction and max_prediction I think, but I have not figured that out yet. 

\n

I may even output in libsvm format so you can do most of this with sklearn or try other solvers like Sofia-ML. 

\n

[quote]

\n

Thanks again for the code, you've helped me out immensely.

\n

[/quote]

\n

No problem! Happy competition!

", "votes": null, "replies": [] }, { "id": 43657, "author_name": "breuker", "author_url": "", "post_date": "05/04/2014 09:03:23", "content": "

As for the choice of the loss function, I tried Triskelions features with all 4 loss functions implemented in Vowpal Wabbit. At first, I also thought logistic regression might be a better fit. Quantile regression scored highest on the leaderboard though.
What kind of problem we are solving is actually not that important afaik, at least in terms of the numbers you want to submit. You can create scores in any range, not just [0,1]. AUC can handle that.

", "votes": null, "replies": [] }, { "id": 44253, "author_name": "decipher", "author_url": "", "post_date": "05/08/2014 10:59:41", "content": "

Hi Triskelion,

\n

Many thanks for the detailed explanation. Really impressed with your knowledge.

\n

Being novice, i am participating in this contest and I am trying to create new features in SAS/R so just wanted to check if what I understood from these posts is correct or not. I think one should try to create new features like How many times person bought from a company/category on offer, what is the total amount amount transacted on company/category etc. Now my question is, are you guys suggesting that we should have this count of transactions/amount of transaction variables for each company present in the offers.csv i.e we should have no_of_trxn_comp1, no_of_trxn_comp2, no_of_trxn_comp3, no_of_trxn_comp4, no_of_trxn_comp5 and so on for each company until comp18 that we have in offers.csv (18 companies) and similarly for 20 categories  in offers.csv file. This way we will so many variables for company and categories. In total my question is that we need to create the variables for each company for whatever parameter (i.e amount, count etc.) we are creating. 

\n

Anyone's help on this would be really appreciated.  Thanks in advance.

", "votes": null, "replies": [] }, { "id": 44259, "author_name": "artghazaryan", "author_url": "", "post_date": "05/08/2014 14:24:32", "content": "

Hi Decipher

\n\n

Your thinking process is absolutely correct. That is a correct approach to extract intelligence and create variables. I have created 114 variables so far.

", "votes": null, "replies": [] }, { "id": 44273, "author_name": "npetitclerc", "author_url": "", "post_date": "05/08/2014 18:22:38", "content": "

@Decipher

\n

Well you do need to calculate all those, but in terms of features to train on you only use the variables for the specific offer.

\n

So if an offer is on company XX you would use the no_of_trxn_compXX variable as a feature. Hope this helps.

", "votes": null, "replies": [] }, { "id": 44396, "author_name": "potato", "author_url": "", "post_date": "05/12/2014 04:24:38", "content": "

Thanks for your sharing, it's a good start

", "votes": null, "replies": [] }, { "id": 44410, "author_name": "user191111", "author_url": "", "post_date": "05/12/2014 16:07:34", "content": "

Did anyone use exactly the same features with logistic regression?

\n

My auc using logistic regression is only 0.53, I'm checking whether there is something wrong with my feature extraction process, or it's due to logistic regression.

", "votes": null, "replies": [] }, { "id": 44411, "author_name": "zachmayer", "author_url": "", "post_date": "05/12/2014 16:10:52", "content": "

[quote=身经百战长者;44410]

\n

Did anyone use exact the same features with logistic regression?

\n

My auc using logistic regression is only 0.53, I'm checking whether there is something wrong with my feature extraction process, or is due to logistic regression.

\n

[/quote]

\n\n

Yes.  Logistic regression using vowpal wabbit is a fair bit worse than quantile regression using vowpal wabbit, and logistic regression using another tool seems to be even worse.

\n

Try quantile regression with your features and see what happens.

\n

There's already a discussion on this topic here.

", "votes": null, "replies": [] }, { "id": 44412, "author_name": "user191111", "author_url": "", "post_date": "05/12/2014 16:16:20", "content": "

Thanks for reply

\n

Are there any tools in python or R for quantile regression? It seems that sklearn does not implement quantile regression

", "votes": null, "replies": [] }, { "id": 44413, "author_name": "abhishek", "author_url": "", "post_date": "05/12/2014 16:17:21", "content": "

statsmodels

", "votes": null, "replies": [] }, { "id": 44414, "author_name": "mathieucliche", "author_url": "", "post_date": "05/12/2014 16:18:47", "content": "

I'm actually using logistic regression with scikit-learn and roughly the same features, and for me it's working well, but you need to use some sort of regularization to avoid over-fitting.

", "votes": null, "replies": [] }, { "id": 44429, "author_name": "user191111", "author_url": "", "post_date": "05/13/2014 03:23:56", "content": "

But I'm still a little confused about applying a regression algorithm(like quantile regression) to a classification problem...

", "votes": null, "replies": [] }, { "id": 44441, "author_name": "decipher", "author_url": "", "post_date": "05/13/2014 08:22:00", "content": "

Can you please explain your feature extraction process?

", "votes": null, "replies": [] }, { "id": 44451, "author_name": "user191111", "author_url": "", "post_date": "05/13/2014 12:55:08", "content": "

It generate the features provided by Triskelion

", "votes": null, "replies": [] }, { "id": 44479, "author_name": "triskelion", "author_url": "", "post_date": "05/13/2014 18:50:06", "content": "

@身经百战长者

\n

> It seems that sklearn does not implement quantile regression.

\n

> ... Confused about using regression for a classification problem.

\n

See: sklearn.ensemble.GradientBoostingRegressor which has quantile loss.

\n

You can treat a classification problem as a regression problem, if you regress between 0 (the negative class) and 1 (the positive class). I don't know if this is unorthodox, probably it is -- It looks like standard sklearn in-memory logistic regression with these features (and some regularization) already outperforms this benchmark.

", "votes": null, "replies": [] }, { "id": 44513, "author_name": "user191111", "author_url": "", "post_date": "05/14/2014 10:14:32", "content": "

Hmm, but in a machine learning MOOC course   . When introducing logistic regression (the first video lecture of logistic regression), the professor showed how bad it is to apply OLS linear regression to a classification problem.

\n

So, I'm amazed that quantile regression can produce good result.

", "votes": null, "replies": [] }, { "id": 44562, "author_name": "triskelion", "author_url": "", "post_date": "05/14/2014 22:58:37", "content": "

@身经百战长者

\n

>So, I'm amazed that quantile regression can produce good result.

\n

Try all, keep the best. Sometimes theoretical knowledge on what is possible or accepted practice can be a burden. Most of the time it is beneficial though. Don't be afraid to experiment or break some rules and you got the best of both worlds!

\n

See: Forum thread and FastML experiment for an opposite example: someone using classification for a regression problem, and ending up in 3rd position.

", "votes": null, "replies": [] }, { "id": 44796, "author_name": "kontoudi", "author_url": "", "post_date": "05/18/2014 23:30:35", "content": "

@身经百战长者

\n

I am following the MOOC you are referring to. Here is how I see the problem.

\n

If I remember correctly the professor explained that linear regression is not good for classification because it is very sensitive to outliers. What linear regression is doing is to estimate the conditional mean so if you add a data point far away from the others this will greatly affect the value of the mean.

\n

According to Wikipedia quantile regression aims at estimating either the conditional median or other quantiles of the response variable.

\n

Now if your features are quite dense around e.g. the median then adding features far away will not affect a lot the value of the median.

\n

If anyone has a better understanding of this please share!

", "votes": null, "replies": [] }, { "id": 47421, "author_name": "sushize", "author_url": "", "post_date": "06/01/2014 01:05:24", "content": "

Dear Triskelion and all friends,

\n

First, thanks for the provided code. Great job!

\n

I am new to Python and I just start this competition.  Previously, I mostly work with R, Matlab and C. Today, I run your posted python code on https://github.com/MLWave/kaggle_acquire-valued-shoppers-challenge, but run into some errors.

\n

My environment is Python 3.4.1 IDLE ( p.s., what's your python programming environment for this posted Python code?). When I run your posted code, I run into two following errors:

\n

1) invalid syntax at line "print e, reduced, datetime.now() - start". I then changed this line to "print (e, reduced, datetime.now() - star)", and then it doesn't report error, but I am not quite sure whether this indeed solve the problem.

\n

2)  after fixing the invalid syntax error, I run the code again, and it showed following error:

\n

Traceback (most recent call last):
File "D:\\ML_DM Competition\\acquire_shopper_challenge\\gen_vw_features.py", line 231, in

\n

<module> reduce_data(loc_offers, loc_transactions, loc_reduced)

\n

File "D:\\ML_DM Competition\\acquire_shopper_challenge\\gen_vw_features.py", line 41, in

\n

reduce_data  outfile.write( line ) #print header

\n

TypeError: 'str' does not support the buffer interface

\n

I have no idea about how to fix this problem. Would you be so kind to give me some hint to solve this problem? Thanks so much!

\n

Besides, I did following things to run the python code. Correct me if I make some stupid mistakes.

\n

1) I install Python 3.4.1 IDLE in C disk.

\n

2) the python code and data files are located in D disk.

\n

3) I start the Python 3.4.1 IDLE, and use open option to open your python code, and use Run Modules option in the IDLE to run the python code, and I run into these two errors.

\n

Any help are pretty appreciated. Thanks!

\n

Best wishes,

\n

Shize

", "votes": null, "replies": [] }, { "id": 47422, "author_name": "sthompson", "author_url": "", "post_date": "06/01/2014 01:21:38", "content": "

Shize, there are two major versions of python, 3.x and 2.x. Try python 2.7. The different versions have quite different syntax.

", "votes": null, "replies": [] }, { "id": 47423, "author_name": "sushize", "author_url": "", "post_date": "06/01/2014 01:29:08", "content": "

Hi Scott,

\n

Thanks for your reply. Besides, does Python 2.7 also have GUI IDLE programming environment? You see, I prefer the user-friendly GUI IDLE environment rather than a DOS-like terminal environment. Thanks, and have a great night!

\n

Best wishes,

\n

Shize

", "votes": null, "replies": [] }, { "id": 47424, "author_name": "sushize", "author_url": "", "post_date": "06/01/2014 02:08:22", "content": "

Dear Scott,

\n

I download Python 2.7.6 and install it on my laptop. Now the Python code works. Thanks so much, and have  a great night!

\n

Best wishes,

\n

Shize

", "votes": null, "replies": [] }, { "id": 48413, "author_name": "sushize", "author_url": "", "post_date": "06/01/2014 03:19:55", "content": "

Dear friends,

\n

I run into another problem when run the posted Python code of Triskelion in Python 2.7.

\n

Actually, when I run the code, it generate the reduce.csv correctly, but when it run the generat_features function, it throw out an error about date format. The error is as follows:

\n

Traceback (most recent call last):
File "D:\\ML_DM Competition\\acquire_shopper_challenge\\gen_features_updated.py", line 232, in <module>

\n


generate_features(loc_train, loc_test, loc_reduced, loc_out_train, loc_out_test)
File "D:\\ML_DM Competition\\acquire_shopper_challenge\\gen_features_updated.py", line 162, in generate_features

\n


date_diff_days = diff_days(row[6],history[-1])
File "D:\\ML_DM Competition\\acquire_shopper_challenge\\gen_features_updated.py", line 57, in diff_days

\n


a = datetime.strptime(s1, date_format)
File "C:\\Python27\\lib\\_strptime.py", line 325, in _strptime
(data_string, format))

\n


ValueError: time data ''6/27/2013'' does not match format '%Y-%m-%d'

\n

Trying to fix this error, I changed the line of Python code "date_format = "%Y-%m-%d"  to "date_format = "%m/%d/%Y", and run the Python code again.

\n

But this time I run into another date format error as follows:

\n

ValueError: time data '2012-03-17' does not match format '%m/%d/%Y'

\n

So it seems that there are more than one kind of date format in the data. I don't have any ideas to solve this problem. Would anyone be so kind to provide some hints? Thanks so much!

\n\n

Best wishes,

\n

Shize

", "votes": null, "replies": [] }, { "id": 48416, "author_name": "neilsummers", "author_url": "", "post_date": "06/01/2014 04:03:04", "content": "

try:

\n

    first date format

\n

except ValueError:

\n

    second date format

", "votes": null, "replies": [] }, { "id": 48417, "author_name": "sushize", "author_url": "", "post_date": "06/01/2014 04:07:48", "content": "

Dear Neil,

\n

Thanks!

\n\n

I am new to Python, and I am not quite sure how to code in Python about your suggestion of 

\n

try:

\n

first date format

\n

except ValueError:

\n

second date format

\n\n

To be specific, how should I change the Python code? Thanks so much, and good night!

\n\n

Best wishes,

\n

Shize

", "votes": null, "replies": [] }, { "id": 48418, "author_name": "neilsummers", "author_url": "", "post_date": "06/01/2014 04:23:47", "content": "

I suggest you read the python documentation on python handling of errors.

\n

https://docs.python.org/2/tutorial/errors.html

\n

What you can do is catch the exception and execute a different piece of code instead of throwing the exception.

\n

I have not looked at the code you are having trouble with but it appears you are getting the exception at the line

\n

a = datetime.strptime(s1, date_format)

\n

so you want to put the try/except clause around this, but you want to change the date_format when you catch the exception. date_format appears to be set in another part of the script, so you need to reset this after you catch the exception. I would set date_format within the try/except clause for clarity, so the code would look something like this (put the most common date format first).

\n

try:

\n

    date_format = "%Y-%m-%d

\n

    a = datetime.strptime(s1, date_format)

\n

except ValueError:

\n

    date_format = "%m/%d/%Y"

\n

    a = datetime.strptime(s1, date_format)

\n\n

Edit: I now see that "a" is set within the one of the python modules. I cannot see all of your code, but you need to traceback to where in your code the exception is thrown and the date_format is set, and wrap the try/except clause around it there.

", "votes": null, "replies": [] }, { "id": 48419, "author_name": "sushize", "author_url": "", "post_date": "06/01/2014 04:33:18", "content": "

Dear Neil,

\n\n

Thanks for your suggestion. It works now. Good night!

\n\n

Best wishes,

\n

Shize

", "votes": null, "replies": [] }, { "id": 48420, "author_name": "sushize", "author_url": "", "post_date": "06/01/2014 04:55:11", "content": "

Dear Friends,

\n

I use Python 2.7.6 and VM 7.6 (Latest version) to run Triskelion's Python code, but I got a leader board score of 0.59104 rather than 0.59410. And I found that there are a couple of other competitors also have  such same score 0.59104. 

\n\n

Does anyone has ideas about what might cause such difference? Thanks in advance.

\n\n

Good night!

\n\n

Best wishes,

\n

Shize

", "votes": null, "replies": [] }, { "id": 48437, "author_name": "odessa", "author_url": "", "post_date": "06/01/2014 16:36:16", "content": "

[quote=Shize Su;48420]

\n

VM 7.6 (Latest version) 

\n

[/quote]

\n

Use VW version 7.1

", "votes": null, "replies": [] }, { "id": 48439, "author_name": "sushize", "author_url": "", "post_date": "06/01/2014 16:56:22", "content": "

Really?

\n

But Zach said that he used VW version 7.6 and got score of 0.59410. See the post at 

\n

http://www.kaggle.com/c/acquire-valued-shoppers-challenge/forums/t/7908/0-59410

\n

Thanks!

\n

Best wishes,

\n

Shize

", "votes": null, "replies": [] }, { "id": 48448, "author_name": "odessa", "author_url": "", "post_date": "06/01/2014 21:17:08", "content": "

Valid point!  I tested my code on version 7.6 and only got 0.59104.

", "votes": null, "replies": [] }, { "id": 48450, "author_name": "sushize", "author_url": "", "post_date": "06/01/2014 21:43:29", "content": "

Hi friends,

\n

I download VW 7.1 and test the code again, but still got the same score 0.59104. 

\n

Dear Odessa, are you sure that you run the code with VW 7.1 and got a score of 0.59410, while you run the same code with VW 7.6 and got a score of 0.59104? I thought there should be some other contributing factors for such difference.

\n\n

Thanks, and have a great day!

\n\n

Best wishes,

\n

Shize

", "votes": null, "replies": [] }, { "id": 48451, "author_name": "odessa", "author_url": "", "post_date": "06/01/2014 22:00:53", "content": "

No, with VW 7.1 I got only 0.59347, not 0.59410. I use 64 bit version of Cygwin for both.

", "votes": null, "replies": [] }, { "id": 48453, "author_name": "sushize", "author_url": "", "post_date": "06/01/2014 22:06:46", "content": "

Hi Odessa,

\n

I also use 64 bit version Cygwin for both, and I got score 0.59104 for both VW 7.1 and VW 7.6.    I have no idea what might cause such difference....

\n

Anyway, thanks!

\n

Best wishes,

\n

Shize

", "votes": null, "replies": [] }, { "id": 48454, "author_name": "triskelion", "author_url": "", "post_date": "06/01/2014 22:28:14", "content": "

[quote=Shize Su;48420]

\n

Does anyone has ideas about what might cause such difference? Thanks in advance.

\n

[/quote]

\n

The difference between Python 2.7 and 3.x should be in changing a few print statements. 0.59410 is caused by newer version of Vowpal Wabbit (equal to using no passes in 7.1 version I think). Since 7.4 VW has holdout functionality and stops after doing 4 passes or so.

\n

Try running any version without doing any passes at all.

", "votes": null, "replies": [] }, { "id": 48457, "author_name": "sushize", "author_url": "", "post_date": "06/01/2014 23:34:27", "content": "

Hi Triskelion,

\n

I tried VW 7.1 with no pass just now but got a score of 0.58717.

\n

As I said in my previous post, I actually once tried the latest version of VW, namely, VW 7.6, but got a score of 0.59104 rather than 0.59410. This is what make me confused, since Zach said that he used VW 7.6 and got a score of 0.59410.

\n

Any other thoughts?

\n\n

Thanks, and have a great day!

\n\n

Best wishes,

\n

Shize

", "votes": null, "replies": [] }, { "id": 48473, "author_name": "zachmayer", "author_url": "", "post_date": "06/02/2014 13:21:57", "content": "

[quote=Shize Su;48439]

\n

Really?

\n

But Zach said that he used VW version 7.6 and got score of 0.59410. See the post at 

\n

http://www.kaggle.com/c/acquire-valued-shoppers-challenge/forums/t/7908/0-59410

\n

Thanks!

\n

Best wishes,

\n

Shize

\n

[/quote]

\n\n

I'm pretty certain I was using 7.6.  Did you run the code exactly as written, or did you make any changes?

\n\n

Try with both 7.1 and 7.6 and see what happens.

", "votes": null, "replies": [] }, { "id": 48476, "author_name": "philculliton", "author_url": "", "post_date": "06/02/2014 13:49:05", "content": "

I'm finding I need to change the learning rate significantly with recent VW builds, in order to get similar results results to a VW version pulled a couple weeks ago.  My guess is folks following the tutorial with very recent builds are going to get very different results from those originally posted.

\n\n

More specifically, with recent builds I seem to get similar results with 0.5 as a learning rate, vice the original recommendation of 0.85.  Unfortunately the loss function seems to be less of a reliable indicator with VW now exiting / converging much earlier than in previous versions.

", "votes": null, "replies": [] }, { "id": 48482, "author_name": "sushize", "author_url": "", "post_date": "06/02/2014 14:39:54", "content": "

Hi Zach, 

\n

I run the code exactly as written, except that I change the paths for the data files (e.g., I change loc_offers = "data/offers.csv" to  loc_offers = "offers.csv".)

\n\n

I don't think such change about the data file paths would cause any difference about the accuracy, so I am a bit confused about what might cause the difference.

\n\n

I do have tried both VW 7.1 and VW 7.6, and they gave me exactly same score 0.59104.

\n\n

Thanks, and have a great day!

\n\n

Best wishes,

\n

Shize

", "votes": null, "replies": [] }, { "id": 48485, "author_name": "sushize", "author_url": "", "post_date": "06/02/2014 14:46:50", "content": "

Hi Phil,

\n\n

I tried learning rate 0.5 with VW 7.6 just now, but it gave me a score of  0.58984. What's the version of your VW?

\n

Thanks!

\n\n

Best wishes,

\n

Shize

", "votes": null, "replies": [] }, { "id": 48486, "author_name": "abhishek", "author_url": "", "post_date": "06/02/2014 14:49:05", "content": "

0.59104 is also good. You might wanna improve on that.

", "votes": null, "replies": [] }, { "id": 48487, "author_name": "zachmayer", "author_url": "", "post_date": "06/02/2014 14:53:09", "content": "

[quote=Shize Su;48482]

\n

Hi Zach, 

\n

I run the code exactly as written, except that I change the paths for the data files (e.g., I change loc_offers = "data/offers.csv" to  loc_offers = "offers.csv".)

\n

I don't think such change about the data file paths would cause any difference about the accuracy, so I am a bit confused about what might cause the difference.

\n

I do have tried both VW 7.1 and VW 7.6, and they gave me exactly same score 0.59104.

\n

Thanks, and have a great day!

\n

Best wishes,

\n

Shize

\n

[/quote]

\n\n

Interesting.  For what it's worth, I'm on a mac.  0.59104 is the score a lot of other people got with the earlier version of VW, and they may have been on windows.

", "votes": null, "replies": [] }, { "id": 48488, "author_name": "sushize", "author_url": "", "post_date": "06/02/2014 14:55:30", "content": "

Hi Abhishek,

\n

I do agree that 0.59104 is good. And currently I don't care about the performance at all since I just start this competition. In fact,  I am just curious about what might cause such difference, since everything seems to be pretty clear.  It seems interesting for me!

\n

Thanks,

\n

Shize

", "votes": null, "replies": [] }, { "id": 48489, "author_name": "sushize", "author_url": "", "post_date": "06/02/2014 14:57:52", "content": "

Hi Zach,

\n

You're right, I do run the code on Windows. Both VW 7.1 and VW 7.6 gave me same score 0.59104. So based on this, maybe the difference between 0.59410 and 0.59104 is simply caused by Mac and Windows? Seems pretty interesting!

\n

Best wishes,

\n

Shize

", "votes": null, "replies": [] }, { "id": 48490, "author_name": "abhishek", "author_url": "", "post_date": "06/02/2014 15:05:54", "content": "

I'm on a mac too and my score with VW in the begining was also 0.59410

", "votes": null, "replies": [] }, { "id": 48494, "author_name": "philculliton", "author_url": "", "post_date": "06/02/2014 15:26:24", "content": "

[quote=Shize Su;48485]

\n

Hi Phil,

\n

I tried learning rate 0.5 with VW 7.6 just now, but it gave me a score of 0.58984. What's the version of your VW?

\n

Thanks!

\n

[/quote]

\n\n

I'm using 7.6.1.  I'm also (mostly) on a Mac, although I'm using Windows and Linux builds as well.

\n

WRT to the score you got - yeah, I haven't quite matched my 7.6 results with 7.6.1.  For now I'm using a 7.6 build for most of my VW work.  0.85 doesn't give me good results in 7.6.1, so anyone using 7.6.1 may want to play with the learning rate a bit.

\n

I suppose my main point is that VW changes often.  People following the (excellent) MLWave install tutorial for Windows are going to end up with brand new code which may or may not behave very differently.

", "votes": null, "replies": [] }, { "id": 48495, "author_name": "abhishek", "author_url": "", "post_date": "06/02/2014 15:27:55", "content": "

AFAIK, Triskelion is using Windows.

", "votes": null, "replies": [] }, { "id": 48499, "author_name": "sushize", "author_url": "", "post_date": "06/02/2014 16:19:24", "content": "

Hi Abhishek,

\n\n

Thanks for your provided information. But maybe Triskelion run the code to get score of 0.59347 a couple of weeks ago, and the reason posted by Phil in this posting might cause the difference between my result of 0.59104 and his result of 0.59347.

\n\n

And Phil, could you let me know your operation system (windows or mac?) and the version of VW you use? Thanks!

\n\n

Anyway, can anyone who uses Windows and VW 7.6 (or VW 7.1) verify that they run the same python code recently and get a score of 0.59410 (or 0.59347)?

\n\n

Thanks in advance.

\n\n

Best wishes,

\n

Shize

", "votes": null, "replies": [] }, { "id": 48514, "author_name": "triskelion", "author_url": "", "post_date": "06/02/2014 19:49:17", "content": "

It is possible that different builds of Vowpal Wabbit give slightly different results. As far as I see this is mostly irrelevant: focus on getting the best score for your build and features. The difference is small enough to be variance in public leaderboard. You can't say for sure which is better.

\n

The settings I gave were for a quick start, they are not the best settings possible.

\n

Vowpal Wabbit's hashing algorithm then also gives different results for little endian and big endian systems: https://github.com/JohnLangford/vowpal_wabbit/blob/master/vowpalwabbit/hash.cc

\n

For the record:

\n

7.1 64-bit Windows = ~0.59347

\n

7.6.1 64-bit Cygwin on Windows = ~0.59410

\n

@Shize Su, I don't know why you are getting 0.59347 with 7.6.1, most are getting 0.59410 with 7.6.1.

\n

I got my current score with 7.1 BTW using just small adjustments (mostly removing things I thought were fancy). I may share these when I optimize my current score some more (if I ever get time for this).

", "votes": null, "replies": [] }, { "id": 48515, "author_name": "sushize", "author_url": "", "post_date": "06/02/2014 20:37:54", "content": "

Hi Triskelion,

\n

Thanks for your reply. Actually, I do agree with you that there is no significant difference between 0.59410 and 0.59104 at all. I was not caring about the score since I just start this competition; instead, I am just quite curious about what subtle reasons might cause such difference since everything seems to be that straight forward and clear for me. 

\n

Anyway, if no one can give an precise answer for this issue, I will just forget about it and start to work on my progress.

\n

P.S.: Triskelion, to clarify, I got score of 0.59104 on both VW 7.6 and VW 7.1, not 0.59347. I use windows operation system.

\n

Thanks, and have a great day!

\n

Best wishes,

\n

Shize

", "votes": null, "replies": [] }, { "id": 48516, "author_name": "philculliton", "author_url": "", "post_date": "06/02/2014 20:45:05", "content": "

Hmm, thanks.  I got wildly different leaderboard scores between 7.6 and 7.6.1 (0.59990 vs 0.59033) on the same data / VW settings / machine, and had assumed it was due to changes made to the learning algorithm, hence my warning.  However, if everybody else is getting very similar scores between versions I may need to dig deeper at some point and find out what's really going on.  :-)

\n\n

Edit: I was able to reproduce it just now - my current high score 0.60393 (IIRC) set, when run against the latest VW, gets ~0.594 or so.  Must be something I'm doing... but this one took place under sterile conditions, everything matched up correctly, so definitely not a one-off mistake.  Odd.

", "votes": null, "replies": [] }, { "id": 48527, "author_name": "alexanderriggers", "author_url": "", "post_date": "06/02/2014 22:43:21", "content": "

Can somebody give me access to the reduced 1.6 GB file? I tried the Python script, but I always get a syntax error

\n\n

Kind regards,

\n

Alex

", "votes": null, "replies": [] }, { "id": 48535, "author_name": "philculliton", "author_url": "", "post_date": "06/03/2014 00:07:20", "content": "

I may be able to provide a link to a reduced file if necessary, but it'd probably be best to get you up and running - the script is *really* useful!  :-)  What kind of syntax error are you getting?

", "votes": null, "replies": [] }, { "id": 48541, "author_name": "sushize", "author_url": "", "post_date": "06/03/2014 01:18:42", "content": "

Hi Alexander,

\n

Are you using Python version 3.* to run the code?

\n

Try to use Python version 2.*, for example, python 2.7 to run the code as written (except changing the paths for the data files), and there should not be any syntax errors.

\n

I once use python 3.4 to run the code and came across the same syntax error problems, and then as suggested by Scott, I turned to python 2.7, and then there were no syntax errors.

\n

Hope this would be helpful.

\n

Have a great night!

\n

Best wishes,

\n

Shize

", "votes": null, "replies": [] }, { "id": 48547, "author_name": "chitrasen", "author_url": "", "post_date": "06/03/2014 05:25:42", "content": "

While VW is used widely for this completion , I  tried with my favorite scikit-learn. With few more features along with what is mentioned by Triskelion's benchmark code, I could fetch 0.59449 (little better than VW benchmark 0.59410). I used RF without any tuning. I am hoping for better scores with tuning the params.

\n\n

My point is, one shouldn't stuck with VW version issues while there are alternate options to try out.

", "votes": null, "replies": [] }, { "id": 48549, "author_name": "dremovd", "author_url": "", "post_date": "06/03/2014 07:03:23", "content": "

I also tried these both, so I can say that it has close results, so we hope to use them both

", "votes": null, "replies": [] }, { "id": 48555, "author_name": "philculliton", "author_url": "", "post_date": "06/03/2014 12:37:53", "content": "

[quote=Chitrasen;48547]

\n

While VW is used widely for this completion , I  tried with my favorite scikit-learn. With few more features along with what is mentioned by Triskelion's benchmark code, I could fetch 0.59449 (little better than VW benchmark 0.59410). I used RF without any tuning. I am hoping for better scores with tuning the params.

\n

My point is, one shouldn't stuck with VW version issues while there are alternate options to try out.

\n

[/quote]

\n

Agreed.  Would you mind posting your early-benchmark sklearn code somewhere to help folks get started down that path?  If it's really simple all the better.

\n

Edit: without the extra features, etc., of course.  Not asking you to give anything away, just the sklearn boilerplate.  :-)

", "votes": null, "replies": [] }, { "id": 48558, "author_name": "alexanderriggers", "author_url": "", "post_date": "06/03/2014 13:20:46", "content": "

Hi Phil, Hi Shize

\n\n

I use Python 2.7 but I am a beginner. I changed the code to my home directory, where I saved my  csv.files:

\n

loc_offers = "offers.csv"
loc_transactions = "transactions.csv"
loc_reduced = "reduced2.csv" # will be created

\n

But after copy and paste the first line of code I get a syntax error:

\n

def reduce_data(loc_offers, loc_transactions, loc_reduced):  start = datetime.now()
  #get all categories on offer in a dict
  offers = {}
  for e, line in enumerate( open(loc_offers) ):
    offers[ line.split(",")[1] ] = 1

\n

Actually I need the 1.6GB csv for a test if I can read 1.6 GB file from S3 to Graphlabb Create:

\n

http://forum.graphlab.com/discussion/251/s3-sframe-read-problems

\n

Kind regards,

\n

Alex

", "votes": null, "replies": [] }, { "id": 48561, "author_name": "philculliton", "author_url": "", "post_date": "06/03/2014 14:07:14", "content": "

If you're doing this in the python console you might be better served by running the whole file all at once:

\n

run "gen_vw_features.py"

\n

Also, if you're not going to provide specific paths, make sure you're running python / ipython from the directory where the data files reside, or set the working directory in some other way.

\n\n

Lastly, here's a link to the reduced csv:

\n

https://dl.dropboxusercontent.com/u/12189505/reduced.csv.zip

\n\n

If anyone decides to put it somewhere more performant, let me know and I'll take the link down.  :-)

", "votes": null, "replies": [] }, { "id": 48595, "author_name": "chitrasen", "author_url": "", "post_date": "06/04/2014 05:22:07", "content": "

[quote=Phil Culliton;48555]

\n\n

Agreed.  Would you mind posting your early-benchmark sklearn code somewhere to help folks get started down that path?  If it's really simple all the better.

\n

Edit: without the extra features, etc., of course.  Not asking you to give anything away, just the sklearn boilerplate.  :-)

\n

[/quote]

\n\n

Here is my skeleton code using Pandas and scikit-learn.

\n

Note: My original features have been removed from the code. Code is extracted from my ipython notebook, hence may not be in proper python coding standard. I am sorry for that.

\n\n\n\n

I also added the code to my blog.

", "votes": null, "replies": [] }, { "id": 48615, "author_name": "rzykov", "author_url": "", "post_date": "06/04/2014 11:39:05", "content": "

Chitrasen, thank you for code sharing. Did you do a cross validation via sklearn?

", "votes": null, "replies": [] }, { "id": 48616, "author_name": "chitrasen", "author_url": "", "post_date": "06/04/2014 11:51:40", "content": "

[quote=rzykov;48615]

\n

Chitrasen, thank you for code sharing. Did you do a cross validation via sklearn?

\n

[/quote]

\n\n

Yes, I did some CV testing, but seems test and train data distributions are either different, or there seems some pattern I am missing to account into. I am getting similar results as mentioned in this thread.

\n\n

I was getting 4 fold CV score of 0.68XXX with the sample code, I provided. (my current CV score improved to 0.715).

", "votes": null, "replies": [] }, { "id": 48617, "author_name": "rzykov", "author_url": "", "post_date": "06/04/2014 12:05:09", "content": "

I do a cross validation over all offers. It is like K-fold CV where each fold is a an offer.

\n

Then calculate mean of CV's errors. It's very close to my test results.

", "votes": null, "replies": [] }, { "id": 48623, "author_name": "paulomagalhaes", "author_url": "", "post_date": "06/04/2014 12:49:27", "content": "

I just wanted to leave my contribution to this great, collaborative, thread.

\n

I used the whole 22Gb transaction data and ran svm + logistic regression with the features as suggested by Triskelion. The result was a 0.59735 score. Running the same set up with the reduced data set produced a 0.58627 score. Therefore, in this case, using more data improved the score by 0.01 . 

", "votes": null, "replies": [] }, { "id": 48630, "author_name": "triskelion", "author_url": "", "post_date": "06/04/2014 14:11:30", "content": "

[quote=Chitrasen;48595]

\n

Here is my skeleton code using Pandas and scikit-learn.

\n

[/quote]

\n

What score does this give on public leaderboard?

", "votes": null, "replies": [] }, { "id": 48632, "author_name": "triskelion", "author_url": "", "post_date": "06/04/2014 15:02:25", "content": "

[quote=Paulo;48623]

\n

I just wanted to leave my contribution to this great, collaborative, thread.

\n

I used the whole 22Gb transaction data and ran svm + logistic regression with the features as suggested by Triskelion. The result was a 0.59735 score. Running the same set up with the reduced data set produced a 0.58627 score. Therefore, in this case, using more data improved the score by 0.01 . 

\n

[/quote]

\n

Cool, this is a useful tip. I wonder why this gives so much of a difference though. Maybe the features that hold "total amount spend" become more indicative?

\n\n

I'll try these with my new settings and report back if it improves. I finally got a little time to optimize my current near-leaderboard score and try out some new features.

", "votes": null, "replies": [] }, { "id": 48669, "author_name": "chitrasen", "author_url": "", "post_date": "06/04/2014 22:41:06", "content": "

[quote=Triskelion;48630]

\n

[quote=Chitrasen;48595]

\n

Here is my skeleton code using Pandas and scikit-learn.

\n

[/quote]

\n

What score does this give on public leaderboard?

\n

[/quote]

\n\n

The additional features like 'median price' and 'median amount' and some others worked better. My initial score with those features and the skeleton code, was 0.59449 on leaderboard.

\n

My current leaderboard score(0.60343) uses some more features and tuning of RF params.

", "votes": null, "replies": [] }, { "id": 48696, "author_name": "mike64", "author_url": "", "post_date": "06/05/2014 09:20:46", "content": "

Only one question,

\n\n

how well the model classifies among repeaters (t) ?

", "votes": null, "replies": [] }, { "id": 48710, "author_name": "philculliton", "author_url": "", "post_date": "06/05/2014 12:21:09", "content": "

Hmm.  I tried your code (exactly as-is, with no changes), Chitrasen, and got a submission error regarding the ids - they were float64 and the submission code wanted them not to be.  I switched them to int64, and got a leaderboard score of 0.54190.  Wonder what's going on there?

", "votes": null, "replies": [] }, { "id": 48716, "author_name": "selfishgene", "author_url": "", "post_date": "06/05/2014 13:24:45", "content": "

I think 0.54190 is OK for this code, it only has features that include only category and company information.

", "votes": null, "replies": [] }, { "id": 48721, "author_name": "philculliton", "author_url": "", "post_date": "06/05/2014 14:08:29", "content": "

[quote=Selfish Gene;48716]

\n

I think 0.54190 is OK for this code, it only has features that include only category and company information.

\n

[/quote]

\n

True, but I figured I'd ask to make sure!  Since newcomers might be using this code, I thought it would be useful to know what the approximate leaderboard score might be, and wanted to double-check my result.  Also, to report the id bug.  :-)

", "votes": null, "replies": [] }, { "id": 48747, "author_name": "nhc191735", "author_url": "", "post_date": "06/06/2014 02:34:43", "content": "

I face this errors...

\n

Can anyone help me "how to solve this problem?"

\n

ValueError: time data '1' does not match format '%Y-%m-%d'

", "votes": null, "replies": [] }, { "id": 48755, "author_name": "neikusc", "author_url": "", "post_date": "06/06/2014 06:26:36", "content": "

[quote=Chitrasen;48669]
The additional features like 'median price' and 'median amount' and some others worked better. My initial score with those features and the skeleton code, was 0.59449 on leaderboard.
My current leaderboard score(0.60343) uses some more features and tuning of RF params.
[/quote]

\n

Hi Chitrasen, I used RF model with 5-fold CV on my training set and got score 0.675xx. But when I applied the same model on my test set, my prediction was just like random, score 0.49476. Did you see any similar thing? Do you have any suggestions? Thanks!

", "votes": null, "replies": [] }, { "id": 48758, "author_name": "chitrasen", "author_url": "", "post_date": "06/06/2014 07:01:27", "content": "

[quote=Kien Trinh;48755]

\n

Hi CHi Chitrasen, I used RF model with 5-fold CV on my training set and got score 0.675xx. But when I applied the same model on my test set, my prediction was just like random, score 0.49476. Did you see any similar thing? Do you have any suggestions? Thanks!

\n

[/quote]

\n

I am also struggling to get a reliable CV testing technique :(

\n

Yes,I see the similar effect as you are getting. To put time effect on CV, I sort the training data with offerdate and then apply K-Fold CV. The CV scores vary from 0.58 to 0.68 and on the leader board it scored 0.572 ( I have used some of my features, not the features mentioned in my previous post on skeleton code).

", "votes": null, "replies": [] }, { "id": 48763, "author_name": "dremovd", "author_url": "", "post_date": "06/06/2014 08:59:34", "content": "

rzykov mentioned very good idea: to train on some offers and to test on others, for those who struggling to get a reliable CV technique it's worth trying. It should be better for sure then any standard CV.

", "votes": null, "replies": [] }, { "id": 48773, "author_name": "mike64", "author_url": "", "post_date": "06/06/2014 12:45:17", "content": "

Christansen,

\n\n

only a very specific question that nobody answers me and it's very easy:

\n\n

there are 43.438 customers who repeated. Well, of these 43.438 customers, how you classify correctly?

\n\n

Kind regards

", "votes": null, "replies": [] }, { "id": 48807, "author_name": "neikusc", "author_url": "", "post_date": "06/06/2014 21:00:40", "content": "

[quote=mike;48773]

\n

Christansen,

\n

only a very specific question that nobody answers me and it's very easy:

\n

there are 43.438 customers who repeated. Well, of these 43.438 customers, how you classify correctly?

\n

Kind regards

\n

[/quote]

\n

Mike - you are doing binary classification {0 or 1}. There are 43438 outcomes { 1 } on training set. You are predicting the probability a shopper will be come a loyal customer { 1 } on test set. You submission file is a list of probability in a range [0,1]. I hope I answer your question. Best!

", "votes": null, "replies": [] }, { "id": 48853, "author_name": "ejlok1", "author_url": "", "post_date": "06/08/2014 02:29:11", "content": "

[quote=nhc;48747]

\n

I face this errors...

\n

Can anyone help me "how to solve this problem?"

\n

ValueError: time data '1' does not match format '%Y-%m-%d'

\n

[/quote]

\n\n

Can you elaborate abit more about the error? I'm going to take a stab here and say that its because the test and train history data has different date format. You'll have to treat them both seperately

", "votes": null, "replies": [] }, { "id": 48857, "author_name": "nhc191735", "author_url": "", "post_date": "06/08/2014 10:49:40", "content": "

thank you Eu Jin Lok!!! I find the problem on my trainHistory Data!!! I make a mistake to check only transaction data...

", "votes": null, "replies": [] }, { "id": 48864, "author_name": "mike64", "author_url": "", "post_date": "06/08/2014 16:47:25", "content": "

thanks for you answer.

\n

I clearly explain bad, my question is: if we consider a probability above 0.5 as repeater, then, how well your model classifies ?

\n\n

bye 

", "votes": null, "replies": [] }, { "id": 48872, "author_name": "omalleyt", "author_url": "", "post_date": "06/08/2014 20:28:47", "content": "

Just a note on this program, as I tried it out the other day (and it's a great and I've learned a lot. Thank you to the submitter!) There are some customers in trainhistory and testhistory who have no data, and from what I can tell they don't end up in the training set or more importantly the prediction set (there are less total rows than in the original trainhistory+testhistory). I don't know how kaggle handles this.

", "votes": null, "replies": [] }, { "id": 48875, "author_name": "neikusc", "author_url": "", "post_date": "06/08/2014 21:20:38", "content": "

[quote=Thomas O'Malley;48872]

\n

Just a note on this program, as I tried it out the other day (and it's a great and I've learned a lot. Thank you to the submitter!) There are some customers in trainhistory and testhistory who have no data, and from what I can tell they don't end up in the training set or more importantly the prediction set (there are less total rows than in the original trainhistory+testhistory). I don't know how kaggle handles this.

\n

[/quote]

\n

If you use transactions.csv file, the number of shoppers is exactly equal to the number of shoppers in two files: trainHistory.csv and testHistory.csv. 

", "votes": null, "replies": [] }, { "id": 48883, "author_name": "namesj", "author_url": "", "post_date": "06/09/2014 06:20:46", "content": "

You can beat the "prior category benchmark" without even using the transaction file.

\n

Just a logistic fit on trainhistory  based on  variables suggested by varImpPlot()  on randomForest  model.   :-)

", "votes": null, "replies": [] }, { "id": 48886, "author_name": "namesj", "author_url": "", "post_date": "06/09/2014 07:19:05", "content": "

for offerdate  , Changing factor to 'as.Date' worked for me   .

", "votes": null, "replies": [] }, { "id": 48895, "author_name": "triskelion", "author_url": "", "post_date": "06/09/2014 12:07:51", "content": "

[quote=Thomas O'Malley;48872]

\n

Just a note on this program, as I tried it out the other day (and it's a great and I've learned a lot. Thank you to the submitter!) There are some customers in trainhistory and testhistory who have no data, and from what I can tell they don't end up in the training set or more importantly the prediction set (there are less total rows than in the original trainhistory+testhistory). I don't know how kaggle handles this.

\n

[/quote]

\n

I handled this in the benchmark by predicting a 0 for these customers. Kaggle submissions need all the entries from the original test set, else it will throw a warning.

", "votes": null, "replies": [] }, { "id": 48896, "author_name": "dremovd", "author_url": "", "post_date": "06/09/2014 12:10:48", "content": "

I think Kien is right.

\n

There are exactly same users in raw transactions as in train + test. So, there are no users that have no data unless you didn't filter transactions first.

", "votes": null, "replies": [] }, { "id": 48996, "author_name": "peemblis", "author_url": "", "post_date": "06/11/2014 13:05:39", "content": "

[quote=Shize Su;48489]

\n

Hi Zach,

\n

You're right, I do run the code on Windows. Both VW 7.1 and VW 7.6 gave me same score 0.59104. So based on this, maybe the difference between 0.59410 and 0.59104 is simply caused by Mac and Windows? Seems pretty interesting!

\n

Best wishes,

\n

Shize

\n

[/quote]

\n\n

Hi Shize Su, I ran Triskelion's method a couple nights ago with VW 7.6 on Windows. I got the same score (0.59104 ) as you.

\n\n

Triskelion, thank you very much for sharing the details of your method.

", "votes": null, "replies": [] }, { "id": 49006, "author_name": "alexanderriggers", "author_url": "", "post_date": "06/11/2014 16:09:37", "content": "

Hi guys

\n

Do you think its possible to run Trischlikons model on a GraphLab Create EC2 instance: 

\n

http://graphlab.com/products/create/docs/generated/graphlab.vowpal_wabbit.create.html#graphlab.vowpal_wabbit.create

\n

I am not quite sure how to transmit the the features from Python to this EC2 instance, but it could be an advantage for the full 20 GB file.

", "votes": null, "replies": [] }, { "id": 49053, "author_name": "dansterba", "author_url": "", "post_date": "06/12/2014 13:43:11", "content": "

Hi All -

\n\n

I've been using R to try and replicate the results that you all have received with Python and VW. Logistic regression using the Lasso method only got me up to .57 with the engineered features, so I wanted to try quantile regression, but am having a hard time figuring out how to run the same kind of quantile regression in R.

\n\n

The most promising package seemed to be 'quantreg', but I've gotten errors with it every time

\n\n

When running: rq(data=tempTR,as.formula(x),tau=.6)   where tempTR is my data.frame with engineered values and 'x' is the text version of my formula, I get the following error

\n

* Error in rq.fit.br(x, y, tau = tau, ...) : Singular design matrix

\n\n

I've also attempted to try other models that looked promising in the 'quantreg' package, such as 'rqProcess': which in the R-help documentation, says: "Computes a standardize quantile regression process for the model specified by the formula, on the partition of [0,1] specified by the taus argument, and standardized according to the argument nullH. Intended for use in KhmaladzeTest.

\n\n

rqProcess(as.formula(x),data=tempTR,taus=c(.5))

\n\n

The problem with this one is it requires I specify a range of 'equally spaced taus' , when I do this:

\n\n

rqProcess(as.formula(x),data=tempTR,taus=c(.2,.4,.6,.8))

\n\n

I get an error saying 'singular design matrix' again.

\n\n

Can anyone advise?

\n\n

My formula is below, and through troubleshooting, I've just narrowed it down to the metrics posted earlier on this thread:

\n\n

"repeater ~ I(last_600_count_brand == 0) + I(last_600_count_brand > 0 & last_600_count_company > 0 & last_600_count_category >0) + I(last_60_count_brand > 0) + I(last_600_count_brand > 0 & last_600_count_company > 0) + I(last_30_count_brand > 0) + I(last_600_count_brand > 0 & last_600_count_category > 0) + I(last_600_count_brand > 0) + last_30_monVal_brand + I(last_90_count_brand > 0) + last_60_monVal_brand + I(last_60_count_company > 0) + last_90_monVal_brand + last_60_q_company + I(last_600_count_category == 0) + last_600_q_brand + last_180_monVal_brand + I(last_180_count_brand > 0) + offervalue + last_30_q_category"

", "votes": null, "replies": [] }, { "id": 49082, "author_name": "sthompson", "author_url": "", "post_date": "06/13/2014 03:44:08", "content": "

ddunder: Having never worked with the quantreg package I can only hazard some guesses.

\n\n

Based on the message returned by R, I'd guess that quantreg has a matrix inversion in it somewhere and the programmers have set the code to detect linear dependence in the design matrix (nice to see some defensive programming). I would reproduce the design matrix (in R: design.matrix) and then check for linear dependence. Given the formula indicated, I'd look for equal columns in the matrix.

", "votes": null, "replies": [] }, { "id": 49126, "author_name": "spimenov", "author_url": "", "post_date": "06/13/2014 20:34:05", "content": "

quantreg works for me... not great;I got only 0.58364:

\n\n

cols <- c("V167", ...)
dt <- data.frame(train[,cols])
dt$y <- idy$repeattrips
model <- rq(y~., data=dt, method='fn', tau=0.5)
submit$repeatProbability <- predict(model, test[,cols])

\n\n

Yes, the feature selection (10 of 350) was done first; the CV (10 folds) was 6.02...

", "votes": null, "replies": [] }, { "id": 49257, "author_name": "alexanderriggers", "author_url": "", "post_date": "06/17/2014 16:49:30", "content": "

Thank you, but when I try to run the gen_vw_features.py file I get:

\n

Alexanders-iMac:~ arigge$ python gen_vw_features.py
File "gen_vw_features.py", line 6
<!DOCTYPE html>
^
SyntaxError: invalid syntax

\n

I placed all my csv files in a folder called data in my home directory arigge. Can somebody tell me what I am missing ?

", "votes": null, "replies": [] }, { "id": 49259, "author_name": "sushize", "author_url": "", "post_date": "06/17/2014 18:20:43", "content": "

Hi Alexander,

\n\n

Are you using Python 2.7 or Python 3.*?

\n

~

\n

To run the posted code by Triskelion, you need to use Python 2.7 version rather than 3.* version. They have quite different syntax.

\n

~

\n\n

Best wishes,

\n

Shize

", "votes": null, "replies": [] }, { "id": 49260, "author_name": "alexanderriggers", "author_url": "", "post_date": "06/17/2014 19:18:51", "content": "

Thank you Shize.

\n\n

I use Python 2.7 which runs on Mac by default.

\n\n

Best,

\n

Alex 

", "votes": null, "replies": [] }, { "id": 49261, "author_name": "sushize", "author_url": "", "post_date": "06/17/2014 19:50:08", "content": "

Hi Alexander,

\n\n

Uh, interesting.

\n

~

\n

For me, I just download the code, change the location for data files, and run it with Python 2.7 on windows, and then everything works.

\n

~

\n

Besides, are you sure that the only change you made to the Python code is just the data file locations? If yes, then I don't know why it doesn't work on your computer. Maybe you could  try to run the same code on another computer and see what happens. It seems to me that there is no reason why it will not work.

\n

~

\n

Have a great day!

\n\n

Best wishes,

\n

Shize

", "votes": null, "replies": [] }, { "id": 49268, "author_name": "nthanhtam", "author_url": "", "post_date": "06/18/2014 03:55:51", "content": "

Hi Alex,

\n

I am also using Mac and Python 2.7 without any problem.

\n\n

Best,

\n

Tam

", "votes": null, "replies": [] }, { "id": 49269, "author_name": "", "author_url": "", "post_date": "06/18/2014 04:33:46", "content": "

Is it important to normalize/standardize the data, since vw seems to be some sort of gradient-based algorithm (plz correct me if I'm wrong)??

\n\n

Triskelion, thanks a million for sharing your solution and introducing us to the powers of vw.

", "votes": null, "replies": [] }, { "id": 49286, "author_name": "rohits", "author_url": "", "post_date": "06/19/2014 03:12:17", "content": "

Check for whitespace. Make sure you don't have any stray spaces on empty lines. Especially line 6.

\n\n

[quote=Alexander Riggers;49257]

\n

Thank you, but when I try to run the gen_vw_features.py file I get:

\n

Alexanders-iMac:~ arigge$ python gen_vw_features.py
File "gen_vw_features.py", line 6
<!DOCTYPE html>
^
SyntaxError: invalid syntax

\n

I placed all my csv files in a folder called data in my home directory arigge. Can somebody tell me what I am missing ?

\n

[/quote]

", "votes": null, "replies": [] }, { "id": 49292, "author_name": "", "author_url": "", "post_date": "06/19/2014 07:57:07", "content": "

May I ask what the "average loss" and "since last" columns converged to towards the end? I've had cases where the predictions are quite different but those 2 columns gave similar results during training. Then how does one tell which one is better?

\n\n

Many thanks in advance!

", "votes": null, "replies": [] }, { "id": 49293, "author_name": "", "author_url": "", "post_date": "06/19/2014 08:02:58", "content": "

Sorry another question: are the Y's used here repeater (i.e. 0s and 1s) or number of repeats?

", "votes": null, "replies": [] }, { "id": 49304, "author_name": "philculliton", "author_url": "", "post_date": "06/19/2014 15:06:04", "content": "

Wanted to note two things quickly:

\n

1) Namespaces!  They're a great feature of Vowpal Wabbit.  I use them to quickly test features - just put each new feature / feature class into its own namespace, then turn them on and off at the vw command line (using --keep or --ignore) rather than rebuilding the train/test files for each possible combination of features.  Other folks are probably already doing this, but I wanted to throw it out there.

\n\n\n

2) If using namespaces, be aware that unless specified, testing (with -t) will use ALL namespaces, and could change your results slightly.

", "votes": null, "replies": [] }, { "id": 49999, "author_name": "philculliton", "author_url": "", "post_date": "07/03/2014 16:01:40", "content": "

By the way - Triskelion, I tried to get in touch with you via your website a few weeks ago about VW.  I'm guessing it didn't go through, and I'm still too much of a Kaggle newbie to be able to email other users, so if you'd like to, please feel free to email me.  :-)

", "votes": null, "replies": [] }, { "id": 50002, "author_name": "triskelion", "author_url": "", "post_date": "07/03/2014 17:51:21", "content": "

[quote=Phil Culliton;49999]

\n

By the way - Triskelion, I tried to get in touch with you via your website a few weeks ago about VW.  I'm guessing it didn't go through, and I'm still too much of a Kaggle newbie to be able to email other users, so if you'd like to, please feel free to email me.  :-)

\n

[/quote]

\n

Whoops, yeah that manual checking of spam vs. ham comments took more time then I expected. Sorry for that, I'll go through the existing queue and install Akismet or something so this won't happen again.

\n\n

I think you can email other users by clicking their profile and then "contact", but maybe you have to be Kaggler for that first. I send you an email over the Kaggle comment system.

\n\n

Thank you for all your comments, here and elsewhere, and helping everyone out!

", "votes": null, "replies": [] }, { "id": 50004, "author_name": "philculliton", "author_url": "", "post_date": "07/03/2014 18:27:16", "content": "

[quote=Triskelion;50002]

\n

Whoops, yeah that manual checking of spam vs. ham comments took more time then I expected. Sorry for that, I'll go through the existing queue and install Akismet or something so this won't happen again.

\n

[/quote]

\n\n

No worries at all, that sort of stuff can creep up on you.

\n\n

[quote=Triskelion;50002]

\n

I think you can email other users by clicking their profile and then "contact", but maybe you have to be Kaggler for that first. I send you an email over the Kaggle comment system.

\n

[/quote]

\n\n

I'm a Kaggler, but I apparently need more points to be able to contact others.  Thanks for the email!

\n\n

[quote=Triskelion;50002]

\n

Thank you for all your comments, here and elsewhere, and helping everyone out!

\n

[/quote]

\n\n

Just taking an example from you, really!  And from many of the other generous folks here.  I've been really impressed by all of the people stepping forward to help folks figure out the basics.  I'd love to help even more myself, if possible.

", "votes": null, "replies": [] }, { "id": 50011, "author_name": "benjaminpost", "author_url": "", "post_date": "07/03/2014 23:00:58", "content": "

[quote=Peemblis;48996]

\n

[quote=Shize Su;48489]

\n

Hi Zach,

\n

You're right, I do run the code on Windows. Both VW 7.1 and VW 7.6 gave me same score 0.59104. So based on this, maybe the difference between 0.59410 and 0.59104 is simply caused by Mac and Windows? Seems pretty interesting!

\n

Best wishes,

\n

Shize

\n

[/quote]

\n

Hi Shize Su, I ran Triskelion's method a couple nights ago with VW 7.6 on Windows. I got the same score (0.59104 ) as you.

\n

Triskelion, thank you very much for sharing the details of your method.

\n

[/quote]

\n

Out of curiosity, do you find you have probability scores that have a truncated decimal string (such as random forest in R)? I have not looked at VW so I am not familiar with the output but when using randomforest in R, I believe it truncates to 3 or 4 decimal places giving many records the same score (especially when using many binary features). A simple change in order upon submission can throw off the final score, even when running the identical model but with a different ordering of input records on each iteration.

", "votes": null, "replies": [] }, { "id": 50056, "author_name": "triskelion", "author_url": "", "post_date": "07/04/2014 23:10:20", "content": "

[quote=Ben Post;50011]

\n

Out of curiosity, do you find you have probability scores that have a truncated decimal string (such as random forest in R)?

\n

[/quote]

\n

It's likely that they use the same script to convert a VW predictions file to a Kaggle submission.

\n

.

\n

It seems that VW can produce different results depending on your CPU type. VW is compiled with ffast-math flag, which is the least conforming but fastest math mode.

", "votes": null, "replies": [] }, { "id": 50058, "author_name": "philculliton", "author_url": "", "post_date": "07/05/2014 02:37:42", "content": "

[quote=Triskelion;50056]

\n

It seems that VW can produce different results depending on your CPU type. VW is compiled with ffast-math flag, which is the least conforming but fastest math mode.

\n

[/quote]

\n

And compiler type!  Every compiler plays with math-heavy code differently, and interprets things like fast-math differently.  I wouldn't be at all surprised to find that VW compiled with GCC would behave slightly differently than with, say, Intel or MSVCPP.

", "votes": null, "replies": [] }, { "id": 50090, "author_name": "pengyan", "author_url": "", "post_date": "07/07/2014 04:05:53", "content": "

i got 0.59474 by Triskelion's feature and vw command line. when i tried to add some features , i got worse scores and curiously found many predict results were zero. after tuned vw parameter lambda，nothing changed.  

", "votes": null, "replies": [] }, { "id": 50210, "author_name": "philculliton", "author_url": "", "post_date": "07/09/2014 15:55:19", "content": "

What features did you try to add?  And what changes did you make to VW parameters?  I've found that Triskelion's initial settings are actually a pretty darned good fit.

\n\n

Also - when did you grab your Vowpal Wabbit build?  Did you pull straight from github?  There are some recent bugs, I think - some nasty issues with not properly updating average loss, etc.

", "votes": null, "replies": [] }, { "id": 50218, "author_name": "alexanderriggers", "author_url": "", "post_date": "07/09/2014 17:21:53", "content": "

I have a question regarding the gen_feature_vw.py output. Why do all entries in the test.vw file have a label of one? 

\n\n

VW input format is:

\n

label [weight] |Namespace Feature ... Feature |Namespace ...

\n\n

In our case 1`lastid ....

\n

While int the train file we also have some 0 labels.

", "votes": null, "replies": [] }, { "id": 50219, "author_name": "philculliton", "author_url": "", "post_date": "07/09/2014 17:36:30", "content": "

VW ignores labels when in test mode - so it doesn't matter what's there.  However, something does need to be there, so we put in some dummy information.  When using the train file we're (most likely) actually training the VW model, so you put real information in the labels.

\n\n

One nice side effect of using all 1s as the dummy labels is that you can use average loss on the test run to gauge how "high" or "low" your current set of features might be pushing the overall results, simply by comparing it to the average testing loss from previous runs.  Since VW doesn't have good support for CV, any kind of meta-data about your current results can be helpful.  :-)

", "votes": null, "replies": [] }, { "id": 116220, "author_name": "leyue260351", "author_url": "", "post_date": "04/22/2016 15:25:03", "content": "

Thanks for sharing!

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "41913": "", "41962": "", "41978": "", "42021": "", "42047": "", "42056": "", "42058": "", "42059": "", "42069": "", "42072": "", "42125": "", "42142": "", "42326": "", "42328": "", "42346": "", "42356": "", "42557": "", "42563": "", "42622": "", "42626": "", "43523": "", "43657": "", "44253": "", "44259": "", "44273": "", "44396": "", "44410": "", "44411": "", "44412": "", "44413": "", "44414": "", "44429": "", "44441": "", "44451": "", "44479": "", "44513": "", "44562": "", "44796": "", "47421": "", "47422": "", "47423": "", "47424": "", "48413": "", "48416": "", "48417": "", "48418": "", "48419": "", "48420": "", "48437": "", "48439": "", "48448": "", "48450": "", "48451": "", "48453": "", "48454": "", "48457": "", "48473": "", "48476": "", "48482": "", "48485": "", "48486": "", "48487": "", "48488": "", "48489": "", "48490": "", "48494": "", "48495": "", "48499": "", "48514": "", "48515": "", "48516": "", "48527": "", "48535": "", "48541": "", "48547": "", "48549": "", "48555": "", "48558": "", "48561": "", "48595": "", "48615": "", "48616": "", "48617": "", "48623": "", "48630": "", "48632": "", "48669": "", "48696": "", "48710": "", "48716": "", "48721": "", "48747": "", "48755": "", "48758": "", "48763": "", "48773": "", "48807": "", "48853": "", "48857": "", "48864": "", "48872": "", "48875": "", "48883": "", "48886": "", "48895": "", "48896": "", "48996": "", "49006": "", "49053": "", "49082": "", "49126": "", "49257": "", "49259": "", "49260": "", "49261": "", "49268": "", "49269": "", "49286": "", "49292": "", "49293": "", "49304": "", "49999": "", "50002": "", "50004": "", "50011": "", "50056": "", "50058": "", "50090": "", "50210": "", "50218": "", "50219": "", "116220": "Thanks for sharing!", "3229372": "nicely explained" }, "source": "meta" }