{ "id": 9750, "title": "Anyone interested in sharing methodologies?", "url": "/competitions/acquire-valued-shoppers-challenge/discussion/9750", "author_name": "", "post_date": "2014-07-15T00:03:33.460Z", "votes": 4, "comment_count": 58, "views": 16538, "content": "

Great work everyone. Just quick pulse on seeing if anyone is interested in sharing methodologies. Similar to like a post mortem. That way we can all benefit from learning what worked and what didn't.

", "messages": [ { "id": "50501", "postDate": "07/15/2014 00:03:33", "content": "

Great work everyone. Just quick pulse on seeing if anyone is interested in sharing methodologies. Similar to like a post mortem. That way we can all benefit from learning what worked and what didn't.

", "rawMarkdown": "", "votes": null }, { "id": "50503", "postDate": "07/15/2014 00:16:45", "content": "

I'm game. :-)

\n

An important key was understanding that many CustIDs did not have a years worth of transactions. I only considered 180 days when doing calculations.

\n

Then I counted the number of visits (i.e., distinct purchase dates) for each CustID for the entire 180 days:

\n\n

I scaled the data (using asinh), and split the data by offer department, and then again by those that had a Tcob > 0 and those that had Tcob = 0.

\n

I used a decision tree classifier to train and test each corresponding dept / Tcob chunk of data.

\n

For the depts in the test data that did not show up on the training data, I did PCA to find depts in the training that were similar in purchase behavior, and used that dept instead.

\n

I placed #36.  :-)

\n

Also - I used mostly Excel and Python, and also did a fair amount of PLS exploration to find important attributes.

", "rawMarkdown": "", "votes": null }, { "id": "50505", "postDate": "07/15/2014 00:28:03", "content": "

I think this is a feature-engineering and feature-selection game. I'd like to hear any unusual features generated.

\n

Also I couldn't get more advanced methods (random forest, gradient boosting) to work better than simple logistic regressions. My best two submissions are blends of logistic regression and random forest models, but it turned out I could get about the same scores without the random forest models.

", "rawMarkdown": "", "votes": null }, { "id": "50506", "postDate": "07/15/2014 00:30:07", "content": "

The most interesting technique I found for this competition was Bayesian Personalized Ranking,  which is a collaborative filtering algorithm that optimizes for rank.  Unfortunately, I didn't get a chance to properly test it with different regularization coefficients, but it gave a .01 point bump when I ran it using only one department at a time and plugged the results into my regression.  I really wanted to try binning by month and making a model based off count-based panel data, but I couldn't find any readily-made packages for such a model able to handle a dataset of this size.

\n

Another interesting collaborative filtering algorithm I came across was Bayesian Probabilistic Tensor Factorization which allows you to incorporate time (or I suppose any other axis) into the PMF.

\n

Papers for the collaborative filtering algorithms if anyone is interested:

\n

http://www.cs.mcgill.ca/~uai2009/papers/UAI2009_0139_48141db02b9f0b02bc7158819ebfa2c7.pdf

\n

http://www.autonlab.org/autonweb/19394/version/4/part/5/data/225_Xiong.pdf?branch=main&language=en

", "rawMarkdown": "", "votes": null }, { "id": "50507", "postDate": "07/15/2014 00:31:10", "content": "

I can't stress enough how important it was to treat high-transaction IDs separate from "regular" IDs. 

\n

I shot up 150 places on the leader board just by doing that.

\n

Then, breaking out training/testing by offer department was what made the rest of the difference.

", "rawMarkdown": "", "votes": null }, { "id": "50508", "postDate": "07/15/2014 00:34:36", "content": "

I would have loved to have considered the data as a time series. That's an obvious step (that I wasn't able to do since I was stuck using a laptop for most of the 2 weeks prior to contest end).

", "rawMarkdown": "", "votes": null }, { "id": "50509", "postDate": "07/15/2014 00:41:02", "content": "

My score (public 0.61070, private 0.60618) was from a weighted average of random forest (mtry around 3 to 5, replace=FALSE, around 800 trees), bagged random forest (sample size of around 40-50k, around 400-500 reps), gbm ("bernoulli", small shrinkage, high interaction depth of around 25, around 11k trees), and bagged lasso (sample 1000 without replacement, 8000 reps).

\n

I didn't do that much feature transformation or selection other than dummy variables for categorical features. I used mostly Triskelion's features (thanks). I had to do a bit of transformation on the bagged lasso output since all the values were so close together. In R this was: " tt4[,2] = pcauchy((tt4[,2]-min(tt4[,2]))*45) "

\n

I did all of this on a virtual machine with about 10 GB ram. Sometimes I leverage AWS and just throw ram at the problem, but I didn't do that in this case. What I learned in this competition is that while feature engineering is probably the most important thing, sometimes just parallel bagging and model averaging can do wonders if you have enough time and computational resources.

", "rawMarkdown": "", "votes": null }, { "id": "50511", "postDate": "07/15/2014 00:46:49", "content": "

Hi, everyone. This is our 2nd Kaggle contest literally and we are very very lucky! Many thanks to Triskelion and all others who contribute codes and ideas. All our predictions except one are based on Triskelion's features. As absolute newbies, we find feature engineering most challenging and difficult. Our model averages three Neural networks models and 1 random forests model. For neural networks (logistic regression) model, we found that the features with value 0 ( e.g. repeaters who never buy the offer's brand ) may cause serious overfitting. That's why we come up with three different neural network models to mitigate the side effect.

\n

1) use all brand, company and category features but the customers who never bought the same brand or company or category as the coupon's product get a discount of all their feature values. (very subjective) 0.59661

\n

2) use all brand, company and category features but the predictions are only based on neurons that have positive weights along the path.  0.59590 

\n

3) use brand, company and category features to train and test separately and average their predictions.  0.59607

\n

We also trained a very basic random forest with 1000 estimators using all features and we get 0.56958. In the end we took a weighted average of the predictions based on public leaderboard score (not good?).

\n

We really love to be a Kaggler and even being a newbie here can be so much fun! There are a lot to learn here and thank you guys very much to be THE great Sensei here.  To be continued.. : )

", "rawMarkdown": "", "votes": null }, { "id": "50515", "postDate": "07/15/2014 01:47:52", "content": "

I improved my benchmark a little bit, by:

\n

- reducing the number of passes to 1,

\n

- doing quantile regression on the "number of repeat trips" for the labels, instead of binary labels.

\n

- Somewhat tighter binning for the time-related features (bought_company_a_170, etc.)

\n

Then I teamed up with Phil Culliton who had made many more effective VW models trained on different and new features. I think he can tell more about those.

\n

Then we basically ensembled all these models. We ended up with a combination of 10 VW models (each trained on their own features) that was a good pick for private leaderboard. Also private public split was good in general (the best submission was only +4 spots).

", "rawMarkdown": "", "votes": null }, { "id": "50516", "postDate": "07/15/2014 01:56:08", "content": "

Did you guys make cross-validation work? I had been frustrated with that all the time in the competition...

", "rawMarkdown": "", "votes": null }, { "id": "50517", "postDate": "07/15/2014 01:59:07", "content": "

I just CVed via the public leaderboard. I usually do this when it's forecasting because I assume non-stationary data unless I see evidence of otherwise. However, you got 5th, so it's likely it wasn't the biggest problem for you?

", "rawMarkdown": "", "votes": null }, { "id": "50518", "postDate": "07/15/2014 02:04:56", "content": "

[quote=Guocong Song;50516]

\n

Did you guys make cross-validation work? I had been frustrated with that all the time in the competition...

\n

[/quote]

\n

The best I could come up with was CV by offer ID. With this approach, there're big differences between folds, but when averaged, it was my best CV method.

", "rawMarkdown": "", "votes": null }, { "id": "50519", "postDate": "07/15/2014 02:08:46", "content": "

@inversion : Offer doesn't have a dept, did I miss something?

", "rawMarkdown": "", "votes": null }, { "id": "50520", "postDate": "07/15/2014 02:12:36", "content": "

The offer department was the first two digits of the offer category. (Or the first digit for a 3-digit category).

\n

So, e.g, for offer 1200578 which had a category of 1703, I analyzed everything for dept 17.

", "rawMarkdown": "", "votes": null }, { "id": "50521", "postDate": "07/15/2014 02:17:29", "content": "

My model was an average of gbm and logistic regression.

\n\n

For gbm, I used similar features as Triskelion but condensed them into fewer features. Rather than counting last30, 60, etc. I weighted each transaction that matched offer cat/comp/brand by how close its date was to the offerdate.

\n

For example, a transaction purchase amount that matched the offer category and was made 5 days before the offer date would be given weight = amount*exp(-5/60). I found 60 days a good decay parameter. Using something like this was the only way I could get gbm to work reasonably (~.605) with params (adaboost, ~1000 trees, depth = 3, shrinkage=~0.01)

\n\n

For logistic regression, I basically used tks' approach with some minor tweaking but got better results with bagging fraction ~0.5.

\n\n

For me, the most significant improvement was combining the models, but not with a straight average of probabilities. Instead, for each model, I ordered each test offer by repeatProbability, then reassigned them with evenly distributed probabilities on a scale of 0 to 1, and then performed the average. I figured given the AUC metric, the ordering was more important than the actual probability from any model. This gave an improvement on the public leaderboard from ~0.605 to ~0.611.

", "rawMarkdown": "", "votes": null }, { "id": "50524", "postDate": "07/15/2014 02:47:47", "content": "

[quote=inversion;50507]

\n

I can't stress enough how important it was to treat high-transaction IDs separate from "regular" IDs.

\n

I shot up 150 places on the leader board just by doing that.

\n

Then, breaking out training/testing by offer department was what made the rest of the difference.

\n

[/quote]

\n

How did you handle the departments when it came time to make the predictions? I also broke my data up by department, but I found that the test set had quite a few offers for departments that weren't in the train set and vice versa.

\n

Also, on a related note, when I broke up the data by time I found that there were many months--as much as half-- where the only purchases made by customers in the test set for the companies of those that made offers were the brands associated with the offer. I have the suspicion that the test and train sets might actually have been collected from two different sources and combined into one. At the very least it would indicate that transactions history from the test data might have been mixed in at a later date and not sampled from the same original pool of data.

", "rawMarkdown": "", "votes": null }, { "id": "50525", "postDate": "07/15/2014 02:55:33", "content": "

[quote=Mike Kim;50517]

\n

I just CVed via the public leaderboard. I usually do this when it's forecasting because I assume non-stationary data unless I see evidence of otherwise. However, you got 5th, so it's likely it wasn't the biggest problem for you?

\n

[/quote]

\n

It was the biggest problem for me. My scores were very fluctuating over time. I generated similar features inversion did, and tried many normalization over offers, which helped lift the score, but the improvement of CV was limited. Since the training time was short, I just kept submitting...

", "rawMarkdown": "", "votes": null }, { "id": "50527", "postDate": "07/15/2014 02:56:50", "content": "

Regarding cross validation:
I think the major problem that made life difficult with building a proper validation set might have been caused due to the fact that the test set and train set had different products offered to them (6 products are in train but not in test, 8 products are both in train and test, and 7 more products are in test but not in train). I've used 'product' as the basic unit since several different offers were actually different discounts offered to the same product.

Anyway, when I tried mimicking these 'product split' ratio's using the 14 products in the train set and tried learning to generalize from a random split of 4 randomly chosen products in train only, 5 randomly chosen products in test only, and 5 remaining products split by time to train and test, I've found that the amounts of 'noise' were huge from fold to fold. for some splits the validation error was in the ranges of 0.55, and for some other splits in the ranges of 0.63 (for my earliest models). I concluded that this is too much noise and that I needed to do about 100 such random splits (folds) and average their results to get a reliable cross validation scheme and decided it would take too much, so instead I tried finding a representative single split. I couldn't find anything really good, but I've found something that was OK, and used that as a general guide to guide me to "the general ballpark".


Regarding the features we've used:
My initial way of thinking about this problem was like a recommender system.
The basic assumption I used was that most people are not aware of all the products that are out there (there are about 120,000 products in the data set), but, that at the same time, people can be characterized by the specific products they tend to buy, and similarities between people can be drawn according to each persons 'long term shopping cart', and similarities between products can be drawn according to the 'the population demographic' that buys different products.

\n

So I constructed the matrix that contains 310K consumers as rows and the 120K products as columns, and filled the (i,j) th element in it to be the amount of times that consumer bought that specific product.
this matrix is of course very very sparse. meaning, most of it's elements are 0.
As I've said, my basic assumption was that some of these 0 elements account for "that person is not interested in that product", but that most of these zeros account for "that person has never heard of that product".

\n

So I wanted to fill in these zeros by "if that person were to know about this product, will this product be part of his long term shopping cart?".

\n

There are several simple ways that I know of that are able to achieve this: consumer kNN (find consumer i's k most similar consumers, and average their j'th column values), product kNN (similar to consumer kNN but with switching the rows and columns), and matrix factorization (construct a low rank matrix approximation to the large matrix by performing SVD or PCA).

\n

Non of these approaches were feasible running on the entire 310000x120000 matrix, so I've subsampled the products and kept only the most popular products across all population (largest amount of consumer that bought them) and those products that were most similar to the query products (the products that were initially offered with discount). so in the end, I had a matrix of about 310000x2000 as the basic feature set to work with. this proved to be quite good.

My partner (who I teamed up with toward the end of the competition) took an entirely different approach and managed to squeeze a lot out of Triskelion's features (and some more features of the same type), by training many different types of learners with the same feature set and mixing them together.

In the end, blending our very different approaches has resulted in quite a large increase in score.

", "rawMarkdown": "", "votes": null }, { "id": "50528", "postDate": "07/15/2014 02:59:14", "content": "

As to the features, besides the basic factor features - brand,category,chain,and company,I also tried to construct additional features from the following business perspectives for forcasting repurchase.

\n

1.Customer's purchasing behiviours in the past.

\n

I used RFM method. RFM stands for
Recency - How recently did the customer purchase?
Frequency - How often do they purchase?
Monetary Value - How much do they spend?

\n

I mainly leveraged Triskelion's work for these variables construction and I also used Triskelion's original score as my base line.

\n

2.Brand's quality - average return rate of a brand

\n

3.BrandPopularity/Influence

\n

BrandPopularity1 - brand's average revenue per month
BrandPopularity2 - brand's average repeat rate

\n

4.Price

\n

Brand's price - average price of the brand
Preferred price for a customer - average price of the customer spent in the same category of a brand
Offer value - offervalue/brand_value

\n

5.Competition - number of brands in the same category

\n

6.Easy to Get - number of total brands or revenue in a chain, to measure the chain's popularity or scale.

\n

However, I found that only the feature of brand's quality can improve the base line (Triskelion's original score) a little bit. 

", "rawMarkdown": "", "votes": null }, { "id": "50531", "postDate": "07/15/2014 03:55:50", "content": "

I discovered the same very effective mixing/ensembling method as cherrybarry. Since with AUC only the ranking matters, and the predictions from different models have very different scale/ranges, so averaging their rankings instead of the prediction values makes more sense and gives much better results.

\n

Once I found the fact that mixing results from two very different models can improve the results significantly, even if the results of individual models are not very impressive. I try to add many different models:

\n

Using a bigger set of features similar to Triskelion's but expanded and also calculated using the full transactions data I tried:

\n

1. Triskelion's VW model

\n

The rest of the models using R packages.

\n

2. R's glmnet model with bagging

\n

3. randomForest with bagging

\n

4. Single gbm model

\n

5. gbm model bagging with very small number of trees

\n

6. knn3, R's k-nearest neighbor model

\n

7. NaiveBayes model

\n

Then using the above mixing/ensemble method to get the final results.

\n

Once I teamed up with Selfish Gene, since there was not much time left, and we are in different parts of the globe, we just independently worked on our own models and once I submitted my models he will download the my best results and using the above stated mixing method to mix his results with mine. We furthur see meaningful improvements from best of either models.

\n

I suspect, that if we mix/ensemble the top 10/20 models of this competitions, we may get again some meaningful improvements from any of the single models, as from the shared ideas from this thread, the approaches are very different. Perfect candidates for mixing/ensembling.

", "rawMarkdown": "", "votes": null }, { "id": "50533", "postDate": "07/15/2014 04:12:55", "content": "", "rawMarkdown": "", "votes": null }, { "id": "50534", "postDate": "07/15/2014 04:28:45", "content": "

Hi,

\n

I am new to this field, and this was my first competition. I am placed at #249

\n

I used Logistic Regression (Feature engineering in Java, and trained my model in Octave). I tried to fine tune my solution using Random Forest, but it did not help much.

\n

I want to share few things which helped me to move up on leader-board rapidly.

\n

1. Removing noise: There were few customers whose feature values (total purchase amount, total number of transactions) were too big. So I removed that data which 8 times more than mean.

\n

2. Polynomial degrees of feature values. (square and cube)

\n

3. After training I found below were high weighted features

\n

Total purchase amount

\n

Total number of transactions

\n

Total quantity purchased

\n

Total purchased amount by same company-category-brand

\n

Total number of transactions by same company-category-brand

\n

Total quantity by same company-category-brand

\n

Thanks.

", "rawMarkdown": "", "votes": null }, { "id": "50537", "postDate": "07/15/2014 05:17:40", "content": "

We started with Triskelion's features and also had the idea of adding recommender based features. We tried item-based collaborative filtering (used mahout) and latent factor models (Koren's work from Netflix prize). Best submissions were item-based.

\n

By pruning the number of most similar items I tuned the number of actual recommendation values (which are always >=1) vs N/A (no similar items found, converted to 0 ), which is effective to  balance between explicit and implicit recommendation (implicit meaning user didn't buy because they didn't like, not  just weren't aware). Adding two such features with different thresholds helped.

\n

We used logistic regression with 10% sub-sampling and bagging 2000 models,  using code of tks . I found taking median was better than average (but now realizes from others that should have tried averaging ranks too).. As tks pointed out, need extreme L2 regularization (lambda~2^13). Strong regularization must be needed due to the large variance but  I'm  puzzled by how big lambda is. Trying elastic net with mix in L1 reduced the optimal lambda but didn't give best result.I'm interested if someone was able to make less regularized  logistic regression to work.

\n

For CV I  used K-folds stratified by  offer labels and averaged. It seems pretty consistent with progress on LB..

\n

.

", "rawMarkdown": "", "votes": null }, { "id": "50544", "postDate": "07/15/2014 07:36:51", "content": "

Hi. And thank you to everyone to share all this information. I’m new and I appreciate this.

\n

The features that I create was basically descriptions features about how customer buy, especially in each department, and features about the offer.

\n

Them I did diferent random forest, and then I esambled it.

\n

The random forest was focusing in the repeater variable, but if I'd had more time I would have tried some glm with repeattrips, using the Poisson distribution.

\n

Did anyone use a glm in that way?

", "rawMarkdown": "", "votes": null }, { "id": "50547", "postDate": "07/15/2014 07:41:45", "content": "

It is interesting has somebody tried to use Assosiation Rules for this task? As for me it`s classic task for Recommendation System, but I couldn't find no significant rule.

", "rawMarkdown": "", "votes": null }, { "id": "50549", "postDate": "07/15/2014 08:52:59", "content": "

I created several new variables and used glmnet with alpha 0 lambda 0.00371. Where the lambda obtained from cv.glmnet from full trainset. It scored 0.61093 on public 0.60313 on private

\n

st_change :OLS cusum test for each customer (to capture structural break)
both_contained_offer: is the offer contained in both train and test set?
ratio_offer_bought_all: offered product (brand+company+category) / numbers of purchased product
ratio_offer_bought_180:offered product (brand+company+category) / numbers of purchased product within 180 days
ratio_offer_bought_90:....
ratio_offer_bought_60:..
ratio_offer_bought_30:...
last_date_purchased: offer date - last date that each customer purchased offered product
mean_pur_amount: daily purchase / total spent
mean_pur_amount_180: daily purchase / total spent within 180days
mean_pur_amount_90:...
mean_pur_amount_60:...
mean_pur_amount_30:...
cate_sensitive: numbers of different brand+category / numbers of different brand+category+category
cate_sensitive_180:numbers of different brand+category / numbers of different brand+category+category within 180 days
cate_sensitive_90
cate_sensitive_60
cate_sensitive_30
cust_cate_sensitive: numbers of different brand / numbers of different brand+category+category
cust_cate_sensitive_180
cust_cate_sensitive_90
cust_cate_sensitive_60
cust_cate_sensitive_30
cust_company_sensitive: numbers of different company/ numbers of different brand+category+category
cust_company_sensitive_180: numbers of different company / numbers of different brand+category+category within 180days
cust_company_sensitive_90
cust_company_sensitive_60
cust_company_sensitive_30
cust_brand_sensitive: numbers of different brand/ numbers of different brand+category+category
cust_brand_sensitive_180
cust_brand_sensitive_90
cust_brand_sensitive_60
cust_brand_sensitive_30

", "rawMarkdown": "", "votes": null }, { "id": "50552", "postDate": "07/15/2014 09:39:11", "content": "

We added in "general" features. In total had 750 features

\n

- customer / dept boolean matrix

\n

- brand / category boolean matrix

\n

- brand / company counts

\n

etc

\n

Training by different subsets and averaging increased the score

\n

XGBOOST and GBM worked the best. Code coming soon

", "rawMarkdown": "", "votes": null }, { "id": "50564", "postDate": "07/15/2014 13:02:23", "content": "

Quick and dirty solution that we implemented in a week with limited computational resources:

\n

1. Created ~100 features that explain users behavior, from transactions. 

\n

2. Created another ~50-100 features from clustering users, to have some more meaningful features for users with very few transactions. Not sure these helped at all.

\n

3. Run average sized RF. Also tried logistic and adaboost. None of the 2 performed as good as RF. RF was also preferable cause there was little or no need to preprocess the features and time was limited.

\n

4. Combined 4-5 methods, with the highest weight put on our best RF. This improved performance just enough to get AUC 0.5997.

", "rawMarkdown": "", "votes": null }, { "id": "50565", "postDate": "07/15/2014 13:38:12", "content": "

[quote=Guocong Song;50516]

\n

Did you guys make cross-validation work? I had been frustrated with that all the time in the competition...

\n

[/quote]

\n

Did you try splitting via offer? E.g train on all offers but one and test on that one. That worked fairly well for me with a little bit more tweaking. E.g to say there should be at least X uplift in order to trust it. For me the was was 0.002 (from AUC 0.610 to AUC 0.612). 

", "rawMarkdown": "", "votes": null }, { "id": "50566", "postDate": "07/15/2014 13:52:35", "content": "

For cross validation, I tried to take into account that observations in the test set were for offers on later dates than the training set, and also that many were for different product categories. I ordered the data by offer date, and used the first half for training (75,000 observations). For the cross-validation set, I used the remaining rows of data, but only selected the categories that had not appeared in the first 75,000 rows, i.e. 2202, 2119, 5616, and 6202.

\n

Of course, this wasn’t perfect, but it did help in identifying larger jumps. I could also do it fairly quickly as opposed to using K folds.

\n

For my score (public: 0.61228, private: 0.60575), I used features similar to Triskelion’s, as well as inversion’s, and trained a single random forest (500 trees, mtry=2, nodesize=5).

\n

I’d be interested to know if anyone had any good feature selection methods. I noticed that my score dropped when adding certain features, and increased when removing some, but I had no systematic way of testing each feature.

", "rawMarkdown": "", "votes": null }, { "id": "50568", "postDate": "07/15/2014 14:10:09", "content": "

For me it was the very first kaggle. My secret was lots of hard work (ended doing exactly 400 submissions all by myself.) I found very specific features to this data set, I don't think anyone will gain from them outside the company running the competition.

", "rawMarkdown": "", "votes": null }, { "id": "50570", "postDate": "07/15/2014 14:31:18", "content": "

[quote=zzspar;50568]

\n

For me it was the very first kaggle. My secret was lots of hard work (ended doing exactly 400 submissions all by myself.) I found very specific features to this data set, I don't think anyone will gain from them outside the company running the competition.

\n

[/quote]

\n

Congratulations! I appreciate it if you could share the methodology to discover golden features, if it's not all about intuitions and prior knowledge. Thank you very much! Again, all by yourself, that's really impressive!

", "rawMarkdown": "", "votes": null }, { "id": "50571", "postDate": "07/15/2014 14:40:13", "content": "

CV was the hardest item also for me. In one of my attempts at inventing a CV method I realized I had a bug in the code but on the other hand this CV was very effective on the leader board so I had to dig into the data to find out what was behind the buggy algorithm

", "rawMarkdown": "", "votes": null }, { "id": "50572", "postDate": "07/15/2014 14:58:40", "content": "

For customers that purchased the same category + company + brand, one feature that seems to have some significance was the discount rate of the coupon. I calculated this based on the median amount per qty.

\n

Lower discount rate seems to have more repeat ability than high.

\n

Other working theories I had was:

\n

1) Brand loyalist to a particular brand may be more likely to try a coupon but less likely to switch brands

\n

2) Whereas brand shifters, maybe more likely to switch brands after trying a coupon.

\n

One thing wanted to try was time-series clustering by customers buying patterns. Was wondering if anyone had luck with this? or found variables with a time dimension was useful?

", "rawMarkdown": "", "votes": null }, { "id": "50579", "postDate": "07/15/2014 16:11:25", "content": "

This was my first Kaggle competition. Big thanks to my teammate turbonerd for introducing me to it and keeping the spirit up! Without him, this would not have happened. Thanks to Kaggle for hosting and excellent competition and to the anonymous data provider for the interesting dataset.

\n

Here are some lessons learned, helpful or not.

\n

1) Spent far too much time worrying about our setup, ending up with a combination of MySQL, python scripts with a task-oriented class structure and csv files. Using SQL limited a bit in what aggregates could be produced (no medians etc), but speeded up the feature engineering process. In hindsight, a clean map-reduce would probably have been better.

\n

2) It took some time for to figure out what the problem was. When, finally, cross-validation started to work (somewhat), dividing up train set by unique product (company,category,brand combination), a lot of time and wasted effort had elapsed.

\n

3) This competition was a lot about feature engineering, and having no experience with that, it took a lot of time and many mistakes were made.

\n

Some of the best features turned out to be hierarchical, trained separately on data from transactions. E.g. a logistic regression-based probability of buying the product on offer, given which of the top100 most popular products this user had purchased (also did this with brand, category, company, department).

\n

Other good ones were simple stuff like which fraction of previous trips the user bought this product on. Other ones tried, besides the ones suggested on forums, fall broadly into these categories: weekday statistics (quantities, amounts, measures purchased on certain days), loyalty to products (how many categories in a department a user bought, how many brands for a company etc – almost all permutations), prior purchase-based (how many times, how often, ), loyalty to store (which weekdays visited, how many times, time period between first and last visit), market, chain, dept-level aggregates of , basket sizes, relative price of offered product to this users usual level for this department, many of above based on 1, 2, 3 months of data prior to offerdate. In total, 214 features.

\n

We also tried some association rule learning, but dropped it pretty quickly.

\n

4) The main mistake was to not be more systematic, which seems obvious in hindsight. We knew that CV was a problem; that offers were very different (repeating ranging 10-40%); that purchase because of coupon and purchase because of prior habits needed to be modeled separately; but somehow did not spend enough time systematically solving these issues, spending most time on feature engineering. A lot more time should have gone into thinking about good ways to train. The absurdity of life is that after all effort with random forests, ridge regression and many others, vowpal vabbit ended up handily beating all out blended attempts at cool models.
Also, more time could have gone into looking and visualizing the data.

\n

5) Not doing feature selection probably hurt the final performance a lot, since some features ended up to be highly correlated. This is a lesson for next time and it would be interesting to hear if someone managed to automate feature selection somehow.

\n

We ended up with a ROC AUC of 0.57485 on the private leaderboard, netting us the 545th place. Hard to say if its good or bad for beginners, at least it was very fun and a learning experience.

\n

Some of the above will probably seem very stupid after reading this thread :)

", "rawMarkdown": "", "votes": null }, { "id": "50580", "postDate": "07/15/2014 16:11:38", "content": "

One problem I noticed sometimes when cross-validating was that although the classifier had good AUC on each offer individually, the classifier overall had much much poorer total AUC. I'm thinking that this was because the relative probabilities of the different offers was not aligned somehow, but that's just a guess. Did anyone find a principled way to deal with this? I looked around for some theoretical approach to this, but could not find any.

", "rawMarkdown": "", "votes": null }, { "id": "50581", "postDate": "07/15/2014 16:15:43", "content": "

[quote=auduno;50580]

\n

One problem I noticed sometimes when cross-validating was that although the classifier had good AUC on each offer individually, the classifier overall had much much poorer total AUC. I'm thinking that this was because the relative probabilities of the different offers was not aligned somehow, but that's just a guess. Did anyone find a principled way to deal with this? I looked around for some theoretical approach to this, but could not find any.

\n

[/quote]

\n\n

You needed 2 models! One to optimize for individual (offer-specific)  AUCs and one for the general AUC :) At least, this is what we did. Also scaling helped in this one.

", "rawMarkdown": "", "votes": null }, { "id": "50589", "postDate": "07/15/2014 16:52:27", "content": "

[quote=KazAnova;50581]

\n

You needed 2 models! One to optimize for individual (offer-specific)  AUCs and one for the general AUC :) At least, this is what we did. Also scaling helped in this one.

\n

[/quote]

\n

Ah, I considered that, but didn't get enough time to try it, unfortunately. Did you blend the two models to get final result, or did you use the second model to adjust the relative predictions of the offer-specific models?

", "rawMarkdown": "", "votes": null }, { "id": "50590", "postDate": "07/15/2014 16:53:20", "content": "

[quote=auduno;50589]

\n

[quote=KazAnova;50581]

\n

You needed 2 models! One to optimize for individual (offer-specific)  AUCs and one for the general AUC :) At least, this is what we did. Also scaling helped in this one.

\n

[/quote]

\n

Ah, I considered that, but didn't get enough time to try it, unfortunately. Did you blend the two models to get final result, or did you use the second model to adjust the relative predictions of the offer-specific models?

\n

[/quote]

\n\n

Blend!

", "rawMarkdown": "", "votes": null }, { "id": "50591", "postDate": "07/15/2014 16:54:28", "content": "

[quote=Grigory;50579]

\n

We ended up with a ROC AUC of 0.57485 on the private leaderboard, netting us the 545th place. Hard to say if its good or bad for beginners, at least it was very fun and a learning experience.

\n

[/quote]

\n

It doesn't matter where you place, it matters what you learned. And it sounds like you learned a lot!

\n

I placed in the bottom 10% my first competition, then top 40%, then top 30%, and now top 5% with this competition.

\n

I'd say you rocked it for your first time.  :-)

", "rawMarkdown": "", "votes": null }, { "id": "50595", "postDate": "07/15/2014 17:05:58", "content": "

[quote=econoraptor;50524]

\n

[quote=inversion;50507]

\n

I can't stress enough how important it was to treat high-transaction IDs separate from "regular" IDs.

\n

I shot up 150 places on the leader board just by doing that.

\n

Then, breaking out training/testing by offer department was what made the rest of the difference.

\n

[/quote]

\n

How did you handle the departments when it came time to make the predictions? I also broke my data up by department, but I found that the test set had quite a few offers for departments that weren't in the train set and vice versa.

\n

[/quote]

\n

For each CustID, I did a pivot (using pandas) of the total amount purchased for each dept. (More accurately, I only counted purchases that were <= 180 days from the date of the offer given to that CustID, filtering out any transactions with a 0 count or negative amount.)

\n

Then I did a PCA on the table. By inspecting the loading plot, I was able to find a dept in the training data that was similarly correlated to a dept in the test data.

\n

There were 2 depts in the test set that didn't correlate well to any in the training set. For those, I just trained on all the depts in aggregate.

", "rawMarkdown": "", "votes": null }, { "id": "50597", "postDate": "07/15/2014 17:46:38", "content": "

[quote=KazAnova;50565]

\n\n

Did you try splitting via offer? E.g train on all offers but one and test on that one. That worked fairly well for me with a little bit more tweaking. E.g to say there should be at least X uplift in order to trust it. For me the was was 0.002 (from AUC 0.610 to AUC 0.612). 

\n

[/quote]

\n

Hmm, I did split by offer, but didn't try leave-one-out CV. I should try it see what happens.

", "rawMarkdown": "", "votes": null }, { "id": "50599", "postDate": "07/15/2014 18:06:02", "content": "

[quote=KazAnova;50565]

\n

[quote=Guocong Song;50516]

\n

Did you guys make cross-validation work? I had been frustrated with that all the time in the competition...

\n

[/quote]

\n

Did you try splitting via offer? E.g train on all offers but one and test on that one. That worked fairly well for me with a little bit more tweaking. E.g to say there should be at least X uplift in order to trust it. For me the was was 0.002 (from AUC 0.610 to AUC 0.612). 

\n

[/quote]

\n\n

I did a quick try in which I used one offer data to train models against the rest, but didn't work out. I should do normal splitting...  Single offer has too large variance:)  

", "rawMarkdown": "", "votes": null }, { "id": "50608", "postDate": "07/15/2014 19:55:34", "content": "

Hi folks, I'm a recent "graduate" of Andrew Ng's machine learning course. I entered this, my first competition, as a training exercise and have thoroughly enjoyed it.

\n


Like Jack Han, I started with RFM (recency, frequency, monetary). Frequency didn't help, but monetary did, especially when binned. I used total spend over the previous 90 days.
I used dummy variables for prod, category and dept in bins of "never bought", 1-15 days ago, 16-30 ago, 31-70 ago. The bins were chosen based on histograms/box-and-whisker plots in Tableau. This is similar to Triskelion's approach, but I didn't use product quantity, just ones and zeros. For brand, I ended up just using the "never bought" bin. I used logistic regression in vw. My features were generated from HP Vertica using SQL, and I used perf to calculate ROC.

\n


I experimented to no avail with various other features to find a proxy for customers who tend to flit between products in the same category: category spend ratios, category entropy, price paid relative to mean. The customer's ratio of distinct products bought per category relative to the average ratio per category worked well on the training set, but harmed my score on the public test set. Like 'inversion', I noticed higher take up for those with high IDs, but it was late in the day and I wasn't able to exploit it with my best model.

\n


In the end, my best submission (0.60291 public and 0.59738 private) was achieved by adding total number of distinct products as a feature. I found that the difference between my train ROC using vw holdout and public ROC varied between 0.47 and 0.57. Is this what others are seeing too?

\n


I had planned to implement a brand loyalty measure (anyone try Guadagni and Little 1983?) and itemset mining. Sadly the day job needed attending to.

", "rawMarkdown": "", "votes": null }, { "id": "50661", "postDate": "07/16/2014 05:43:20", "content": "

Similar to Jack Han, for each customer I computed the following score

\n

customer_value_score = total_spend_per_day * trxn_per_trip * (1.0/recency) * lifetime * (1.0/avg_interpurchase_time)

\n

where:

\n

trxn_per_trip = number of transactions per trip

\n

recency = days between last purchase date relative to last date in data

\n

lifetime = time elapsed from first purchase date to last purchase date

\n

avg_interpurchase_time = is the average time gap between purchases

\n\n

Being new to Kaggle competitions, I benefited a great deal from combining Triskelion's features with tks's glmnet code + bagging (thanks to meet thakar and clustifier). Additionally, I added return events (quantity < 0) as features. I used the full 22GB dataset. 

\n\n

A great many thanks to Triskelion, tks, culstifier, meet thakar and everyone else for sharing their ideas, insights, and code.

\n

Cheers!

", "rawMarkdown": "", "votes": null }, { "id": "50740", "postDate": "07/16/2014 22:37:04", "content": "

[quote=KazAnova;50581]

\n

You needed 2 models! One to optimize for individual (offer-specific)  AUCs and one for the general AUC :) At least, this is what we did. Also scaling helped in this one.

\n

[/quote]

\n

I'm guessing you might share the code eventually, but I'm curious : how did you optimize for individual offer-specific AUCs? Most of the offers in the testset were not in the training set, so did you just optimize for those that were in the training set, or did you use some method similar to inversion's method, i.e. finding some combination of the closest "similar" offers to the ones in testset and optimizing for those?

", "rawMarkdown": "", "votes": null }, { "id": "50741", "postDate": "07/16/2014 22:43:06", "content": "

[quote=auduno;50740]

\n

 how did you optimize for individual offer-specific AUCs? Most of the offers in the testset were not in the training set, so did you just optimize for those that were in the training set

\n

[/quote]

\n\n

For those in the training set only. When I optimized for individual AUCs, I was ignoring the sample size of the offer and I was just averaging the results of the AUCs of the different offers ( 1-out format), When we were optimizing for the total AUC we were appending the results of each offer's prediction in an array and calculating the total AUC once all results had been appended. I hope that helps. 

", "rawMarkdown": "", "votes": null }, { "id": "50765", "postDate": "07/17/2014 11:25:11", "content": "

[quote=inversion;50507]

\n

For each CustID, I did a pivot (using pandas) of the total amount purchased for each dept. (More accurately, I only counted purchases that were <= 180 days from the date of the offer given to that CustID, filtering out any transactions with a 0 count or negative amount.)

\n

Then I did a PCA on the table. By inspecting the loading plot, I was able to find a dept in the training data that was similarly correlated to a dept in the test data.

\n

There were 2 depts in the test set that didn't correlate well to any in the training set. For those, I just trained on all the depts in aggregate.

\n

[/quote]

\n

Hello,
Interesting approach. We have applied a Kolmogorov-Smirnov test and selected only features with similar distributions in de train and test sets.
Can you give more details about your PCA procedure?

", "rawMarkdown": "", "votes": null }, { "id": "50779", "postDate": "07/17/2014 13:52:25", "content": "

[quote=inversion;50507]

\n

I can't stress enough how important it was to treat high-transaction IDs separate from "regular" IDs.

\n

I shot up 150 places on the leader board just by doing that.

\n

Then, breaking out training/testing by offer department was what made the rest of the difference.

\n

[/quote]

\n

I read your previous post about your method. I didn't understand how you treat the high-transactions IDs and low-transactions IDs. Can you explain more on it?

", "rawMarkdown": "", "votes": null }, { "id": "50786", "postDate": "07/17/2014 15:21:59", "content": "

[quote=inversion;50507]

\n

That's a very wise move. I wonder if a clustering algorithm to assign customers to different groups based on similarities in their shopping habits would have been helpful; then each group from the test set and train set would have been treated individually. I guess I will never know  ;)

", "rawMarkdown": "", "votes": null }, { "id": "50810", "postDate": "07/18/2014 00:17:08", "content": "

[quote=Yiqun Hu;50779]

\n

I read your previous post about your method. I didn't understand how you treat the high-transactions IDs and low-transactions IDs. Can you explain more on it?

\n

[/quote]

\n

Looking at the customer id data (training set), there were ~1,300 ids that had more than 4,000 total transactions. (The highest had 2,647,164 transactions!) As was mentioned in the forum earlier, this was probably a "default" card that is used at check-out, e.g., when a shopper doesn't have a loyalty card the store scans a "default" card at the register.

\n

If you reverse sort the training data by the number of transactions, you'll see that those ids were almost always repeat shoppers. (See the image below.) The fact that these are repeat buyers is an artifact of including these "default" cards in with normal shoppers.

\n

I simply removed these high-transaction ids and trained without them. For the test set, any id that had a high transaction count, I manually set it to "1" for the estimated repeater probability. Doing this was a "freebie" improvement in score.

\n

\"Repeaters\"

", "rawMarkdown": "", "votes": null }, { "id": "50811", "postDate": "07/18/2014 00:33:49", "content": "

[quote=inversion;50810]

\n

I simply removed these high-transaction ids and trained without them. For the test set, any id that had a high transaction count, I manually set it to "1" for the estimated repeater probability. Doing this was a "freebie" improvement in score.

\n

[/quote]

\n

How much improvement did you get from this ?

\n

Another possible "freebie" is to train the offers that are in both train and test sets separately. However I didn't use this in the end because the improvement was small, and the CV unreliable.

", "rawMarkdown": "", "votes": null }, { "id": "50814", "postDate": "07/18/2014 01:21:20", "content": "

[quote=B Yang;50811]

\n

[quote=inversion;50810]

\n

I simply removed these high-transaction ids and trained without them. For the test set, any id that had a high transaction count, I manually set it to "1" for the estimated repeater probability. Doing this was a "freebie" improvement in score.

\n

[/quote]

\n

How much improvement did you get from this ?

\n

[/quote]

\n

From 0.59507 to 0.61017

\n

+150 positions on the leader board or so.

", "rawMarkdown": "", "votes": null }, { "id": "50816", "postDate": "07/18/2014 01:36:58", "content": "

[quote=RW;50765]

\n

[quote=inversion;50507]

\n

For each CustID, I did a pivot (using pandas) of the total amount purchased for each dept. (More accurately, I only counted purchases that were <= 180 days from the date of the offer given to that CustID, filtering out any transactions with a 0 count or negative amount.)

\n

Then I did a PCA on the table. By inspecting the loading plot, I was able to find a dept in the training data that was similarly correlated to a dept in the test data.

\n

There were 2 depts in the test set that didn't correlate well to any in the training set. For those, I just trained on all the depts in aggregate.

\n

[/quote]

\n

Hello,
Interesting approach. We have applied a Kolmogorov-Smirnov test and selected only features with similar distributions in de train and test sets.
Can you give more details about your PCA procedure?

\n

[/quote]

\n

Sure. My data table looked like this. (Rows = CustIDs, Columns = Depts, Values = sum of customer transactions for each dept [truncated at 180 days from offer date and ignoring negative transaction amounts]) The amounts were transformed with arcsinh for scaling.

\n

\"Dept

\n

I then did a 2-component PCA on the table. (I don't have an image of the loading plot handy, otherwise I'd show it. I used 3 different computers . . . so my files are all over the place. If I stumble upon it I'll post it later.)

The loading plot showed a position for each department. 

\n

For departments that were in the test set but not in the train set, I looked at the loading plot to find a department that was close to that point that was also in the training set.

\n

For instance, I trained on Dept 44 to test for Dept 7. For Dept 45 and 51, I used Dept 55 for training.

\n

Departments 72 and 91 didn't covary well with any departments in the training set, so I used the entire training set to predict those departments.

", "rawMarkdown": "", "votes": null }, { "id": "50861", "postDate": "07/18/2014 16:43:47", "content": "

Like many mention before the CV problem with he public leader board came from the very strong "Offer ID" bias (some offers were much more successful then others). The way I dealt with it was to find a "repeat ratio" for each "Offer ID" and use that as a feature. I also did the same for all the offers info (Market, dept, brand, etc..). These were all quite strong features for me. This method had the obvious issue that some offers were not in the training data (or had very low statistics), but that's the best I could come up with.

\n

The other features I didn't see people talk about, but that I though were clever, was a measure of how faithful a customer was to products in general. For that I when through all the products a customer ever bought and found how many times he bought it again. From that data I found the usual statistics (min/max, mean. median, standard deviation, etc...), and more importantly a "repeater ratio", which is basically how many purchases they made divided by the number of different products they bought. I liked that feature the most because:

\n

1 - It makes sense - people that are faithful to their products (or don't bother looking for new ones) are more likely to become faithful to a  new product they try.    

\n

2 - It turned out to be quite a strong feature.

\n

My best model was using Scikit-learn with AdaBoost (n_estimators=200) with Random Forest (n_estimators=200, max_depth=2) for base_estimator.

\n

PS. If someone is good with Neural Networks, I would be very curious to see how my set of features would do with that. It was my next step, but I never had a chance to try it.

", "rawMarkdown": "", "votes": null }, { "id": "50866", "postDate": "07/18/2014 17:46:23", "content": "

[quote=cherrybarry;50521]

\n

For me, the most significant improvement was combining the models, but not with a straight average of probabilities. Instead, for each model, I ordered each test offer by repeatProbability, then reassigned them with evenly distributed probabilities on a scale of 0 to 1, and then performed the average. I figured given the AUC metric, the ordering was more important than the actual probability from any model. This gave an improvement on the public leaderboard from ~0.605 to ~0.611.

\n

[/quote]

\n\n

Thank you for sharing your method. I have struggled on how to ensemble different models. Can you elaborate on how you assigned scores from different models evenly distributed probabilities on a scale of 0 to 1? 

\n

Thank you for your help. 

", "rawMarkdown": "", "votes": null }, { "id": "50867", "postDate": "07/18/2014 17:50:26", "content": "

[quote=Wei Wu;50531]

\n

I discovered the same very effective mixing/ensembling method as cherrybarry. Since with AUC only the ranking matters, and the predictions from different models have very different scale/ranges, so averaging their rankings instead of the prediction values makes more sense and gives much better results.

\n

[/quote]

\n

How did you average the rankings and transform that into probabilities? Could you explain a little bit more? Thank you! 

", "rawMarkdown": "", "votes": null }, { "id": "50882", "postDate": "07/18/2014 21:12:46", "content": "

[quote=Iris Ren;50866]

\n

[quote=cherrybarry;50521]

\n

For me, the most significant improvement was combining the models, but not with a straight average of probabilities. Instead, for each model, I ordered each test offer by repeatProbability, then reassigned them with evenly distributed probabilities on a scale of 0 to 1, and then performed the average. I figured given the AUC metric, the ordering was more important than the actual probability from any model. This gave an improvement on the public leaderboard from ~0.605 to ~0.611.

\n

[/quote]

\n

Thank you for sharing your method. I have struggled on how to ensemble different models. Can you elaborate on how you assigned scores from different models evenly distributed probabilities on a scale of 0 to 1? 

\n

Thank you for your help. 

\n

[/quote]

\n

I can't speak for cherrybarry, but here is how I would do it:

\n\n

You will now have evenly distributed probabilities between 0 and 1. Do that for each model, and then to ensemble, take a straight average of the adjustedProbability: (model 1 + model 2) / 2

\n

Or, you could weight the models differently: (model1 + 2*model 2 + 3*model 3) / 6

", "rawMarkdown": "", "votes": null }, { "id": "1343061", "postDate": "06/10/2021 02:08:52", "content": "

Hi , can you explain what is TCOB? I have been breaking my head over this.
\nAlso, can this be done without Python?

", "rawMarkdown": "Hi , can you explain what is TCOB? I have been breaking my head over this. \nAlso, can this be done without Python?", "votes": null } ], "comments": [ { "id": 50503, "author_name": "inversion", "author_url": "", "post_date": "07/15/2014 00:16:45", "content": "

I'm game. :-)

\n

An important key was understanding that many CustIDs did not have a years worth of transactions. I only considered 180 days when doing calculations.

\n

Then I counted the number of visits (i.e., distinct purchase dates) for each CustID for the entire 180 days:

\n\n

I scaled the data (using asinh), and split the data by offer department, and then again by those that had a Tcob > 0 and those that had Tcob = 0.

\n

I used a decision tree classifier to train and test each corresponding dept / Tcob chunk of data.

\n

For the depts in the test data that did not show up on the training data, I did PCA to find depts in the training that were similar in purchase behavior, and used that dept instead.

\n

I placed #36.  :-)

\n

Also - I used mostly Excel and Python, and also did a fair amount of PLS exploration to find important attributes.

", "votes": null, "replies": [ { "id": 1343061, "author_name": "scottrobertson7772", "author_url": "", "post_date": "06/10/2021 02:08:52", "content": "

Hi , can you explain what is TCOB? I have been breaking my head over this.
\nAlso, can this be done without Python?

", "votes": null, "replies": [] } ] }, { "id": 50505, "author_name": "byang1", "author_url": "", "post_date": "07/15/2014 00:28:03", "content": "

I think this is a feature-engineering and feature-selection game. I'd like to hear any unusual features generated.

\n

Also I couldn't get more advanced methods (random forest, gradient boosting) to work better than simple logistic regressions. My best two submissions are blends of logistic regression and random forest models, but it turned out I could get about the same scores without the random forest models.

", "votes": null, "replies": [] }, { "id": 50506, "author_name": "econoraptor", "author_url": "", "post_date": "07/15/2014 00:30:07", "content": "

The most interesting technique I found for this competition was Bayesian Personalized Ranking,  which is a collaborative filtering algorithm that optimizes for rank.  Unfortunately, I didn't get a chance to properly test it with different regularization coefficients, but it gave a .01 point bump when I ran it using only one department at a time and plugged the results into my regression.  I really wanted to try binning by month and making a model based off count-based panel data, but I couldn't find any readily-made packages for such a model able to handle a dataset of this size.

\n

Another interesting collaborative filtering algorithm I came across was Bayesian Probabilistic Tensor Factorization which allows you to incorporate time (or I suppose any other axis) into the PMF.

\n

Papers for the collaborative filtering algorithms if anyone is interested:

\n

http://www.cs.mcgill.ca/~uai2009/papers/UAI2009_0139_48141db02b9f0b02bc7158819ebfa2c7.pdf

\n

http://www.autonlab.org/autonweb/19394/version/4/part/5/data/225_Xiong.pdf?branch=main&language=en

", "votes": null, "replies": [] }, { "id": 50507, "author_name": "inversion", "author_url": "", "post_date": "07/15/2014 00:31:10", "content": "

I can't stress enough how important it was to treat high-transaction IDs separate from "regular" IDs. 

\n

I shot up 150 places on the leader board just by doing that.

\n

Then, breaking out training/testing by offer department was what made the rest of the difference.

", "votes": null, "replies": [] }, { "id": 50508, "author_name": "inversion", "author_url": "", "post_date": "07/15/2014 00:34:36", "content": "

I would have loved to have considered the data as a time series. That's an obvious step (that I wasn't able to do since I was stuck using a laptop for most of the 2 weeks prior to contest end).

", "votes": null, "replies": [] }, { "id": 50509, "author_name": "mikeskim", "author_url": "", "post_date": "07/15/2014 00:41:02", "content": "

My score (public 0.61070, private 0.60618) was from a weighted average of random forest (mtry around 3 to 5, replace=FALSE, around 800 trees), bagged random forest (sample size of around 40-50k, around 400-500 reps), gbm ("bernoulli", small shrinkage, high interaction depth of around 25, around 11k trees), and bagged lasso (sample 1000 without replacement, 8000 reps).

\n

I didn't do that much feature transformation or selection other than dummy variables for categorical features. I used mostly Triskelion's features (thanks). I had to do a bit of transformation on the bagged lasso output since all the values were so close together. In R this was: " tt4[,2] = pcauchy((tt4[,2]-min(tt4[,2]))*45) "

\n

I did all of this on a virtual machine with about 10 GB ram. Sometimes I leverage AWS and just throw ram at the problem, but I didn't do that in this case. What I learned in this competition is that while feature engineering is probably the most important thing, sometimes just parallel bagging and model averaging can do wonders if you have enough time and computational resources.

", "votes": null, "replies": [] }, { "id": 50511, "author_name": "xueerchen", "author_url": "", "post_date": "07/15/2014 00:46:49", "content": "

Hi, everyone. This is our 2nd Kaggle contest literally and we are very very lucky! Many thanks to Triskelion and all others who contribute codes and ideas. All our predictions except one are based on Triskelion's features. As absolute newbies, we find feature engineering most challenging and difficult. Our model averages three Neural networks models and 1 random forests model. For neural networks (logistic regression) model, we found that the features with value 0 ( e.g. repeaters who never buy the offer's brand ) may cause serious overfitting. That's why we come up with three different neural network models to mitigate the side effect.

\n

1) use all brand, company and category features but the customers who never bought the same brand or company or category as the coupon's product get a discount of all their feature values. (very subjective) 0.59661

\n

2) use all brand, company and category features but the predictions are only based on neurons that have positive weights along the path.  0.59590 

\n

3) use brand, company and category features to train and test separately and average their predictions.  0.59607

\n

We also trained a very basic random forest with 1000 estimators using all features and we get 0.56958. In the end we took a weighted average of the predictions based on public leaderboard score (not good?).

\n

We really love to be a Kaggler and even being a newbie here can be so much fun! There are a lot to learn here and thank you guys very much to be THE great Sensei here.  To be continued.. : )

", "votes": null, "replies": [] }, { "id": 50515, "author_name": "triskelion", "author_url": "", "post_date": "07/15/2014 01:47:52", "content": "

I improved my benchmark a little bit, by:

\n

- reducing the number of passes to 1,

\n

- doing quantile regression on the "number of repeat trips" for the labels, instead of binary labels.

\n

- Somewhat tighter binning for the time-related features (bought_company_a_170, etc.)

\n

Then I teamed up with Phil Culliton who had made many more effective VW models trained on different and new features. I think he can tell more about those.

\n

Then we basically ensembled all these models. We ended up with a combination of 10 VW models (each trained on their own features) that was a good pick for private leaderboard. Also private public split was good in general (the best submission was only +4 spots).

", "votes": null, "replies": [] }, { "id": 50516, "author_name": "songgc", "author_url": "", "post_date": "07/15/2014 01:56:08", "content": "

Did you guys make cross-validation work? I had been frustrated with that all the time in the competition...

", "votes": null, "replies": [] }, { "id": 50517, "author_name": "mikeskim", "author_url": "", "post_date": "07/15/2014 01:59:07", "content": "

I just CVed via the public leaderboard. I usually do this when it's forecasting because I assume non-stationary data unless I see evidence of otherwise. However, you got 5th, so it's likely it wasn't the biggest problem for you?

", "votes": null, "replies": [] }, { "id": 50518, "author_name": "byang1", "author_url": "", "post_date": "07/15/2014 02:04:56", "content": "

[quote=Guocong Song;50516]

\n

Did you guys make cross-validation work? I had been frustrated with that all the time in the competition...

\n

[/quote]

\n

The best I could come up with was CV by offer ID. With this approach, there're big differences between folds, but when averaged, it was my best CV method.

", "votes": null, "replies": [] }, { "id": 50519, "author_name": "paulperry", "author_url": "", "post_date": "07/15/2014 02:08:46", "content": "

@inversion : Offer doesn't have a dept, did I miss something?

", "votes": null, "replies": [] }, { "id": 50520, "author_name": "inversion", "author_url": "", "post_date": "07/15/2014 02:12:36", "content": "

The offer department was the first two digits of the offer category. (Or the first digit for a 3-digit category).

\n

So, e.g, for offer 1200578 which had a category of 1703, I analyzed everything for dept 17.

", "votes": null, "replies": [] }, { "id": 50521, "author_name": "bshang", "author_url": "", "post_date": "07/15/2014 02:17:29", "content": "

My model was an average of gbm and logistic regression.

\n\n

For gbm, I used similar features as Triskelion but condensed them into fewer features. Rather than counting last30, 60, etc. I weighted each transaction that matched offer cat/comp/brand by how close its date was to the offerdate.

\n

For example, a transaction purchase amount that matched the offer category and was made 5 days before the offer date would be given weight = amount*exp(-5/60). I found 60 days a good decay parameter. Using something like this was the only way I could get gbm to work reasonably (~.605) with params (adaboost, ~1000 trees, depth = 3, shrinkage=~0.01)

\n\n

For logistic regression, I basically used tks' approach with some minor tweaking but got better results with bagging fraction ~0.5.

\n\n

For me, the most significant improvement was combining the models, but not with a straight average of probabilities. Instead, for each model, I ordered each test offer by repeatProbability, then reassigned them with evenly distributed probabilities on a scale of 0 to 1, and then performed the average. I figured given the AUC metric, the ordering was more important than the actual probability from any model. This gave an improvement on the public leaderboard from ~0.605 to ~0.611.

", "votes": null, "replies": [] }, { "id": 50524, "author_name": "econoraptor", "author_url": "", "post_date": "07/15/2014 02:47:47", "content": "

[quote=inversion;50507]

\n

I can't stress enough how important it was to treat high-transaction IDs separate from "regular" IDs.

\n

I shot up 150 places on the leader board just by doing that.

\n

Then, breaking out training/testing by offer department was what made the rest of the difference.

\n

[/quote]

\n

How did you handle the departments when it came time to make the predictions? I also broke my data up by department, but I found that the test set had quite a few offers for departments that weren't in the train set and vice versa.

\n

Also, on a related note, when I broke up the data by time I found that there were many months--as much as half-- where the only purchases made by customers in the test set for the companies of those that made offers were the brands associated with the offer. I have the suspicion that the test and train sets might actually have been collected from two different sources and combined into one. At the very least it would indicate that transactions history from the test data might have been mixed in at a later date and not sampled from the same original pool of data.

", "votes": null, "replies": [] }, { "id": 50525, "author_name": "songgc", "author_url": "", "post_date": "07/15/2014 02:55:33", "content": "

[quote=Mike Kim;50517]

\n

I just CVed via the public leaderboard. I usually do this when it's forecasting because I assume non-stationary data unless I see evidence of otherwise. However, you got 5th, so it's likely it wasn't the biggest problem for you?

\n

[/quote]

\n

It was the biggest problem for me. My scores were very fluctuating over time. I generated similar features inversion did, and tried many normalization over offers, which helped lift the score, but the improvement of CV was limited. Since the training time was short, I just kept submitting...

", "votes": null, "replies": [] }, { "id": 50527, "author_name": "selfishgene", "author_url": "", "post_date": "07/15/2014 02:56:50", "content": "

Regarding cross validation:
I think the major problem that made life difficult with building a proper validation set might have been caused due to the fact that the test set and train set had different products offered to them (6 products are in train but not in test, 8 products are both in train and test, and 7 more products are in test but not in train). I've used 'product' as the basic unit since several different offers were actually different discounts offered to the same product.

Anyway, when I tried mimicking these 'product split' ratio's using the 14 products in the train set and tried learning to generalize from a random split of 4 randomly chosen products in train only, 5 randomly chosen products in test only, and 5 remaining products split by time to train and test, I've found that the amounts of 'noise' were huge from fold to fold. for some splits the validation error was in the ranges of 0.55, and for some other splits in the ranges of 0.63 (for my earliest models). I concluded that this is too much noise and that I needed to do about 100 such random splits (folds) and average their results to get a reliable cross validation scheme and decided it would take too much, so instead I tried finding a representative single split. I couldn't find anything really good, but I've found something that was OK, and used that as a general guide to guide me to "the general ballpark".


Regarding the features we've used:
My initial way of thinking about this problem was like a recommender system.
The basic assumption I used was that most people are not aware of all the products that are out there (there are about 120,000 products in the data set), but, that at the same time, people can be characterized by the specific products they tend to buy, and similarities between people can be drawn according to each persons 'long term shopping cart', and similarities between products can be drawn according to the 'the population demographic' that buys different products.

\n

So I constructed the matrix that contains 310K consumers as rows and the 120K products as columns, and filled the (i,j) th element in it to be the amount of times that consumer bought that specific product.
this matrix is of course very very sparse. meaning, most of it's elements are 0.
As I've said, my basic assumption was that some of these 0 elements account for "that person is not interested in that product", but that most of these zeros account for "that person has never heard of that product".

\n

So I wanted to fill in these zeros by "if that person were to know about this product, will this product be part of his long term shopping cart?".

\n

There are several simple ways that I know of that are able to achieve this: consumer kNN (find consumer i's k most similar consumers, and average their j'th column values), product kNN (similar to consumer kNN but with switching the rows and columns), and matrix factorization (construct a low rank matrix approximation to the large matrix by performing SVD or PCA).

\n

Non of these approaches were feasible running on the entire 310000x120000 matrix, so I've subsampled the products and kept only the most popular products across all population (largest amount of consumer that bought them) and those products that were most similar to the query products (the products that were initially offered with discount). so in the end, I had a matrix of about 310000x2000 as the basic feature set to work with. this proved to be quite good.

My partner (who I teamed up with toward the end of the competition) took an entirely different approach and managed to squeeze a lot out of Triskelion's features (and some more features of the same type), by training many different types of learners with the same feature set and mixing them together.

In the end, blending our very different approaches has resulted in quite a large increase in score.

", "votes": null, "replies": [] }, { "id": 50528, "author_name": "jackhan1", "author_url": "", "post_date": "07/15/2014 02:59:14", "content": "

As to the features, besides the basic factor features - brand,category,chain,and company,I also tried to construct additional features from the following business perspectives for forcasting repurchase.

\n

1.Customer's purchasing behiviours in the past.

\n

I used RFM method. RFM stands for
Recency - How recently did the customer purchase?
Frequency - How often do they purchase?
Monetary Value - How much do they spend?

\n

I mainly leveraged Triskelion's work for these variables construction and I also used Triskelion's original score as my base line.

\n

2.Brand's quality - average return rate of a brand

\n

3.BrandPopularity/Influence

\n

BrandPopularity1 - brand's average revenue per month
BrandPopularity2 - brand's average repeat rate

\n

4.Price

\n

Brand's price - average price of the brand
Preferred price for a customer - average price of the customer spent in the same category of a brand
Offer value - offervalue/brand_value

\n

5.Competition - number of brands in the same category

\n

6.Easy to Get - number of total brands or revenue in a chain, to measure the chain's popularity or scale.

\n

However, I found that only the feature of brand's quality can improve the base line (Triskelion's original score) a little bit. 

", "votes": null, "replies": [] }, { "id": 50531, "author_name": "weiwunyc", "author_url": "", "post_date": "07/15/2014 03:55:50", "content": "

I discovered the same very effective mixing/ensembling method as cherrybarry. Since with AUC only the ranking matters, and the predictions from different models have very different scale/ranges, so averaging their rankings instead of the prediction values makes more sense and gives much better results.

\n

Once I found the fact that mixing results from two very different models can improve the results significantly, even if the results of individual models are not very impressive. I try to add many different models:

\n

Using a bigger set of features similar to Triskelion's but expanded and also calculated using the full transactions data I tried:

\n

1. Triskelion's VW model

\n

The rest of the models using R packages.

\n

2. R's glmnet model with bagging

\n

3. randomForest with bagging

\n

4. Single gbm model

\n

5. gbm model bagging with very small number of trees

\n

6. knn3, R's k-nearest neighbor model

\n

7. NaiveBayes model

\n

Then using the above mixing/ensemble method to get the final results.

\n

Once I teamed up with Selfish Gene, since there was not much time left, and we are in different parts of the globe, we just independently worked on our own models and once I submitted my models he will download the my best results and using the above stated mixing method to mix his results with mine. We furthur see meaningful improvements from best of either models.

\n

I suspect, that if we mix/ensemble the top 10/20 models of this competitions, we may get again some meaningful improvements from any of the single models, as from the shared ideas from this thread, the approaches are very different. Perfect candidates for mixing/ensembling.

", "votes": null, "replies": [] }, { "id": 50533, "author_name": "vijay2110", "author_url": "", "post_date": "07/15/2014 04:12:55", "content": "", "votes": null, "replies": [] }, { "id": 50534, "author_name": "vijay2110", "author_url": "", "post_date": "07/15/2014 04:28:45", "content": "

Hi,

\n

I am new to this field, and this was my first competition. I am placed at #249

\n

I used Logistic Regression (Feature engineering in Java, and trained my model in Octave). I tried to fine tune my solution using Random Forest, but it did not help much.

\n

I want to share few things which helped me to move up on leader-board rapidly.

\n

1. Removing noise: There were few customers whose feature values (total purchase amount, total number of transactions) were too big. So I removed that data which 8 times more than mean.

\n

2. Polynomial degrees of feature values. (square and cube)

\n

3. After training I found below were high weighted features

\n

Total purchase amount

\n

Total number of transactions

\n

Total quantity purchased

\n

Total purchased amount by same company-category-brand

\n

Total number of transactions by same company-category-brand

\n

Total quantity by same company-category-brand

\n

Thanks.

", "votes": null, "replies": [] }, { "id": 50537, "author_name": "shaironen", "author_url": "", "post_date": "07/15/2014 05:17:40", "content": "

We started with Triskelion's features and also had the idea of adding recommender based features. We tried item-based collaborative filtering (used mahout) and latent factor models (Koren's work from Netflix prize). Best submissions were item-based.

\n

By pruning the number of most similar items I tuned the number of actual recommendation values (which are always >=1) vs N/A (no similar items found, converted to 0 ), which is effective to  balance between explicit and implicit recommendation (implicit meaning user didn't buy because they didn't like, not  just weren't aware). Adding two such features with different thresholds helped.

\n

We used logistic regression with 10% sub-sampling and bagging 2000 models,  using code of tks . I found taking median was better than average (but now realizes from others that should have tried averaging ranks too).. As tks pointed out, need extreme L2 regularization (lambda~2^13). Strong regularization must be needed due to the large variance but  I'm  puzzled by how big lambda is. Trying elastic net with mix in L1 reduced the optimal lambda but didn't give best result.I'm interested if someone was able to make less regularized  logistic regression to work.

\n

For CV I  used K-folds stratified by  offer labels and averaged. It seems pretty consistent with progress on LB..

\n

.

", "votes": null, "replies": [] }, { "id": 50544, "author_name": "fidelcastillo", "author_url": "", "post_date": "07/15/2014 07:36:51", "content": "

Hi. And thank you to everyone to share all this information. I’m new and I appreciate this.

\n

The features that I create was basically descriptions features about how customer buy, especially in each department, and features about the offer.

\n

Them I did diferent random forest, and then I esambled it.

\n

The random forest was focusing in the repeater variable, but if I'd had more time I would have tried some glm with repeattrips, using the Poisson distribution.

\n

Did anyone use a glm in that way?

", "votes": null, "replies": [] }, { "id": 50547, "author_name": "westsider", "author_url": "", "post_date": "07/15/2014 07:41:45", "content": "

It is interesting has somebody tried to use Assosiation Rules for this task? As for me it`s classic task for Recommendation System, but I couldn't find no significant rule.

", "votes": null, "replies": [] }, { "id": 50549, "author_name": "tayshin", "author_url": "", "post_date": "07/15/2014 08:52:59", "content": "

I created several new variables and used glmnet with alpha 0 lambda 0.00371. Where the lambda obtained from cv.glmnet from full trainset. It scored 0.61093 on public 0.60313 on private

\n

st_change :OLS cusum test for each customer (to capture structural break)
both_contained_offer: is the offer contained in both train and test set?
ratio_offer_bought_all: offered product (brand+company+category) / numbers of purchased product
ratio_offer_bought_180:offered product (brand+company+category) / numbers of purchased product within 180 days
ratio_offer_bought_90:....
ratio_offer_bought_60:..
ratio_offer_bought_30:...
last_date_purchased: offer date - last date that each customer purchased offered product
mean_pur_amount: daily purchase / total spent
mean_pur_amount_180: daily purchase / total spent within 180days
mean_pur_amount_90:...
mean_pur_amount_60:...
mean_pur_amount_30:...
cate_sensitive: numbers of different brand+category / numbers of different brand+category+category
cate_sensitive_180:numbers of different brand+category / numbers of different brand+category+category within 180 days
cate_sensitive_90
cate_sensitive_60
cate_sensitive_30
cust_cate_sensitive: numbers of different brand / numbers of different brand+category+category
cust_cate_sensitive_180
cust_cate_sensitive_90
cust_cate_sensitive_60
cust_cate_sensitive_30
cust_company_sensitive: numbers of different company/ numbers of different brand+category+category
cust_company_sensitive_180: numbers of different company / numbers of different brand+category+category within 180days
cust_company_sensitive_90
cust_company_sensitive_60
cust_company_sensitive_30
cust_brand_sensitive: numbers of different brand/ numbers of different brand+category+category
cust_brand_sensitive_180
cust_brand_sensitive_90
cust_brand_sensitive_60
cust_brand_sensitive_30

", "votes": null, "replies": [] }, { "id": 50552, "author_name": "domcastro", "author_url": "", "post_date": "07/15/2014 09:39:11", "content": "

We added in "general" features. In total had 750 features

\n

- customer / dept boolean matrix

\n

- brand / category boolean matrix

\n

- brand / company counts

\n

etc

\n

Training by different subsets and averaging increased the score

\n

XGBOOST and GBM worked the best. Code coming soon

", "votes": null, "replies": [] }, { "id": 50564, "author_name": "drk195517", "author_url": "", "post_date": "07/15/2014 13:02:23", "content": "

Quick and dirty solution that we implemented in a week with limited computational resources:

\n

1. Created ~100 features that explain users behavior, from transactions. 

\n

2. Created another ~50-100 features from clustering users, to have some more meaningful features for users with very few transactions. Not sure these helped at all.

\n

3. Run average sized RF. Also tried logistic and adaboost. None of the 2 performed as good as RF. RF was also preferable cause there was little or no need to preprocess the features and time was limited.

\n

4. Combined 4-5 methods, with the highest weight put on our best RF. This improved performance just enough to get AUC 0.5997.

", "votes": null, "replies": [] }, { "id": 50565, "author_name": "kazanova", "author_url": "", "post_date": "07/15/2014 13:38:12", "content": "

[quote=Guocong Song;50516]

\n

Did you guys make cross-validation work? I had been frustrated with that all the time in the competition...

\n

[/quote]

\n

Did you try splitting via offer? E.g train on all offers but one and test on that one. That worked fairly well for me with a little bit more tweaking. E.g to say there should be at least X uplift in order to trust it. For me the was was 0.002 (from AUC 0.610 to AUC 0.612). 

", "votes": null, "replies": [] }, { "id": 50566, "author_name": "klapod", "author_url": "", "post_date": "07/15/2014 13:52:35", "content": "

For cross validation, I tried to take into account that observations in the test set were for offers on later dates than the training set, and also that many were for different product categories. I ordered the data by offer date, and used the first half for training (75,000 observations). For the cross-validation set, I used the remaining rows of data, but only selected the categories that had not appeared in the first 75,000 rows, i.e. 2202, 2119, 5616, and 6202.

\n

Of course, this wasn’t perfect, but it did help in identifying larger jumps. I could also do it fairly quickly as opposed to using K folds.

\n

For my score (public: 0.61228, private: 0.60575), I used features similar to Triskelion’s, as well as inversion’s, and trained a single random forest (500 trees, mtry=2, nodesize=5).

\n

I’d be interested to know if anyone had any good feature selection methods. I noticed that my score dropped when adding certain features, and increased when removing some, but I had no systematic way of testing each feature.

", "votes": null, "replies": [] }, { "id": 50568, "author_name": "udibr1", "author_url": "", "post_date": "07/15/2014 14:10:09", "content": "

For me it was the very first kaggle. My secret was lots of hard work (ended doing exactly 400 submissions all by myself.) I found very specific features to this data set, I don't think anyone will gain from them outside the company running the competition.

", "votes": null, "replies": [] }, { "id": 50570, "author_name": "jiweiliu", "author_url": "", "post_date": "07/15/2014 14:31:18", "content": "

[quote=zzspar;50568]

\n

For me it was the very first kaggle. My secret was lots of hard work (ended doing exactly 400 submissions all by myself.) I found very specific features to this data set, I don't think anyone will gain from them outside the company running the competition.

\n

[/quote]

\n

Congratulations! I appreciate it if you could share the methodology to discover golden features, if it's not all about intuitions and prior knowledge. Thank you very much! Again, all by yourself, that's really impressive!

", "votes": null, "replies": [] }, { "id": 50571, "author_name": "udibr1", "author_url": "", "post_date": "07/15/2014 14:40:13", "content": "

CV was the hardest item also for me. In one of my attempts at inventing a CV method I realized I had a bug in the code but on the other hand this CV was very effective on the leader board so I had to dig into the data to find out what was behind the buggy algorithm

", "votes": null, "replies": [] }, { "id": 50572, "author_name": "davidzwoo", "author_url": "", "post_date": "07/15/2014 14:58:40", "content": "

For customers that purchased the same category + company + brand, one feature that seems to have some significance was the discount rate of the coupon. I calculated this based on the median amount per qty.

\n

Lower discount rate seems to have more repeat ability than high.

\n

Other working theories I had was:

\n

1) Brand loyalist to a particular brand may be more likely to try a coupon but less likely to switch brands

\n

2) Whereas brand shifters, maybe more likely to switch brands after trying a coupon.

\n

One thing wanted to try was time-series clustering by customers buying patterns. Was wondering if anyone had luck with this? or found variables with a time dimension was useful?

", "votes": null, "replies": [] }, { "id": 50579, "author_name": "grigorydymov", "author_url": "", "post_date": "07/15/2014 16:11:25", "content": "

This was my first Kaggle competition. Big thanks to my teammate turbonerd for introducing me to it and keeping the spirit up! Without him, this would not have happened. Thanks to Kaggle for hosting and excellent competition and to the anonymous data provider for the interesting dataset.

\n

Here are some lessons learned, helpful or not.

\n

1) Spent far too much time worrying about our setup, ending up with a combination of MySQL, python scripts with a task-oriented class structure and csv files. Using SQL limited a bit in what aggregates could be produced (no medians etc), but speeded up the feature engineering process. In hindsight, a clean map-reduce would probably have been better.

\n

2) It took some time for to figure out what the problem was. When, finally, cross-validation started to work (somewhat), dividing up train set by unique product (company,category,brand combination), a lot of time and wasted effort had elapsed.

\n

3) This competition was a lot about feature engineering, and having no experience with that, it took a lot of time and many mistakes were made.

\n

Some of the best features turned out to be hierarchical, trained separately on data from transactions. E.g. a logistic regression-based probability of buying the product on offer, given which of the top100 most popular products this user had purchased (also did this with brand, category, company, department).

\n

Other good ones were simple stuff like which fraction of previous trips the user bought this product on. Other ones tried, besides the ones suggested on forums, fall broadly into these categories: weekday statistics (quantities, amounts, measures purchased on certain days), loyalty to products (how many categories in a department a user bought, how many brands for a company etc – almost all permutations), prior purchase-based (how many times, how often, ), loyalty to store (which weekdays visited, how many times, time period between first and last visit), market, chain, dept-level aggregates of , basket sizes, relative price of offered product to this users usual level for this department, many of above based on 1, 2, 3 months of data prior to offerdate. In total, 214 features.

\n

We also tried some association rule learning, but dropped it pretty quickly.

\n

4) The main mistake was to not be more systematic, which seems obvious in hindsight. We knew that CV was a problem; that offers were very different (repeating ranging 10-40%); that purchase because of coupon and purchase because of prior habits needed to be modeled separately; but somehow did not spend enough time systematically solving these issues, spending most time on feature engineering. A lot more time should have gone into thinking about good ways to train. The absurdity of life is that after all effort with random forests, ridge regression and many others, vowpal vabbit ended up handily beating all out blended attempts at cool models.
Also, more time could have gone into looking and visualizing the data.

\n

5) Not doing feature selection probably hurt the final performance a lot, since some features ended up to be highly correlated. This is a lesson for next time and it would be interesting to hear if someone managed to automate feature selection somehow.

\n

We ended up with a ROC AUC of 0.57485 on the private leaderboard, netting us the 545th place. Hard to say if its good or bad for beginners, at least it was very fun and a learning experience.

\n

Some of the above will probably seem very stupid after reading this thread :)

", "votes": null, "replies": [] }, { "id": 50580, "author_name": "auduno", "author_url": "", "post_date": "07/15/2014 16:11:38", "content": "

One problem I noticed sometimes when cross-validating was that although the classifier had good AUC on each offer individually, the classifier overall had much much poorer total AUC. I'm thinking that this was because the relative probabilities of the different offers was not aligned somehow, but that's just a guess. Did anyone find a principled way to deal with this? I looked around for some theoretical approach to this, but could not find any.

", "votes": null, "replies": [] }, { "id": 50581, "author_name": "kazanova", "author_url": "", "post_date": "07/15/2014 16:15:43", "content": "

[quote=auduno;50580]

\n

One problem I noticed sometimes when cross-validating was that although the classifier had good AUC on each offer individually, the classifier overall had much much poorer total AUC. I'm thinking that this was because the relative probabilities of the different offers was not aligned somehow, but that's just a guess. Did anyone find a principled way to deal with this? I looked around for some theoretical approach to this, but could not find any.

\n

[/quote]

\n\n

You needed 2 models! One to optimize for individual (offer-specific)  AUCs and one for the general AUC :) At least, this is what we did. Also scaling helped in this one.

", "votes": null, "replies": [] }, { "id": 50589, "author_name": "auduno", "author_url": "", "post_date": "07/15/2014 16:52:27", "content": "

[quote=KazAnova;50581]

\n

You needed 2 models! One to optimize for individual (offer-specific)  AUCs and one for the general AUC :) At least, this is what we did. Also scaling helped in this one.

\n

[/quote]

\n

Ah, I considered that, but didn't get enough time to try it, unfortunately. Did you blend the two models to get final result, or did you use the second model to adjust the relative predictions of the offer-specific models?

", "votes": null, "replies": [] }, { "id": 50590, "author_name": "kazanova", "author_url": "", "post_date": "07/15/2014 16:53:20", "content": "

[quote=auduno;50589]

\n

[quote=KazAnova;50581]

\n

You needed 2 models! One to optimize for individual (offer-specific)  AUCs and one for the general AUC :) At least, this is what we did. Also scaling helped in this one.

\n

[/quote]

\n

Ah, I considered that, but didn't get enough time to try it, unfortunately. Did you blend the two models to get final result, or did you use the second model to adjust the relative predictions of the offer-specific models?

\n

[/quote]

\n\n

Blend!

", "votes": null, "replies": [] }, { "id": 50591, "author_name": "inversion", "author_url": "", "post_date": "07/15/2014 16:54:28", "content": "

[quote=Grigory;50579]

\n

We ended up with a ROC AUC of 0.57485 on the private leaderboard, netting us the 545th place. Hard to say if its good or bad for beginners, at least it was very fun and a learning experience.

\n

[/quote]

\n

It doesn't matter where you place, it matters what you learned. And it sounds like you learned a lot!

\n

I placed in the bottom 10% my first competition, then top 40%, then top 30%, and now top 5% with this competition.

\n

I'd say you rocked it for your first time.  :-)

", "votes": null, "replies": [] }, { "id": 50595, "author_name": "inversion", "author_url": "", "post_date": "07/15/2014 17:05:58", "content": "

[quote=econoraptor;50524]

\n

[quote=inversion;50507]

\n

I can't stress enough how important it was to treat high-transaction IDs separate from "regular" IDs.

\n

I shot up 150 places on the leader board just by doing that.

\n

Then, breaking out training/testing by offer department was what made the rest of the difference.

\n

[/quote]

\n

How did you handle the departments when it came time to make the predictions? I also broke my data up by department, but I found that the test set had quite a few offers for departments that weren't in the train set and vice versa.

\n

[/quote]

\n

For each CustID, I did a pivot (using pandas) of the total amount purchased for each dept. (More accurately, I only counted purchases that were <= 180 days from the date of the offer given to that CustID, filtering out any transactions with a 0 count or negative amount.)

\n

Then I did a PCA on the table. By inspecting the loading plot, I was able to find a dept in the training data that was similarly correlated to a dept in the test data.

\n

There were 2 depts in the test set that didn't correlate well to any in the training set. For those, I just trained on all the depts in aggregate.

", "votes": null, "replies": [] }, { "id": 50597, "author_name": "byang1", "author_url": "", "post_date": "07/15/2014 17:46:38", "content": "

[quote=KazAnova;50565]

\n\n

Did you try splitting via offer? E.g train on all offers but one and test on that one. That worked fairly well for me with a little bit more tweaking. E.g to say there should be at least X uplift in order to trust it. For me the was was 0.002 (from AUC 0.610 to AUC 0.612). 

\n

[/quote]

\n

Hmm, I did split by offer, but didn't try leave-one-out CV. I should try it see what happens.

", "votes": null, "replies": [] }, { "id": 50599, "author_name": "songgc", "author_url": "", "post_date": "07/15/2014 18:06:02", "content": "

[quote=KazAnova;50565]

\n

[quote=Guocong Song;50516]

\n

Did you guys make cross-validation work? I had been frustrated with that all the time in the competition...

\n

[/quote]

\n

Did you try splitting via offer? E.g train on all offers but one and test on that one. That worked fairly well for me with a little bit more tweaking. E.g to say there should be at least X uplift in order to trust it. For me the was was 0.002 (from AUC 0.610 to AUC 0.612). 

\n

[/quote]

\n\n

I did a quick try in which I used one offer data to train models against the rest, but didn't work out. I should do normal splitting...  Single offer has too large variance:)  

", "votes": null, "replies": [] }, { "id": 50608, "author_name": "chinga", "author_url": "", "post_date": "07/15/2014 19:55:34", "content": "

Hi folks, I'm a recent "graduate" of Andrew Ng's machine learning course. I entered this, my first competition, as a training exercise and have thoroughly enjoyed it.

\n


Like Jack Han, I started with RFM (recency, frequency, monetary). Frequency didn't help, but monetary did, especially when binned. I used total spend over the previous 90 days.
I used dummy variables for prod, category and dept in bins of "never bought", 1-15 days ago, 16-30 ago, 31-70 ago. The bins were chosen based on histograms/box-and-whisker plots in Tableau. This is similar to Triskelion's approach, but I didn't use product quantity, just ones and zeros. For brand, I ended up just using the "never bought" bin. I used logistic regression in vw. My features were generated from HP Vertica using SQL, and I used perf to calculate ROC.

\n


I experimented to no avail with various other features to find a proxy for customers who tend to flit between products in the same category: category spend ratios, category entropy, price paid relative to mean. The customer's ratio of distinct products bought per category relative to the average ratio per category worked well on the training set, but harmed my score on the public test set. Like 'inversion', I noticed higher take up for those with high IDs, but it was late in the day and I wasn't able to exploit it with my best model.

\n


In the end, my best submission (0.60291 public and 0.59738 private) was achieved by adding total number of distinct products as a feature. I found that the difference between my train ROC using vw holdout and public ROC varied between 0.47 and 0.57. Is this what others are seeing too?

\n


I had planned to implement a brand loyalty measure (anyone try Guadagni and Little 1983?) and itemset mining. Sadly the day job needed attending to.

", "votes": null, "replies": [] }, { "id": 50661, "author_name": "ephrance", "author_url": "", "post_date": "07/16/2014 05:43:20", "content": "

Similar to Jack Han, for each customer I computed the following score

\n

customer_value_score = total_spend_per_day * trxn_per_trip * (1.0/recency) * lifetime * (1.0/avg_interpurchase_time)

\n

where:

\n

trxn_per_trip = number of transactions per trip

\n

recency = days between last purchase date relative to last date in data

\n

lifetime = time elapsed from first purchase date to last purchase date

\n

avg_interpurchase_time = is the average time gap between purchases

\n\n

Being new to Kaggle competitions, I benefited a great deal from combining Triskelion's features with tks's glmnet code + bagging (thanks to meet thakar and clustifier). Additionally, I added return events (quantity < 0) as features. I used the full 22GB dataset. 

\n\n

A great many thanks to Triskelion, tks, culstifier, meet thakar and everyone else for sharing their ideas, insights, and code.

\n

Cheers!

", "votes": null, "replies": [] }, { "id": 50740, "author_name": "auduno", "author_url": "", "post_date": "07/16/2014 22:37:04", "content": "

[quote=KazAnova;50581]

\n

You needed 2 models! One to optimize for individual (offer-specific)  AUCs and one for the general AUC :) At least, this is what we did. Also scaling helped in this one.

\n

[/quote]

\n

I'm guessing you might share the code eventually, but I'm curious : how did you optimize for individual offer-specific AUCs? Most of the offers in the testset were not in the training set, so did you just optimize for those that were in the training set, or did you use some method similar to inversion's method, i.e. finding some combination of the closest "similar" offers to the ones in testset and optimizing for those?

", "votes": null, "replies": [] }, { "id": 50741, "author_name": "kazanova", "author_url": "", "post_date": "07/16/2014 22:43:06", "content": "

[quote=auduno;50740]

\n

 how did you optimize for individual offer-specific AUCs? Most of the offers in the testset were not in the training set, so did you just optimize for those that were in the training set

\n

[/quote]

\n\n

For those in the training set only. When I optimized for individual AUCs, I was ignoring the sample size of the offer and I was just averaging the results of the AUCs of the different offers ( 1-out format), When we were optimizing for the total AUC we were appending the results of each offer's prediction in an array and calculating the total AUC once all results had been appended. I hope that helps. 

", "votes": null, "replies": [] }, { "id": 50765, "author_name": "ricgama", "author_url": "", "post_date": "07/17/2014 11:25:11", "content": "

[quote=inversion;50507]

\n

For each CustID, I did a pivot (using pandas) of the total amount purchased for each dept. (More accurately, I only counted purchases that were <= 180 days from the date of the offer given to that CustID, filtering out any transactions with a 0 count or negative amount.)

\n

Then I did a PCA on the table. By inspecting the loading plot, I was able to find a dept in the training data that was similarly correlated to a dept in the test data.

\n

There were 2 depts in the test set that didn't correlate well to any in the training set. For those, I just trained on all the depts in aggregate.

\n

[/quote]

\n

Hello,
Interesting approach. We have applied a Kolmogorov-Smirnov test and selected only features with similar distributions in de train and test sets.
Can you give more details about your PCA procedure?

", "votes": null, "replies": [] }, { "id": 50779, "author_name": "codingneo", "author_url": "", "post_date": "07/17/2014 13:52:25", "content": "

[quote=inversion;50507]

\n

I can't stress enough how important it was to treat high-transaction IDs separate from "regular" IDs.

\n

I shot up 150 places on the leader board just by doing that.

\n

Then, breaking out training/testing by offer department was what made the rest of the difference.

\n

[/quote]

\n

I read your previous post about your method. I didn't understand how you treat the high-transactions IDs and low-transactions IDs. Can you explain more on it?

", "votes": null, "replies": [] }, { "id": 50786, "author_name": "ash208948", "author_url": "", "post_date": "07/17/2014 15:21:59", "content": "

[quote=inversion;50507]

\n

That's a very wise move. I wonder if a clustering algorithm to assign customers to different groups based on similarities in their shopping habits would have been helpful; then each group from the test set and train set would have been treated individually. I guess I will never know  ;)

", "votes": null, "replies": [] }, { "id": 50810, "author_name": "inversion", "author_url": "", "post_date": "07/18/2014 00:17:08", "content": "

[quote=Yiqun Hu;50779]

\n

I read your previous post about your method. I didn't understand how you treat the high-transactions IDs and low-transactions IDs. Can you explain more on it?

\n

[/quote]

\n

Looking at the customer id data (training set), there were ~1,300 ids that had more than 4,000 total transactions. (The highest had 2,647,164 transactions!) As was mentioned in the forum earlier, this was probably a "default" card that is used at check-out, e.g., when a shopper doesn't have a loyalty card the store scans a "default" card at the register.

\n

If you reverse sort the training data by the number of transactions, you'll see that those ids were almost always repeat shoppers. (See the image below.) The fact that these are repeat buyers is an artifact of including these "default" cards in with normal shoppers.

\n

I simply removed these high-transaction ids and trained without them. For the test set, any id that had a high transaction count, I manually set it to "1" for the estimated repeater probability. Doing this was a "freebie" improvement in score.

\n

\"Repeaters\"

", "votes": null, "replies": [] }, { "id": 50811, "author_name": "byang1", "author_url": "", "post_date": "07/18/2014 00:33:49", "content": "

[quote=inversion;50810]

\n

I simply removed these high-transaction ids and trained without them. For the test set, any id that had a high transaction count, I manually set it to "1" for the estimated repeater probability. Doing this was a "freebie" improvement in score.

\n

[/quote]

\n

How much improvement did you get from this ?

\n

Another possible "freebie" is to train the offers that are in both train and test sets separately. However I didn't use this in the end because the improvement was small, and the CV unreliable.

", "votes": null, "replies": [] }, { "id": 50814, "author_name": "inversion", "author_url": "", "post_date": "07/18/2014 01:21:20", "content": "

[quote=B Yang;50811]

\n

[quote=inversion;50810]

\n

I simply removed these high-transaction ids and trained without them. For the test set, any id that had a high transaction count, I manually set it to "1" for the estimated repeater probability. Doing this was a "freebie" improvement in score.

\n

[/quote]

\n

How much improvement did you get from this ?

\n

[/quote]

\n

From 0.59507 to 0.61017

\n

+150 positions on the leader board or so.

", "votes": null, "replies": [] }, { "id": 50816, "author_name": "inversion", "author_url": "", "post_date": "07/18/2014 01:36:58", "content": "

[quote=RW;50765]

\n

[quote=inversion;50507]

\n

For each CustID, I did a pivot (using pandas) of the total amount purchased for each dept. (More accurately, I only counted purchases that were <= 180 days from the date of the offer given to that CustID, filtering out any transactions with a 0 count or negative amount.)

\n

Then I did a PCA on the table. By inspecting the loading plot, I was able to find a dept in the training data that was similarly correlated to a dept in the test data.

\n

There were 2 depts in the test set that didn't correlate well to any in the training set. For those, I just trained on all the depts in aggregate.

\n

[/quote]

\n

Hello,
Interesting approach. We have applied a Kolmogorov-Smirnov test and selected only features with similar distributions in de train and test sets.
Can you give more details about your PCA procedure?

\n

[/quote]

\n

Sure. My data table looked like this. (Rows = CustIDs, Columns = Depts, Values = sum of customer transactions for each dept [truncated at 180 days from offer date and ignoring negative transaction amounts]) The amounts were transformed with arcsinh for scaling.

\n

\"Dept

\n

I then did a 2-component PCA on the table. (I don't have an image of the loading plot handy, otherwise I'd show it. I used 3 different computers . . . so my files are all over the place. If I stumble upon it I'll post it later.)

The loading plot showed a position for each department. 

\n

For departments that were in the test set but not in the train set, I looked at the loading plot to find a department that was close to that point that was also in the training set.

\n

For instance, I trained on Dept 44 to test for Dept 7. For Dept 45 and 51, I used Dept 55 for training.

\n

Departments 72 and 91 didn't covary well with any departments in the training set, so I used the entire training set to predict those departments.

", "votes": null, "replies": [] }, { "id": 50861, "author_name": "npetitclerc", "author_url": "", "post_date": "07/18/2014 16:43:47", "content": "

Like many mention before the CV problem with he public leader board came from the very strong "Offer ID" bias (some offers were much more successful then others). The way I dealt with it was to find a "repeat ratio" for each "Offer ID" and use that as a feature. I also did the same for all the offers info (Market, dept, brand, etc..). These were all quite strong features for me. This method had the obvious issue that some offers were not in the training data (or had very low statistics), but that's the best I could come up with.

\n

The other features I didn't see people talk about, but that I though were clever, was a measure of how faithful a customer was to products in general. For that I when through all the products a customer ever bought and found how many times he bought it again. From that data I found the usual statistics (min/max, mean. median, standard deviation, etc...), and more importantly a "repeater ratio", which is basically how many purchases they made divided by the number of different products they bought. I liked that feature the most because:

\n

1 - It makes sense - people that are faithful to their products (or don't bother looking for new ones) are more likely to become faithful to a  new product they try.    

\n

2 - It turned out to be quite a strong feature.

\n

My best model was using Scikit-learn with AdaBoost (n_estimators=200) with Random Forest (n_estimators=200, max_depth=2) for base_estimator.

\n

PS. If someone is good with Neural Networks, I would be very curious to see how my set of features would do with that. It was my next step, but I never had a chance to try it.

", "votes": null, "replies": [] }, { "id": 50866, "author_name": "irisren", "author_url": "", "post_date": "07/18/2014 17:46:23", "content": "

[quote=cherrybarry;50521]

\n

For me, the most significant improvement was combining the models, but not with a straight average of probabilities. Instead, for each model, I ordered each test offer by repeatProbability, then reassigned them with evenly distributed probabilities on a scale of 0 to 1, and then performed the average. I figured given the AUC metric, the ordering was more important than the actual probability from any model. This gave an improvement on the public leaderboard from ~0.605 to ~0.611.

\n

[/quote]

\n\n

Thank you for sharing your method. I have struggled on how to ensemble different models. Can you elaborate on how you assigned scores from different models evenly distributed probabilities on a scale of 0 to 1? 

\n

Thank you for your help. 

", "votes": null, "replies": [] }, { "id": 50867, "author_name": "irisren", "author_url": "", "post_date": "07/18/2014 17:50:26", "content": "

[quote=Wei Wu;50531]

\n

I discovered the same very effective mixing/ensembling method as cherrybarry. Since with AUC only the ranking matters, and the predictions from different models have very different scale/ranges, so averaging their rankings instead of the prediction values makes more sense and gives much better results.

\n

[/quote]

\n

How did you average the rankings and transform that into probabilities? Could you explain a little bit more? Thank you! 

", "votes": null, "replies": [] }, { "id": 50882, "author_name": "justmarkham", "author_url": "", "post_date": "07/18/2014 21:12:46", "content": "

[quote=Iris Ren;50866]

\n

[quote=cherrybarry;50521]

\n

For me, the most significant improvement was combining the models, but not with a straight average of probabilities. Instead, for each model, I ordered each test offer by repeatProbability, then reassigned them with evenly distributed probabilities on a scale of 0 to 1, and then performed the average. I figured given the AUC metric, the ordering was more important than the actual probability from any model. This gave an improvement on the public leaderboard from ~0.605 to ~0.611.

\n

[/quote]

\n

Thank you for sharing your method. I have struggled on how to ensemble different models. Can you elaborate on how you assigned scores from different models evenly distributed probabilities on a scale of 0 to 1? 

\n

Thank you for your help. 

\n

[/quote]

\n

I can't speak for cherrybarry, but here is how I would do it:

\n\n

You will now have evenly distributed probabilities between 0 and 1. Do that for each model, and then to ensemble, take a straight average of the adjustedProbability: (model 1 + model 2) / 2

\n

Or, you could weight the models differently: (model1 + 2*model 2 + 3*model 3) / 6

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "50501": "", "50503": "", "50505": "", "50506": "", "50507": "", "50508": "", "50509": "", "50511": "", "50515": "", "50516": "", "50517": "", "50518": "", "50519": "", "50520": "", "50521": "", "50524": "", "50525": "", "50527": "", "50528": "", "50531": "", "50533": "", "50534": "", "50537": "", "50544": "", "50547": "", "50549": "", "50552": "", "50564": "", "50565": "", "50566": "", "50568": "", "50570": "", "50571": "", "50572": "", "50579": "", "50580": "", "50581": "", "50589": "", "50590": "", "50591": "", "50595": "", "50597": "", "50599": "", "50608": "", "50661": "", "50740": "", "50741": "", "50765": "", "50779": "", "50786": "", "50810": "", "50811": "", "50814": "", "50816": "", "50861": "", "50866": "", "50867": "", "50882": "", "1343061": "Hi , can you explain what is TCOB? I have been breaking my head over this. \nAlso, can this be done without Python?" }, "source": "meta" }