{ "id": 12460, "title": "Congrats to the winners!", "url": "/competitions/avazu-ctr-prediction/discussion/12460", "author_name": "", "post_date": "2015-02-10T00:06:11.063Z", "votes": 8, "comment_count": 52, "views": 22228, "content": "

This was a LONG competition. Well played.

", "messages": [ { "id": "63845", "postDate": "02/10/2015 00:06:11", "content": "

This was a LONG competition. Well played.

", "rawMarkdown": "", "votes": null }, { "id": "63846", "postDate": "02/10/2015 00:19:56", "content": "

Congrats. this is a particular hard problem. cannot wait to see the winning solutions.

", "rawMarkdown": "", "votes": null }, { "id": "63850", "postDate": "02/10/2015 00:33:35", "content": "

Congrats! Looking forward to learn something new!

\n

Special thanks to @Tinrtgu !

", "rawMarkdown": "", "votes": null }, { "id": "63852", "postDate": "02/10/2015 00:37:48", "content": "

Congratulations to the folks on the nn2 team, guestwalk, Michael Jahrer, yolicat and mandora, to Owen, and to infinitezxc and machine learner from RandomWalker on their great finishes. I look forward to hearing about what set them apart. Also, congratulations to tingrtu on that impressive beat-the-benchmark code. I used it as part of my solution and I know lots of other people did too.

", "rawMarkdown": "", "votes": null }, { "id": "63866", "postDate": "02/10/2015 03:41:30", "content": "

Congratulations! 

", "rawMarkdown": "", "votes": null }, { "id": "63870", "postDate": "02/10/2015 03:59:51", "content": "

Thanks guys. :) This competition is really long but fun.

\n

And special thanks to tingrtu and 3 idiots for their awesome code for ftrl, fm and gbdt. Our method is mainly built on their code with moderate modifications. But maybe it is feature engineering that is the most important.

\n

We make up about 10 numerical features by counting the times of impression to one person and time intervals. How to locate a person is an interesting question though. Device_id has too many nulls and device_ip is not that accurate. We use device_id if it is not null and device_ip+device_model for others. Then we use gbdt trained on numerical features to better "bin" these features and get some edge.

\n

Our final solution is the ensemble of several ftrl and fm models. In ftrl selected interactions and gbdt features are used and in fm both the model with gbdt features and without gbdt features are used. We will post the code and detailed document soon.

", "rawMarkdown": "", "votes": null }, { "id": "63871", "postDate": "02/10/2015 04:16:59", "content": "

Congratulations!

", "rawMarkdown": "", "votes": null }, { "id": "63872", "postDate": "02/10/2015 04:17:16", "content": "

Congratulation to the winners!

\n

My solution is similar to RandomWalkers, except I did not use gbrt. The key points of my solution:

\n

1. A lot of feature engineering: count features, combinations of several categorical features, several types of likelihood features, time features.

\n

2. FTRL code with batches: here is the simple idea how to essentially improve FTRL algorithm: sort dataset by app_id, site_id and run Python code suggested by tingrtu. To get different algorithms, I sorted dataset using different set of categorical features.

\n

3. My final model contained 1 fm and 3 FTRL models with batches, ensembled with geometric average. The best score for fm model on public lb is 0.38358, for FTRL model - 0.38557.

", "rawMarkdown": "", "votes": null }, { "id": "63873", "postDate": "02/10/2015 04:27:12", "content": "

[quote=Dmitry Efimov;63872]

\n

2. FTRL code with batches: here is the simple idea how to essentially improve FTRL algorithm: sort dataset by app_id, site_id and run Python code suggested by tingrtu. To get different algorithms, I sorted dataset using different set of categorical features.

\n

[/quote]

\n

Thank you for sharing! Does it mean same app_id => same batch?

", "rawMarkdown": "", "votes": null }, { "id": "63875", "postDate": "02/10/2015 04:36:52", "content": "

Yes, basically, it works in the following way: train weights on the first app_id, make predictions on this app_id, train weights on the second app_id (starting from previous app_id weights) and so on. I suppose, there are a lot of ways to improve this idea, for example, initialize weights after some batches, but I did not have time to try it.

", "rawMarkdown": "", "votes": null }, { "id": "63876", "postDate": "02/10/2015 04:43:45", "content": "

Congratulations winners. That was an impressive amount of separation in an otherwise tight leaderboard.

\n

Something simple that helped me was to adapt the FTRL to have a decaying learning rate per epoch. So my single best was a 0.3894276, and that was achieved with 3 epochs, where the learning rate was adjusted to 85% of its previous rate. The first time I did that, it was worth 100 spots or so. I'll post the slight update in that thread for others to try out--a simple method in the FTRL class and a call to that after the epoch loop is done. Hopefully others get some use out of it.

\n

For features, I started with a few things I thought would help classify new users, which was to measure the number of records divided by the number of hours, and the number of records divided by the number of unique site/apps. Logging and truncating that provided really nice separation, and it looked similar using it on sites/apps as well so I had two more--four in all. Aside from that, there were a bunch of carefully selected interactions--trying not to put elements of a hierarchy together, which was an idea referenced in one of the Criteo winner's post.

\n

Toward the end, I started running segmented models by site/app. I think 5 sites captured 40% of the volume, so I had a model for each. Those models helped, but not near as much as anything listed above.

", "rawMarkdown": "", "votes": null }, { "id": "63877", "postDate": "02/10/2015 05:09:19", "content": "

Congratulations to the winners especially Michael Jahrer ( We have worked together in our employer's projects but were competing separately here ) !! My solution is based on a single code, ftrl (thanks to Tinrtgu, you are awesome) and a lot of feature engineering from my end (thanks to pandas). Happy that I got my first top 10 finish in such a crazily hard competition. I thought of implementing 3 idiot's solution of Criteo to make an ensemble but didnt have time for it and I really wanted to know if it is possible to enter top 10 with a single code, all Pythonic . :) 

", "rawMarkdown": "", "votes": null }, { "id": "63878", "postDate": "02/10/2015 05:27:41", "content": "

Congrats to all!

\n

I ended up getting the most mileage from feature engineering time features. I tried to capture as much about the dynamics of each visitor's history of visits as I could. E.g. time since last visit, number of visits the prior hour, number of visits N days back, etc. I never checked to see which combinations time features were most useful, but in general they improved my LB score a lot. I think more clever feature engineering of time features might have gotten a better score, but ironically, I ran out of time.

\n

@Dmitry Efimov, and @infinitezxc, I'm curious if you could share details on how you got your FM to work. How did you deal with overfitting? I played around with a FM, but ran into some challenges with overfitting.

", "rawMarkdown": "", "votes": null }, { "id": "63881", "postDate": "02/10/2015 06:10:13", "content": "

[quote=Daniel Yoo;63878]

\n

@Dmitry Efimov, and @infinitezxc, I'm curious if you could share details on how you got your FM to work. How did you deal with overfitting? I played around with a FM, but ran into some challenges with overfitting.

\n

[/quote]

\n

We got the FM to work by tuning lambda and number of epochs through local validation that takes the last two days for testing. The model tends to overfit with smaller lambda and more epochs, so we just used the best parameters for validation and got 0.3834 for our best single fm model.

", "rawMarkdown": "", "votes": null }, { "id": "63888", "postDate": "02/10/2015 07:58:57", "content": "

Although I like building algorithms I realised along the way that this competetition was more about finding additional features. I used FM and FTRL although I tried a few others.

\n

Working method

\n\n

Feature engineering

\n\n

Software and tuning

\n\n


What did not help (for me).

\n\n

Results/conclusions

\n\n

Congrats to the winners and a shoutout to Trintu and the 3 idioits and Micheal Jahrer and stephen rendle for their great contributions in terms of algoritms.

Personal shoutout to Dmitry Efimov.. I felt pretty lame with my last-effort submission just before the deadline..

", "rawMarkdown": "", "votes": null }, { "id": "63890", "postDate": "02/10/2015 08:00:25", "content": "

Thanks @infinitezxc!!! It seems almost too easy when you put it that way :-).

\n

Did you mostly manually tune your lambda (I'm assuming L2 regularizer) and # epochs? Did you optimize the degree of the factorization or the random initialization of the factorization?

\n

Perhaps I never found a good factorization model because I was searching over too many dimensions of hyper parameters.

\n

I'd love to hear how others also optimized over their model hyper parameters considering how long it could take to pass through the data. I found myself forgoing much hyper parameter optimization in favor of ensembling a random mix of hyper parameters.

", "rawMarkdown": "", "votes": null }, { "id": "63906", "postDate": "02/10/2015 09:57:20", "content": "

[quote=Julian de Wit;63888]

\n\n\n

[/quote]

\n

This is interesting, any hint about how to implement an online FM?

", "rawMarkdown": "", "votes": null }, { "id": "63907", "postDate": "02/10/2015 10:04:12", "content": "

[quote=Daniel Yoo;63890]

\n

Did you mostly manually tune your lambda (I'm assuming L2 regularizer) and # epochs? Did you optimize the degree of the factorization or the random initialization of the factorization?

\n

[/quote]

\n

For # epochs, just give it a large number and see when loss goes up in validation. 

\n

We also tried tuning other hyper parameters, and adjusting the random initialization got a ~0.0001 improvement. BTW we also made use of the first order features in FM. 

", "rawMarkdown": "", "votes": null }, { "id": "63908", "postDate": "02/10/2015 10:05:57", "content": "

Congratulations to all winners, and thanks for sharing.

\n

I used the SGD code posted by Abhishek (thanks!) and did greedy selection of interaction terms (depth first) using 5% of the data. I selected interaction terms that improved the prediction for holdout day 10, to identify interactions that are 'stable in time'. As features I had raw data and univariate logtransformed counts.

", "rawMarkdown": "", "votes": null }, { "id": "63909", "postDate": "02/10/2015 10:34:40", "content": "

[quote=Daniel Yoo;63878]

\n

@Dmitry Efimov, and @infinitezxc, I'm curious if you could share details on how you got your FM to work. How did you deal with overfitting? I played around with a FM, but ran into some challenges with overfitting.

\n

[/quote]

\n

I am going to write the detailed description of my solution soon. Regarding FM, I used the code of 3 idiots, with learning rate 0.02, 20 epochs and 20 latent factors. I have also tried Stephen Rendle's libfm, but I got only 0.38640 on lb with it. Some hints to avoid overfitting:

\n

1. Remove low-frequent levels (I used threshold 100).

\n

2. For continuous features I transformed it to binary features using discretisation or rounding. Regarding discretisation, there is a very nice package infotheo in R with function discretize there.

\n

3. The main problem was to add likelihood features to FM without overfitting, I constructed them in different ways, for example, for current day I used the previous day only. Likelihood features gave 0.002 improvement on the public lb.

", "rawMarkdown": "", "votes": null }, { "id": "63913", "postDate": "02/10/2015 11:23:44", "content": "

[quote=Nishant Kumar;63877]

\n

My solution is based on a single code, ftrl (thanks to Tinrtgu, you are awesome) and a lot of feature engineering from my end (thanks to pandas).

\n

[/quote]

\n\n

Congratulations! And please share more info on what exactly you did :-)

", "rawMarkdown": "", "votes": null }, { "id": "63915", "postDate": "02/10/2015 12:10:44", "content": "

Can someone post the libFM pipeline code (from data transformation into sh run)? I tried to use it as a hastily modified 3 idiots code, and could not get it to perform well. 

", "rawMarkdown": "", "votes": null }, { "id": "63921", "postDate": "02/10/2015 14:09:33", "content": "

@Julian de Wit great explanation mate !

\n

I will talk for my part (which would probably have finished top 10)

\n

Based on some discussions I had recently with colleagues, academics and other data science enthusiasts , I came into some sort of collision by claiming that on many different problems at present (and with the advances of machine learning) you can do really well without even inspecting  the data - just by 

\n

1) knowing the tools

\n

2) cross- validating properly when you make features' selections, hyper parameter optimization etc

\n

3) ensembling many different models to capture different information

\n

In Light of this I did exactly what I tried in Tradeshift plus a couple of more models since we had more time.  I only looked at the header of the data and once I understood the set is divided by days , I did a simple frequency count in that column. I also found  the row that constitutes the beginning of day 30 (row 36,210,029) and looked at that. That is the end of my data inspection.

\n

I did NOT divide my set in device or not, did NOT look at what constitutes a "user id"(did not bother to understand that either), did not make features' selections (apart from setting interactions=true to Tingru's code or doing the equivalent for other techniques) .  I made 17 different models based on the given categorical variables :

\n

-(4) Tingru-based (thanks Tingru!) for l1 only, l2 only, l1 only + interactions, l2 plus interactions cv at day 30

\n

-(7) sparse data with small samples, and l1 , l2 logits and LinearSVC  with or without interactions ( 2 x )  and  XGBoost in the same sparse data/  cv day 30

\n

-(3) converting categories to counts  and running Extratrees, XGboost, Gradient Boosting with scikit  / cv day 30 

\n

-(2) using days <29 to calculate ratios of goods vs bads for each category which I applied on day 30 and 31. Then use gradient boosting of scikit or XGBoost to model these / internal random split 50-50 cv 

\n

-(1) LibFM with 5 latent features and l2 for all features  / cv day 30

\n

I did all of these in a manner that I made predictions for days 30 and 31. Then I used Extra trees and XGBoost as meta-models to combine these and to get my final submission. 

", "rawMarkdown": "", "votes": null }, { "id": "63949", "postDate": "02/10/2015 18:26:57", "content": "

Hello Kazanova,
What do you think.. Did any of the methods capture the historical features ?
Perhaps (3) ? Or do you think that some manual features would add to the score ?

@Cyberduck
Online Fm was just trintgu's code with a changed the 'update' and 'predict' methods to account for the extra interactions-latent-factor-magic-term as specified in the paper by Stephen Rendle. 

", "rawMarkdown": "", "votes": null }, { "id": "63957", "postDate": "02/10/2015 19:42:07", "content": "

@Julian de Wit, thanks for your post! I also tried an online fm based on tinrtgu's code. Did you use FTRL for the FM? or did you go with a standard stochastic gradient descent?

\n

@Cyberduck, I've put up my factorization machine code on github. It's completely based of tinrtgu's code. Of course, I never was able to get it to work very well :-), so there might be 0+ bugs. I'll add some better documentation and readability later:

\n

https://github.com/comadan/FM_FTRL

\n\n

EDIT: updated the github repository with better documentation of code and an example of running the code.

", "rawMarkdown": "", "votes": null }, { "id": "63968", "postDate": "02/10/2015 21:11:10", "content": "

[quote=Julian de Wit;63949]

\n

Hello Kazanova,
What do you think.. Did any of the methods capture the historical features ?
Perhaps (3) ? Or do you think that some manual features would add to the score ?

\n

[/quote]

\n

Obviously the counts and the ratios of 1s and 0s allowed the trees to explore very deep interactions (in fact max_depth was higher than 20 ) historical-or-not, while the linear models gave good baselines (averages) of the main categories. I think deeper relationships were captured by the meta-models too. A weighted average of my models (based on leaderboard feedback) was 0.388+ but could get down to 0.3860 with the meta modelling with forests. Interestingly a  logistic regression meta-model could come close to my single weighted average but nowhere as good as the meta-model with forests (which where quite deep too). It seemed to me like the most efficient logic IMHO was like " hey, when my probabilities of linear and non-linear models do not align, look at the nonlinear models, when they are quite close, look at the linear ones " and so on.

\n

I am fairly certain that having introduced a user-id idea (conceptually at least) would have added some good new information and maybe allowed my LibFM model score better ( albeit was quite poor). I guess some food for thought...I have not spent much time in this competition (in comparison to other comps) - my computer has !  

", "rawMarkdown": "", "votes": null }, { "id": "63969", "postDate": "02/10/2015 21:12:50", "content": "

[quote=Daniel Yoo;63957]

\n

I've put up my factorization machine code on github.

\n

[/quote]

\n

+1 for not using regularization in the first epoch for the latent features

", "rawMarkdown": "", "votes": null }, { "id": "63974", "postDate": "02/10/2015 21:53:41", "content": "

[quote=Dmitry Efimov;63909]

\n

[quote=Daniel Yoo;63878]

\n

@Dmitry Efimov, and @infinitezxc, I'm curious if you could share details on how you got your FM to work. How did you deal with overfitting? I played around with a FM, but ran into some challenges with overfitting.

\n

[/quote]

\n

I am going to write the detailed description of my solution soon. Regarding FM, I used the code of 3 idiots, with learning rate 0.02, 20 epochs and 20 latent factors. I have also tried Stephen Rendle's libfm, but I got only 0.38640 on lb with it. Some hints to avoid overfitting:

\n

1. Remove low-frequent levels (I used threshold 100).

\n

2. For continuous features I transformed it to binary features using discretisation or rounding. Regarding discretisation, there is a very nice package infotheo in R with function discretize there.

\n

3. The main problem was to add likelihood features to FM without overfitting, I constructed them in different ways, for example, for current day I used the previous day only. Likelihood features gave 0.002 improvement on the public lb.

\n

[/quote]

\n

Would you mind expanding on the likelihood feature design?

", "rawMarkdown": "", "votes": null }, { "id": "64016", "postDate": "02/11/2015 10:55:21", "content": "

[quote=marbel;63974]

\n

Would you mind expanding on the likelihood feature design?

\n

[/quote]

\n

Actually, this is the first algorithm I tried in this competition. It gave 0.39400 on the public leaderboard with just 11 features with little feature engineering.

\n

Here is the algorithm:

\n

1) Choose the sequence of features, in my case I took the sequence:

\n

f = "site_id", "site_domain", "app_id", "banner_pos", "device_id_count", "device_conn_type", "C16", "device_ip_count", "device_id_site_id_count", "device_ip_site_id_count", "device_model".

\n

2) Define the parameter M - reliable size of group (in my case I defined M = 28 by running grid search for different M).

\n

3) Define the sequence of jumps: j = 11, 9, 8, 7, 4, 2.

\n

4) Start iteration by j:

\n

define likelihood inside the group: L = N1/N, where group is defined as f[1:j], N1 - number of clicks in the group, N - number of samples in the group. 

\n

define weight for the group: W = 1/(1+exp(-N+M)).

\n

After this step for each jump j I had likelihood L_j and weight W_j.

\n

5) Start iteration by reversed j:

\n

5.1) pred = global average of clicks;

\n

5.2) pred = W_j * L_j + (1-W_j)*pred.

", "rawMarkdown": "", "votes": null }, { "id": "64117", "postDate": "02/12/2015 15:49:44", "content": "

My solution(at 10th place) is as follows.

\n

Working method:

\n\n

- hour, device_ip and C17 were removed from my feature set, because they did not help.
- quadratic interactions between major features and all others.
  The features which had greater impacts on my score were 
                 device_model, app_id, site_id, site_domain and banner_pos.
- cubic interactions among major features.
- three types of count features (inspired by additions by Yannick Martel for tinrtgu code.    Thanks Martel and tinrtgu):

\n
              \n
            1. device_ip, device_id, device_ip * device_id and so on counted in all datasets.
              2. \n
            2. the same features counted in past 24 hours.
              3. \n
            3. the same features counted in reverse event order in 24 hours ahead.
              4. \n
\n

              - click rate for each feature with Laplace smoothing.

\n\n

- holdout was days 29 and 30

\n

- parameter tuning of vowpal wabbit: --l1, -l, -b and --passes
- blending a model with click features and one without them

\n

 

\n

Tried but not working methods:

\n\n

                 At first I used their codes as is, but oddly enough, the codes did not break 0.4 score barrier.

\n

                I am not sure why, but I guess my preprocessed data files were corrupted or wrongly formatted.

\n\n

                I understand the lrq algorithm in vowpal wabbit has the same philosophy

\n

                 as the factorization machine(FM)(if I am wrong, correct me!). Since lrq did not work,

\n

                I did not try 3 idiots fm codes or libFM. Surprisingly (for me), it seems FM successfully

\n

                worked in this competition. I should have tried FM!

\n", "rawMarkdown": "", "votes": null }, { "id": "64140", "postDate": "02/12/2015 20:09:11", "content": "

[quote=Julian de Wit;63888]

\n\n

[/quote]

\n

@Julian de Wit

\n

Interesting that you mentioned it. For this comp, I implemented an FM with drop out applying to the latent factors. This helps a lot in regularizing the FM and I was able to train it with a fairly high number of latent factors (48-64, took ages though, which is why I am also interested to see if it will work on a GPU, but I can't figure out how to align the sparse data.) Of course, once I did that I realize that I am treating the latent factors like hidden units in a neural net so I tried to stick another 'layer' in the back of these latent factors and apply an activation to turn it into a mini net. Unfortunately, these additions didn't really help me in this contest. 

", "rawMarkdown": "", "votes": null }, { "id": "64177", "postDate": "02/13/2015 18:57:54", "content": "

My approach to 18th place:

\n

Feature engineering:

\n

user id by combining device id / device ip / device model + intraday frequency for user id. Removed all the features that were not present in test set.

\n

Models:

\n

Lots of different logistic regressions. I couldn't make FTRL work as good as a simple AdaGrad (thanks to Abnishek's code from Criteo!). I got 0.3663 on private leaderboard with a simple AdaGrad! That was a shock for me.

\n

I tried single hidden layer perceptron with no success. And after that I set out to build a working Stochastic Gradient Boosting Machine with weights from logistic regression as one of the inputs and it could improve the score. But mainly it didn't give me as good result as I was hoping for. An ensemble of it + best AdaGrad gave me 0.3856 on the private leaderboard.

\n\n

I would say that feature engineering and ensembling could get me much higher while I was putting to much effort into building and testing different alogithms. Anyway, It was real fun and I leant a lot from it. Thanks everybody!

", "rawMarkdown": "", "votes": null }, { "id": "64196", "postDate": "02/13/2015 22:13:55", "content": "

[quote=KazAnova;63921]

\n

I did all of these in a manner that I made predictions for days 30 and 31. Then I used Extra trees and XGBoost as meta-models to combine these and to get my final submission. 

\n

[/quote]

\n

Am I understanding this correctly: you take the output click probabilities of a number of models and feed them to XGBoost to get a final probability?

\n

I'm curious how many models and what improvement from your best single model you got.

\n

Cheers!

", "rawMarkdown": "", "votes": null }, { "id": "64205", "postDate": "02/14/2015 01:03:27", "content": "

[quote=laserwolf;64196]

\n

Am I understanding this correctly: you take the output click probabilities of a number of models and feed them to XGBoost to get a final probability?

\n

[/quote]

\n

So all my models were trained with the following logic:

\n

1) Choose a classifier

\n

2) Build a model with the selected classifier using data from 20-29 days

\n

3) make predictions (in probabilities) for day 30

\n

4) save the labels that correspond to day 30

\n

5) Build a model with the selected classifier using data from 20-30 days

\n

6)  make predictions (in probabilities) for day 31

\n

7) once I have number of predictions for different classifiers, I use as new training set my predictions for day 30 and as test set the predictions of the different models for day 31

\n

8) i used XGBOOST to make this meta-model (that uses as inputs the predictions of all the models)

\n

A weak point of this approach is that the models are combined based on predictions for day 30 which might be slightly different than day 31

\n

[quote=laserwolf;64196]

\n

I'm curious how many models and what improvement from your best single model you got.

\n

[/quote]

\n

I made 17 models

\n

my best single model was 0.392

\n

The weighted average of my models was around 0.388+

\n

The XGBoost ensemble took me down to 0.385+

", "rawMarkdown": "", "votes": null }, { "id": "64218", "postDate": "02/14/2015 07:21:25", "content": "

I'm really interested in how far this really could be pushed.
For instance.. for me 29+30 was a better CV than 30 only. Perhaps that alone might improve the scores. I always hope to acquire general knowledge from a competition. Many feature engineering tricks just work once for one comp and not for the other. This ensembling approach is general knowledge. Of course it is known but I really thought it was not working in this competition.
But I'm worried of you can solve the problem of, say, converting the date-of-birth to an age..

But perhaps one of the first jobs that datascientists automate is their own job :)

\n


@mymo

\n

Of course I was wildly speculating but I was thinking the same thing.
Next to the "normal" neurons, have a number of latent factor-neurons... When you think of it. finding interactions is very hard for NN's. It's just not a natural fit. This would help them. Another idea was to put the FM on top of the network as the last layer..


", "rawMarkdown": "", "votes": null }, { "id": "64340", "postDate": "02/16/2015 14:42:27", "content": "

[quote=KazAnova;64205]

\n

I made 17 models

\n

my best single model was 0.392

\n

The weighted average of my models was around 0.388+

\n

The XGBoost ensemble took me down to 0.385+

\n

[/quote]

\n

Thanks so much KazAnova for this awesome post!

\n

I just can't seem to get XGBoost to ensemble better than a simple weighted average though. are you  binary:logistic or something else? if you could post your configs that would be awesome.

", "rawMarkdown": "", "votes": null }, { "id": "64344", "postDate": "02/16/2015 15:08:46", "content": "

SEED=10

\n

param = {}
eta=0.04
max_depth=11
subs=1.0
thread=30
trees=600
param['objective'] = 'binary:logistic'
param['bst:eta'] = eta
param['bst:max_depth'] = max_depth
param['eval_metric'] = 'auc'
#param['bst:min_child_weight']=1
param['silent'] = 1
param['seed'] = SEED
param['nthread'] = thread
param['bst:subsample'] = subs

", "rawMarkdown": "", "votes": null }, { "id": "64376", "postDate": "02/17/2015 01:41:59", "content": "

[quote=infinitezxc;63870]

\n

Thanks guys. :) This competition is really long but fun.

\n

And special thanks to tingrtu and 3 idiots for their awesome code for ftrl, fm and gbdt. Our method is mainly built on their code with moderate modifications. But maybe it is feature engineering that is the most important.

\n

We make up about 10 numerical features by counting the times of impression to one person and time intervals. How to locate a person is an interesting question though. Device_id has too many nulls and device_ip is not that accurate. We use device_id if it is not null and device_ip+device_model for others. Then we use gbdt trained on numerical features to better "bin" these features and get some edge.

\n

Our final solution is the ensemble of several ftrl and fm models. In ftrl selected interactions and gbdt features are used and in fm both the model with gbdt features and without gbdt features are used. We will post the code and detailed document soon.

\n

[/quote]

\n

@infinitezxc

\n

亲,在哪里能得到您的文档,如果发放出来的话,请通知我们一下好吗?期待..

\n

顺便问一下,亲在哪里高就?我们想邀请亲来我们公司做一下讲座.如果您日程能安排的话,请回复我一下.谢谢.

\n

我的邮箱:306388406@qq.com

", "rawMarkdown": "", "votes": null }, { "id": "64382", "postDate": "02/17/2015 04:58:00", "content": "

一般要在比赛结束后两周之内上传文档到论坛，我们也好想看啊👀， 太厉害了你们@infinitezxc！

", "rawMarkdown": "", "votes": null }, { "id": "64386", "postDate": "02/17/2015 09:53:40", "content": "

@chang yangfan @Xueer Chen 文档已经差不多整理好了，马上会根据要求放出来的。:）可以邮箱联系 infinitezxc@gmail.com

", "rawMarkdown": "", "votes": null }, { "id": "64463", "postDate": "02/18/2015 04:43:23", "content": "

The problem appears to be tough for those who are new to ML. (Unfortunately) some students in my Institute have taken up this as a graduation Project and want to stick to it. I will be grateful if some simple and working code that achieves good enough results can be posted so they can further work upon it. Thanks.

", "rawMarkdown": "", "votes": null }, { "id": "64474", "postDate": "02/18/2015 09:11:20", "content": "

[quote=ashok k harnal;64463]

\n

I will be grateful if some simple and working code that achieves good enough results can be posted so they can further work upon it.

\n

[/quote]

\n

There are already several good examples on the forums: tinrtgu's FTRL in python; a few vowpal wabbit examples; and xgboost/R in the recent thread.

", "rawMarkdown": "", "votes": null }, { "id": "64476", "postDate": "02/18/2015 09:15:29", "content": "

@chang yangfan @Xueer Chen @infinitezxc

\n

今天才知道。。原来可以打中文。。。

", "rawMarkdown": "", "votes": null }, { "id": "64480", "postDate": "02/18/2015 09:40:16", "content": "

Thanks Lewis. Vowpal wabbit examples do appear easy to follow.

", "rawMarkdown": "", "votes": null }, { "id": "64549", "postDate": "02/19/2015 04:28:05", "content": "

For those deep learners among you - how would you use neural networks in this competition?

", "rawMarkdown": "", "votes": null }, { "id": "64799", "postDate": "02/24/2015 04:37:35", "content": "

Thanks everyone for sharing. Here is my solution:

\n\n

https://github.com/owenzhang/kaggle-avazu

\n\n

It is largely based 3 idiots' winning solution for criteo. 

", "rawMarkdown": "", "votes": null }, { "id": "64855", "postDate": "02/25/2015 10:27:19", "content": "

[quote=Owen;64799]

\n

Thanks everyone for sharing. Here is my solution:

\n

https://github.com/owenzhang/kaggle-avazu

\n

It is largely based 3 idiots' winning solution for criteo. 

\n

[/quote]

\n\n

Thanks so much!

", "rawMarkdown": "", "votes": null }, { "id": "64866", "postDate": "02/25/2015 14:40:09", "content": "

Time to learn from the best :D

\n

thank you Owen and 4 idiots for sharing their solution !

", "rawMarkdown": "", "votes": null }, { "id": "65015", "postDate": "02/27/2015 09:18:38", "content": "

[quote=infinitezxc;64386]

\n

@chang yangfan @Xueer Chen 文档已经差不多整理好了，马上会根据要求放出来的。:）我还是清华计算机研一的学生...可以邮箱联系 infinitezxc@gmail.com

\n

[/quote]

\n

发送了邮件,不知道是否收到!

", "rawMarkdown": "", "votes": null }, { "id": "65223", "postDate": "03/02/2015 08:05:41", "content": "

Hi All,

\n

We have prepared our code and document here.

\n

Your comments are very welcome. :)

", "rawMarkdown": "", "votes": null }, { "id": "65282", "postDate": "03/02/2015 19:32:08", "content": "

[quote=infinitezxc;65223]

\n

Hi All,

\n

We have prepared our code and document here.

\n

Your comments are very welcome. :)

\n

[/quote]

\n

Great work!

\n

I have a question. I see you added the first order terms into FFM. In my experience, removing first order terms works slightly better. (I can't remember the exact difference.) Did you observe any improvement by adding the first order terms back? Thanks!

", "rawMarkdown": "", "votes": null }, { "id": "65312", "postDate": "03/03/2015 02:02:49", "content": "

[quote=guestwalk;65282]

\n

I have a question. I see you added the first order terms into FFM. In my experience, removing first order terms works slightly better. (I can't remember the exact difference.) Did you observe any improvement by adding the first order terms back? Thanks!

\n

[/quote]

\n

Yes. We got ~0.001 improvement on this dataset by adding first order terms at an early stage and we keep using it regarding the positive result.

\n

Thanks again for the excellent work in the two competitions, we learned a lot from your solutions. :)

", "rawMarkdown": "", "votes": null }, { "id": "65323", "postDate": "03/03/2015 11:22:28", "content": "

[quote=infinitezxc;65312]

\n

[quote=guestwalk;65282]

\n

I have a question. I see you added the first order terms into FFM. In my experience, removing first order terms works slightly better. (I can't remember the exact difference.) Did you observe any improvement by adding the first order terms back? Thanks!

\n

[/quote]

\n

Yes. We got ~0.001 improvement on this dataset by adding first order terms at an early stage and we keep using it regarding the positive result.

\n

Thanks again for the excellent work in the two competitions, we learned a lot from your solutions. :)

\n

[/quote]

\n

I see. 0.001 is a significant improvement. I may need to check this issue again. Thanks a lot for your answer!  :)

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 63846, "author_name": "jiweiliu", "author_url": "", "post_date": "02/10/2015 00:19:56", "content": "

Congrats. this is a particular hard problem. cannot wait to see the winning solutions.

", "votes": null, "replies": [] }, { "id": 63850, "author_name": "superfan123", "author_url": "", "post_date": "02/10/2015 00:33:35", "content": "

Congrats! Looking forward to learn something new!

\n

Special thanks to @Tinrtgu !

", "votes": null, "replies": [] }, { "id": 63852, "author_name": "davidthaler", "author_url": "", "post_date": "02/10/2015 00:37:48", "content": "

Congratulations to the folks on the nn2 team, guestwalk, Michael Jahrer, yolicat and mandora, to Owen, and to infinitezxc and machine learner from RandomWalker on their great finishes. I look forward to hearing about what set them apart. Also, congratulations to tingrtu on that impressive beat-the-benchmark code. I used it as part of my solution and I know lots of other people did too.

", "votes": null, "replies": [] }, { "id": 63866, "author_name": "supernova117", "author_url": "", "post_date": "02/10/2015 03:41:30", "content": "

Congratulations! 

", "votes": null, "replies": [] }, { "id": 63870, "author_name": "infinite", "author_url": "", "post_date": "02/10/2015 03:59:51", "content": "

Thanks guys. :) This competition is really long but fun.

\n

And special thanks to tingrtu and 3 idiots for their awesome code for ftrl, fm and gbdt. Our method is mainly built on their code with moderate modifications. But maybe it is feature engineering that is the most important.

\n

We make up about 10 numerical features by counting the times of impression to one person and time intervals. How to locate a person is an interesting question though. Device_id has too many nulls and device_ip is not that accurate. We use device_id if it is not null and device_ip+device_model for others. Then we use gbdt trained on numerical features to better "bin" these features and get some edge.

\n

Our final solution is the ensemble of several ftrl and fm models. In ftrl selected interactions and gbdt features are used and in fm both the model with gbdt features and without gbdt features are used. We will post the code and detailed document soon.

", "votes": null, "replies": [] }, { "id": 63871, "author_name": "chuan92", "author_url": "", "post_date": "02/10/2015 04:16:59", "content": "

Congratulations!

", "votes": null, "replies": [] }, { "id": 63872, "author_name": "efimov", "author_url": "", "post_date": "02/10/2015 04:17:16", "content": "

Congratulation to the winners!

\n

My solution is similar to RandomWalkers, except I did not use gbrt. The key points of my solution:

\n

1. A lot of feature engineering: count features, combinations of several categorical features, several types of likelihood features, time features.

\n

2. FTRL code with batches: here is the simple idea how to essentially improve FTRL algorithm: sort dataset by app_id, site_id and run Python code suggested by tingrtu. To get different algorithms, I sorted dataset using different set of categorical features.

\n

3. My final model contained 1 fm and 3 FTRL models with batches, ensembled with geometric average. The best score for fm model on public lb is 0.38358, for FTRL model - 0.38557.

", "votes": null, "replies": [] }, { "id": 63873, "author_name": "jiweiliu", "author_url": "", "post_date": "02/10/2015 04:27:12", "content": "

[quote=Dmitry Efimov;63872]

\n

2. FTRL code with batches: here is the simple idea how to essentially improve FTRL algorithm: sort dataset by app_id, site_id and run Python code suggested by tingrtu. To get different algorithms, I sorted dataset using different set of categorical features.

\n

[/quote]

\n

Thank you for sharing! Does it mean same app_id => same batch?

", "votes": null, "replies": [] }, { "id": 63875, "author_name": "efimov", "author_url": "", "post_date": "02/10/2015 04:36:52", "content": "

Yes, basically, it works in the following way: train weights on the first app_id, make predictions on this app_id, train weights on the second app_id (starting from previous app_id weights) and so on. I suppose, there are a lot of ways to improve this idea, for example, initialize weights after some batches, but I did not have time to try it.

", "votes": null, "replies": [] }, { "id": 63876, "author_name": "mlandry", "author_url": "", "post_date": "02/10/2015 04:43:45", "content": "

Congratulations winners. That was an impressive amount of separation in an otherwise tight leaderboard.

\n

Something simple that helped me was to adapt the FTRL to have a decaying learning rate per epoch. So my single best was a 0.3894276, and that was achieved with 3 epochs, where the learning rate was adjusted to 85% of its previous rate. The first time I did that, it was worth 100 spots or so. I'll post the slight update in that thread for others to try out--a simple method in the FTRL class and a call to that after the epoch loop is done. Hopefully others get some use out of it.

\n

For features, I started with a few things I thought would help classify new users, which was to measure the number of records divided by the number of hours, and the number of records divided by the number of unique site/apps. Logging and truncating that provided really nice separation, and it looked similar using it on sites/apps as well so I had two more--four in all. Aside from that, there were a bunch of carefully selected interactions--trying not to put elements of a hierarchy together, which was an idea referenced in one of the Criteo winner's post.

\n

Toward the end, I started running segmented models by site/app. I think 5 sites captured 40% of the volume, so I had a model for each. Those models helped, but not near as much as anything listed above.

", "votes": null, "replies": [] }, { "id": 63877, "author_name": "nishkgp", "author_url": "", "post_date": "02/10/2015 05:09:19", "content": "

Congratulations to the winners especially Michael Jahrer ( We have worked together in our employer's projects but were competing separately here ) !! My solution is based on a single code, ftrl (thanks to Tinrtgu, you are awesome) and a lot of feature engineering from my end (thanks to pandas). Happy that I got my first top 10 finish in such a crazily hard competition. I thought of implementing 3 idiot's solution of Criteo to make an ensemble but didnt have time for it and I really wanted to know if it is possible to enter top 10 with a single code, all Pythonic . :) 

", "votes": null, "replies": [] }, { "id": 63878, "author_name": "danielyoo", "author_url": "", "post_date": "02/10/2015 05:27:41", "content": "

Congrats to all!

\n

I ended up getting the most mileage from feature engineering time features. I tried to capture as much about the dynamics of each visitor's history of visits as I could. E.g. time since last visit, number of visits the prior hour, number of visits N days back, etc. I never checked to see which combinations time features were most useful, but in general they improved my LB score a lot. I think more clever feature engineering of time features might have gotten a better score, but ironically, I ran out of time.

\n

@Dmitry Efimov, and @infinitezxc, I'm curious if you could share details on how you got your FM to work. How did you deal with overfitting? I played around with a FM, but ran into some challenges with overfitting.

", "votes": null, "replies": [] }, { "id": 63881, "author_name": "infinite", "author_url": "", "post_date": "02/10/2015 06:10:13", "content": "

[quote=Daniel Yoo;63878]

\n

@Dmitry Efimov, and @infinitezxc, I'm curious if you could share details on how you got your FM to work. How did you deal with overfitting? I played around with a FM, but ran into some challenges with overfitting.

\n

[/quote]

\n

We got the FM to work by tuning lambda and number of epochs through local validation that takes the last two days for testing. The model tends to overfit with smaller lambda and more epochs, so we just used the best parameters for validation and got 0.3834 for our best single fm model.

", "votes": null, "replies": [] }, { "id": 63888, "author_name": "juliandewit", "author_url": "", "post_date": "02/10/2015 07:58:57", "content": "

Although I like building algorithms I realised along the way that this competetition was more about finding additional features. I used FM and FTRL although I tried a few others.

\n

Working method

\n\n

Feature engineering

\n\n

Software and tuning

\n\n


What did not help (for me).

\n\n

Results/conclusions

\n\n

Congrats to the winners and a shoutout to Trintu and the 3 idioits and Micheal Jahrer and stephen rendle for their great contributions in terms of algoritms.

Personal shoutout to Dmitry Efimov.. I felt pretty lame with my last-effort submission just before the deadline..

", "votes": null, "replies": [] }, { "id": 63890, "author_name": "danielyoo", "author_url": "", "post_date": "02/10/2015 08:00:25", "content": "

Thanks @infinitezxc!!! It seems almost too easy when you put it that way :-).

\n

Did you mostly manually tune your lambda (I'm assuming L2 regularizer) and # epochs? Did you optimize the degree of the factorization or the random initialization of the factorization?

\n

Perhaps I never found a good factorization model because I was searching over too many dimensions of hyper parameters.

\n

I'd love to hear how others also optimized over their model hyper parameters considering how long it could take to pass through the data. I found myself forgoing much hyper parameter optimization in favor of ensembling a random mix of hyper parameters.

", "votes": null, "replies": [] }, { "id": 63906, "author_name": "antoniom", "author_url": "", "post_date": "02/10/2015 09:57:20", "content": "

[quote=Julian de Wit;63888]

\n\n\n

[/quote]

\n

This is interesting, any hint about how to implement an online FM?

", "votes": null, "replies": [] }, { "id": 63907, "author_name": "infinite", "author_url": "", "post_date": "02/10/2015 10:04:12", "content": "

[quote=Daniel Yoo;63890]

\n

Did you mostly manually tune your lambda (I'm assuming L2 regularizer) and # epochs? Did you optimize the degree of the factorization or the random initialization of the factorization?

\n

[/quote]

\n

For # epochs, just give it a large number and see when loss goes up in validation. 

\n

We also tried tuning other hyper parameters, and adjusting the random initialization got a ~0.0001 improvement. BTW we also made use of the first order features in FM. 

", "votes": null, "replies": [] }, { "id": 63908, "author_name": "gertjac", "author_url": "", "post_date": "02/10/2015 10:05:57", "content": "

Congratulations to all winners, and thanks for sharing.

\n

I used the SGD code posted by Abhishek (thanks!) and did greedy selection of interaction terms (depth first) using 5% of the data. I selected interaction terms that improved the prediction for holdout day 10, to identify interactions that are 'stable in time'. As features I had raw data and univariate logtransformed counts.

", "votes": null, "replies": [] }, { "id": 63909, "author_name": "efimov", "author_url": "", "post_date": "02/10/2015 10:34:40", "content": "

[quote=Daniel Yoo;63878]

\n

@Dmitry Efimov, and @infinitezxc, I'm curious if you could share details on how you got your FM to work. How did you deal with overfitting? I played around with a FM, but ran into some challenges with overfitting.

\n

[/quote]

\n

I am going to write the detailed description of my solution soon. Regarding FM, I used the code of 3 idiots, with learning rate 0.02, 20 epochs and 20 latent factors. I have also tried Stephen Rendle's libfm, but I got only 0.38640 on lb with it. Some hints to avoid overfitting:

\n

1. Remove low-frequent levels (I used threshold 100).

\n

2. For continuous features I transformed it to binary features using discretisation or rounding. Regarding discretisation, there is a very nice package infotheo in R with function discretize there.

\n

3. The main problem was to add likelihood features to FM without overfitting, I constructed them in different ways, for example, for current day I used the previous day only. Likelihood features gave 0.002 improvement on the public lb.

", "votes": null, "replies": [] }, { "id": 63913, "author_name": "tobycheese", "author_url": "", "post_date": "02/10/2015 11:23:44", "content": "

[quote=Nishant Kumar;63877]

\n

My solution is based on a single code, ftrl (thanks to Tinrtgu, you are awesome) and a lot of feature engineering from my end (thanks to pandas).

\n

[/quote]

\n\n

Congratulations! And please share more info on what exactly you did :-)

", "votes": null, "replies": [] }, { "id": 63915, "author_name": "mikeskim", "author_url": "", "post_date": "02/10/2015 12:10:44", "content": "

Can someone post the libFM pipeline code (from data transformation into sh run)? I tried to use it as a hastily modified 3 idiots code, and could not get it to perform well. 

", "votes": null, "replies": [] }, { "id": 63921, "author_name": "kazanova", "author_url": "", "post_date": "02/10/2015 14:09:33", "content": "

@Julian de Wit great explanation mate !

\n

I will talk for my part (which would probably have finished top 10)

\n

Based on some discussions I had recently with colleagues, academics and other data science enthusiasts , I came into some sort of collision by claiming that on many different problems at present (and with the advances of machine learning) you can do really well without even inspecting  the data - just by 

\n

1) knowing the tools

\n

2) cross- validating properly when you make features' selections, hyper parameter optimization etc

\n

3) ensembling many different models to capture different information

\n

In Light of this I did exactly what I tried in Tradeshift plus a couple of more models since we had more time.  I only looked at the header of the data and once I understood the set is divided by days , I did a simple frequency count in that column. I also found  the row that constitutes the beginning of day 30 (row 36,210,029) and looked at that. That is the end of my data inspection.

\n

I did NOT divide my set in device or not, did NOT look at what constitutes a "user id"(did not bother to understand that either), did not make features' selections (apart from setting interactions=true to Tingru's code or doing the equivalent for other techniques) .  I made 17 different models based on the given categorical variables :

\n

-(4) Tingru-based (thanks Tingru!) for l1 only, l2 only, l1 only + interactions, l2 plus interactions cv at day 30

\n

-(7) sparse data with small samples, and l1 , l2 logits and LinearSVC  with or without interactions ( 2 x )  and  XGBoost in the same sparse data/  cv day 30

\n

-(3) converting categories to counts  and running Extratrees, XGboost, Gradient Boosting with scikit  / cv day 30 

\n

-(2) using days <29 to calculate ratios of goods vs bads for each category which I applied on day 30 and 31. Then use gradient boosting of scikit or XGBoost to model these / internal random split 50-50 cv 

\n

-(1) LibFM with 5 latent features and l2 for all features  / cv day 30

\n

I did all of these in a manner that I made predictions for days 30 and 31. Then I used Extra trees and XGBoost as meta-models to combine these and to get my final submission. 

", "votes": null, "replies": [] }, { "id": 63949, "author_name": "juliandewit", "author_url": "", "post_date": "02/10/2015 18:26:57", "content": "

Hello Kazanova,
What do you think.. Did any of the methods capture the historical features ?
Perhaps (3) ? Or do you think that some manual features would add to the score ?

@Cyberduck
Online Fm was just trintgu's code with a changed the 'update' and 'predict' methods to account for the extra interactions-latent-factor-magic-term as specified in the paper by Stephen Rendle. 

", "votes": null, "replies": [] }, { "id": 63957, "author_name": "danielyoo", "author_url": "", "post_date": "02/10/2015 19:42:07", "content": "

@Julian de Wit, thanks for your post! I also tried an online fm based on tinrtgu's code. Did you use FTRL for the FM? or did you go with a standard stochastic gradient descent?

\n

@Cyberduck, I've put up my factorization machine code on github. It's completely based of tinrtgu's code. Of course, I never was able to get it to work very well :-), so there might be 0+ bugs. I'll add some better documentation and readability later:

\n

https://github.com/comadan/FM_FTRL

\n\n

EDIT: updated the github repository with better documentation of code and an example of running the code.

", "votes": null, "replies": [] }, { "id": 63968, "author_name": "kazanova", "author_url": "", "post_date": "02/10/2015 21:11:10", "content": "

[quote=Julian de Wit;63949]

\n

Hello Kazanova,
What do you think.. Did any of the methods capture the historical features ?
Perhaps (3) ? Or do you think that some manual features would add to the score ?

\n

[/quote]

\n

Obviously the counts and the ratios of 1s and 0s allowed the trees to explore very deep interactions (in fact max_depth was higher than 20 ) historical-or-not, while the linear models gave good baselines (averages) of the main categories. I think deeper relationships were captured by the meta-models too. A weighted average of my models (based on leaderboard feedback) was 0.388+ but could get down to 0.3860 with the meta modelling with forests. Interestingly a  logistic regression meta-model could come close to my single weighted average but nowhere as good as the meta-model with forests (which where quite deep too). It seemed to me like the most efficient logic IMHO was like " hey, when my probabilities of linear and non-linear models do not align, look at the nonlinear models, when they are quite close, look at the linear ones " and so on.

\n

I am fairly certain that having introduced a user-id idea (conceptually at least) would have added some good new information and maybe allowed my LibFM model score better ( albeit was quite poor). I guess some food for thought...I have not spent much time in this competition (in comparison to other comps) - my computer has !  

", "votes": null, "replies": [] }, { "id": 63969, "author_name": "kazanova", "author_url": "", "post_date": "02/10/2015 21:12:50", "content": "

[quote=Daniel Yoo;63957]

\n

I've put up my factorization machine code on github.

\n

[/quote]

\n

+1 for not using regularization in the first epoch for the latent features

", "votes": null, "replies": [] }, { "id": 63974, "author_name": "marbel", "author_url": "", "post_date": "02/10/2015 21:53:41", "content": "

[quote=Dmitry Efimov;63909]

\n

[quote=Daniel Yoo;63878]

\n

@Dmitry Efimov, and @infinitezxc, I'm curious if you could share details on how you got your FM to work. How did you deal with overfitting? I played around with a FM, but ran into some challenges with overfitting.

\n

[/quote]

\n

I am going to write the detailed description of my solution soon. Regarding FM, I used the code of 3 idiots, with learning rate 0.02, 20 epochs and 20 latent factors. I have also tried Stephen Rendle's libfm, but I got only 0.38640 on lb with it. Some hints to avoid overfitting:

\n

1. Remove low-frequent levels (I used threshold 100).

\n

2. For continuous features I transformed it to binary features using discretisation or rounding. Regarding discretisation, there is a very nice package infotheo in R with function discretize there.

\n

3. The main problem was to add likelihood features to FM without overfitting, I constructed them in different ways, for example, for current day I used the previous day only. Likelihood features gave 0.002 improvement on the public lb.

\n

[/quote]

\n

Would you mind expanding on the likelihood feature design?

", "votes": null, "replies": [] }, { "id": 64016, "author_name": "efimov", "author_url": "", "post_date": "02/11/2015 10:55:21", "content": "

[quote=marbel;63974]

\n

Would you mind expanding on the likelihood feature design?

\n

[/quote]

\n

Actually, this is the first algorithm I tried in this competition. It gave 0.39400 on the public leaderboard with just 11 features with little feature engineering.

\n

Here is the algorithm:

\n

1) Choose the sequence of features, in my case I took the sequence:

\n

f = "site_id", "site_domain", "app_id", "banner_pos", "device_id_count", "device_conn_type", "C16", "device_ip_count", "device_id_site_id_count", "device_ip_site_id_count", "device_model".

\n

2) Define the parameter M - reliable size of group (in my case I defined M = 28 by running grid search for different M).

\n

3) Define the sequence of jumps: j = 11, 9, 8, 7, 4, 2.

\n

4) Start iteration by j:

\n

define likelihood inside the group: L = N1/N, where group is defined as f[1:j], N1 - number of clicks in the group, N - number of samples in the group. 

\n

define weight for the group: W = 1/(1+exp(-N+M)).

\n

After this step for each jump j I had likelihood L_j and weight W_j.

\n

5) Start iteration by reversed j:

\n

5.1) pred = global average of clicks;

\n

5.2) pred = W_j * L_j + (1-W_j)*pred.

", "votes": null, "replies": [] }, { "id": 64117, "author_name": "cydonia", "author_url": "", "post_date": "02/12/2015 15:49:44", "content": "

My solution(at 10th place) is as follows.

\n

Working method:

\n\n

- hour, device_ip and C17 were removed from my feature set, because they did not help.
- quadratic interactions between major features and all others.
  The features which had greater impacts on my score were 
                 device_model, app_id, site_id, site_domain and banner_pos.
- cubic interactions among major features.
- three types of count features (inspired by additions by Yannick Martel for tinrtgu code.    Thanks Martel and tinrtgu):

\n
              \n
            1. device_ip, device_id, device_ip * device_id and so on counted in all datasets.
              2. \n
            2. the same features counted in past 24 hours.
              3. \n
            3. the same features counted in reverse event order in 24 hours ahead.
              4. \n
\n

              - click rate for each feature with Laplace smoothing.

\n\n

- holdout was days 29 and 30

\n

- parameter tuning of vowpal wabbit: --l1, -l, -b and --passes
- blending a model with click features and one without them

\n

 

\n

Tried but not working methods:

\n\n

                 At first I used their codes as is, but oddly enough, the codes did not break 0.4 score barrier.

\n

                I am not sure why, but I guess my preprocessed data files were corrupted or wrongly formatted.

\n\n

                I understand the lrq algorithm in vowpal wabbit has the same philosophy

\n

                 as the factorization machine(FM)(if I am wrong, correct me!). Since lrq did not work,

\n

                I did not try 3 idiots fm codes or libFM. Surprisingly (for me), it seems FM successfully

\n

                worked in this competition. I should have tried FM!

\n", "votes": null, "replies": [] }, { "id": 64140, "author_name": "mamymo", "author_url": "", "post_date": "02/12/2015 20:09:11", "content": "

[quote=Julian de Wit;63888]

\n\n

[/quote]

\n

@Julian de Wit

\n

Interesting that you mentioned it. For this comp, I implemented an FM with drop out applying to the latent factors. This helps a lot in regularizing the FM and I was able to train it with a fairly high number of latent factors (48-64, took ages though, which is why I am also interested to see if it will work on a GPU, but I can't figure out how to align the sparse data.) Of course, once I did that I realize that I am treating the latent factors like hidden units in a neural net so I tried to stick another 'layer' in the back of these latent factors and apply an activation to turn it into a mini net. Unfortunately, these additions didn't really help me in this contest. 

", "votes": null, "replies": [] }, { "id": 64177, "author_name": "ivanlobov", "author_url": "", "post_date": "02/13/2015 18:57:54", "content": "

My approach to 18th place:

\n

Feature engineering:

\n

user id by combining device id / device ip / device model + intraday frequency for user id. Removed all the features that were not present in test set.

\n

Models:

\n

Lots of different logistic regressions. I couldn't make FTRL work as good as a simple AdaGrad (thanks to Abnishek's code from Criteo!). I got 0.3663 on private leaderboard with a simple AdaGrad! That was a shock for me.

\n

I tried single hidden layer perceptron with no success. And after that I set out to build a working Stochastic Gradient Boosting Machine with weights from logistic regression as one of the inputs and it could improve the score. But mainly it didn't give me as good result as I was hoping for. An ensemble of it + best AdaGrad gave me 0.3856 on the private leaderboard.

\n\n

I would say that feature engineering and ensembling could get me much higher while I was putting to much effort into building and testing different alogithms. Anyway, It was real fun and I leant a lot from it. Thanks everybody!

", "votes": null, "replies": [] }, { "id": 64196, "author_name": "vvarkey", "author_url": "", "post_date": "02/13/2015 22:13:55", "content": "

[quote=KazAnova;63921]

\n

I did all of these in a manner that I made predictions for days 30 and 31. Then I used Extra trees and XGBoost as meta-models to combine these and to get my final submission. 

\n

[/quote]

\n

Am I understanding this correctly: you take the output click probabilities of a number of models and feed them to XGBoost to get a final probability?

\n

I'm curious how many models and what improvement from your best single model you got.

\n

Cheers!

", "votes": null, "replies": [] }, { "id": 64205, "author_name": "kazanova", "author_url": "", "post_date": "02/14/2015 01:03:27", "content": "

[quote=laserwolf;64196]

\n

Am I understanding this correctly: you take the output click probabilities of a number of models and feed them to XGBoost to get a final probability?

\n

[/quote]

\n

So all my models were trained with the following logic:

\n

1) Choose a classifier

\n

2) Build a model with the selected classifier using data from 20-29 days

\n

3) make predictions (in probabilities) for day 30

\n

4) save the labels that correspond to day 30

\n

5) Build a model with the selected classifier using data from 20-30 days

\n

6)  make predictions (in probabilities) for day 31

\n

7) once I have number of predictions for different classifiers, I use as new training set my predictions for day 30 and as test set the predictions of the different models for day 31

\n

8) i used XGBOOST to make this meta-model (that uses as inputs the predictions of all the models)

\n

A weak point of this approach is that the models are combined based on predictions for day 30 which might be slightly different than day 31

\n

[quote=laserwolf;64196]

\n

I'm curious how many models and what improvement from your best single model you got.

\n

[/quote]

\n

I made 17 models

\n

my best single model was 0.392

\n

The weighted average of my models was around 0.388+

\n

The XGBoost ensemble took me down to 0.385+

", "votes": null, "replies": [] }, { "id": 64218, "author_name": "juliandewit", "author_url": "", "post_date": "02/14/2015 07:21:25", "content": "

I'm really interested in how far this really could be pushed.
For instance.. for me 29+30 was a better CV than 30 only. Perhaps that alone might improve the scores. I always hope to acquire general knowledge from a competition. Many feature engineering tricks just work once for one comp and not for the other. This ensembling approach is general knowledge. Of course it is known but I really thought it was not working in this competition.
But I'm worried of you can solve the problem of, say, converting the date-of-birth to an age..

But perhaps one of the first jobs that datascientists automate is their own job :)

\n


@mymo

\n

Of course I was wildly speculating but I was thinking the same thing.
Next to the "normal" neurons, have a number of latent factor-neurons... When you think of it. finding interactions is very hard for NN's. It's just not a natural fit. This would help them. Another idea was to put the FM on top of the network as the last layer..


", "votes": null, "replies": [] }, { "id": 64340, "author_name": "vvarkey", "author_url": "", "post_date": "02/16/2015 14:42:27", "content": "

[quote=KazAnova;64205]

\n

I made 17 models

\n

my best single model was 0.392

\n

The weighted average of my models was around 0.388+

\n

The XGBoost ensemble took me down to 0.385+

\n

[/quote]

\n

Thanks so much KazAnova for this awesome post!

\n

I just can't seem to get XGBoost to ensemble better than a simple weighted average though. are you  binary:logistic or something else? if you could post your configs that would be awesome.

", "votes": null, "replies": [] }, { "id": 64344, "author_name": "kazanova", "author_url": "", "post_date": "02/16/2015 15:08:46", "content": "

SEED=10

\n

param = {}
eta=0.04
max_depth=11
subs=1.0
thread=30
trees=600
param['objective'] = 'binary:logistic'
param['bst:eta'] = eta
param['bst:max_depth'] = max_depth
param['eval_metric'] = 'auc'
#param['bst:min_child_weight']=1
param['silent'] = 1
param['seed'] = SEED
param['nthread'] = thread
param['bst:subsample'] = subs

", "votes": null, "replies": [] }, { "id": 64376, "author_name": "changyangfan", "author_url": "", "post_date": "02/17/2015 01:41:59", "content": "

[quote=infinitezxc;63870]

\n

Thanks guys. :) This competition is really long but fun.

\n

And special thanks to tingrtu and 3 idiots for their awesome code for ftrl, fm and gbdt. Our method is mainly built on their code with moderate modifications. But maybe it is feature engineering that is the most important.

\n

We make up about 10 numerical features by counting the times of impression to one person and time intervals. How to locate a person is an interesting question though. Device_id has too many nulls and device_ip is not that accurate. We use device_id if it is not null and device_ip+device_model for others. Then we use gbdt trained on numerical features to better "bin" these features and get some edge.

\n

Our final solution is the ensemble of several ftrl and fm models. In ftrl selected interactions and gbdt features are used and in fm both the model with gbdt features and without gbdt features are used. We will post the code and detailed document soon.

\n

[/quote]

\n

@infinitezxc

\n

亲,在哪里能得到您的文档,如果发放出来的话,请通知我们一下好吗?期待..

\n

顺便问一下,亲在哪里高就?我们想邀请亲来我们公司做一下讲座.如果您日程能安排的话,请回复我一下.谢谢.

\n

我的邮箱:306388406@qq.com

", "votes": null, "replies": [] }, { "id": 64382, "author_name": "xueerchen", "author_url": "", "post_date": "02/17/2015 04:58:00", "content": "

一般要在比赛结束后两周之内上传文档到论坛，我们也好想看啊👀， 太厉害了你们@infinitezxc！

", "votes": null, "replies": [] }, { "id": 64386, "author_name": "infinite", "author_url": "", "post_date": "02/17/2015 09:53:40", "content": "

@chang yangfan @Xueer Chen 文档已经差不多整理好了，马上会根据要求放出来的。:）可以邮箱联系 infinitezxc@gmail.com

", "votes": null, "replies": [] }, { "id": 64463, "author_name": "harnalashok", "author_url": "", "post_date": "02/18/2015 04:43:23", "content": "

The problem appears to be tough for those who are new to ML. (Unfortunately) some students in my Institute have taken up this as a graduation Project and want to stick to it. I will be grateful if some simple and working code that achieves good enough results can be posted so they can further work upon it. Thanks.

", "votes": null, "replies": [] }, { "id": 64474, "author_name": "cartographic", "author_url": "", "post_date": "02/18/2015 09:11:20", "content": "

[quote=ashok k harnal;64463]

\n

I will be grateful if some simple and working code that achieves good enough results can be posted so they can further work upon it.

\n

[/quote]

\n

There are already several good examples on the forums: tinrtgu's FTRL in python; a few vowpal wabbit examples; and xgboost/R in the recent thread.

", "votes": null, "replies": [] }, { "id": 64476, "author_name": "gangbeng", "author_url": "", "post_date": "02/18/2015 09:15:29", "content": "

@chang yangfan @Xueer Chen @infinitezxc

\n

今天才知道。。原来可以打中文。。。

", "votes": null, "replies": [] }, { "id": 64480, "author_name": "harnalashok", "author_url": "", "post_date": "02/18/2015 09:40:16", "content": "

Thanks Lewis. Vowpal wabbit examples do appear easy to follow.

", "votes": null, "replies": [] }, { "id": 64549, "author_name": "orirafael", "author_url": "", "post_date": "02/19/2015 04:28:05", "content": "

For those deep learners among you - how would you use neural networks in this competition?

", "votes": null, "replies": [] }, { "id": 64799, "author_name": "owenzhang1", "author_url": "", "post_date": "02/24/2015 04:37:35", "content": "

Thanks everyone for sharing. Here is my solution:

\n\n

https://github.com/owenzhang/kaggle-avazu

\n\n

It is largely based 3 idiots' winning solution for criteo. 

", "votes": null, "replies": [] }, { "id": 64855, "author_name": "yejiming", "author_url": "", "post_date": "02/25/2015 10:27:19", "content": "

[quote=Owen;64799]

\n

Thanks everyone for sharing. Here is my solution:

\n

https://github.com/owenzhang/kaggle-avazu

\n

It is largely based 3 idiots' winning solution for criteo. 

\n

[/quote]

\n\n

Thanks so much!

", "votes": null, "replies": [] }, { "id": 64866, "author_name": "jiweiliu", "author_url": "", "post_date": "02/25/2015 14:40:09", "content": "

Time to learn from the best :D

\n

thank you Owen and 4 idiots for sharing their solution !

", "votes": null, "replies": [] }, { "id": 65015, "author_name": "changyangfan", "author_url": "", "post_date": "02/27/2015 09:18:38", "content": "

[quote=infinitezxc;64386]

\n

@chang yangfan @Xueer Chen 文档已经差不多整理好了，马上会根据要求放出来的。:）我还是清华计算机研一的学生...可以邮箱联系 infinitezxc@gmail.com

\n

[/quote]

\n

发送了邮件,不知道是否收到!

", "votes": null, "replies": [] }, { "id": 65223, "author_name": "infinite", "author_url": "", "post_date": "03/02/2015 08:05:41", "content": "

Hi All,

\n

We have prepared our code and document here.

\n

Your comments are very welcome. :)

", "votes": null, "replies": [] }, { "id": 65282, "author_name": "guestwalk", "author_url": "", "post_date": "03/02/2015 19:32:08", "content": "

[quote=infinitezxc;65223]

\n

Hi All,

\n

We have prepared our code and document here.

\n

Your comments are very welcome. :)

\n

[/quote]

\n

Great work!

\n

I have a question. I see you added the first order terms into FFM. In my experience, removing first order terms works slightly better. (I can't remember the exact difference.) Did you observe any improvement by adding the first order terms back? Thanks!

", "votes": null, "replies": [] }, { "id": 65312, "author_name": "infinite", "author_url": "", "post_date": "03/03/2015 02:02:49", "content": "

[quote=guestwalk;65282]

\n

I have a question. I see you added the first order terms into FFM. In my experience, removing first order terms works slightly better. (I can't remember the exact difference.) Did you observe any improvement by adding the first order terms back? Thanks!

\n

[/quote]

\n

Yes. We got ~0.001 improvement on this dataset by adding first order terms at an early stage and we keep using it regarding the positive result.

\n

Thanks again for the excellent work in the two competitions, we learned a lot from your solutions. :)

", "votes": null, "replies": [] }, { "id": 65323, "author_name": "guestwalk", "author_url": "", "post_date": "03/03/2015 11:22:28", "content": "

[quote=infinitezxc;65312]

\n

[quote=guestwalk;65282]

\n

I have a question. I see you added the first order terms into FFM. In my experience, removing first order terms works slightly better. (I can't remember the exact difference.) Did you observe any improvement by adding the first order terms back? Thanks!

\n

[/quote]

\n

Yes. We got ~0.001 improvement on this dataset by adding first order terms at an early stage and we keep using it regarding the positive result.

\n

Thanks again for the excellent work in the two competitions, we learned a lot from your solutions. :)

\n

[/quote]

\n

I see. 0.001 is a significant improvement. I may need to check this issue again. Thanks a lot for your answer!  :)

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "63845": "", "63846": "", "63850": "", "63852": "", "63866": "", "63870": "", "63871": "", "63872": "", "63873": "", "63875": "", "63876": "", "63877": "", "63878": "", "63881": "", "63888": "", "63890": "", "63906": "", "63907": "", "63908": "", "63909": "", "63913": "", "63915": "", "63921": "", "63949": "", "63957": "", "63968": "", "63969": "", "63974": "", "64016": "", "64117": "", "64140": "", "64177": "", "64196": "", "64205": "", "64218": "", "64340": "", "64344": "", "64376": "", "64382": "", "64386": "", "64463": "", "64474": "", "64476": "", "64480": "", "64549": "", "64799": "", "64855": "", "64866": "", "65015": "", "65223": "", "65282": "", "65312": "", "65323": "" }, "source": "meta" }