{ "id": 11315, "title": "not asking for your secret sauce recipe but....", "url": "/competitions/avazu-ctr-prediction/discussion/11315", "author_name": "", "post_date": "2014-12-22T16:44:38.117Z", "votes": 4, "comment_count": 13, "views": 5998, "content": "

Hi there,

\n

I wondered if anyone tried manufacturing synthetic features using a decision tree-like algorithm.  The winner solution at the Criteo competition used that technique, and there is a known paper from facebook on the topic (http://quinonero.net/Publications/predicting-clicks-facebook.pdf).

\n

Well, the results I got following this recipe were absolutely disappointing.  It might have well been that I implemented this strategy poorly, but I did try to tweak it a few times and no results to show for.

\n

Has anyone else ventured into this road? anything you can share?

\n

Thanks!

\n

A

", "messages": [ { "id": "61675", "postDate": "12/22/2014 16:44:38", "content": "

Hi there,

\n

I wondered if anyone tried manufacturing synthetic features using a decision tree-like algorithm.  The winner solution at the Criteo competition used that technique, and there is a known paper from facebook on the topic (http://quinonero.net/Publications/predicting-clicks-facebook.pdf).

\n

Well, the results I got following this recipe were absolutely disappointing.  It might have well been that I implemented this strategy poorly, but I did try to tweak it a few times and no results to show for.

\n

Has anyone else ventured into this road? anything you can share?

\n

Thanks!

\n

A

", "rawMarkdown": "", "votes": null }, { "id": "61990", "postDate": "12/29/2014 04:30:27", "content": "

What did you use to create and pull the tree features? I also wanted to try this but using python I didn't find any .apply function for the GradientBoostingClassifier like there is for RandomForest. 

", "rawMarkdown": "", "votes": null }, { "id": "61999", "postDate": "12/29/2014 09:19:39", "content": "

[quote=Florian Muellerklein;61990]

\n

What did you use to create and pull the tree features? I also wanted to try this but using python I didn't find any .apply function for the GradientBoostingClassifier like there is for RandomForest. 

\n

[/quote]

\n

You can use sklearn (see attached file) or xgboost for instance

", "rawMarkdown": "", "votes": null }, { "id": "62004", "postDate": "12/29/2014 12:05:51", "content": "

Wow! You are my hero, I wish that I could buy you a beer. 

\n

I kept trying indices.append(tree.apply(x.astype(np.float32)) but that wouldn't work. As a python noob, I can't figure out why your method worked and mine didn't but I'm really grateful for your solution. Thanks again. 

", "rawMarkdown": "", "votes": null }, { "id": "62006", "postDate": "12/29/2014 12:21:31", "content": "

Hi there, Faron, I second Florian's comments.  Thanks so much for providing the code.

\n

@Florian: I tried sklearn.ensemble.RandomTreesEmbedding, method fit_transform() but it appears that the "random" in the library name is there for one reason.  Conclusion: the tree indexes appear to be assigned totally at random, and the log loss does not improve at all after adding the tree indexes.

\n

I thought about playing a bit with the code of sklearn.ensemble.GradientBoostingRegressor and change it to return the tree indexes, but no luck so far.

\n

@ Faron or anyone else, feel free to comment, especially on changing the code of GradientBoostingRegressor to return the tree indexes.

\n

Thanks a lot!

", "rawMarkdown": "", "votes": null }, { "id": "62009", "postDate": "12/29/2014 13:44:47", "content": "

[quote=Faron;61999]

\n

[quote=Florian Muellerklein;61990]

\n

What did you use to create and pull the tree features? I also wanted to try this but using python I didn't find any .apply function for the GradientBoostingClassifier like there is for RandomForest. 

\n

[/quote]

\n

You can use sklearn (see attached file) or xgboost for instance

\n

[/quote]

\n

What about xgboost? 

\n

Does the python interface also can access the trees?

", "rawMarkdown": "", "votes": null }, { "id": "62010", "postDate": "12/29/2014 14:14:45", "content": "

Isn't this technique particularly useful for binning numerical features to categoricals ?

\n

Using trees to "bin" values is better than ie. cutting the space up in even parts.

\n

Perhaps that's why it's not so effective in this challenge..

", "rawMarkdown": "", "votes": null }, { "id": "62012", "postDate": "12/29/2014 14:51:50", "content": "

[quote=clustifier;62009]

\n

What about xgboost? 

\n

Does the python interface also can access the trees?

\n

[/quote]

\n

see: https://github.com/tqchen/xgboost/blob/unity/demo/guide-python/predict_leaf_indices.py

", "rawMarkdown": "", "votes": null }, { "id": "62029", "postDate": "12/29/2014 19:13:58", "content": "

[quote=Faron;62012]

\n

[quote=clustifier;62009]

\n

What about xgboost? 

\n

Does the python interface also can access the trees?

\n

[/quote]

\n

see: https://github.com/tqchen/xgboost/blob/unity/demo/guide-python/predict_leaf_indices.py

\n

[/quote]

\n

Thanks @Faron!

\n

Look like it isn't supported any more (or supported differently). I'm getting that error:

\n

TypeError: predict() got an unexpected keyword argument 'pred_leaf'

\n

This is how the predict function currently defined:

\n

def predict(self, data, output_margin=False, ntree_limit=0)

", "rawMarkdown": "", "votes": null }, { "id": "62036", "postDate": "12/29/2014 19:49:09", "content": "

[quote=clustifier;62029]

\n

[quote=Faron;62012]

\n

[quote=clustifier;62009]

\n

What about xgboost? 

\n

Does the python interface also can access the trees?

\n

[/quote]

\n

see: https://github.com/tqchen/xgboost/blob/unity/demo/guide-python/predict_leaf_indices.py

\n

[/quote]

\n

Thanks @Faron!

\n

Look like it isn't supported any more (or supported differently). I'm getting that error:

\n

TypeError: predict() got an unexpected keyword argument 'pred_leaf'

\n

This is how the predict function currently defined:

\n

def predict(self, data, output_margin=False, ntree_limit=0)

\n

[/quote]

\n

this feature is only present in the "unity" branch (not merged into "master" yet).

", "rawMarkdown": "", "votes": null }, { "id": "62058", "postDate": "12/29/2014 22:00:52", "content": "", "rawMarkdown": "", "votes": null }, { "id": "62087", "postDate": "12/30/2014 05:38:38", "content": "

[quote=Faron;62036]

\n

this feature is only present in the "unity" branch (not merged into "master" yet).

\n

[/quote]

\n\n

Hi Faron - would you know if we can install the unity branch then? I was trying to install, but "make" spits out an error that "rabit.h" is not found... Is the branch still in development that is not ready for users? 

", "rawMarkdown": "", "votes": null }, { "id": "62156", "postDate": "12/31/2014 22:19:07", "content": "

[quote=AS;61675]

\n

Hi there,

\n

I wondered if anyone tried manufacturing synthetic features using a decision tree-like algorithm.  The winner solution at the Criteo competition used that technique, and there is a known paper from facebook on the topic (http://quinonero.net/Publications/predicting-clicks-facebook.pdf).

\n

Well, the results I got following this recipe were absolutely disappointing.  It might have well been that I implemented this strategy poorly, but I did try to tweak it a few times and no results to show for.

\n

Has anyone else ventured into this road? anything you can share?

\n

Thanks!

\n

A

\n

[/quote]

\n

It is funny how this was "rediscovered". It dates back to at least 1998, albeit using CART instead of an ensemble of trees. http://media.salford-systems.com/pdf/the-hybrid-cart-logit-model-in-classification-and-data%20mining-1998.pdf

", "rawMarkdown": "", "votes": null }, { "id": "63303", "postDate": "01/29/2015 13:10:42", "content": "

[quote=DerekZH;62087]

\n

[quote=Faron;62036]

\n

this feature is only present in the "unity" branch (not merged into "master" yet).

\n

[/quote]

\n

Hi Faron - would you know if we can install the unity branch then? I was trying to install, but "make" spits out an error that "rabit.h" is not found... Is the branch still in development that is not ready for users? 

\n

[/quote]

\n

It's in the master branch now.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 61990, "author_name": "florianm", "author_url": "", "post_date": "12/29/2014 04:30:27", "content": "

What did you use to create and pull the tree features? I also wanted to try this but using python I didn't find any .apply function for the GradientBoostingClassifier like there is for RandomForest. 

", "votes": null, "replies": [] }, { "id": 61999, "author_name": "mmueller", "author_url": "", "post_date": "12/29/2014 09:19:39", "content": "

[quote=Florian Muellerklein;61990]

\n

What did you use to create and pull the tree features? I also wanted to try this but using python I didn't find any .apply function for the GradientBoostingClassifier like there is for RandomForest. 

\n

[/quote]

\n

You can use sklearn (see attached file) or xgboost for instance

", "votes": null, "replies": [] }, { "id": 62004, "author_name": "florianm", "author_url": "", "post_date": "12/29/2014 12:05:51", "content": "

Wow! You are my hero, I wish that I could buy you a beer. 

\n

I kept trying indices.append(tree.apply(x.astype(np.float32)) but that wouldn't work. As a python noob, I can't figure out why your method worked and mine didn't but I'm really grateful for your solution. Thanks again. 

", "votes": null, "replies": [] }, { "id": 62006, "author_name": "asimkievich", "author_url": "", "post_date": "12/29/2014 12:21:31", "content": "

Hi there, Faron, I second Florian's comments.  Thanks so much for providing the code.

\n

@Florian: I tried sklearn.ensemble.RandomTreesEmbedding, method fit_transform() but it appears that the "random" in the library name is there for one reason.  Conclusion: the tree indexes appear to be assigned totally at random, and the log loss does not improve at all after adding the tree indexes.

\n

I thought about playing a bit with the code of sklearn.ensemble.GradientBoostingRegressor and change it to return the tree indexes, but no luck so far.

\n

@ Faron or anyone else, feel free to comment, especially on changing the code of GradientBoostingRegressor to return the tree indexes.

\n

Thanks a lot!

", "votes": null, "replies": [] }, { "id": 62009, "author_name": "clustifier", "author_url": "", "post_date": "12/29/2014 13:44:47", "content": "

[quote=Faron;61999]

\n

[quote=Florian Muellerklein;61990]

\n

What did you use to create and pull the tree features? I also wanted to try this but using python I didn't find any .apply function for the GradientBoostingClassifier like there is for RandomForest. 

\n

[/quote]

\n

You can use sklearn (see attached file) or xgboost for instance

\n

[/quote]

\n

What about xgboost? 

\n

Does the python interface also can access the trees?

", "votes": null, "replies": [] }, { "id": 62010, "author_name": "juliandewit", "author_url": "", "post_date": "12/29/2014 14:14:45", "content": "

Isn't this technique particularly useful for binning numerical features to categoricals ?

\n

Using trees to "bin" values is better than ie. cutting the space up in even parts.

\n

Perhaps that's why it's not so effective in this challenge..

", "votes": null, "replies": [] }, { "id": 62012, "author_name": "mmueller", "author_url": "", "post_date": "12/29/2014 14:51:50", "content": "

[quote=clustifier;62009]

\n

What about xgboost? 

\n

Does the python interface also can access the trees?

\n

[/quote]

\n

see: https://github.com/tqchen/xgboost/blob/unity/demo/guide-python/predict_leaf_indices.py

", "votes": null, "replies": [] }, { "id": 62029, "author_name": "clustifier", "author_url": "", "post_date": "12/29/2014 19:13:58", "content": "

[quote=Faron;62012]

\n

[quote=clustifier;62009]

\n

What about xgboost? 

\n

Does the python interface also can access the trees?

\n

[/quote]

\n

see: https://github.com/tqchen/xgboost/blob/unity/demo/guide-python/predict_leaf_indices.py

\n

[/quote]

\n

Thanks @Faron!

\n

Look like it isn't supported any more (or supported differently). I'm getting that error:

\n

TypeError: predict() got an unexpected keyword argument 'pred_leaf'

\n

This is how the predict function currently defined:

\n

def predict(self, data, output_margin=False, ntree_limit=0)

", "votes": null, "replies": [] }, { "id": 62036, "author_name": "mmueller", "author_url": "", "post_date": "12/29/2014 19:49:09", "content": "

[quote=clustifier;62029]

\n

[quote=Faron;62012]

\n

[quote=clustifier;62009]

\n

What about xgboost? 

\n

Does the python interface also can access the trees?

\n

[/quote]

\n

see: https://github.com/tqchen/xgboost/blob/unity/demo/guide-python/predict_leaf_indices.py

\n

[/quote]

\n

Thanks @Faron!

\n

Look like it isn't supported any more (or supported differently). I'm getting that error:

\n

TypeError: predict() got an unexpected keyword argument 'pred_leaf'

\n

This is how the predict function currently defined:

\n

def predict(self, data, output_margin=False, ntree_limit=0)

\n

[/quote]

\n

this feature is only present in the "unity" branch (not merged into "master" yet).

", "votes": null, "replies": [] }, { "id": 62058, "author_name": "inspector", "author_url": "", "post_date": "12/29/2014 22:00:52", "content": "", "votes": null, "replies": [] }, { "id": 62087, "author_name": "derekzh", "author_url": "", "post_date": "12/30/2014 05:38:38", "content": "

[quote=Faron;62036]

\n

this feature is only present in the "unity" branch (not merged into "master" yet).

\n

[/quote]

\n\n

Hi Faron - would you know if we can install the unity branch then? I was trying to install, but "make" spits out an error that "rabit.h" is not found... Is the branch still in development that is not ready for users? 

", "votes": null, "replies": [] }, { "id": 62156, "author_name": "inspector", "author_url": "", "post_date": "12/31/2014 22:19:07", "content": "

[quote=AS;61675]

\n

Hi there,

\n

I wondered if anyone tried manufacturing synthetic features using a decision tree-like algorithm.  The winner solution at the Criteo competition used that technique, and there is a known paper from facebook on the topic (http://quinonero.net/Publications/predicting-clicks-facebook.pdf).

\n

Well, the results I got following this recipe were absolutely disappointing.  It might have well been that I implemented this strategy poorly, but I did try to tweak it a few times and no results to show for.

\n

Has anyone else ventured into this road? anything you can share?

\n

Thanks!

\n

A

\n

[/quote]

\n

It is funny how this was "rediscovered". It dates back to at least 1998, albeit using CART instead of an ensemble of trees. http://media.salford-systems.com/pdf/the-hybrid-cart-logit-model-in-classification-and-data%20mining-1998.pdf

", "votes": null, "replies": [] }, { "id": 63303, "author_name": "mmueller", "author_url": "", "post_date": "01/29/2015 13:10:42", "content": "

[quote=DerekZH;62087]

\n

[quote=Faron;62036]

\n

this feature is only present in the "unity" branch (not merged into "master" yet).

\n

[/quote]

\n

Hi Faron - would you know if we can install the unity branch then? I was trying to install, but "make" spits out an error that "rabit.h" is not found... Is the branch still in development that is not ready for users? 

\n

[/quote]

\n

It's in the master branch now.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "61675": "", "61990": "", "61999": "", "62004": "", "62006": "", "62009": "", "62010": "", "62012": "", "62029": "", "62036": "", "62058": "", "62087": "", "62156": "", "63303": "" }, "source": "meta" }