{ "id": 11150, "title": "Beating the Benchmark with Vowpal Wabbit LB- 0.4031", "url": "/competitions/avazu-ctr-prediction/discussion/11150", "author_name": "", "post_date": "2014-12-05T20:03:00.510Z", "votes": 14, "comment_count": 17, "views": 6637, "content": "

Taking inspiration from Triskelion and Zygmuntz, here is my solution using Vowpal wabbit which scored a 0.4031825 without much tuning whatsoever

\n

mlwhiz.com/blog/2014/12/01/Exploring_Vowpal_Wabbit_with_Avazu/

", "messages": [ { "id": "59638", "postDate": "12/05/2014 20:03:00", "content": "

Taking inspiration from Triskelion and Zygmuntz, here is my solution using Vowpal wabbit which scored a 0.4031825 without much tuning whatsoever

\n

mlwhiz.com/blog/2014/12/01/Exploring_Vowpal_Wabbit_with_Avazu/

", "rawMarkdown": "", "votes": null }, { "id": "59641", "postDate": "12/05/2014 20:42:50", "content": "

Very cool and helpful. Thank you for sharing!

", "rawMarkdown": "", "votes": null }, { "id": "59645", "postDate": "12/05/2014 21:05:08", "content": "

Thanks triskelion. I am really happy that you liked it :)

", "rawMarkdown": "", "votes": null }, { "id": "60025", "postDate": "12/11/2014 17:11:36", "content": "

Hi I'm really new to python. I'm trying to learn this code. I'm not sure how to install or apply the vw to the converted vw file. I'm using the ipython book in Anaconda3.4. I would greatly appreciate your assistance!

\n

Thank you!

", "rawMarkdown": "", "votes": null }, { "id": "60067", "postDate": "12/12/2014 02:41:04", "content": "

Hi heather you just have to install vowpal wabbit from the command line. You cab see the install instructions from the vowpal wabbit repository on github. 

\n

After you installedvw, there are actually three things that th script does.

\n

1 convert data to VW format. That should work fine with ipython notebook.

\n

2 run VWon the terminal. You should paste the code ditectly on the terminal.

\n

3 convert the file back to kaggle submission using the last snippet of code.

\n

One thing I can think of that you might be trying to run vw via the ipython which will not work.

\n

Or there could be problems with the relative paths.

", "rawMarkdown": "", "votes": null }, { "id": "60406", "postDate": "12/16/2014 22:38:03", "content": "

Hi Rahul - what are the memory requirements and the time it takes to run this?

", "rawMarkdown": "", "votes": null }, { "id": "60445", "postDate": "12/17/2014 08:45:21", "content": "

The memory footprint can be controlled via the vw command line, I think the default only uses a few hundred mb.

\n

As for speed, on my low end hardware the vw route was similar to Tinrtgu's code via pypy - about 20 mins for one pass. The vw route does require the one-off data conversion. My awk code for that takes about 30mins. I haven't tried the python provided here.

", "rawMarkdown": "", "votes": null }, { "id": "60459", "postDate": "12/17/2014 10:20:57", "content": "

Hi,

\n

Just one point. If you pass "--link logistic" to VW it automatically convert the output predictions to a 0-1 range and you won't need to do it yourself.

\n

I'm currently using also VW for this competition. With more tunning I managed to achieve 0.4004741 but I'm struggling to make the LB score go down. I tried it with 2-features interactions, and also with a neural net layer with some regularization. I haven't done many feature processing, that's what I'm focusing right now.

\n

It seems that VW overfits quite easily, event if with some regularization. Did anyone observe also this behavior?

", "rawMarkdown": "", "votes": null }, { "id": "60461", "postDate": "12/17/2014 10:52:46", "content": "

Yes, I have had the same overfit problem, and vw-hypersearch did not help: it was able to control overfit on same day data, but we need to tune L1 / L2 based on a cross-validation on a separate day.

\n

My best cv score on Vowpal is also around 0.40, and at the moment is obtained with straight regression and no feature interactions... which probably means I am overfitting and should increase regularization.

", "rawMarkdown": "", "votes": null }, { "id": "60467", "postDate": "12/17/2014 11:35:40", "content": "

Hi Guillem,

\n

Thats a nice little trick, I didnt know of. Will try that for sure.

\n

Apart from that I have also been able to tune vowpal to around the same.

\n

I tried many things but couldnt improve it more than that. But one thing i have noticed is that this model works very well in ensembles and so it is a part of my ensemble which is giving me .395 LB.

", "rawMarkdown": "", "votes": null }, { "id": "60470", "postDate": "12/17/2014 12:24:45", "content": "

I think the "-t" flag for vw-hypersearch allows you to test for cross validation. I haven't tried that yet but I will sure do. here is the reference page:

\n

https://github.com/JohnLangford/vowpal_wabbit/wiki/Using-vw-hypersearch

\n

I was also thinking to do an ensemble of classifiers. I tried to do it by training a single model per day and combine all of them but that didn't help (around 0.41). I'm almost certain that regularization is the one responsible for these bad results.

\n

Since in the test set there are a bunch of variables not seen in training, now I'm adding additional features to indicate if a feature has been seen or not before. I hope that this will help to generalize.

", "rawMarkdown": "", "votes": null }, { "id": "61416", "postDate": "12/17/2014 19:25:33", "content": "

@lewis ml - what does your awk code looks like?

", "rawMarkdown": "", "votes": null }, { "id": "61417", "postDate": "12/17/2014 19:57:35", "content": "

Here's my python code to convert an input file to vowpal wabbit. It does not contain any feature engineering, so you might want to add some :-)

\n

https://gist.github.com/anonymous/83187ed984fa6ed78080

\n

Edit: updated gist with a version that also works for the test file

\n

Edit2: fixed typo that caused train files not being recognised correctly

", "rawMarkdown": "", "votes": null }, { "id": "61429", "postDate": "12/17/2014 21:27:49", "content": "

Max -

\n

If you can live with hard coded field numbers then the awk code can be

\n

awk -F , 'NR > 1 {print $2 * 2 -1, "|s", "id_"$6, "d_"$7, "c_"$8}' train.csv > train.vw

\n

The above only covers the click indicator and one namespace - add as many others as you like. And don't forget to reduce the field numbers by one for the test file and replace the click calculation with "0".

\n

[edited]

", "rawMarkdown": "", "votes": null }, { "id": "61430", "postDate": "12/17/2014 21:34:12", "content": "

and you should be able to use substr() for processing the hour field.

", "rawMarkdown": "", "votes": null }, { "id": "63055", "postDate": "01/23/2015 14:26:25", "content": "

Hi All

\n

I have validate following data in vw validator. The Id (ie tag) is act as Importance Weight.

\n

Please any one suggest , Is it correct or not

\n\n

-1 '1000009418151094272 |hr 00 |day Tuesday|c3 1005|c4 0|c5 1fbe01fe|c6 f3845767|c7 28905ebd |c8 ecad2386 |c9 7801e8d9 |c10 07d7df22|c11 a99f214a|c12 ddd2926e|c13 44956a

\n\n

Report

\n

example #1) Example “-1 '1000009418151094272 |hr 00 |day Tuesday |c3 1005 |c4 0 |c5 1fbe01fe |c6 f3845767”.
(example #1) Found “-1 '1000009418151094272 ” ambiguous prefix with 2 elements.
(example #1) Example label / response / class is “-1”.
(example #1) Importance weight “'1000009418151094272” must be a float!
(example #1) Example has default “0” base.

\n\n

Thank in advance

", "rawMarkdown": "", "votes": null }, { "id": "63057", "postDate": "01/23/2015 14:39:16", "content": "

@Tamilselvan Palani

\n

You need to make sure that all namespaces are space-seperated.

\n

|c10 07d7df22|c11 a99f214a|c12 ddd2926e|c13 44956a

\n

Should be:

\n

|c10 07d7df22 |c11 a99f214a |c12 ddd2926e |c13 44956a

\n

Also, I believe the namespacing is mapped to the first letter of the namespace:

\n

|train foo |test bar

\n

will both be mapped to "t". In your example all categorical namespaces are mapped to "c".

\n

Other than that, I see nothing why it should not work.

\n

-1 '1000009418151094272 |hr 00 |day Tuesday |c3 1005 |c4 0 |c5 1fbe01fe |c6 f3845767 |c7 28905ebd |c8 ecad2386 |c9 7801e8d9 |c10 07d7df22 |c11 a99f214a |c12 ddd2926e |c13 44956a

\n

In validator works fine:

\n

(example #1) Example “-1 '100000941815… 926e |c13 44956a”.
(example #1) Found “[label] ['tag] |…” prefix format.
(example #1) Example label / response / class is “-1”.
(example #1) Example has default “1.0” importance weight.
(example #1) Example has default “0” base.
(example #1) Tag is “1000009418151094272”.
(example #1, namespace #1) Namespace found with name “hr”.

\n

[...]

", "rawMarkdown": "", "votes": null }, { "id": "63058", "postDate": "01/23/2015 14:43:21", "content": "

Thanks Triskelion

\n\n

I have already done  

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 59641, "author_name": "triskelion", "author_url": "", "post_date": "12/05/2014 20:42:50", "content": "

Very cool and helpful. Thank you for sharing!

", "votes": null, "replies": [] }, { "id": 59645, "author_name": "mlwhiz", "author_url": "", "post_date": "12/05/2014 21:05:08", "content": "

Thanks triskelion. I am really happy that you liked it :)

", "votes": null, "replies": [] }, { "id": 60025, "author_name": "heatherpatterson", "author_url": "", "post_date": "12/11/2014 17:11:36", "content": "

Hi I'm really new to python. I'm trying to learn this code. I'm not sure how to install or apply the vw to the converted vw file. I'm using the ipython book in Anaconda3.4. I would greatly appreciate your assistance!

\n

Thank you!

", "votes": null, "replies": [] }, { "id": 60067, "author_name": "mlwhiz", "author_url": "", "post_date": "12/12/2014 02:41:04", "content": "

Hi heather you just have to install vowpal wabbit from the command line. You cab see the install instructions from the vowpal wabbit repository on github. 

\n

After you installedvw, there are actually three things that th script does.

\n

1 convert data to VW format. That should work fine with ipython notebook.

\n

2 run VWon the terminal. You should paste the code ditectly on the terminal.

\n

3 convert the file back to kaggle submission using the last snippet of code.

\n

One thing I can think of that you might be trying to run vw via the ipython which will not work.

\n

Or there could be problems with the relative paths.

", "votes": null, "replies": [] }, { "id": 60406, "author_name": "maxsop", "author_url": "", "post_date": "12/16/2014 22:38:03", "content": "

Hi Rahul - what are the memory requirements and the time it takes to run this?

", "votes": null, "replies": [] }, { "id": 60445, "author_name": "cartographic", "author_url": "", "post_date": "12/17/2014 08:45:21", "content": "

The memory footprint can be controlled via the vw command line, I think the default only uses a few hundred mb.

\n

As for speed, on my low end hardware the vw route was similar to Tinrtgu's code via pypy - about 20 mins for one pass. The vw route does require the one-off data conversion. My awk code for that takes about 30mins. I haven't tried the python provided here.

", "votes": null, "replies": [] }, { "id": 60459, "author_name": "guillempalou", "author_url": "", "post_date": "12/17/2014 10:20:57", "content": "

Hi,

\n

Just one point. If you pass "--link logistic" to VW it automatically convert the output predictions to a 0-1 range and you won't need to do it yourself.

\n

I'm currently using also VW for this competition. With more tunning I managed to achieve 0.4004741 but I'm struggling to make the LB score go down. I tried it with 2-features interactions, and also with a neural net layer with some regularization. I haven't done many feature processing, that's what I'm focusing right now.

\n

It seems that VW overfits quite easily, event if with some regularization. Did anyone observe also this behavior?

", "votes": null, "replies": [] }, { "id": 60461, "author_name": "martely", "author_url": "", "post_date": "12/17/2014 10:52:46", "content": "

Yes, I have had the same overfit problem, and vw-hypersearch did not help: it was able to control overfit on same day data, but we need to tune L1 / L2 based on a cross-validation on a separate day.

\n

My best cv score on Vowpal is also around 0.40, and at the moment is obtained with straight regression and no feature interactions... which probably means I am overfitting and should increase regularization.

", "votes": null, "replies": [] }, { "id": 60467, "author_name": "mlwhiz", "author_url": "", "post_date": "12/17/2014 11:35:40", "content": "

Hi Guillem,

\n

Thats a nice little trick, I didnt know of. Will try that for sure.

\n

Apart from that I have also been able to tune vowpal to around the same.

\n

I tried many things but couldnt improve it more than that. But one thing i have noticed is that this model works very well in ensembles and so it is a part of my ensemble which is giving me .395 LB.

", "votes": null, "replies": [] }, { "id": 60470, "author_name": "guillempalou", "author_url": "", "post_date": "12/17/2014 12:24:45", "content": "

I think the "-t" flag for vw-hypersearch allows you to test for cross validation. I haven't tried that yet but I will sure do. here is the reference page:

\n

https://github.com/JohnLangford/vowpal_wabbit/wiki/Using-vw-hypersearch

\n

I was also thinking to do an ensemble of classifiers. I tried to do it by training a single model per day and combine all of them but that didn't help (around 0.41). I'm almost certain that regularization is the one responsible for these bad results.

\n

Since in the test set there are a bunch of variables not seen in training, now I'm adding additional features to indicate if a feature has been seen or not before. I hope that this will help to generalize.

", "votes": null, "replies": [] }, { "id": 61416, "author_name": "maxsop", "author_url": "", "post_date": "12/17/2014 19:25:33", "content": "

@lewis ml - what does your awk code looks like?

", "votes": null, "replies": [] }, { "id": 61417, "author_name": "tobycheese", "author_url": "", "post_date": "12/17/2014 19:57:35", "content": "

Here's my python code to convert an input file to vowpal wabbit. It does not contain any feature engineering, so you might want to add some :-)

\n

https://gist.github.com/anonymous/83187ed984fa6ed78080

\n

Edit: updated gist with a version that also works for the test file

\n

Edit2: fixed typo that caused train files not being recognised correctly

", "votes": null, "replies": [] }, { "id": 61429, "author_name": "cartographic", "author_url": "", "post_date": "12/17/2014 21:27:49", "content": "

Max -

\n

If you can live with hard coded field numbers then the awk code can be

\n

awk -F , 'NR > 1 {print $2 * 2 -1, "|s", "id_"$6, "d_"$7, "c_"$8}' train.csv > train.vw

\n

The above only covers the click indicator and one namespace - add as many others as you like. And don't forget to reduce the field numbers by one for the test file and replace the click calculation with "0".

\n

[edited]

", "votes": null, "replies": [] }, { "id": 61430, "author_name": "cartographic", "author_url": "", "post_date": "12/17/2014 21:34:12", "content": "

and you should be able to use substr() for processing the hour field.

", "votes": null, "replies": [] }, { "id": 63055, "author_name": "tamilselvan", "author_url": "", "post_date": "01/23/2015 14:26:25", "content": "

Hi All

\n

I have validate following data in vw validator. The Id (ie tag) is act as Importance Weight.

\n

Please any one suggest , Is it correct or not

\n\n

-1 '1000009418151094272 |hr 00 |day Tuesday|c3 1005|c4 0|c5 1fbe01fe|c6 f3845767|c7 28905ebd |c8 ecad2386 |c9 7801e8d9 |c10 07d7df22|c11 a99f214a|c12 ddd2926e|c13 44956a

\n\n

Report

\n

example #1) Example “-1 '1000009418151094272 |hr 00 |day Tuesday |c3 1005 |c4 0 |c5 1fbe01fe |c6 f3845767”.
(example #1) Found “-1 '1000009418151094272 ” ambiguous prefix with 2 elements.
(example #1) Example label / response / class is “-1”.
(example #1) Importance weight “'1000009418151094272” must be a float!
(example #1) Example has default “0” base.

\n\n

Thank in advance

", "votes": null, "replies": [] }, { "id": 63057, "author_name": "triskelion", "author_url": "", "post_date": "01/23/2015 14:39:16", "content": "

@Tamilselvan Palani

\n

You need to make sure that all namespaces are space-seperated.

\n

|c10 07d7df22|c11 a99f214a|c12 ddd2926e|c13 44956a

\n

Should be:

\n

|c10 07d7df22 |c11 a99f214a |c12 ddd2926e |c13 44956a

\n

Also, I believe the namespacing is mapped to the first letter of the namespace:

\n

|train foo |test bar

\n

will both be mapped to "t". In your example all categorical namespaces are mapped to "c".

\n

Other than that, I see nothing why it should not work.

\n

-1 '1000009418151094272 |hr 00 |day Tuesday |c3 1005 |c4 0 |c5 1fbe01fe |c6 f3845767 |c7 28905ebd |c8 ecad2386 |c9 7801e8d9 |c10 07d7df22 |c11 a99f214a |c12 ddd2926e |c13 44956a

\n

In validator works fine:

\n

(example #1) Example “-1 '100000941815… 926e |c13 44956a”.
(example #1) Found “[label] ['tag] |…” prefix format.
(example #1) Example label / response / class is “-1”.
(example #1) Example has default “1.0” importance weight.
(example #1) Example has default “0” base.
(example #1) Tag is “1000009418151094272”.
(example #1, namespace #1) Namespace found with name “hr”.

\n

[...]

", "votes": null, "replies": [] }, { "id": 63058, "author_name": "tamilselvan", "author_url": "", "post_date": "01/23/2015 14:43:21", "content": "

Thanks Triskelion

\n\n

I have already done  

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "59638": "", "59641": "", "59645": "", "60025": "", "60067": "", "60406": "", "60445": "", "60459": "", "60461": "", "60467": "", "60470": "", "61416": "", "61417": "", "61429": "", "61430": "", "63055": "", "63057": "", "63058": "" }, "source": "meta" }