Having used it for the first time here, I think I'll remember this competition for properly introducing me to Vowpal Wabbit. And thanks again to Triskelion for all the valuable Kaggle posts and blog entries.
\nWith that in mind, would anyone care to share what did and didn't work for them with vw? I'm not really thinking of top-of-the-leaderboard scores. It would just be useful to know more about the different vw parameters or data preparation steps that helped and which approaches were easier to implement.
\nIn my case, I found it too difficult (impossible?) to just throw in all quadratic interactions and rely on regularisation. So I created separate namespaces for each variable and used a small sample to identify the most promising interactions to add manually (I also needed to remove the pesky hidden line endings in the raw data files).
\nWhich meant my code amounted to:
\ntr -d '\\r' < Data/train.csv > Data/train_clean.csv
awk -F, 'NR > 1 {print ($2*2)-1, "|a", "a_"$4, "|b", "b_"$5, "|c", "c_"$6, "|d", "d_"$7, "|e", "e_"$8, "|f", "f_"$9, "|g", "g_"$10, "|h", "h_"$11, "|i", "i_"$12, "|j", "j_"$13, "|k", "k_"$14, "|l", "l_"$15, "|m", "m_"$16, "|n", "n_"$17, "|o", "o_"$18, "|p", "p_"$19, "|q", "q_"$20, "|r", "r_"$21, "|s", "s_"$22, "|t", "t_"$23, "|u", "u_"$24, "|v", "v_"substr($3, 7, 2)}' Data/train_clean.csv > Vowpal/train.vw
vw -d Vowpal/train.vw -f Vowpal/model_sub3.vw --loss_function logistic -l 0.07 -c -k --passes 4 -b 25 --holdout_after 32377421 --hash all --l1 1e-8 --l2 1e-8 -q d: -q f: -q bk --cubic bdk --cubic bfk --ignore l --ignore v
Which gave a public leaderboard score of 0.3918 after re-running with holdout_off. The interactions relied mainly on site_domain (d) and app_id (f).
\nNot sure if I made a mistake, but feature_mask didn't work for me. And after re-categorising rare features into 'other', I found the neural network option on vw to be too slow on my hardware.
", "messages": [ { "id": "64007", "postDate": "02/11/2015 09:03:29", "content": "Having used it for the first time here, I think I'll remember this competition for properly introducing me to Vowpal Wabbit. And thanks again to Triskelion for all the valuable Kaggle posts and blog entries.
\nWith that in mind, would anyone care to share what did and didn't work for them with vw? I'm not really thinking of top-of-the-leaderboard scores. It would just be useful to know more about the different vw parameters or data preparation steps that helped and which approaches were easier to implement.
\nIn my case, I found it too difficult (impossible?) to just throw in all quadratic interactions and rely on regularisation. So I created separate namespaces for each variable and used a small sample to identify the most promising interactions to add manually (I also needed to remove the pesky hidden line endings in the raw data files).
\nWhich meant my code amounted to:
\ntr -d '\\r' < Data/train.csv > Data/train_clean.csv
awk -F, 'NR > 1 {print ($2*2)-1, "|a", "a_"$4, "|b", "b_"$5, "|c", "c_"$6, "|d", "d_"$7, "|e", "e_"$8, "|f", "f_"$9, "|g", "g_"$10, "|h", "h_"$11, "|i", "i_"$12, "|j", "j_"$13, "|k", "k_"$14, "|l", "l_"$15, "|m", "m_"$16, "|n", "n_"$17, "|o", "o_"$18, "|p", "p_"$19, "|q", "q_"$20, "|r", "r_"$21, "|s", "s_"$22, "|t", "t_"$23, "|u", "u_"$24, "|v", "v_"substr($3, 7, 2)}' Data/train_clean.csv > Vowpal/train.vw
vw -d Vowpal/train.vw -f Vowpal/model_sub3.vw --loss_function logistic -l 0.07 -c -k --passes 4 -b 25 --holdout_after 32377421 --hash all --l1 1e-8 --l2 1e-8 -q d: -q f: -q bk --cubic bdk --cubic bfk --ignore l --ignore v
Which gave a public leaderboard score of 0.3918 after re-running with holdout_off. The interactions relied mainly on site_domain (d) and app_id (f).
\nNot sure if I made a mistake, but feature_mask didn't work for me. And after re-categorising rare features into 'other', I found the neural network option on vw to be too slow on my hardware.
", "rawMarkdown": "", "votes": null }, { "id": "64010", "postDate": "02/11/2015 09:22:47", "content": "I've got 0.3893116 on public and 0.3871975 on private with single VW model. It was a fork of VW with several my improvements some of which I'm now trying to contribute back to VW master. They are described in my feedback on vw's github page: https://github.com/JohnLangford/vowpal_wabbit/issues/510
\nIf you be able (it's really long) to read suggestions #1-6 and 9 that mostly describe my model design flow. And final VW commands become understandable: http://pastebin.com/SnGwiCRi
\nsed is used due to #5
\ntrain_freq.vw looks like that http://pastebin.com/ZEZec0eL
\nwhere x_freq_y features are rond(log(count_of_this_feature_in_dataset)).
", "rawMarkdown": "", "votes": null }, { "id": "64013", "postDate": "02/11/2015 10:33:18", "content": "Wow - that's more than I expected and I'm sure the vw team will appreciate all the feedback. I can see the case for more powerful approaches to namespaces and interaction terms, and the record of time taken would certainly come in handy.
", "rawMarkdown": "", "votes": null }, { "id": "64015", "postDate": "02/11/2015 10:54:52", "content": "I can't recall exactly, but training on full dataset and prediction (2 passes + testing) shall took about 15-17 hours and 5 Gb RAM. There is no improvement with more passes or regularization applied. I also don't expect significant improvements with more memory (-b > 28). I tried it on my laptop with Intel i7 2.9GHz 4-cores, 8Gb RAM and Kubuntu 14.
", "rawMarkdown": "", "votes": null }, { "id": "64017", "postDate": "02/11/2015 11:02:25", "content": "Hi All, after using SGDClassifier in scikit-learn and reaching the 0.397 score on Public LB, I moved to VW obtaining on public LB 0.3939311 and private 0.3919883 (which I suspect is the VW entry and not the scikit one).
\nI combined the features in preparing the data for VW format creating new features per 'categories': app, site, domain, id...that is something like this:
\napp=app_id+app_domain+app_category
\n....
\nthen I run vw with the following command line:
\nvw train.vw.gz -f model.vw.gz --loss_function=logistic --hash strings -b 26 --learning_rate=0.6 --compressed --ftrl --ftrl_alpha=0.1 --ftrl_beta=1.0 --nn=10
", "rawMarkdown": "", "votes": null }, { "id": "64021", "postDate": "02/11/2015 12:42:40", "content": "[quote=truf;64010]
\nI've got 0.3893116 on public and 0.3871975 on private with single VW model. It was a fork of VW with several my improvements some of which I'm now trying to contribute back to VW master. [/quote]
\nYou get this result by tweaking VW or by custom features? Can you explain your approach, please?
\n\nI also have a lot of troubles with VW, especially with --rank option
", "rawMarkdown": "", "votes": null }, { "id": "64025", "postDate": "02/11/2015 13:59:53", "content": "[quote=Mikhail Trofimov;64021]
\nYou get this result by tweaking VW or by custom features? Can you explain your approach, please?
\nI also have a lot of troubles with VW, especially with --rank option
\n[/quote]
\nBoth. As for tweaks, they are described in my github post. As for feature engineering - it looks like i used that Mr. Gert means by 'logtransformed counts'. I suspect our solutions are similar as I used high degrees of interaction between selected features too. For some even more than 3 (which is --cubic in vw).
\n\nAs for --rank - it didn't helped me, but I spent some time investigating its implementation. If VWs behavior didn't change much since my fork was done then --rank by itself is pretty useless as it launches the code in gd_mf.cc This implementation overwrites gradient descent parameters and use it without adaptive, normalazed and other enhancements which is ON by default for regular GD in VW. You can try VW's GD without them - it's much worse than default one. So if you just add --rank to your command line you'll get significantly worse results. There is a better implementation of matrix factorization in VW in mf.cc which is triggered if you add --new_mf key together with --rank. That implementation sits on top of your existing GD. So it just reuse GD you already use with all its parameters and modes. Thus --new_mf --rank behaves much better.
\nAnd using --new_mf --rank you might face with problem which I described on github in I.12 - random values used for feature weights initialization might be too big for your dataset. In this case you need to initialize them with smaller value with --initial_weight.
\nAnyway, I couldn't improve my model with MF. Perhaps I would get better results if I ensembled it with my solution.
", "rawMarkdown": "", "votes": null }, { "id": "64029", "postDate": "02/11/2015 14:22:39", "content": "[quote=Elena Cuoco;64017]
\nHi All, after using SGDClassifier in scikit-learn and reaching the 0.397 score on Public LB, I moved to VW obtaining on public LB 0.3939311 and private 0.3919883 (which I suspect is the VW entry and not the scikit one).
\n[/quote]
\nOn the competition site under "my submissions" you can check both the public and private LB score for each of your submissions (provided you can map your submissions to your attempts by filename, comment or timestamp).
", "rawMarkdown": "", "votes": null }, { "id": "64034", "postDate": "02/11/2015 15:06:00", "content": "[quote=Toby Cheese;64029]
\nOn the competition site under "my submissions" you can check both the public and private LB score for each of your submissions (provided you can map your submissions to your attempts by filename, comment or timestamp).
\n[/quote]
\nThanks a lot Toby! I relized that I have a better result for VW (but just on the last decimal digit) than the one I selected and that all my attempts with scikit were not so bad, reaching a private score around 0.396 with best result of 0.3961003.
\nI will put my scikit solution on github as soon as I have free time.
", "rawMarkdown": "", "votes": null }, { "id": "64036", "postDate": "02/11/2015 15:12:20", "content": "This doesn't really give insight into the various parameters available in vw, but I found the vw-hypersearch utility for tuning pretty useful. It needs data for cross-validation, otherwise parameter values are determined based on progressive validation, which underperforms.
", "rawMarkdown": "", "votes": null }, { "id": "65406", "postDate": "03/04/2015 13:22:56", "content": "The competition exposed me to Vowpal Wabbit. After some experiments in which I finally scored 0.3976030 (post closure), I wrote a blog explaining all my steps. VW is indeed fast and uses less resources.
", "rawMarkdown": "", "votes": null }, { "id": "79893", "postDate": "05/24/2015 09:05:30", "content": "Hi pals!
\nSorry for reanimate this dead thread. Just want to say that most of VW improvements I made during this competition were finally merged into VW master codebase. So it may worth to `git pull` these changes. The changes are:
\n* --redefine param to rename/group namespaces on fly
\n* --interactions param for interactions of any length (not only 2 in -q and 3 in --cubic)
\n* simple combinations instead of permutations of interacting features of the same namespace allows to generate less new features making model train faster. For example `-q aa` where a holds 3 features will generate only 3 new features instead of 9.
\n* Slightly better hash function for interacting features
", "rawMarkdown": "", "votes": null }, { "id": "79898", "postDate": "05/24/2015 11:17:06", "content": "Truf, thanks a lot for adding to VW and the extensive user feedback! Much appreciated.
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 64010, "author_name": "trufanov", "author_url": "", "post_date": "02/11/2015 09:22:47", "content": "I've got 0.3893116 on public and 0.3871975 on private with single VW model. It was a fork of VW with several my improvements some of which I'm now trying to contribute back to VW master. They are described in my feedback on vw's github page: https://github.com/JohnLangford/vowpal_wabbit/issues/510
\nIf you be able (it's really long) to read suggestions #1-6 and 9 that mostly describe my model design flow. And final VW commands become understandable: http://pastebin.com/SnGwiCRi
\nsed is used due to #5
\ntrain_freq.vw looks like that http://pastebin.com/ZEZec0eL
\nwhere x_freq_y features are rond(log(count_of_this_feature_in_dataset)).
", "votes": null, "replies": [] }, { "id": 64013, "author_name": "cartographic", "author_url": "", "post_date": "02/11/2015 10:33:18", "content": "Wow - that's more than I expected and I'm sure the vw team will appreciate all the feedback. I can see the case for more powerful approaches to namespaces and interaction terms, and the record of time taken would certainly come in handy.
", "votes": null, "replies": [] }, { "id": 64015, "author_name": "trufanov", "author_url": "", "post_date": "02/11/2015 10:54:52", "content": "I can't recall exactly, but training on full dataset and prediction (2 passes + testing) shall took about 15-17 hours and 5 Gb RAM. There is no improvement with more passes or regularization applied. I also don't expect significant improvements with more memory (-b > 28). I tried it on my laptop with Intel i7 2.9GHz 4-cores, 8Gb RAM and Kubuntu 14.
", "votes": null, "replies": [] }, { "id": 64017, "author_name": "elenacuoco", "author_url": "", "post_date": "02/11/2015 11:02:25", "content": "Hi All, after using SGDClassifier in scikit-learn and reaching the 0.397 score on Public LB, I moved to VW obtaining on public LB 0.3939311 and private 0.3919883 (which I suspect is the VW entry and not the scikit one).
\nI combined the features in preparing the data for VW format creating new features per 'categories': app, site, domain, id...that is something like this:
\napp=app_id+app_domain+app_category
\n....
\nthen I run vw with the following command line:
\nvw train.vw.gz -f model.vw.gz --loss_function=logistic --hash strings -b 26 --learning_rate=0.6 --compressed --ftrl --ftrl_alpha=0.1 --ftrl_beta=1.0 --nn=10
", "votes": null, "replies": [] }, { "id": 64021, "author_name": "mikhailtrofimov", "author_url": "", "post_date": "02/11/2015 12:42:40", "content": "[quote=truf;64010]
\nI've got 0.3893116 on public and 0.3871975 on private with single VW model. It was a fork of VW with several my improvements some of which I'm now trying to contribute back to VW master. [/quote]
\nYou get this result by tweaking VW or by custom features? Can you explain your approach, please?
\n\nI also have a lot of troubles with VW, especially with --rank option
", "votes": null, "replies": [] }, { "id": 64025, "author_name": "trufanov", "author_url": "", "post_date": "02/11/2015 13:59:53", "content": "[quote=Mikhail Trofimov;64021]
\nYou get this result by tweaking VW or by custom features? Can you explain your approach, please?
\nI also have a lot of troubles with VW, especially with --rank option
\n[/quote]
\nBoth. As for tweaks, they are described in my github post. As for feature engineering - it looks like i used that Mr. Gert means by 'logtransformed counts'. I suspect our solutions are similar as I used high degrees of interaction between selected features too. For some even more than 3 (which is --cubic in vw).
\n\nAs for --rank - it didn't helped me, but I spent some time investigating its implementation. If VWs behavior didn't change much since my fork was done then --rank by itself is pretty useless as it launches the code in gd_mf.cc This implementation overwrites gradient descent parameters and use it without adaptive, normalazed and other enhancements which is ON by default for regular GD in VW. You can try VW's GD without them - it's much worse than default one. So if you just add --rank to your command line you'll get significantly worse results. There is a better implementation of matrix factorization in VW in mf.cc which is triggered if you add --new_mf key together with --rank. That implementation sits on top of your existing GD. So it just reuse GD you already use with all its parameters and modes. Thus --new_mf --rank behaves much better.
\nAnd using --new_mf --rank you might face with problem which I described on github in I.12 - random values used for feature weights initialization might be too big for your dataset. In this case you need to initialize them with smaller value with --initial_weight.
\nAnyway, I couldn't improve my model with MF. Perhaps I would get better results if I ensembled it with my solution.
", "votes": null, "replies": [] }, { "id": 64029, "author_name": "tobycheese", "author_url": "", "post_date": "02/11/2015 14:22:39", "content": "[quote=Elena Cuoco;64017]
\nHi All, after using SGDClassifier in scikit-learn and reaching the 0.397 score on Public LB, I moved to VW obtaining on public LB 0.3939311 and private 0.3919883 (which I suspect is the VW entry and not the scikit one).
\n[/quote]
\nOn the competition site under "my submissions" you can check both the public and private LB score for each of your submissions (provided you can map your submissions to your attempts by filename, comment or timestamp).
", "votes": null, "replies": [] }, { "id": 64034, "author_name": "elenacuoco", "author_url": "", "post_date": "02/11/2015 15:06:00", "content": "[quote=Toby Cheese;64029]
\nOn the competition site under "my submissions" you can check both the public and private LB score for each of your submissions (provided you can map your submissions to your attempts by filename, comment or timestamp).
\n[/quote]
\nThanks a lot Toby! I relized that I have a better result for VW (but just on the last decimal digit) than the one I selected and that all my attempts with scikit were not so bad, reaching a private score around 0.396 with best result of 0.3961003.
\nI will put my scikit solution on github as soon as I have free time.
", "votes": null, "replies": [] }, { "id": 64036, "author_name": "jwjohnson314", "author_url": "", "post_date": "02/11/2015 15:12:20", "content": "This doesn't really give insight into the various parameters available in vw, but I found the vw-hypersearch utility for tuning pretty useful. It needs data for cross-validation, otherwise parameter values are determined based on progressive validation, which underperforms.
", "votes": null, "replies": [] }, { "id": 65406, "author_name": "harnalashok", "author_url": "", "post_date": "03/04/2015 13:22:56", "content": "The competition exposed me to Vowpal Wabbit. After some experiments in which I finally scored 0.3976030 (post closure), I wrote a blog explaining all my steps. VW is indeed fast and uses less resources.
", "votes": null, "replies": [] }, { "id": 79893, "author_name": "trufanov", "author_url": "", "post_date": "05/24/2015 09:05:30", "content": "Hi pals!
\nSorry for reanimate this dead thread. Just want to say that most of VW improvements I made during this competition were finally merged into VW master codebase. So it may worth to `git pull` these changes. The changes are:
\n* --redefine param to rename/group namespaces on fly
\n* --interactions param for interactions of any length (not only 2 in -q and 3 in --cubic)
\n* simple combinations instead of permutations of interacting features of the same namespace allows to generate less new features making model train faster. For example `-q aa` where a holds 3 features will generate only 3 new features instead of 9.
\n* Slightly better hash function for interacting features
", "votes": null, "replies": [] }, { "id": 79898, "author_name": "triskelion", "author_url": "", "post_date": "05/24/2015 11:17:06", "content": "Truf, thanks a lot for adding to VW and the extensive user feedback! Much appreciated.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "64007": "", "64010": "", "64013": "", "64015": "", "64017": "", "64021": "", "64025": "", "64029": "", "64034": "", "64036": "", "65406": "", "79893": "", "79898": "" }, "source": "meta" }