{ "id": 10120, "title": "Beating the benchmark", "url": "/competitions/avito-prohibited-content/discussion/10120", "author_name": "", "post_date": "2014-08-26T20:26:20.640Z", "votes": 25, "comment_count": 35, "views": 11951, "content": "

Good news, everyone! Here's the code that got me ~0.971:

\n

https://github.com/zygmuntz/kaggle-avito

\n

It uses Vowpal Wabbit and comes with instructions. Remember to click thanks if you find it interesting or useful.

", "messages": [ { "id": "52477", "postDate": "08/26/2014 20:26:20", "content": "

Good news, everyone! Here's the code that got me ~0.971:

\n

https://github.com/zygmuntz/kaggle-avito

\n

It uses Vowpal Wabbit and comes with instructions. Remember to click thanks if you find it interesting or useful.

", "rawMarkdown": "", "votes": null }, { "id": "52478", "postDate": "08/26/2014 21:09:43", "content": "

You stole my copyrighted topic name :P lol

", "rawMarkdown": "", "votes": null }, { "id": "52479", "postDate": "08/26/2014 21:49:43", "content": "

[quote=Abhishek;52478]

\n

You stole my copyrighted topic name :P lol

\n

[/quote]

\n

Man, you compete too much, your memory is failing :P

", "rawMarkdown": "", "votes": null }, { "id": "52480", "postDate": "08/26/2014 22:20:47", "content": "

[quote=Abhishek;52478]

\n

You stole my copyrighted topic name :P lol

\n

[/quote]

\n

No, your copyrighted topic name is "Beating the benchmark :-)"

\n

;-)

", "rawMarkdown": "", "votes": null }, { "id": "52483", "postDate": "08/26/2014 23:49:24", "content": "

[quote=Foxtrot;52477]

\n

Good news, everyone! Here's the code that got me ~0.971:

\n

https://github.com/zygmuntz/kaggle-avito

\n

It uses Vowpal Wabbit and comes with instructions. Remember to click thanks if you find it interesting or useful.

\n

[/quote]

\n

Additional friendly advice: splitting and training for subcategories might produce a better score (by merging the results). The trouble is that some subcategories don't have enough training examples. 

", "rawMarkdown": "", "votes": null }, { "id": "52512", "postDate": "08/27/2014 16:21:59", "content": "

[quote=Foxtrot;52477]

\n

Good news, everyone! Here's the code that got me ~0.971:

\n

https://github.com/zygmuntz/kaggle-avito

\n

It uses Vowpal Wabbit and comes with instructions. Remember to click thanks if you find it interesting or useful.

\n

[/quote]

\n

Hi Zygmunt, can you explain the logic behind setting label to -1 when it's 0 in the train data? Is this is a VW convention or a choice you made?

", "rawMarkdown": "", "votes": null }, { "id": "52513", "postDate": "08/27/2014 16:26:34", "content": "

[quote=Momchil Georgiev;52512]

\n

Hi Zygmunt, can you explain the logic behind setting label to -1 when it's 0 in the train data? Is this is a VW convention or a choice you made?

\n

[/quote]

\n

Vowpal Wabbit needs 1/-1 labels for binary classification, otherwise you get an error.

", "rawMarkdown": "", "votes": null }, { "id": "52514", "postDate": "08/27/2014 16:27:16", "content": "

Thats a very high benchmark!  :(

", "rawMarkdown": "", "votes": null }, { "id": "52515", "postDate": "08/27/2014 16:28:30", "content": "

[quote=Momchil Georgiev;52512]

\n

[quote=Foxtrot;52477]

\n

Good news, everyone! Here's the code that got me ~0.971:

\n

https://github.com/zygmuntz/kaggle-avito

\n

It uses Vowpal Wabbit and comes with instructions. Remember to click thanks if you find it interesting or useful.

\n

[/quote]

\n

Hi Zygmunt, can you explain the logic behind setting label to -1 when it's 0 in the train data? Is this is a VW convention or a choice you made?

\n

[/quote]

\n

Logistic regression (and hinge loss) needs labels to be [-1,1] for binary classification tasks. So it is aVW convention.

", "rawMarkdown": "", "votes": null }, { "id": "52516", "postDate": "08/27/2014 16:46:51", "content": "

[quote=Foxtrot;52513]

\n

[quote=Momchil Georgiev;52512]

\n

Hi Zygmunt, can you explain the logic behind setting label to -1 when it's 0 in the train data? Is this is a VW convention or a choice you made?

\n

[/quote]

\n

Vowpal Wabbit needs 1/-1 labels for binary classification, otherwise you get an error.

\n

[/quote]

\n

Thanks! I'm very new to VW but it looks like an awesome tool so I'm trying to learn it. What about the -holdout_off parameter? On my version of VW it complained that it was not a valid flag.

", "rawMarkdown": "", "votes": null }, { "id": "52517", "postDate": "08/27/2014 17:03:50", "content": "

[quote=Momchil Georgiev;52516]

\n

Thanks! I'm very new to VW but it looks like an awesome tool so I'm trying to learn it. What about the -holdout_off parameter? On my version of VW it complained that it was not a valid flag.

\n

[/quote]

\n

This option was introduced recently. When doing multiple passes over data, VW reserves 10% for validation - that's a holdout set. The idea is to turn it off to use all available examples for training. However in case of this competition the difference in the score is minor.

\n

P.S. Moar here: http://fastml.com/vowpal-wabbit-eats-big-data-from-the-criteo-competition-for-breakfast/

", "rawMarkdown": "", "votes": null }, { "id": "52529", "postDate": "08/27/2014 18:40:57", "content": "

Thanks for sharing.

\n

I would love to learn about VW and tried running your code, but it got stuck at predicting stage. When I ran vw -t -i model -d test.vw -p predicshuns.txt, it only shows following logs but nothing more:

\n

For more information use: vw --help
Num weight bits = 18
learning rate = 0.5
initial_t = 0
power_t = 0.5
using no cache
Reading datafile =
num sources = 1
average since example example current current current
loss last counter weight label predict features

", "rawMarkdown": "", "votes": null }, { "id": "52563", "postDate": "08/28/2014 12:14:27", "content": "

Vowpal is just amazing. The more I use it, the more impressed I get. Online learning is in a different class of its own. @Foxtrot, thanks for sharing your insights and code examples, it is very helpful in understanding how VW works.

", "rawMarkdown": "", "votes": null }, { "id": "52567", "postDate": "08/28/2014 12:49:39", "content": "

@xbsd You're welcome.

\n

@Apophenia Check if you have the files (model, test.vw) in place.

", "rawMarkdown": "", "votes": null }, { "id": "52581", "postDate": "08/28/2014 14:28:53", "content": "

Question @Foxtrot, just wondering what is the purpose of not using holdout_off in the final training. I believe this is due to the fact that we want to use the entire training set for the final training and for the initial run (where we do not specify a -f model) we use that to measure the average loss only before we do the final training.

\n\n

Since in the case where we do not specify a -f model or use -holdout_off VW is already measuring the performance using the holdout set, is there a need for a separate cross-validation (like in R/Python where it is more batch learning) ? Thanks.

", "rawMarkdown": "", "votes": null }, { "id": "52585", "postDate": "08/28/2014 15:19:29", "content": "

@Foxtrot Yes, I have them all in place.

\n

kaggle-avito-master/
├── LICENSE
├── README.md
├── load_attribs.py
├── model
├── predict.py
├── score.py
├── test.vw
├── train.vw
├── train.vw.cache
└── tsv2vw.py

\n

0 directories, 10 files

", "rawMarkdown": "", "votes": null }, { "id": "52595", "postDate": "08/28/2014 17:18:15", "content": "

@xbsd The purpose of the initial run is mainly to find the optimal number of passes, and yes, the error. There is no need for manual validation, unless you want to do it differently (for example, use 20% for validation or perform cross-validation).

\n

@Apophenia Then I don't know, maybe you have some old version of VW. Also the printout indicates -b 18 (default), this may mean VW doesn't read the model if you have specified more bits for training.

", "rawMarkdown": "", "votes": null }, { "id": "52599", "postDate": "08/28/2014 19:48:05", "content": "

Thanks for your reply.

\n

I'm using version 7.4. And I tried training using default number of bits but still had the same problem.

\n

-----problem resolved-----

\n

It seems that I didn't build vw correctly. It worked after I reinstall vw using Homebrew. So, for OSX users, install vw by running brew install vowpal-wabbit instead of building it by yourself.

", "rawMarkdown": "", "votes": null }, { "id": "52610", "postDate": "08/29/2014 04:14:40", "content": "

@Foxtrot,

\n

The passes arg is tuned w.r.t the log-loss. Say I want to tune it w.r.t to the true evaluation metric, i.e., AP@k, what approach would you recommend?

", "rawMarkdown": "", "votes": null }, { "id": "52630", "postDate": "08/29/2014 12:57:36", "content": "

@yr Good point. I'd say log-loss is a pretty good proxy. You could try using quantile loss, specifically for ranking, it gave me similiar, slightly worse, results. Or a more involved approach: train for one pass, validate, resume training. Again, see this:

\n

http://fastml.com/vowpal-wabbit-eats-big-data-from-the-criteo-competition-for-breakfast/

", "rawMarkdown": "", "votes": null }, { "id": "52631", "postDate": "08/29/2014 13:57:55", "content": "

[quote=Apophenia;52599]

\n

Thanks for your reply.

\n

I'm using version 7.4. And I tried training using default number of bits but still had the same problem.

\n

-----problem resolved-----

\n

It seems that I didn't build vw correctly. It worked after I reinstall vw using Homebrew. So, for OSX users, install vw by running brew install vowpal-wabbit instead of building it by yourself.

\n

[/quote]

\n

Hmm.  VW builds fine on OSX for me.  You just git clone'd and then ran make and make install, correct?

\n\n

git clone git://github.com/JohnLangford/vowpal_wabbit.git

\n

cd vowpal_wabbit

\n

./autogen.sh

\n

make

\n

make install

\n\n

Should do the trick.  You do need to have Xcode+command line tools already.

\n

Brew installs are fine, too, it's just not as easy to pull in fixes and the latest updates.

\n\n

Edited to add the tip from xbsd...

", "rawMarkdown": "", "votes": null }, { "id": "52636", "postDate": "08/29/2014 15:23:18", "content": "

[quote=Phil Culliton;52631]

\n

[quote=Apophenia;52599]

\n

Thanks for your reply.

\n

I'm using version 7.4. And I tried training using default number of bits but still had the same problem.

\n

-----problem resolved-----

\n

It seems that I didn't build vw correctly. It worked after I reinstall vw using Homebrew. So, for OSX users, install vw by running brew install vowpal-wabbit instead of building it by yourself.

\n

[/quote]

\n

Hmm.  VW builds fine on OSX for me.  You just git clone'd and then ran make and make install, correct?

\n

git clone git://github.com/JohnLangford/vowpal_wabbit.git

\n

cd vowpal_wabbit

\n

make

\n

make install

\n

Should do the trick.  You do need to have Xcode+command line tools already.

\n

Brew installs are fine, too, it's just not as easy to pull in fixes and the latest updates.

\n

[/quote]

\n

Thanks. I tried this first, but some dependencies required by vw were not correctly installed so that vw failed(or falsely succeeded) to build.

", "rawMarkdown": "", "votes": null }, { "id": "52637", "postDate": "08/29/2014 15:28:58", "content": "

git clone

\n

Run ./autogen.sh

\n

Then run make; make install

\n

You will need Boost if it is not already there but autogen should give some indication

", "rawMarkdown": "", "votes": null }, { "id": "52638", "postDate": "08/29/2014 15:40:44", "content": "

Thanks!  Forgot I did that initially.

", "rawMarkdown": "", "votes": null }, { "id": "52646", "postDate": "08/29/2014 18:05:12", "content": "

It would be great if you could wait the end of the competition before giving a solution that gives results way above the benchmark. It's kind of annoying to see the LB spoiled.

", "rawMarkdown": "", "votes": null }, { "id": "52686", "postDate": "08/30/2014 11:23:54", "content": "

@Pourquoipas I can see your point but I view the matter differently. I profoundly enjoy "beating the benchmark" threads, whether I'm the author or someone posts a solution better than mine. And after the contest it's time for the winners to show their hand (if they choose to do so).

", "rawMarkdown": "", "votes": null }, { "id": "52725", "postDate": "08/30/2014 23:06:11", "content": "

Hi  friends,

\n

When I ran the given code (without any change) to transfer tsv file to vw file, I came across the following syntax error. Any ideas? Thanks in advance!

\n

$ python tsv2vw.py train.tsv train.vw
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
100000
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
200000
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
300000
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
Traceback (most recent call last):
File "tsv2vw.py", line 20, in <module>
new_line = construct_line( line )
File "/home/Shize/vowpal_wabbit/vowpalwabbit/vw/construct_line.py", line 54, in construct_line
attrs = eval( line['attrs'] )
File "<string>", line 1
{"Тип объявления":"Сдам", "Количество комнат":"2", "Срок аренды":"На длительный срок", "Этаж":"Не первый и не последний", "Этаж":"7", "Этажей в доме":"9-12", "Этажей в доме":"10", "Тип дома":"Кирпичный", "Адрес":"ул.10-й пятилетки,64 (м/н /"Турист/" /"}

\n

^
SyntaxError: invalid syntax

\n

P.S.: running this command indeed output a vw file in correct form, but with much smaller size (with only about 300,000 examples)

\n

Best wishes,

\n

Shize

", "rawMarkdown": "", "votes": null }, { "id": "52726", "postDate": "08/31/2014 01:24:15", "content": "

Yes, you have to hand-edit some of the attrs. For eg, ""name"":""some text /""some escaped text/"""" ...will become ""name"":""some text some escaped text"" ... basically remove the occurrences of /"" or /" within the value of the key-value pair. Alternatively, you could write some regex, but hand-editing the file incrementally was faster.

", "rawMarkdown": "", "votes": null }, { "id": "52742", "postDate": "08/31/2014 10:06:32", "content": "

[quote=Foxtrot;52686]

\n

@Pourquoipas I can see your point but I view the matter differently. I profoundly enjoy "beating the benchmark" threads, whether I'm the author or someone posts a solution better than mine. And after the contest it's time for the winners to show their hand (if they choose to do so).

\n

[/quote]

\n

I also like beating the benchmark threads when posted at the begining of the competition. It's a good way to start. However, when posted at the end, it can penalize people who've worked on the competition for a long time.

", "rawMarkdown": "", "votes": null }, { "id": "52746", "postDate": "08/31/2014 12:32:15", "content": "

"json.loads() failed, trying eval()..." is actually output from the script, just for information. Don't know about the syntax error at the end!

", "rawMarkdown": "", "votes": null }, { "id": "52748", "postDate": "08/31/2014 13:41:24", "content": "

--passes in vw

\n

The more the better.
Is it right?

", "rawMarkdown": "", "votes": null }, { "id": "52752", "postDate": "08/31/2014 14:25:21", "content": "

no if you want to avoid overfitting.

\n

there is an early-stopping (relatively new feature) but I'm not sure exactly how it works.

", "rawMarkdown": "", "votes": null }, { "id": "52754", "postDate": "08/31/2014 15:03:33", "content": "

Hm... I can't overfit in my local tests...

", "rawMarkdown": "", "votes": null }, { "id": "52755", "postDate": "08/31/2014 15:13:49", "content": "

[quote=Alexander D'yakonov;52748]

\n

--passes in vw

\n

The more the better.
Is it right?

\n

[/quote]

\n

[quote=clustifier;52752]

\n

no if you want to avoid overfitting.

\n

[/quote]

\n

It depends.

\n

Like other SG-methods, with high learning rate and\\or weak regularization its possible to overfit. With small learning rate and\\or strong regularization - more is better.

\n\n

Is it right? =)

", "rawMarkdown": "", "votes": null }, { "id": "52854", "postDate": "09/01/2014 14:38:48", "content": "

@clustifier I have written about early stopping earlier in this thread, look for "holdout".

\n

@Alexander Here's how to overfit:

\n

$ vw -b 29 --loss_function logistic -c -P 1e6 train.vw --passes 30 --holdout_off

\n

...

\n

0.020554 0.014088 110000000 110000000.0 -1.0000 -16.3854 132
0.020494 0.013932 111000000 111000000.0 -1.0000 -16.0693 46
0.020437 0.014074 112000000 112000000.0 -1.0000 -6.4644 19
0.020377 0.013637 113000000 113000000.0 -1.0000 -5.1424 9
0.020320 0.013909 114000000 114000000.0 -1.0000 -8.8841 39
0.020263 0.013750 115000000 115000000.0 -1.0000 -12.8807 27
0.020208 0.013882 116000000 116000000.0 -1.0000 -3.4141 26
0.020150 0.013475 117000000 117000000.0 -1.0000 -22.4390 31
0.020096 0.013758 118000000 118000000.0 -1.0000 -23.9844 187
0.020041 0.013557 119000000 119000000.0 -1.0000 -6.8467 17

\n

finished run
number of examples per pass = 3995804
passes used = 30
weighted example sum = 1.19874e+08
weighted label sum = -1.03374e+08
average loss = 0.0199952
best constant = -0.862358
total feature number = 6113723760

\n

With early stopping the holdout score won't get beneath 0.03.

\n

@Mikhail I'm not sure how learning rate relates to overfitting... I'd imagine a smaller learning rate would make overfitting easier in the end.

", "rawMarkdown": "", "votes": null }, { "id": "52863", "postDate": "09/01/2014 16:26:41", "content": "

I don't know if neural network reduction is better now, or if it always worked this great, anyway, a lot of gain was found in the --nn parameter. Maybe not surprising since the inspiration to add it to VW was to win some Kaggle competitions with VW. http://www.machinedlearnings.com/2013/02/one-louder.html http://www.machinedlearnings.com/2012/11/unpimp-your-sigmoid.html

\n

Also: Namespacing features did not help, but hamper our score. We did not treat the dataset with much respect: One bag of features, trying to encode non-text tokens as floats, and encoding all tokens as categorical: year:2009 year_2009:1 category_cars etc.

\n

using ngrams or nskips then almost becomes a lightweight version of -q quadratic features. With the right regularization and high enough bitsize, I think VW could even handle those (perhaps cubic is a step too far).

\n

As for learning rate and overfitting: I think after so many passes the learning rate is so low, so a few extra passes do not matter much for adjusting feature weights (and hence overfit). With older versions of VW I ran 300 passes for a slight increase in the leaderboards. I thought hold_out was more to get a realistic score of average loss (not a skewed value approaching 0). Using bootstrap and nn functionality seems to do well with fewer or even single passes.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 52478, "author_name": "abhishek", "author_url": "", "post_date": "08/26/2014 21:09:43", "content": "

You stole my copyrighted topic name :P lol

", "votes": null, "replies": [] }, { "id": 52479, "author_name": "zygmunt", "author_url": "", "post_date": "08/26/2014 21:49:43", "content": "

[quote=Abhishek;52478]

\n

You stole my copyrighted topic name :P lol

\n

[/quote]

\n

Man, you compete too much, your memory is failing :P

", "votes": null, "replies": [] }, { "id": 52480, "author_name": "adjgiulio", "author_url": "", "post_date": "08/26/2014 22:20:47", "content": "

[quote=Abhishek;52478]

\n

You stole my copyrighted topic name :P lol

\n

[/quote]

\n

No, your copyrighted topic name is "Beating the benchmark :-)"

\n

;-)

", "votes": null, "replies": [] }, { "id": 52483, "author_name": "snisioi", "author_url": "", "post_date": "08/26/2014 23:49:24", "content": "

[quote=Foxtrot;52477]

\n

Good news, everyone! Here's the code that got me ~0.971:

\n

https://github.com/zygmuntz/kaggle-avito

\n

It uses Vowpal Wabbit and comes with instructions. Remember to click thanks if you find it interesting or useful.

\n

[/quote]

\n

Additional friendly advice: splitting and training for subcategories might produce a better score (by merging the results). The trouble is that some subcategories don't have enough training examples. 

", "votes": null, "replies": [] }, { "id": 52512, "author_name": "sirguessalot", "author_url": "", "post_date": "08/27/2014 16:21:59", "content": "

[quote=Foxtrot;52477]

\n

Good news, everyone! Here's the code that got me ~0.971:

\n

https://github.com/zygmuntz/kaggle-avito

\n

It uses Vowpal Wabbit and comes with instructions. Remember to click thanks if you find it interesting or useful.

\n

[/quote]

\n

Hi Zygmunt, can you explain the logic behind setting label to -1 when it's 0 in the train data? Is this is a VW convention or a choice you made?

", "votes": null, "replies": [] }, { "id": 52513, "author_name": "zygmunt", "author_url": "", "post_date": "08/27/2014 16:26:34", "content": "

[quote=Momchil Georgiev;52512]

\n

Hi Zygmunt, can you explain the logic behind setting label to -1 when it's 0 in the train data? Is this is a VW convention or a choice you made?

\n

[/quote]

\n

Vowpal Wabbit needs 1/-1 labels for binary classification, otherwise you get an error.

", "votes": null, "replies": [] }, { "id": 52514, "author_name": "abhishek", "author_url": "", "post_date": "08/27/2014 16:27:16", "content": "

Thats a very high benchmark!  :(

", "votes": null, "replies": [] }, { "id": 52515, "author_name": "triskelion", "author_url": "", "post_date": "08/27/2014 16:28:30", "content": "

[quote=Momchil Georgiev;52512]

\n

[quote=Foxtrot;52477]

\n

Good news, everyone! Here's the code that got me ~0.971:

\n

https://github.com/zygmuntz/kaggle-avito

\n

It uses Vowpal Wabbit and comes with instructions. Remember to click thanks if you find it interesting or useful.

\n

[/quote]

\n

Hi Zygmunt, can you explain the logic behind setting label to -1 when it's 0 in the train data? Is this is a VW convention or a choice you made?

\n

[/quote]

\n

Logistic regression (and hinge loss) needs labels to be [-1,1] for binary classification tasks. So it is aVW convention.

", "votes": null, "replies": [] }, { "id": 52516, "author_name": "sirguessalot", "author_url": "", "post_date": "08/27/2014 16:46:51", "content": "

[quote=Foxtrot;52513]

\n

[quote=Momchil Georgiev;52512]

\n

Hi Zygmunt, can you explain the logic behind setting label to -1 when it's 0 in the train data? Is this is a VW convention or a choice you made?

\n

[/quote]

\n

Vowpal Wabbit needs 1/-1 labels for binary classification, otherwise you get an error.

\n

[/quote]

\n

Thanks! I'm very new to VW but it looks like an awesome tool so I'm trying to learn it. What about the -holdout_off parameter? On my version of VW it complained that it was not a valid flag.

", "votes": null, "replies": [] }, { "id": 52517, "author_name": "zygmunt", "author_url": "", "post_date": "08/27/2014 17:03:50", "content": "

[quote=Momchil Georgiev;52516]

\n

Thanks! I'm very new to VW but it looks like an awesome tool so I'm trying to learn it. What about the -holdout_off parameter? On my version of VW it complained that it was not a valid flag.

\n

[/quote]

\n

This option was introduced recently. When doing multiple passes over data, VW reserves 10% for validation - that's a holdout set. The idea is to turn it off to use all available examples for training. However in case of this competition the difference in the score is minor.

\n

P.S. Moar here: http://fastml.com/vowpal-wabbit-eats-big-data-from-the-criteo-competition-for-breakfast/

", "votes": null, "replies": [] }, { "id": 52529, "author_name": "apophenia", "author_url": "", "post_date": "08/27/2014 18:40:57", "content": "

Thanks for sharing.

\n

I would love to learn about VW and tried running your code, but it got stuck at predicting stage. When I ran vw -t -i model -d test.vw -p predicshuns.txt, it only shows following logs but nothing more:

\n

For more information use: vw --help
Num weight bits = 18
learning rate = 0.5
initial_t = 0
power_t = 0.5
using no cache
Reading datafile =
num sources = 1
average since example example current current current
loss last counter weight label predict features

", "votes": null, "replies": [] }, { "id": 52563, "author_name": "dasgupta", "author_url": "", "post_date": "08/28/2014 12:14:27", "content": "

Vowpal is just amazing. The more I use it, the more impressed I get. Online learning is in a different class of its own. @Foxtrot, thanks for sharing your insights and code examples, it is very helpful in understanding how VW works.

", "votes": null, "replies": [] }, { "id": 52567, "author_name": "zygmunt", "author_url": "", "post_date": "08/28/2014 12:49:39", "content": "

@xbsd You're welcome.

\n

@Apophenia Check if you have the files (model, test.vw) in place.

", "votes": null, "replies": [] }, { "id": 52581, "author_name": "dasgupta", "author_url": "", "post_date": "08/28/2014 14:28:53", "content": "

Question @Foxtrot, just wondering what is the purpose of not using holdout_off in the final training. I believe this is due to the fact that we want to use the entire training set for the final training and for the initial run (where we do not specify a -f model) we use that to measure the average loss only before we do the final training.

\n\n

Since in the case where we do not specify a -f model or use -holdout_off VW is already measuring the performance using the holdout set, is there a need for a separate cross-validation (like in R/Python where it is more batch learning) ? Thanks.

", "votes": null, "replies": [] }, { "id": 52585, "author_name": "apophenia", "author_url": "", "post_date": "08/28/2014 15:19:29", "content": "

@Foxtrot Yes, I have them all in place.

\n

kaggle-avito-master/
├── LICENSE
├── README.md
├── load_attribs.py
├── model
├── predict.py
├── score.py
├── test.vw
├── train.vw
├── train.vw.cache
└── tsv2vw.py

\n

0 directories, 10 files

", "votes": null, "replies": [] }, { "id": 52595, "author_name": "zygmunt", "author_url": "", "post_date": "08/28/2014 17:18:15", "content": "

@xbsd The purpose of the initial run is mainly to find the optimal number of passes, and yes, the error. There is no need for manual validation, unless you want to do it differently (for example, use 20% for validation or perform cross-validation).

\n

@Apophenia Then I don't know, maybe you have some old version of VW. Also the printout indicates -b 18 (default), this may mean VW doesn't read the model if you have specified more bits for training.

", "votes": null, "replies": [] }, { "id": 52599, "author_name": "apophenia", "author_url": "", "post_date": "08/28/2014 19:48:05", "content": "

Thanks for your reply.

\n

I'm using version 7.4. And I tried training using default number of bits but still had the same problem.

\n

-----problem resolved-----

\n

It seems that I didn't build vw correctly. It worked after I reinstall vw using Homebrew. So, for OSX users, install vw by running brew install vowpal-wabbit instead of building it by yourself.

", "votes": null, "replies": [] }, { "id": 52610, "author_name": "chenglongchen", "author_url": "", "post_date": "08/29/2014 04:14:40", "content": "

@Foxtrot,

\n

The passes arg is tuned w.r.t the log-loss. Say I want to tune it w.r.t to the true evaluation metric, i.e., AP@k, what approach would you recommend?

", "votes": null, "replies": [] }, { "id": 52630, "author_name": "zygmunt", "author_url": "", "post_date": "08/29/2014 12:57:36", "content": "

@yr Good point. I'd say log-loss is a pretty good proxy. You could try using quantile loss, specifically for ranking, it gave me similiar, slightly worse, results. Or a more involved approach: train for one pass, validate, resume training. Again, see this:

\n

http://fastml.com/vowpal-wabbit-eats-big-data-from-the-criteo-competition-for-breakfast/

", "votes": null, "replies": [] }, { "id": 52631, "author_name": "philculliton", "author_url": "", "post_date": "08/29/2014 13:57:55", "content": "

[quote=Apophenia;52599]

\n

Thanks for your reply.

\n

I'm using version 7.4. And I tried training using default number of bits but still had the same problem.

\n

-----problem resolved-----

\n

It seems that I didn't build vw correctly. It worked after I reinstall vw using Homebrew. So, for OSX users, install vw by running brew install vowpal-wabbit instead of building it by yourself.

\n

[/quote]

\n

Hmm.  VW builds fine on OSX for me.  You just git clone'd and then ran make and make install, correct?

\n\n

git clone git://github.com/JohnLangford/vowpal_wabbit.git

\n

cd vowpal_wabbit

\n

./autogen.sh

\n

make

\n

make install

\n\n

Should do the trick.  You do need to have Xcode+command line tools already.

\n

Brew installs are fine, too, it's just not as easy to pull in fixes and the latest updates.

\n\n

Edited to add the tip from xbsd...

", "votes": null, "replies": [] }, { "id": 52636, "author_name": "apophenia", "author_url": "", "post_date": "08/29/2014 15:23:18", "content": "

[quote=Phil Culliton;52631]

\n

[quote=Apophenia;52599]

\n

Thanks for your reply.

\n

I'm using version 7.4. And I tried training using default number of bits but still had the same problem.

\n

-----problem resolved-----

\n

It seems that I didn't build vw correctly. It worked after I reinstall vw using Homebrew. So, for OSX users, install vw by running brew install vowpal-wabbit instead of building it by yourself.

\n

[/quote]

\n

Hmm.  VW builds fine on OSX for me.  You just git clone'd and then ran make and make install, correct?

\n

git clone git://github.com/JohnLangford/vowpal_wabbit.git

\n

cd vowpal_wabbit

\n

make

\n

make install

\n

Should do the trick.  You do need to have Xcode+command line tools already.

\n

Brew installs are fine, too, it's just not as easy to pull in fixes and the latest updates.

\n

[/quote]

\n

Thanks. I tried this first, but some dependencies required by vw were not correctly installed so that vw failed(or falsely succeeded) to build.

", "votes": null, "replies": [] }, { "id": 52637, "author_name": "dasgupta", "author_url": "", "post_date": "08/29/2014 15:28:58", "content": "

git clone

\n

Run ./autogen.sh

\n

Then run make; make install

\n

You will need Boost if it is not already there but autogen should give some indication

", "votes": null, "replies": [] }, { "id": 52638, "author_name": "philculliton", "author_url": "", "post_date": "08/29/2014 15:40:44", "content": "

Thanks!  Forgot I did that initially.

", "votes": null, "replies": [] }, { "id": 52646, "author_name": "coucouv", "author_url": "", "post_date": "08/29/2014 18:05:12", "content": "

It would be great if you could wait the end of the competition before giving a solution that gives results way above the benchmark. It's kind of annoying to see the LB spoiled.

", "votes": null, "replies": [] }, { "id": 52686, "author_name": "zygmunt", "author_url": "", "post_date": "08/30/2014 11:23:54", "content": "

@Pourquoipas I can see your point but I view the matter differently. I profoundly enjoy "beating the benchmark" threads, whether I'm the author or someone posts a solution better than mine. And after the contest it's time for the winners to show their hand (if they choose to do so).

", "votes": null, "replies": [] }, { "id": 52725, "author_name": "sushize", "author_url": "", "post_date": "08/30/2014 23:06:11", "content": "

Hi  friends,

\n

When I ran the given code (without any change) to transfer tsv file to vw file, I came across the following syntax error. Any ideas? Thanks in advance!

\n

$ python tsv2vw.py train.tsv train.vw
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
100000
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
200000
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
300000
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
json.loads() failed, trying eval()...
Traceback (most recent call last):
File "tsv2vw.py", line 20, in <module>
new_line = construct_line( line )
File "/home/Shize/vowpal_wabbit/vowpalwabbit/vw/construct_line.py", line 54, in construct_line
attrs = eval( line['attrs'] )
File "<string>", line 1
{"Тип объявления":"Сдам", "Количество комнат":"2", "Срок аренды":"На длительный срок", "Этаж":"Не первый и не последний", "Этаж":"7", "Этажей в доме":"9-12", "Этажей в доме":"10", "Тип дома":"Кирпичный", "Адрес":"ул.10-й пятилетки,64 (м/н /"Турист/" /"}

\n

^
SyntaxError: invalid syntax

\n

P.S.: running this command indeed output a vw file in correct form, but with much smaller size (with only about 300,000 examples)

\n

Best wishes,

\n

Shize

", "votes": null, "replies": [] }, { "id": 52726, "author_name": "dasgupta", "author_url": "", "post_date": "08/31/2014 01:24:15", "content": "

Yes, you have to hand-edit some of the attrs. For eg, ""name"":""some text /""some escaped text/"""" ...will become ""name"":""some text some escaped text"" ... basically remove the occurrences of /"" or /" within the value of the key-value pair. Alternatively, you could write some regex, but hand-editing the file incrementally was faster.

", "votes": null, "replies": [] }, { "id": 52742, "author_name": "coucouv", "author_url": "", "post_date": "08/31/2014 10:06:32", "content": "

[quote=Foxtrot;52686]

\n

@Pourquoipas I can see your point but I view the matter differently. I profoundly enjoy "beating the benchmark" threads, whether I'm the author or someone posts a solution better than mine. And after the contest it's time for the winners to show their hand (if they choose to do so).

\n

[/quote]

\n

I also like beating the benchmark threads when posted at the begining of the competition. It's a good way to start. However, when posted at the end, it can penalize people who've worked on the competition for a long time.

", "votes": null, "replies": [] }, { "id": 52746, "author_name": "zygmunt", "author_url": "", "post_date": "08/31/2014 12:32:15", "content": "

"json.loads() failed, trying eval()..." is actually output from the script, just for information. Don't know about the syntax error at the end!

", "votes": null, "replies": [] }, { "id": 52748, "author_name": "dyakonov", "author_url": "", "post_date": "08/31/2014 13:41:24", "content": "

--passes in vw

\n

The more the better.
Is it right?

", "votes": null, "replies": [] }, { "id": 52752, "author_name": "clustifier", "author_url": "", "post_date": "08/31/2014 14:25:21", "content": "

no if you want to avoid overfitting.

\n

there is an early-stopping (relatively new feature) but I'm not sure exactly how it works.

", "votes": null, "replies": [] }, { "id": 52754, "author_name": "dyakonov", "author_url": "", "post_date": "08/31/2014 15:03:33", "content": "

Hm... I can't overfit in my local tests...

", "votes": null, "replies": [] }, { "id": 52755, "author_name": "mikhailtrofimov", "author_url": "", "post_date": "08/31/2014 15:13:49", "content": "

[quote=Alexander D'yakonov;52748]

\n

--passes in vw

\n

The more the better.
Is it right?

\n

[/quote]

\n

[quote=clustifier;52752]

\n

no if you want to avoid overfitting.

\n

[/quote]

\n

It depends.

\n

Like other SG-methods, with high learning rate and\\or weak regularization its possible to overfit. With small learning rate and\\or strong regularization - more is better.

\n\n

Is it right? =)

", "votes": null, "replies": [] }, { "id": 52854, "author_name": "zygmunt", "author_url": "", "post_date": "09/01/2014 14:38:48", "content": "

@clustifier I have written about early stopping earlier in this thread, look for "holdout".

\n

@Alexander Here's how to overfit:

\n

$ vw -b 29 --loss_function logistic -c -P 1e6 train.vw --passes 30 --holdout_off

\n

...

\n

0.020554 0.014088 110000000 110000000.0 -1.0000 -16.3854 132
0.020494 0.013932 111000000 111000000.0 -1.0000 -16.0693 46
0.020437 0.014074 112000000 112000000.0 -1.0000 -6.4644 19
0.020377 0.013637 113000000 113000000.0 -1.0000 -5.1424 9
0.020320 0.013909 114000000 114000000.0 -1.0000 -8.8841 39
0.020263 0.013750 115000000 115000000.0 -1.0000 -12.8807 27
0.020208 0.013882 116000000 116000000.0 -1.0000 -3.4141 26
0.020150 0.013475 117000000 117000000.0 -1.0000 -22.4390 31
0.020096 0.013758 118000000 118000000.0 -1.0000 -23.9844 187
0.020041 0.013557 119000000 119000000.0 -1.0000 -6.8467 17

\n

finished run
number of examples per pass = 3995804
passes used = 30
weighted example sum = 1.19874e+08
weighted label sum = -1.03374e+08
average loss = 0.0199952
best constant = -0.862358
total feature number = 6113723760

\n

With early stopping the holdout score won't get beneath 0.03.

\n

@Mikhail I'm not sure how learning rate relates to overfitting... I'd imagine a smaller learning rate would make overfitting easier in the end.

", "votes": null, "replies": [] }, { "id": 52863, "author_name": "triskelion", "author_url": "", "post_date": "09/01/2014 16:26:41", "content": "

I don't know if neural network reduction is better now, or if it always worked this great, anyway, a lot of gain was found in the --nn parameter. Maybe not surprising since the inspiration to add it to VW was to win some Kaggle competitions with VW. http://www.machinedlearnings.com/2013/02/one-louder.html http://www.machinedlearnings.com/2012/11/unpimp-your-sigmoid.html

\n

Also: Namespacing features did not help, but hamper our score. We did not treat the dataset with much respect: One bag of features, trying to encode non-text tokens as floats, and encoding all tokens as categorical: year:2009 year_2009:1 category_cars etc.

\n

using ngrams or nskips then almost becomes a lightweight version of -q quadratic features. With the right regularization and high enough bitsize, I think VW could even handle those (perhaps cubic is a step too far).

\n

As for learning rate and overfitting: I think after so many passes the learning rate is so low, so a few extra passes do not matter much for adjusting feature weights (and hence overfit). With older versions of VW I ran 300 passes for a slight increase in the leaderboards. I thought hold_out was more to get a realistic score of average loss (not a skewed value approaching 0). Using bootstrap and nn functionality seems to do well with fewer or even single passes.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "52477": "", "52478": "", "52479": "", "52480": "", "52483": "", "52512": "", "52513": "", "52514": "", "52515": "", "52516": "", "52517": "", "52529": "", "52563": "", "52567": "", "52581": "", "52585": "", "52595": "", "52599": "", "52610": "", "52630": "", "52631": "", "52636": "", "52637": "", "52638": "", "52646": "", "52686": "", "52725": "", "52726": "", "52742": "", "52746": "", "52748": "", "52752": "", "52754": "", "52755": "", "52854": "", "52863": "" }, "source": "meta" }