{ "id": 10178, "title": "Congrats Barisumog & Giulio", "url": "/competitions/avito-prohibited-content/discussion/10178", "author_name": "", "post_date": "2014-09-01T00:22:48.823Z", "votes": 4, "comment_count": 38, "views": 14978, "content": "

Congrats guys! You deserved it!!!

\n

Special congrats to Barisumog for becoming a Master!!!

\n

Looking forward to reading about your approach!

", "messages": [ { "id": "52795", "postDate": "09/01/2014 00:22:48", "content": "

Congrats guys! You deserved it!!!

\n

Special congrats to Barisumog for becoming a Master!!!

\n

Looking forward to reading about your approach!

", "rawMarkdown": "", "votes": null }, { "id": "52798", "postDate": "09/01/2014 00:31:07", "content": "

Congratulations! 

", "rawMarkdown": "", "votes": null }, { "id": "52800", "postDate": "09/01/2014 00:42:28", "content": "

Grats all. Grats to Abhishek for becoming top 10 Kaggler!

\n

And grats to Barisumog for becoming Master Kaggler. That's absolutely what that impressive profile deserved!

\n

Grats to my team members Phil Culliton and Jules, for reaching top 10 position together, and getting Master Kaggler status.

", "rawMarkdown": "", "votes": null }, { "id": "52802", "postDate": "09/01/2014 00:59:48", "content": "

Great work guys! Very curious to see what you found that led to the separation.

\n\n

Did anyone find a good way to use the is_proved targets? I couldn't get them to add any value to my models.

", "rawMarkdown": "", "votes": null }, { "id": "52806", "postDate": "09/01/2014 01:21:00", "content": "

Congrat winners! Congrat Barisumog with Master Tier!

", "rawMarkdown": "", "votes": null }, { "id": "52807", "postDate": "09/01/2014 01:33:31", "content": "

[quote=dkay;52802]

\n

Great work guys! Very curious to see what you found that led to the separation.

\n

Did anyone find a good way to use the is_proved targets? I couldn't get them to add any value to my models.

\n

[/quote]

\n

Jules found two ways: train a VW model where the positive class is samples where is_blocked ==1 AND is_proved == 1.

\n

Also as part of an illicit score to use for (quantile) regression: Something like: is_blocked + 0.4*is_proved + close_hours_score.

\n

This improved ensembles where we simply averaged the ranks of the submission files.

", "rawMarkdown": "", "votes": null }, { "id": "52809", "postDate": "09/01/2014 02:33:55", "content": "

Congrat winners! It's a wonderful game,i also learned a lot,Thanks to the organizers

", "rawMarkdown": "", "votes": null }, { "id": "52810", "postDate": "09/01/2014 02:41:44", "content": "

[quote=dkay;52802]

\n

Great work guys! Very curious to see what you found that led to the separation.

\n

Did anyone find a good way to use the is_proved targets? I couldn't get them to add any value to my models.

\n

[/quote]

\n

I trained two models, with all the blocked ads (mixed model) and only blocked ads labeled by experienced moderator (proved model). This improve my results by 0.001~0.002. My best private score is 0.98589 which is simple averaging by 2 mixed models and 2 proved models of 2 different set of features.

\n

This is the first time I come so close to top3!

\n

Congrats to everyone! Congrats to Barisumog & Giulio! Well done!

\n

Update: I initially entered this competition to familiarize myself with VW and Linux & Shell (I used to be a Windows user). If anyone interested in how to get to 0.985x using VW, I have put all my code here. 

\n

Finally, I would also want to thank @Triskelion and @Foxtrot for introducing me to VW :)

", "rawMarkdown": "", "votes": null }, { "id": "52811", "postDate": "09/01/2014 02:51:14", "content": "

[quote=Triskelion;52807]

\n

[quote=dkay;52802]

\n

Great work guys! Very curious to see what you found that led to the separation.

\n

Did anyone find a good way to use the is_proved targets? I couldn't get them to add any value to my models.

\n

[/quote]

\n

Jules found two ways: train a VW model where the positive class is samples where is_blocked ==1 AND is_proved == 1.

\n

Also as part of an illicit score to use for (quantile) regression: Something like: is_blocked + 0.4*is_proved + close_hours_score.

\n

This improved ensembles where we simply averaged the ranks of the submission files.

\n

[/quote]

\n\n

Very interesting. Just curious, did you use VW exclusively for model building ? And, re: training for examples where is_blocked and is_blocked == 1, did you use any method to find this. Thanks.

", "rawMarkdown": "", "votes": null }, { "id": "52812", "postDate": "09/01/2014 02:58:03", "content": "

Great competition, lots of fun and awesome learning opportunity! Thanks Kaggle and Avito!

\n

I’ve got to first congratulate everyone who fought hard throughout this competition, other winners, top-10, and new Masters! Well done everyone! 
As for the approach, I think that what we did will not come as a shocker to anyone.

\n

My model was essentially a series of SGD on pieces of the text data (one for title, one for description, one for title+description, one for attrs). This alone gets you around .97 on the public LB. By adding to the tfidf matrices one-hot-encoded category and subcategory the models score well above .97.
The outputs from all the SGDs were then fed into a tree model (RF worked better than GBM) along with some other features, most importantly category and subcategory, but also email, url and phone number counts. This piece really helped the model re-learn the differences among categories and subcategories and re-sort the original SGD predictions. This added another .002-3ish and takes you in .983 territory.
Since plain accuracy was so high across the dataset, semi-supervised test data added in the order of .003 to the train data alone bringing my best model around .986.
All of this was built on the entire data minus real estate, which barisimug found out could be removed not only without any loss, but with a sizeable increase in performance.

\n

I'll let barisumog describe his models but in his approach he modelled each category and subcategory independently, later stacking results.

\n

Our winner model is a simple rank based weighed average of my best RF andGBM, and barisumog's ensemble of SVMs.

\n

We used ranks to blend our models mostly because, from a practical perspective, our model outputs were on different scales (mine were probabilities, barisumog’s were distances from the SVM separating hyperplane). In reality I have observed that even amongst my models (who all produced probabilities) ranks were still as good as if not better than probabilities for blends.
My intuition, with no scientific evidence for it, is that when using semi-supervised learning you end up getting many extreme predictions for the test set, with lots of observations clustering around 1 and 0. That makes intuitive sense since you’re predicting observations the algorithm has already seen in training. With these big clusters of 1’s, .9995’s, .9993’s… ranking is introducing some (admittedly arbitrary) randomization, which might act as a good counterweight to the extreme decisiveness of semi-supervised approaches.

\n

Another surprising thing for me was the fact I was getting much less than in other competitions out of ensembles. It really took two very distinct approaches to get decent value out of our team ensemble. And that was also, the key to our success. I'm really curious to see what other approaches folks have come up with, but essentially we had two very good models covering two very different approaches.

\n

Things that did not work for me:
-is_proven added no value to our models
-feature engineering has provided little to no value. I tried a bunch of features derived from text (i.e. count of !, ?, mixed words, length of text,...) and none seemed to add much.
-for the attributes feature, anything more fancy that running it through a tfidf added nothing.
-I tried a second level model where I’d take the top 20% observations and re-model those alone to see if it could help further minimize false positives. Even at different cutoffs, that did not help.

\n

Things that did work for me:
-no need to do anything fancy with the russian text. Just feed it into a tfidf vectorizer and you’re already in better shape than the benchmark code.
-simple semi-supervised learning performed the best. I tried using only portions of the scored test set (i.e. observation with predicted probabilities above .9 or below .1), tried many cutoffs, but nothing was better than using all test. I even tried to use all the test set but weighted its observations based on how close to 0 and 1 the probabilities were, but that also did not add value.

\n

One part of the competition that was a first, and very interesting for someone like me who enjoys the competitive aspect of Kaggle, was the strategic aspect of managing a high LB rank. This is where machine learning becomes strategic/competitive machine learning. :-)

", "rawMarkdown": "", "votes": null }, { "id": "52813", "postDate": "09/01/2014 03:24:43", "content": "

Nice work. It sounds like my method was similar to bariusmog's approach.

\n\n

I used the title + description, and also just title on it's own and blended them together. For both of these I trained with all category data, then separated into single models by category, and also single models based on subcategory in order to get some sort of diversity.

\n\n

With combining in this way, it gave me an extra ~.002  above any on their own. I also used some semi-supervised learning with the test set, but it only gave me ~.001.  I tried some feature engineering as well, but it gave little value.

", "rawMarkdown": "", "votes": null }, { "id": "52814", "postDate": "09/01/2014 04:08:08", "content": "

And congrats David T. on becoming Kaggle's new #2 overall. Seattle should be pretty proud these days :-)

", "rawMarkdown": "", "votes": null }, { "id": "52815", "postDate": "09/01/2014 04:20:31", "content": "

My 2 cents on text features:

\n

Other than unigrams, bigrams were also helpful, but not trigrams. Stemming and correcting words didn't help.

", "rawMarkdown": "", "votes": null }, { "id": "52818", "postDate": "09/01/2014 04:49:24", "content": "

Congratulations to Giulio and Barisumog for winning this interesting competition! And for Abhishek for entering the top 10! Very interesting competition, after a long time I had to use VW again.

", "rawMarkdown": "", "votes": null }, { "id": "52820", "postDate": "09/01/2014 05:44:27", "content": "

Thank you all for the work done! We are carefully watching the competition process and the results are really impressive! We hope that the competition was interesting and honest. Now it's our turn to work hard to implement something like that in our production models...

", "rawMarkdown": "", "votes": null }, { "id": "52824", "postDate": "09/01/2014 06:40:26", "content": "

Congrats winners! Congrat Barisumog for becoming Master!

\n

Congrats Abhishek for entering  into the top 10!

", "rawMarkdown": "", "votes": null }, { "id": "52825", "postDate": "09/01/2014 07:28:42", "content": "

I’ve got to congratulate everyone who participate and especially everyone who break 0.984 =)

\n

Our approach is quite similar to discribed by Giulio. We use different pieces of data (title, title+description, title+description+attrs, title+attrs) and made 3 levels of details for each (top100k word, all word, all pair of words). For all of this features-sets was trained SVM, for some - additional LibFM models. Solely they give 0.97 - 0.983.

\n

All this models was blended by RF, so it gave 0.986.

\n

[quote=Giulio;52812]

\n

simple semi-supervised learning performed the best. I tried using only portions of the scored test set (i.e. observation with predicted probabilities above .9 or below .1), tried many cutoffs, but nothing was better than using all test. I even tried to use all the test set but weighted its observations based on how close to 0 and 1 the probabilities were, but that also did not add value.

\n

[/quote]

\n

Can you explain this in details? It is thing I want to learn! =)

\n

I've tried to use SSL, but got no profit. You predict whole test set, added it to the train, learn from concatenation and again predict test set?

", "rawMarkdown": "", "votes": null }, { "id": "52828", "postDate": "09/01/2014 08:38:48", "content": "

Congrats to everyone who fought hard to the end! And thanks to Kaggle and Avito, it was a fun competition.

\n

About my approach: I don't speak Russian, but running numerous data through Google translate, I could see that some categories used very different language than others. Also the ratio of illicit content varied widely between categories (and subcategories). So I decided to tackle the problem on a category / subcategory basis.

\n

This is what I did, in steps:

\n
    \n
  1. separate the raw data into categories and subcategories
    2. \n
  2. ignore the Real Estate category and related subcategories (vastly different language than other categories, and very tiny ratio of illicit content)
    3. \n
  3. extract raw text from each post by concatenating the title, description and attributes sections (We tried many other features, some worked for Giulio, but none for me. I used only textual features)
    4. \n
  4. for each category and subcategory, create 3 tf-idf matrices: one with raw text, one with stemming, and one with stop words (separately, they gave similar results, but I noticed they improved the score a bit and became more stable when combined)
    5. \n
  5. for each category and subcategory, train 2 sets of SVCs with different C parameters on each tf-idf (again, similar results separately, but slightly better when combined)
    6. \n
  6. so now I have 2 x 3 SVCs for every category, and 2 x 3 SVCs for every subcategory (12 models to use for every data point)
    7. \n
  7. apply semi-supervised learning, which was Giulio's idea and worked quite well. Use the trained models on test data to predict classes. Concatenate train+test, and labels+prediction. Retrain all models on this new merged data set.
    8. \n
  8. finally, use the models to make predictions on the test data. use the SVC output of distance from hyperplane to rank the individual posts
    9. \n
\n

Then Giulio did some of his magic to combine my ranks with his own models. As he already pointed out, the difference of approach between our models resulted in a nice boost when ensembled.

", "rawMarkdown": "", "votes": null }, { "id": "52834", "postDate": "09/01/2014 11:44:21", "content": "

Thanks to the organisers and congrats to all the top finishers!

\n

Special congratulations to winners who avoided overfitting.

\n

Поздравляю лучшую Российскую команду Михаила и Дмитрия!  

", "rawMarkdown": "", "votes": null }, { "id": "52836", "postDate": "09/01/2014 12:14:22", "content": "

[quote=xbsd;52811]

\n

Very interesting. Just curious, did you use VW exclusively for model building ? And, re: training for examples where is_blocked and is_blocked == 1, did you use any method to find this. Thanks.

\n

[/quote]

\n

Yes. Single VW models gave around 97.7, with a lot of tuning single VW models became around 98. Averaging the ranks of a hinge and logloss model improved the score to well over 98, etc. Reading through yr's code (very well done on the VW!) it's possible to get a very high score with very little memory resources and time. So for us no elaborate complex GBM or RF models (though we very much would have used them if we managed or saw improvement).

\n

So for implementation, I think VW could be a solid and fast choice, with simple model introspection. But it's not like these models need retraining every minute, so more complex models are possible too.

\n

I believe Jules found that trick through a hunch. It makes sense that a model trained on different features/samples or labels will hold different information for an ensemble to use.

\n

Vowpal Wabbit eats Russian spam for breakfast. But just to make sure we latinized all Russian characters, so we could dream to avoid further encoding issues on Windows.

\n

The semi-supervised trick from Giulio is interesting. Makes perfect sense to try it out, but I was thinking about this for other competitions, not this one. I think it has its own risks associated with it (I call it over-learning).

", "rawMarkdown": "", "votes": null }, { "id": "52837", "postDate": "09/01/2014 12:26:03", "content": "

A single libfm on raw data (one-hot with minOccurence=5 of category,subcategory,price,phones_cnt,emails_cnt,urls_cnt) gave me 0.98073 public and 0.98103 private.

\n

Pretty good for such a simple attempt.

", "rawMarkdown": "", "votes": null }, { "id": "52838", "postDate": "09/01/2014 12:46:26", "content": "

[quote=Michael Jahrer;52837]

\n

A single libfm on raw data (one-hot with minOccurence=5 of category,subcategory,price,phones_cnt,emails_cnt,urls_cnt) gave me 0.98073 public and 0.98103 private.

\n

Pretty good for such a simple attempt.

\n

[/quote]

\n

Wow, that is really good. Do I understand you correctly in that you did not use the text at all?

", "rawMarkdown": "", "votes": null }, { "id": "52841", "postDate": "09/01/2014 12:55:22", "content": "

[quote=Triskelion;52838]

\n

[quote=Michael Jahrer;52837]

\n

A single libfm on raw data (one-hot with minOccurence=5 of category,subcategory,price,phones_cnt,emails_cnt,urls_cnt) gave me 0.98073 public and 0.98103 private.

\n

Pretty good for such a simple attempt.

\n

[/quote]

\n

Wow, that is really good. Do I understand you correctly in that you did not use the text at all?

\n

[/quote]

\n

I checked my feature generation code (several weeks ago), and yes sorry I forgot to mentioned the tokens in these 3 cols: title,description,attr. So yes I used this text tokens.

", "rawMarkdown": "", "votes": null }, { "id": "52843", "postDate": "09/01/2014 13:03:20", "content": "

I used vw exclusively, experimenting with the parameters, especially --passes and --decay_learning_rate. Surprisingly, the number of passes had a large effect, even with a very steep decay. For example, the difference between 15 and 16 passes, with the decay at 0.5, could be as much as .002. Also, the average variance between the LB and PB scores was about .002 and was quite erratic. My final two choices were my best LB model (.98461), which scored only .98200 on the PB, and an ensemble of 10 vw models, which scored .98404 on the LB and .98291 on the PB. My best PB score was the same ensemble, but mixed based on rank instead of raw scores, which scored .98431 on the PB but only .98317 on the LB.

\n

All in all, I had a great time learning about vw, which is an incredibly powerful and fast tool. It is difficult, however, to set up a robust cross-validation system with vw. For example, I found that the  built-in stopping mechanism on passes tended to underestimate the optimal number of passes. For example, my models generally stopped after 8, or sometimes 9, passes, but I found that letting it run for 14-16 passes on the full data achieved better results. I didn't think to try semi-supervised learning and I wonder how that would work within the vw framework. Did anyone else try ss with vw?

", "rawMarkdown": "", "votes": null }, { "id": "52844", "postDate": "09/01/2014 13:11:13", "content": "

[quote=Michael Jahrer;52837]

\n

A single libfm on raw data (one-hot with minOccurence=5 of category,subcategory,price,phones_cnt,emails_cnt,urls_cnt) gave me 0.98073 public and 0.98103 private.

\n

Pretty good for such a simple attempt.

\n

[/quote]

\n

Wow~ I will definitely give libfm a shot for Display comp if I have time!

", "rawMarkdown": "", "votes": null }, { "id": "52847", "postDate": "09/01/2014 13:17:57", "content": "

[quote=yr;52844]

\n

Wow~ I will definitely give libfm a shot for Display comp if I have time!

\n

[/quote]

\n

I did. It looks very promising (also with the results it got for KDD-cup 2012 Ad click prediction). Would love to see a benchmark in LibFM.

\n

Can't get it to build to hold more than 2GB of memory though, is 64-bit LibFM on Windows possible?

", "rawMarkdown": "", "votes": null }, { "id": "52848", "postDate": "09/01/2014 13:19:47", "content": "

[quote=yr;52844]

\n

[quote=Michael Jahrer;52837]

\n

A single libfm on raw data (one-hot with minOccurence=5 of category,subcategory,price,phones_cnt,emails_cnt,urls_cnt) gave me 0.98073 public and 0.98103 private.

\n

Pretty good for such a simple attempt.

\n

[/quote]

\n

Wow~ I will definitely give libfm a shot for Display comp if I have time!

\n

[/quote]

\n

yep - works on display ad challenge, one part of my solution there ;)

", "rawMarkdown": "", "votes": null }, { "id": "52856", "postDate": "09/01/2014 14:53:37", "content": "

Congratulations to Giulio and Barisumog. After posting my solution I trained a separate model for each category, following sergiu's suggestion: 0.971 -> 0.978.

", "rawMarkdown": "", "votes": null }, { "id": "52862", "postDate": "09/01/2014 16:13:45", "content": "

[quote=Mikhail Trofimov;52825]

\n

[quote=Giulio;52812]

\n

simple semi-supervised learning performed the best. I tried using only portions of the scored test set (i.e. observation with predicted probabilities above .9 or below .1), tried many cutoffs, but nothing was better than using all test. I even tried to use all the test set but weighted its observations based on how close to 0 and 1 the probabilities were, but that also did not add value.

\n

[/quote]

\n

Can you explain this in details? It is thing I want to learn! =)

\n

I've tried to use SSL, but got no profit. You predict whole test set, added it to the train, learn from concatenation and again predict test set?

\n

[/quote]

\n

In my case I had a two stage model. In the first stage I used several SGD as initial predictions. I fed those into a RF. I used the RF to predict test (as in 0/1, not probabilities). I then stacked train and test, target and predictions and retrained on the whole set. That improved my RF from .982-ish to .9859.

", "rawMarkdown": "", "votes": null }, { "id": "52871", "postDate": "09/01/2014 17:47:20", "content": "

What were the engineered features like for everyone?

\n

title_char_length, body_char_length, category_count, sub_category_count, and numerous char frequencies.

\n

all seemed informative features.

\n

We tried building an RF model with no text tokens, but word_frequencies of words like: gun, pistol, knife, diploma, thesis, medical, m2, massage. This did ok for so few features ~80%.

\n

We got these words by studying duplicates or near-duplicates in test and train set. Could have been more gain there, as there where a lot of title-duplicates and even more near duplicates. Perhaps that is why KNN worked well on this challenge.

\n

Also dreamed about training opensource spamfilters like CRM114 on both train and test data, to generate features this way. Maybe another time. 

\n

Did you notice the strange distribution of the IDs in good submission files? ID's starting with 9 ranked near the top, and ID's starting with 1 ranked near the bottom. A curious artifact, but no we found no leakage in this.

\n

I am a bit bummed I could not understand Russian, and had to rely on translate. What were the funny illicit advertisements you found in the dataset? I think I saw some military equipment advertised like sniper rifles and war medals.

", "rawMarkdown": "", "votes": null }, { "id": "52873", "postDate": "09/01/2014 18:04:50", "content": "

[quote=Triskelion;52871]

\n

Did you notice the strange distribution of the IDs in good submission files? ID's starting with 9 ranked near the top, and ID's starting with 1 ranked near the bottom. A curious artifact, but no we found no leakage in this.

\n

[/quote]

\n

I happen to find this too. Have a look at the first attachment (with "p"). Something like this:

\n

Top:

\n

id
99997125
99996057
99994331
99989755
99989702
99983497
99982520
99981089
99975380
99974721
99962972
Bottom:

\n

10000998
10000962
10000930
10000831
10000816
10000612
10000395
10000387
10000196
10000175
10000124
10000074

\n

with public LB: 0.97625 and private LB: 0.97812. It is generated when I used squared loss in VW but with -p arg for prediction.

\n

With -r arg instead, the corresponding (corrected) version yields public LB: 0.98104 and private LB: 0.97953 (The second attachment with "r"). Not that different. I am curious why it is so. 

\n

Edited: Upload the file twice... Pls ignore the .rar file.

", "rawMarkdown": "", "votes": null }, { "id": "52874", "postDate": "09/01/2014 18:26:24", "content": "

[quote=yr;52873]

\n

[quote=Triskelion;52871]

\n

Did you notice the strange distribution of the IDs in good submission files? ID's starting with 9 ranked near the top, and ID's starting with 1 ranked near the bottom. A curious artifact, but no we found no leakage in this.

\n

[/quote]

\n

I happen to find this too. Have a look at the first attachment (with "p"). Something like this:

\n

[/quote]

\n

Mine aren't like this at all. Did you guys bucket the predictions and then use itemid as a secondary sort?

", "rawMarkdown": "", "votes": null }, { "id": "52875", "postDate": "09/01/2014 18:28:50", "content": "

[quote=dkay;52874]

\n

[quote=yr;52873]

\n

[quote=Triskelion;52871]

\n

Did you notice the strange distribution of the IDs in good submission files? ID's starting with 9 ranked near the top, and ID's starting with 1 ranked near the bottom. A curious artifact, but no we found no leakage in this.

\n

[/quote]

\n

I happen to find this too. Have a look at the first attachment (with "p"). Something like this:

\n

[/quote]

\n

Mine aren't like this at all. Did you guys bucket the predictions and then use itemid as a secondary sort?

\n

[/quote]

\n

No, we didn't sort on ID. This just appeared when ensembling different models together. But even individual model files will have a higher than normal distribution of IDs starting with 5-9 in the first 1000 or so lines. At least with VW.

", "rawMarkdown": "", "votes": null }, { "id": "52877", "postDate": "09/01/2014 19:01:05", "content": "

[quote=Triskelion;52875]

\n

[quote=dkay;52874]

\n

[quote=yr;52873]

\n

[quote=Triskelion;52871]

\n

Did you notice the strange distribution of the IDs in good submission files? ID's starting with 9 ranked near the top, and ID's starting with 1 ranked near the bottom. A curious artifact, but no we found no leakage in this.

\n

[/quote]

\n

I happen to find this too. Have a look at the first attachment (with "p"). Something like this:

\n

[/quote]

\n

Mine aren't like this at all. Did you guys bucket the predictions and then use itemid as a secondary sort?

\n

[/quote]

\n

No, we didn't sort on ID. This just appeared when ensembling different models together. But even individual model files will have a higher than normal distribution of IDs starting with 5-9 in the first 1000 or so lines. At least with VW.

\n

[/quote]

\n

I just checked my own rankings, and also the combined rankings with Giulio, but I can't see a similar pattern. The first digits of the top 1000 seem pretty random.

", "rawMarkdown": "", "votes": null }, { "id": "52878", "postDate": "09/01/2014 19:06:50", "content": "

[quote=barisumog;52877]

\n

[quote=Triskelion;52875]

\n

[quote=dkay;52874]

\n

[quote=yr;52873]

\n

[quote=Triskelion;52871]

\n

Did you notice the strange distribution of the IDs in good submission files? ID's starting with 9 ranked near the top, and ID's starting with 1 ranked near the bottom. A curious artifact, but no we found no leakage in this.

\n

[/quote]

\n

I happen to find this too. Have a look at the first attachment (with "p"). Something like this:

\n

[/quote]

\n

Mine aren't like this at all. Did you guys bucket the predictions and then use itemid as a secondary sort?

\n

[/quote]

\n

No, we didn't sort on ID. This just appeared when ensembling different models together. But even individual model files will have a higher than normal distribution of IDs starting with 5-9 in the first 1000 or so lines. At least with VW.

\n

[/quote]

\n

I just checked my own rankings, and also the combined rankings with Giulio, but I can't see a similar pattern. The first digits of the top 1000 seem pretty random.

\n

[/quote]

\n

Yes. Most of my submissions seem random too. But the one I attached with weird ID order can yield high score 0.97x. That I can't understand.

", "rawMarkdown": "", "votes": null }, { "id": "52880", "postDate": "09/01/2014 19:09:35", "content": "

Pretty weird then. Something specific to VW being very good at remembering duplicates?

\n

The effect slowly appears, but is quite evident:

\n

Id
89473945
94127948
99606052
94989648
79129076
99849589
98818087
82182051
95990314
86040175

\n

...

\n

12564162
13689647
15085030
15669423
12604276
10813971
12198103
11358305
15096447
10287928

\n

Or even almost fully sorted, like yr's submission shows. Since the solution files were still well performing on both public and private leaderboard, I don't think it's a sorting/ensembling bug.

\n

There could however be small leakage: find high ID's like 99997125 that are ranked below line 35k and put them back on top. Perhaps we dodged a bullet?

", "rawMarkdown": "", "votes": null }, { "id": "52881", "postDate": "09/01/2014 19:20:47", "content": "

I noticed that in this competition RFs tend to produce that type of distribution when you let the full tree grow. In my case, I was feeding prediction from SGD into forests, and that was by far the most important feature. Since there were so few features in the RF model, even the full grown tree produces a very "discrete" solution. If I recall correctly, I was getting something like 1000 distinct leaves. That means, you'll get few clusters of probabilities. Depending on how you create your submission, the id could end up being sorted in each cluster of probabilities, thus giving the impression that 9xxxx ids are ranked higher. But if you look further down you'll see 9x, 8x, 7x,..1x repeat in cycles.

\n

I solved this by using a high min_sample_leaf, which stopped the tree much earlier and provided more "continuous" outcomes.

", "rawMarkdown": "", "votes": null }, { "id": "52902", "postDate": "09/02/2014 06:01:39", "content": "

Congrats to barisumog & Giulio, Mikhail & Dmitry and Feng & Hang & Jeong-Yoon.

\n

I used vw, trying several different parameter combinations particularly quadratic (cat/subcat with title / description / attrib), learning_rate and ngram (2 and 3 were beneficial beyond that it tended to overfit). Like Silogram, I found the number of passes could make a difference but found the auto stop kicked in at 20-30 passes. One of my earlier submissions, I tested beyond the auto stop and ended up with a submission that badly overfit, so I avoided the technique. Sounds like I should investigate that more. I did find a small amount of l1 regularisation helped.

\n

A substantial portion of my time was spent trying to get a system for cross-validating and learning vw and python.

", "rawMarkdown": "", "votes": null }, { "id": "52927", "postDate": "09/02/2014 18:09:02", "content": "

Congrats to Barisumog & Giulio for keeping a lead within a tight margin.

\n\n

Like many others, I decided to learn vw when starting this competition.

\n

I ended up blending vw logistic, neural network and 2 gram.

\n\n

What setting did you use for l1?

\n

Setting regularization only seemed to lower my score.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 52798, "author_name": "kevinandhu", "author_url": "", "post_date": "09/01/2014 00:31:07", "content": "

Congratulations! 

", "votes": null, "replies": [] }, { "id": 52800, "author_name": "triskelion", "author_url": "", "post_date": "09/01/2014 00:42:28", "content": "

Grats all. Grats to Abhishek for becoming top 10 Kaggler!

\n

And grats to Barisumog for becoming Master Kaggler. That's absolutely what that impressive profile deserved!

\n

Grats to my team members Phil Culliton and Jules, for reaching top 10 position together, and getting Master Kaggler status.

", "votes": null, "replies": [] }, { "id": 52802, "author_name": "dkaylor", "author_url": "", "post_date": "09/01/2014 00:59:48", "content": "

Great work guys! Very curious to see what you found that led to the separation.

\n\n

Did anyone find a good way to use the is_proved targets? I couldn't get them to add any value to my models.

", "votes": null, "replies": [] }, { "id": 52806, "author_name": "mikhailtrofimov", "author_url": "", "post_date": "09/01/2014 01:21:00", "content": "

Congrat winners! Congrat Barisumog with Master Tier!

", "votes": null, "replies": [] }, { "id": 52807, "author_name": "triskelion", "author_url": "", "post_date": "09/01/2014 01:33:31", "content": "

[quote=dkay;52802]

\n

Great work guys! Very curious to see what you found that led to the separation.

\n

Did anyone find a good way to use the is_proved targets? I couldn't get them to add any value to my models.

\n

[/quote]

\n

Jules found two ways: train a VW model where the positive class is samples where is_blocked ==1 AND is_proved == 1.

\n

Also as part of an illicit score to use for (quantile) regression: Something like: is_blocked + 0.4*is_proved + close_hours_score.

\n

This improved ensembles where we simply averaged the ranks of the submission files.

", "votes": null, "replies": [] }, { "id": 52809, "author_name": "magicjin", "author_url": "", "post_date": "09/01/2014 02:33:55", "content": "

Congrat winners! It's a wonderful game,i also learned a lot,Thanks to the organizers

", "votes": null, "replies": [] }, { "id": 52810, "author_name": "chenglongchen", "author_url": "", "post_date": "09/01/2014 02:41:44", "content": "

[quote=dkay;52802]

\n

Great work guys! Very curious to see what you found that led to the separation.

\n

Did anyone find a good way to use the is_proved targets? I couldn't get them to add any value to my models.

\n

[/quote]

\n

I trained two models, with all the blocked ads (mixed model) and only blocked ads labeled by experienced moderator (proved model). This improve my results by 0.001~0.002. My best private score is 0.98589 which is simple averaging by 2 mixed models and 2 proved models of 2 different set of features.

\n

This is the first time I come so close to top3!

\n

Congrats to everyone! Congrats to Barisumog & Giulio! Well done!

\n

Update: I initially entered this competition to familiarize myself with VW and Linux & Shell (I used to be a Windows user). If anyone interested in how to get to 0.985x using VW, I have put all my code here. 

\n

Finally, I would also want to thank @Triskelion and @Foxtrot for introducing me to VW :)

", "votes": null, "replies": [] }, { "id": 52811, "author_name": "dasgupta", "author_url": "", "post_date": "09/01/2014 02:51:14", "content": "

[quote=Triskelion;52807]

\n

[quote=dkay;52802]

\n

Great work guys! Very curious to see what you found that led to the separation.

\n

Did anyone find a good way to use the is_proved targets? I couldn't get them to add any value to my models.

\n

[/quote]

\n

Jules found two ways: train a VW model where the positive class is samples where is_blocked ==1 AND is_proved == 1.

\n

Also as part of an illicit score to use for (quantile) regression: Something like: is_blocked + 0.4*is_proved + close_hours_score.

\n

This improved ensembles where we simply averaged the ranks of the submission files.

\n

[/quote]

\n\n

Very interesting. Just curious, did you use VW exclusively for model building ? And, re: training for examples where is_blocked and is_blocked == 1, did you use any method to find this. Thanks.

", "votes": null, "replies": [] }, { "id": 52812, "author_name": "adjgiulio", "author_url": "", "post_date": "09/01/2014 02:58:03", "content": "

Great competition, lots of fun and awesome learning opportunity! Thanks Kaggle and Avito!

\n

I’ve got to first congratulate everyone who fought hard throughout this competition, other winners, top-10, and new Masters! Well done everyone! 
As for the approach, I think that what we did will not come as a shocker to anyone.

\n

My model was essentially a series of SGD on pieces of the text data (one for title, one for description, one for title+description, one for attrs). This alone gets you around .97 on the public LB. By adding to the tfidf matrices one-hot-encoded category and subcategory the models score well above .97.
The outputs from all the SGDs were then fed into a tree model (RF worked better than GBM) along with some other features, most importantly category and subcategory, but also email, url and phone number counts. This piece really helped the model re-learn the differences among categories and subcategories and re-sort the original SGD predictions. This added another .002-3ish and takes you in .983 territory.
Since plain accuracy was so high across the dataset, semi-supervised test data added in the order of .003 to the train data alone bringing my best model around .986.
All of this was built on the entire data minus real estate, which barisimug found out could be removed not only without any loss, but with a sizeable increase in performance.

\n

I'll let barisumog describe his models but in his approach he modelled each category and subcategory independently, later stacking results.

\n

Our winner model is a simple rank based weighed average of my best RF andGBM, and barisumog's ensemble of SVMs.

\n

We used ranks to blend our models mostly because, from a practical perspective, our model outputs were on different scales (mine were probabilities, barisumog’s were distances from the SVM separating hyperplane). In reality I have observed that even amongst my models (who all produced probabilities) ranks were still as good as if not better than probabilities for blends.
My intuition, with no scientific evidence for it, is that when using semi-supervised learning you end up getting many extreme predictions for the test set, with lots of observations clustering around 1 and 0. That makes intuitive sense since you’re predicting observations the algorithm has already seen in training. With these big clusters of 1’s, .9995’s, .9993’s… ranking is introducing some (admittedly arbitrary) randomization, which might act as a good counterweight to the extreme decisiveness of semi-supervised approaches.

\n

Another surprising thing for me was the fact I was getting much less than in other competitions out of ensembles. It really took two very distinct approaches to get decent value out of our team ensemble. And that was also, the key to our success. I'm really curious to see what other approaches folks have come up with, but essentially we had two very good models covering two very different approaches.

\n

Things that did not work for me:
-is_proven added no value to our models
-feature engineering has provided little to no value. I tried a bunch of features derived from text (i.e. count of !, ?, mixed words, length of text,...) and none seemed to add much.
-for the attributes feature, anything more fancy that running it through a tfidf added nothing.
-I tried a second level model where I’d take the top 20% observations and re-model those alone to see if it could help further minimize false positives. Even at different cutoffs, that did not help.

\n

Things that did work for me:
-no need to do anything fancy with the russian text. Just feed it into a tfidf vectorizer and you’re already in better shape than the benchmark code.
-simple semi-supervised learning performed the best. I tried using only portions of the scored test set (i.e. observation with predicted probabilities above .9 or below .1), tried many cutoffs, but nothing was better than using all test. I even tried to use all the test set but weighted its observations based on how close to 0 and 1 the probabilities were, but that also did not add value.

\n

One part of the competition that was a first, and very interesting for someone like me who enjoys the competitive aspect of Kaggle, was the strategic aspect of managing a high LB rank. This is where machine learning becomes strategic/competitive machine learning. :-)

", "votes": null, "replies": [] }, { "id": 52813, "author_name": "dkaylor", "author_url": "", "post_date": "09/01/2014 03:24:43", "content": "

Nice work. It sounds like my method was similar to bariusmog's approach.

\n\n

I used the title + description, and also just title on it's own and blended them together. For both of these I trained with all category data, then separated into single models by category, and also single models based on subcategory in order to get some sort of diversity.

\n\n

With combining in this way, it gave me an extra ~.002  above any on their own. I also used some semi-supervised learning with the test set, but it only gave me ~.001.  I tried some feature engineering as well, but it gave little value.

", "votes": null, "replies": [] }, { "id": 52814, "author_name": "adjgiulio", "author_url": "", "post_date": "09/01/2014 04:08:08", "content": "

And congrats David T. on becoming Kaggle's new #2 overall. Seattle should be pretty proud these days :-)

", "votes": null, "replies": [] }, { "id": 52815, "author_name": "apophenia", "author_url": "", "post_date": "09/01/2014 04:20:31", "content": "

My 2 cents on text features:

\n

Other than unigrams, bigrams were also helpful, but not trigrams. Stemming and correcting words didn't help.

", "votes": null, "replies": [] }, { "id": 52818, "author_name": "lucamassaron", "author_url": "", "post_date": "09/01/2014 04:49:24", "content": "

Congratulations to Giulio and Barisumog for winning this interesting competition! And for Abhishek for entering the top 10! Very interesting competition, after a long time I had to use VW again.

", "votes": null, "replies": [] }, { "id": 52820, "author_name": "vasilyleksin", "author_url": "", "post_date": "09/01/2014 05:44:27", "content": "

Thank you all for the work done! We are carefully watching the competition process and the results are really impressive! We hope that the competition was interesting and honest. Now it's our turn to work hard to implement something like that in our production models...

", "votes": null, "replies": [] }, { "id": 52824, "author_name": "dixitgaurav", "author_url": "", "post_date": "09/01/2014 06:40:26", "content": "

Congrats winners! Congrat Barisumog for becoming Master!

\n

Congrats Abhishek for entering  into the top 10!

", "votes": null, "replies": [] }, { "id": 52825, "author_name": "mikhailtrofimov", "author_url": "", "post_date": "09/01/2014 07:28:42", "content": "

I’ve got to congratulate everyone who participate and especially everyone who break 0.984 =)

\n

Our approach is quite similar to discribed by Giulio. We use different pieces of data (title, title+description, title+description+attrs, title+attrs) and made 3 levels of details for each (top100k word, all word, all pair of words). For all of this features-sets was trained SVM, for some - additional LibFM models. Solely they give 0.97 - 0.983.

\n

All this models was blended by RF, so it gave 0.986.

\n

[quote=Giulio;52812]

\n

simple semi-supervised learning performed the best. I tried using only portions of the scored test set (i.e. observation with predicted probabilities above .9 or below .1), tried many cutoffs, but nothing was better than using all test. I even tried to use all the test set but weighted its observations based on how close to 0 and 1 the probabilities were, but that also did not add value.

\n

[/quote]

\n

Can you explain this in details? It is thing I want to learn! =)

\n

I've tried to use SSL, but got no profit. You predict whole test set, added it to the train, learn from concatenation and again predict test set?

", "votes": null, "replies": [] }, { "id": 52828, "author_name": "barisumog", "author_url": "", "post_date": "09/01/2014 08:38:48", "content": "

Congrats to everyone who fought hard to the end! And thanks to Kaggle and Avito, it was a fun competition.

\n

About my approach: I don't speak Russian, but running numerous data through Google translate, I could see that some categories used very different language than others. Also the ratio of illicit content varied widely between categories (and subcategories). So I decided to tackle the problem on a category / subcategory basis.

\n

This is what I did, in steps:

\n
    \n
  1. separate the raw data into categories and subcategories
    2. \n
  2. ignore the Real Estate category and related subcategories (vastly different language than other categories, and very tiny ratio of illicit content)
    3. \n
  3. extract raw text from each post by concatenating the title, description and attributes sections (We tried many other features, some worked for Giulio, but none for me. I used only textual features)
    4. \n
  4. for each category and subcategory, create 3 tf-idf matrices: one with raw text, one with stemming, and one with stop words (separately, they gave similar results, but I noticed they improved the score a bit and became more stable when combined)
    5. \n
  5. for each category and subcategory, train 2 sets of SVCs with different C parameters on each tf-idf (again, similar results separately, but slightly better when combined)
    6. \n
  6. so now I have 2 x 3 SVCs for every category, and 2 x 3 SVCs for every subcategory (12 models to use for every data point)
    7. \n
  7. apply semi-supervised learning, which was Giulio's idea and worked quite well. Use the trained models on test data to predict classes. Concatenate train+test, and labels+prediction. Retrain all models on this new merged data set.
    8. \n
  8. finally, use the models to make predictions on the test data. use the SVC output of distance from hyperplane to rank the individual posts
    9. \n
\n

Then Giulio did some of his magic to combine my ranks with his own models. As he already pointed out, the difference of approach between our models resulted in a nice boost when ensembled.

", "votes": null, "replies": [] }, { "id": 52834, "author_name": "dyakonov", "author_url": "", "post_date": "09/01/2014 11:44:21", "content": "

Thanks to the organisers and congrats to all the top finishers!

\n

Special congratulations to winners who avoided overfitting.

\n

Поздравляю лучшую Российскую команду Михаила и Дмитрия!  

", "votes": null, "replies": [] }, { "id": 52836, "author_name": "triskelion", "author_url": "", "post_date": "09/01/2014 12:14:22", "content": "

[quote=xbsd;52811]

\n

Very interesting. Just curious, did you use VW exclusively for model building ? And, re: training for examples where is_blocked and is_blocked == 1, did you use any method to find this. Thanks.

\n

[/quote]

\n

Yes. Single VW models gave around 97.7, with a lot of tuning single VW models became around 98. Averaging the ranks of a hinge and logloss model improved the score to well over 98, etc. Reading through yr's code (very well done on the VW!) it's possible to get a very high score with very little memory resources and time. So for us no elaborate complex GBM or RF models (though we very much would have used them if we managed or saw improvement).

\n

So for implementation, I think VW could be a solid and fast choice, with simple model introspection. But it's not like these models need retraining every minute, so more complex models are possible too.

\n

I believe Jules found that trick through a hunch. It makes sense that a model trained on different features/samples or labels will hold different information for an ensemble to use.

\n

Vowpal Wabbit eats Russian spam for breakfast. But just to make sure we latinized all Russian characters, so we could dream to avoid further encoding issues on Windows.

\n

The semi-supervised trick from Giulio is interesting. Makes perfect sense to try it out, but I was thinking about this for other competitions, not this one. I think it has its own risks associated with it (I call it over-learning).

", "votes": null, "replies": [] }, { "id": 52837, "author_name": "mjahrer", "author_url": "", "post_date": "09/01/2014 12:26:03", "content": "

A single libfm on raw data (one-hot with minOccurence=5 of category,subcategory,price,phones_cnt,emails_cnt,urls_cnt) gave me 0.98073 public and 0.98103 private.

\n

Pretty good for such a simple attempt.

", "votes": null, "replies": [] }, { "id": 52838, "author_name": "triskelion", "author_url": "", "post_date": "09/01/2014 12:46:26", "content": "

[quote=Michael Jahrer;52837]

\n

A single libfm on raw data (one-hot with minOccurence=5 of category,subcategory,price,phones_cnt,emails_cnt,urls_cnt) gave me 0.98073 public and 0.98103 private.

\n

Pretty good for such a simple attempt.

\n

[/quote]

\n

Wow, that is really good. Do I understand you correctly in that you did not use the text at all?

", "votes": null, "replies": [] }, { "id": 52841, "author_name": "mjahrer", "author_url": "", "post_date": "09/01/2014 12:55:22", "content": "

[quote=Triskelion;52838]

\n

[quote=Michael Jahrer;52837]

\n

A single libfm on raw data (one-hot with minOccurence=5 of category,subcategory,price,phones_cnt,emails_cnt,urls_cnt) gave me 0.98073 public and 0.98103 private.

\n

Pretty good for such a simple attempt.

\n

[/quote]

\n

Wow, that is really good. Do I understand you correctly in that you did not use the text at all?

\n

[/quote]

\n

I checked my feature generation code (several weeks ago), and yes sorry I forgot to mentioned the tokens in these 3 cols: title,description,attr. So yes I used this text tokens.

", "votes": null, "replies": [] }, { "id": 52843, "author_name": "psilogram", "author_url": "", "post_date": "09/01/2014 13:03:20", "content": "

I used vw exclusively, experimenting with the parameters, especially --passes and --decay_learning_rate. Surprisingly, the number of passes had a large effect, even with a very steep decay. For example, the difference between 15 and 16 passes, with the decay at 0.5, could be as much as .002. Also, the average variance between the LB and PB scores was about .002 and was quite erratic. My final two choices were my best LB model (.98461), which scored only .98200 on the PB, and an ensemble of 10 vw models, which scored .98404 on the LB and .98291 on the PB. My best PB score was the same ensemble, but mixed based on rank instead of raw scores, which scored .98431 on the PB but only .98317 on the LB.

\n

All in all, I had a great time learning about vw, which is an incredibly powerful and fast tool. It is difficult, however, to set up a robust cross-validation system with vw. For example, I found that the  built-in stopping mechanism on passes tended to underestimate the optimal number of passes. For example, my models generally stopped after 8, or sometimes 9, passes, but I found that letting it run for 14-16 passes on the full data achieved better results. I didn't think to try semi-supervised learning and I wonder how that would work within the vw framework. Did anyone else try ss with vw?

", "votes": null, "replies": [] }, { "id": 52844, "author_name": "chenglongchen", "author_url": "", "post_date": "09/01/2014 13:11:13", "content": "

[quote=Michael Jahrer;52837]

\n

A single libfm on raw data (one-hot with minOccurence=5 of category,subcategory,price,phones_cnt,emails_cnt,urls_cnt) gave me 0.98073 public and 0.98103 private.

\n

Pretty good for such a simple attempt.

\n

[/quote]

\n

Wow~ I will definitely give libfm a shot for Display comp if I have time!

", "votes": null, "replies": [] }, { "id": 52847, "author_name": "triskelion", "author_url": "", "post_date": "09/01/2014 13:17:57", "content": "

[quote=yr;52844]

\n

Wow~ I will definitely give libfm a shot for Display comp if I have time!

\n

[/quote]

\n

I did. It looks very promising (also with the results it got for KDD-cup 2012 Ad click prediction). Would love to see a benchmark in LibFM.

\n

Can't get it to build to hold more than 2GB of memory though, is 64-bit LibFM on Windows possible?

", "votes": null, "replies": [] }, { "id": 52848, "author_name": "mjahrer", "author_url": "", "post_date": "09/01/2014 13:19:47", "content": "

[quote=yr;52844]

\n

[quote=Michael Jahrer;52837]

\n

A single libfm on raw data (one-hot with minOccurence=5 of category,subcategory,price,phones_cnt,emails_cnt,urls_cnt) gave me 0.98073 public and 0.98103 private.

\n

Pretty good for such a simple attempt.

\n

[/quote]

\n

Wow~ I will definitely give libfm a shot for Display comp if I have time!

\n

[/quote]

\n

yep - works on display ad challenge, one part of my solution there ;)

", "votes": null, "replies": [] }, { "id": 52856, "author_name": "zygmunt", "author_url": "", "post_date": "09/01/2014 14:53:37", "content": "

Congratulations to Giulio and Barisumog. After posting my solution I trained a separate model for each category, following sergiu's suggestion: 0.971 -> 0.978.

", "votes": null, "replies": [] }, { "id": 52862, "author_name": "adjgiulio", "author_url": "", "post_date": "09/01/2014 16:13:45", "content": "

[quote=Mikhail Trofimov;52825]

\n

[quote=Giulio;52812]

\n

simple semi-supervised learning performed the best. I tried using only portions of the scored test set (i.e. observation with predicted probabilities above .9 or below .1), tried many cutoffs, but nothing was better than using all test. I even tried to use all the test set but weighted its observations based on how close to 0 and 1 the probabilities were, but that also did not add value.

\n

[/quote]

\n

Can you explain this in details? It is thing I want to learn! =)

\n

I've tried to use SSL, but got no profit. You predict whole test set, added it to the train, learn from concatenation and again predict test set?

\n

[/quote]

\n

In my case I had a two stage model. In the first stage I used several SGD as initial predictions. I fed those into a RF. I used the RF to predict test (as in 0/1, not probabilities). I then stacked train and test, target and predictions and retrained on the whole set. That improved my RF from .982-ish to .9859.

", "votes": null, "replies": [] }, { "id": 52871, "author_name": "triskelion", "author_url": "", "post_date": "09/01/2014 17:47:20", "content": "

What were the engineered features like for everyone?

\n

title_char_length, body_char_length, category_count, sub_category_count, and numerous char frequencies.

\n

all seemed informative features.

\n

We tried building an RF model with no text tokens, but word_frequencies of words like: gun, pistol, knife, diploma, thesis, medical, m2, massage. This did ok for so few features ~80%.

\n

We got these words by studying duplicates or near-duplicates in test and train set. Could have been more gain there, as there where a lot of title-duplicates and even more near duplicates. Perhaps that is why KNN worked well on this challenge.

\n

Also dreamed about training opensource spamfilters like CRM114 on both train and test data, to generate features this way. Maybe another time. 

\n

Did you notice the strange distribution of the IDs in good submission files? ID's starting with 9 ranked near the top, and ID's starting with 1 ranked near the bottom. A curious artifact, but no we found no leakage in this.

\n

I am a bit bummed I could not understand Russian, and had to rely on translate. What were the funny illicit advertisements you found in the dataset? I think I saw some military equipment advertised like sniper rifles and war medals.

", "votes": null, "replies": [] }, { "id": 52873, "author_name": "chenglongchen", "author_url": "", "post_date": "09/01/2014 18:04:50", "content": "

[quote=Triskelion;52871]

\n

Did you notice the strange distribution of the IDs in good submission files? ID's starting with 9 ranked near the top, and ID's starting with 1 ranked near the bottom. A curious artifact, but no we found no leakage in this.

\n

[/quote]

\n

I happen to find this too. Have a look at the first attachment (with "p"). Something like this:

\n

Top:

\n

id
99997125
99996057
99994331
99989755
99989702
99983497
99982520
99981089
99975380
99974721
99962972
Bottom:

\n

10000998
10000962
10000930
10000831
10000816
10000612
10000395
10000387
10000196
10000175
10000124
10000074

\n

with public LB: 0.97625 and private LB: 0.97812. It is generated when I used squared loss in VW but with -p arg for prediction.

\n

With -r arg instead, the corresponding (corrected) version yields public LB: 0.98104 and private LB: 0.97953 (The second attachment with "r"). Not that different. I am curious why it is so. 

\n

Edited: Upload the file twice... Pls ignore the .rar file.

", "votes": null, "replies": [] }, { "id": 52874, "author_name": "dkaylor", "author_url": "", "post_date": "09/01/2014 18:26:24", "content": "

[quote=yr;52873]

\n

[quote=Triskelion;52871]

\n

Did you notice the strange distribution of the IDs in good submission files? ID's starting with 9 ranked near the top, and ID's starting with 1 ranked near the bottom. A curious artifact, but no we found no leakage in this.

\n

[/quote]

\n

I happen to find this too. Have a look at the first attachment (with "p"). Something like this:

\n

[/quote]

\n

Mine aren't like this at all. Did you guys bucket the predictions and then use itemid as a secondary sort?

", "votes": null, "replies": [] }, { "id": 52875, "author_name": "triskelion", "author_url": "", "post_date": "09/01/2014 18:28:50", "content": "

[quote=dkay;52874]

\n

[quote=yr;52873]

\n

[quote=Triskelion;52871]

\n

Did you notice the strange distribution of the IDs in good submission files? ID's starting with 9 ranked near the top, and ID's starting with 1 ranked near the bottom. A curious artifact, but no we found no leakage in this.

\n

[/quote]

\n

I happen to find this too. Have a look at the first attachment (with "p"). Something like this:

\n

[/quote]

\n

Mine aren't like this at all. Did you guys bucket the predictions and then use itemid as a secondary sort?

\n

[/quote]

\n

No, we didn't sort on ID. This just appeared when ensembling different models together. But even individual model files will have a higher than normal distribution of IDs starting with 5-9 in the first 1000 or so lines. At least with VW.

", "votes": null, "replies": [] }, { "id": 52877, "author_name": "barisumog", "author_url": "", "post_date": "09/01/2014 19:01:05", "content": "

[quote=Triskelion;52875]

\n

[quote=dkay;52874]

\n

[quote=yr;52873]

\n

[quote=Triskelion;52871]

\n

Did you notice the strange distribution of the IDs in good submission files? ID's starting with 9 ranked near the top, and ID's starting with 1 ranked near the bottom. A curious artifact, but no we found no leakage in this.

\n

[/quote]

\n

I happen to find this too. Have a look at the first attachment (with "p"). Something like this:

\n

[/quote]

\n

Mine aren't like this at all. Did you guys bucket the predictions and then use itemid as a secondary sort?

\n

[/quote]

\n

No, we didn't sort on ID. This just appeared when ensembling different models together. But even individual model files will have a higher than normal distribution of IDs starting with 5-9 in the first 1000 or so lines. At least with VW.

\n

[/quote]

\n

I just checked my own rankings, and also the combined rankings with Giulio, but I can't see a similar pattern. The first digits of the top 1000 seem pretty random.

", "votes": null, "replies": [] }, { "id": 52878, "author_name": "chenglongchen", "author_url": "", "post_date": "09/01/2014 19:06:50", "content": "

[quote=barisumog;52877]

\n

[quote=Triskelion;52875]

\n

[quote=dkay;52874]

\n

[quote=yr;52873]

\n

[quote=Triskelion;52871]

\n

Did you notice the strange distribution of the IDs in good submission files? ID's starting with 9 ranked near the top, and ID's starting with 1 ranked near the bottom. A curious artifact, but no we found no leakage in this.

\n

[/quote]

\n

I happen to find this too. Have a look at the first attachment (with "p"). Something like this:

\n

[/quote]

\n

Mine aren't like this at all. Did you guys bucket the predictions and then use itemid as a secondary sort?

\n

[/quote]

\n

No, we didn't sort on ID. This just appeared when ensembling different models together. But even individual model files will have a higher than normal distribution of IDs starting with 5-9 in the first 1000 or so lines. At least with VW.

\n

[/quote]

\n

I just checked my own rankings, and also the combined rankings with Giulio, but I can't see a similar pattern. The first digits of the top 1000 seem pretty random.

\n

[/quote]

\n

Yes. Most of my submissions seem random too. But the one I attached with weird ID order can yield high score 0.97x. That I can't understand.

", "votes": null, "replies": [] }, { "id": 52880, "author_name": "triskelion", "author_url": "", "post_date": "09/01/2014 19:09:35", "content": "

Pretty weird then. Something specific to VW being very good at remembering duplicates?

\n

The effect slowly appears, but is quite evident:

\n

Id
89473945
94127948
99606052
94989648
79129076
99849589
98818087
82182051
95990314
86040175

\n

...

\n

12564162
13689647
15085030
15669423
12604276
10813971
12198103
11358305
15096447
10287928

\n

Or even almost fully sorted, like yr's submission shows. Since the solution files were still well performing on both public and private leaderboard, I don't think it's a sorting/ensembling bug.

\n

There could however be small leakage: find high ID's like 99997125 that are ranked below line 35k and put them back on top. Perhaps we dodged a bullet?

", "votes": null, "replies": [] }, { "id": 52881, "author_name": "adjgiulio", "author_url": "", "post_date": "09/01/2014 19:20:47", "content": "

I noticed that in this competition RFs tend to produce that type of distribution when you let the full tree grow. In my case, I was feeding prediction from SGD into forests, and that was by far the most important feature. Since there were so few features in the RF model, even the full grown tree produces a very "discrete" solution. If I recall correctly, I was getting something like 1000 distinct leaves. That means, you'll get few clusters of probabilities. Depending on how you create your submission, the id could end up being sorted in each cluster of probabilities, thus giving the impression that 9xxxx ids are ranked higher. But if you look further down you'll see 9x, 8x, 7x,..1x repeat in cycles.

\n

I solved this by using a high min_sample_leaf, which stopped the tree much earlier and provided more "continuous" outcomes.

", "votes": null, "replies": [] }, { "id": 52902, "author_name": "sthompson", "author_url": "", "post_date": "09/02/2014 06:01:39", "content": "

Congrats to barisumog & Giulio, Mikhail & Dmitry and Feng & Hang & Jeong-Yoon.

\n

I used vw, trying several different parameter combinations particularly quadratic (cat/subcat with title / description / attrib), learning_rate and ngram (2 and 3 were beneficial beyond that it tended to overfit). Like Silogram, I found the number of passes could make a difference but found the auto stop kicked in at 20-30 passes. One of my earlier submissions, I tested beyond the auto stop and ended up with a submission that badly overfit, so I avoided the technique. Sounds like I should investigate that more. I did find a small amount of l1 regularisation helped.

\n

A substantial portion of my time was spent trying to get a system for cross-validating and learning vw and python.

", "votes": null, "replies": [] }, { "id": 52927, "author_name": "devcon", "author_url": "", "post_date": "09/02/2014 18:09:02", "content": "

Congrats to Barisumog & Giulio for keeping a lead within a tight margin.

\n\n

Like many others, I decided to learn vw when starting this competition.

\n

I ended up blending vw logistic, neural network and 2 gram.

\n\n

What setting did you use for l1?

\n

Setting regularization only seemed to lower my score.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "52795": "", "52798": "", "52800": "", "52802": "", "52806": "", "52807": "", "52809": "", "52810": "", "52811": "", "52812": "", "52813": "", "52814": "", "52815": "", "52818": "", "52820": "", "52824": "", "52825": "", "52828": "", "52834": "", "52836": "", "52837": "", "52838": "", "52841": "", "52843": "", "52844": "", "52847": "", "52848": "", "52856": "", "52862": "", "52871": "", "52873": "", "52874": "", "52875": "", "52877": "", "52878": "", "52880": "", "52881": "", "52902": "", "52927": "" }, "source": "meta" }