First of all, congratulations to the winners.
\nSecond, post your code! Here is mine (R); I scored 0.92401 on the private leaderboard (higher AUC than I had on the public one, but I dropped several positions anyway). I have written a brief blog post detailing some of the stuff I did.
\nSee you on the next challenge.
", "messages": [ { "id": "81298", "postDate": "06/09/2015 01:37:30", "content": "First of all, congratulations to the winners.
\nSecond, post your code! Here is mine (R); I scored 0.92401 on the private leaderboard (higher AUC than I had on the public one, but I dropped several positions anyway). I have written a brief blog post detailing some of the stuff I did.
\nSee you on the next challenge.
", "rawMarkdown": "", "votes": null }, { "id": "81305", "postDate": "06/09/2015 02:15:27", "content": "Here's my code for a basic model I used within my ensemble.
", "rawMarkdown": "", "votes": null }, { "id": "81311", "postDate": "06/09/2015 03:13:56", "content": "I think the important part for this competition should be feature engineering, even though the ML could be equal important as well. The features that I found useful (at least I used in my model) are below:
\n1. Total bids for each bidder
\n2. Total auctions for each bidder
\n3. Total bids in each country for each bidder
\n4. Total urls for each bidder
\n5. Total unique times for each bidder
\n6. Total devices for each bidder
\n7. Max number of bids that share the same left-most-5 digits of time for each bidder
\n8. Max number of bids that share the same left-most-4 digits of time for each bidder
\n9. Max number of bids that share the same left-most-6 digits of time for each bidder
\n10. The median time difference of all consecuitive bids for each bidder
\n11. Total ips for each bidder
\n12. Total ips in Indian for each bidder
\n13. Total bids of each merchandise for each bidder
\n14. The median of each auction's time for each bidder.
\nThe ML models I used are XGB, adaboost and svm ensembled by ranking (weights around 1.0, 0.9, 0.8 for each).
\n1. setting class.weights for SVM are quite helpful. ( class.weights=c('0'=0.1,'1'=1.0) )
\n2. For adaboost, I used svm and rf as base models, and adaboost as second stage stacking.
\n3. XGB itself gives the best performance of the three. (around 0.92 in CV)
\nFinal models are the ranking average of the three with weights around 1.0*xgb + 0.9*ada + 0.8*svm, and use log(1+x) to scale the features.
\nI got 0.92176 for Public LB and 0.93426 for Private (I tried not to overfit too much, and in the end I selected one best CV model and one best LB model as my final submission)
\nEDIT: I just found out that my 2-stage bagged adaboost (stacked on svm and rm as base classifier, as in part 2 above), or meta bagging as Mike Kim called it, gives a single private score of 0.936, which is better than my final combined model. So I attached it here.
\nAt least I can say that bagging & stacking of adaboost in this kind of competition could work well. (For bagging, I used 75% WITHOUT replacement as bag, and the remaining 25% as OOB data, instead of sample WITH replacement as in normal bagging; also the adabag package has weird issue when you don't have target column for prediction data set, which is why I added a dummy target col. )
", "rawMarkdown": "", "votes": null }, { "id": "81335", "postDate": "06/09/2015 11:57:33", "content": "[quote=Mike Kim;81305]
\nHere's my code for a basic model I used within my ensemble.
\n[/quote]
\n\nMike,
\nWhen using Xgboost, the eval metric of auc always prints out numbers around 0.5. The end result is good (0.92 on leaderboard), but it prints out uninformative auc values. Does Xgboost print reasonable AUC values for you? I ended up training "blind" with Xgboost, and then running separate code to check how the AUC was doing. Worked out, but would like to do it an easier way next time.
\n\n-Rich
", "rawMarkdown": "", "votes": null }, { "id": "81359", "postDate": "06/09/2015 15:37:11", "content": "I think it's probably just a version issue with xgboost. I'm using this version:
https://www.kaggle.com/c/otto-group-product-classification-challenge/forums/t/12947/achieve-0-50776-on-the-leaderboard-in-a-minute-with-xgboost
For cv you can use something like:
xgb.cv()
But I think the CV code above over estimates AUC compared to my 300 fold random 2/3 subset CV by a large margin.
I got really into this competition and I had tons of fun, thanks to you all for respecting the rules about sharing that make this kind of competitions special.
\nSo here is my solution for this challenge, with a score of ~0.927 with some comments and some visualizations in R markdown.
\n\nThe full repository is here in case somebody wants to have a peek. It still contains parts from models that weren't included in the final result (ex. linear models) and EDAs but they gave important insights nonetheless.
\nhttps://github.com/wacax/Facebook-Recruiting-IV
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 81305, "author_name": "mikeskim", "author_url": "", "post_date": "06/09/2015 02:15:27", "content": "Here's my code for a basic model I used within my ensemble.
", "votes": null, "replies": [] }, { "id": 81311, "author_name": "weimin", "author_url": "", "post_date": "06/09/2015 03:13:56", "content": "I think the important part for this competition should be feature engineering, even though the ML could be equal important as well. The features that I found useful (at least I used in my model) are below:
\n1. Total bids for each bidder
\n2. Total auctions for each bidder
\n3. Total bids in each country for each bidder
\n4. Total urls for each bidder
\n5. Total unique times for each bidder
\n6. Total devices for each bidder
\n7. Max number of bids that share the same left-most-5 digits of time for each bidder
\n8. Max number of bids that share the same left-most-4 digits of time for each bidder
\n9. Max number of bids that share the same left-most-6 digits of time for each bidder
\n10. The median time difference of all consecuitive bids for each bidder
\n11. Total ips for each bidder
\n12. Total ips in Indian for each bidder
\n13. Total bids of each merchandise for each bidder
\n14. The median of each auction's time for each bidder.
\nThe ML models I used are XGB, adaboost and svm ensembled by ranking (weights around 1.0, 0.9, 0.8 for each).
\n1. setting class.weights for SVM are quite helpful. ( class.weights=c('0'=0.1,'1'=1.0) )
\n2. For adaboost, I used svm and rf as base models, and adaboost as second stage stacking.
\n3. XGB itself gives the best performance of the three. (around 0.92 in CV)
\nFinal models are the ranking average of the three with weights around 1.0*xgb + 0.9*ada + 0.8*svm, and use log(1+x) to scale the features.
\nI got 0.92176 for Public LB and 0.93426 for Private (I tried not to overfit too much, and in the end I selected one best CV model and one best LB model as my final submission)
\nEDIT: I just found out that my 2-stage bagged adaboost (stacked on svm and rm as base classifier, as in part 2 above), or meta bagging as Mike Kim called it, gives a single private score of 0.936, which is better than my final combined model. So I attached it here.
\nAt least I can say that bagging & stacking of adaboost in this kind of competition could work well. (For bagging, I used 75% WITHOUT replacement as bag, and the remaining 25% as OOB data, instead of sample WITH replacement as in normal bagging; also the adabag package has weird issue when you don't have target column for prediction data set, which is why I added a dummy target col. )
", "votes": null, "replies": [] }, { "id": 81335, "author_name": "richardepstein", "author_url": "", "post_date": "06/09/2015 11:57:33", "content": "[quote=Mike Kim;81305]
\nHere's my code for a basic model I used within my ensemble.
\n[/quote]
\n\nMike,
\nWhen using Xgboost, the eval metric of auc always prints out numbers around 0.5. The end result is good (0.92 on leaderboard), but it prints out uninformative auc values. Does Xgboost print reasonable AUC values for you? I ended up training "blind" with Xgboost, and then running separate code to check how the AUC was doing. Worked out, but would like to do it an easier way next time.
\n\n-Rich
", "votes": null, "replies": [] }, { "id": 81359, "author_name": "mikeskim", "author_url": "", "post_date": "06/09/2015 15:37:11", "content": "I think it's probably just a version issue with xgboost. I'm using this version:
https://www.kaggle.com/c/otto-group-product-classification-challenge/forums/t/12947/achieve-0-50776-on-the-leaderboard-in-a-minute-with-xgboost
For cv you can use something like:
xgb.cv()
But I think the CV code above over estimates AUC compared to my 300 fold random 2/3 subset CV by a large margin.
I got really into this competition and I had tons of fun, thanks to you all for respecting the rules about sharing that make this kind of competitions special.
\nSo here is my solution for this challenge, with a score of ~0.927 with some comments and some visualizations in R markdown.
\n\nThe full repository is here in case somebody wants to have a peek. It still contains parts from models that weren't included in the final result (ex. linear models) and EDAs but they gave important insights nonetheless.
\n", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "81298": "", "81305": "", "81311": "", "81335": "", "81359": "", "81444": "" }, "source": "meta" }