Here's a little blurb about how I approached feature generation and cross-validation for this problem:
\nhttp://small-yellow-duck.github.io/auction.html
\ncode: https://github.com/small-yellow-duck/facebook_auction
\nI used sklearn's RandomForestClassifier. The most useful features I identified were:
\nCross-validation was a bit tricky: I did CV 100 times on different 80-20 splits. My mean CV score was almost the same (+/- 0.003) as my final leaderboard score.
\n\nedit to answer some further questions:
\nIn order to convert the time scale to hours, I made a bids/unit time histogram and looked at the periodicity - bidding volumes cycled with a period of one day. There are pretty graphs and more details on my github page.
\nA concrete example of the entropy is the IP entropy: N!/(N_{IP1}! N{IP2}!... N{IPn}!). N is the total number of bids and N_{IPn} is the total number of bids placed from the nth IP. (Because the entropy is a large number, I used the log of the entropy.) The entropy is a measure of how both randomly distributed the bids are, how many IPs there are and how many bids there are. A bidder that places all their bids from the same IP has an entropy of N!/N! = 1.
\nI used the simple average of five RandomForestClassifiers (each with a different value for the initialization parameter, random_state) to generate the predictions.
\n\nThanks to everyone else who shared their approaches! I thought that leaving out the bots with only one bid was a clever idea. And I was surprised to see that people got mileage out of the device type.
", "messages": [ { "id": "81297", "postDate": "06/09/2015 01:33:54", "content": "Here's a little blurb about how I approached feature generation and cross-validation for this problem:
\nhttp://small-yellow-duck.github.io/auction.html
\ncode: https://github.com/small-yellow-duck/facebook_auction
\nI used sklearn's RandomForestClassifier. The most useful features I identified were:
\nCross-validation was a bit tricky: I did CV 100 times on different 80-20 splits. My mean CV score was almost the same (+/- 0.003) as my final leaderboard score.
\n\nedit to answer some further questions:
\nIn order to convert the time scale to hours, I made a bids/unit time histogram and looked at the periodicity - bidding volumes cycled with a period of one day. There are pretty graphs and more details on my github page.
\nA concrete example of the entropy is the IP entropy: N!/(N_{IP1}! N{IP2}!... N{IPn}!). N is the total number of bids and N_{IPn} is the total number of bids placed from the nth IP. (Because the entropy is a large number, I used the log of the entropy.) The entropy is a measure of how both randomly distributed the bids are, how many IPs there are and how many bids there are. A bidder that places all their bids from the same IP has an entropy of N!/N! = 1.
\nI used the simple average of five RandomForestClassifiers (each with a different value for the initialization parameter, random_state) to generate the predictions.
\n\nThanks to everyone else who shared their approaches! I thought that leaving out the bots with only one bid was a clever idea. And I was surprised to see that people got mileage out of the device type.
", "rawMarkdown": "", "votes": null }, { "id": "81300", "postDate": "06/09/2015 02:03:53", "content": "Nice feature engineering... well done. Pushed the random forest and deep learning models as far as possible but could not get above 0.90, as I principally relied on ratios among the various counts per bidder. Thanks for sharing!
", "rawMarkdown": "", "votes": null }, { "id": "81302", "postDate": "06/09/2015 02:08:41", "content": "Beat me by 3 minutes :D
\nWell, here is my code and here is a brief blog entry I wrote about this competition.
", "rawMarkdown": "", "votes": null }, { "id": "81303", "postDate": "06/09/2015 02:13:33", "content": "Thanks for sharing, this is great! Quick question: how were you able to parse the time into 20-minute intervals and also days of the week? Was that intuition or did I miss an explanation somewhere?
\n\nThanks again, this is extremely helpful!
", "rawMarkdown": "", "votes": null }, { "id": "81304", "postDate": "06/09/2015 02:15:00", "content": "thanks for sharing, and nice job! I had 48 features, thought I had enough features, but turns out you have some features that I don't have which might explain our difference in score.
\n\n[quote=small yellow duck;81297]
\nHere's a little blurb about how I approached feature generation and cross-validation for this problem:
\nhttp://small-yellow-duck.github.io/auction.html
\nI used sklearn's RandomForestClassifier. The most useful features I identified were:
\n- the median time between a user's bid and that user's previous bid
\n- the mean number of bids a user made per auction
\n- the entropy for how many bids a user placed on each day of the week
\n- the means of the per-auction URL entropy and IP entropy for each user
\n- the maximum number of bids in a 20 min span
\n- the total number of bids placed by the user
\n- the average number of bids a user placed per referring URL
\n- the number of bids placed by the user on each of the three weekdays in the data
\n- the minimum and median times between a user's bid and the previous bid by another user in the same auction.
\nCross-validation was a bit tricky: I did CV 100 times on different 80-20 splits. My mean CV score was almost the same (+/- 0.003) as my final leaderboard score.
\n[/quote]
", "rawMarkdown": "", "votes": null }, { "id": "81306", "postDate": "06/09/2015 02:23:15", "content": "Interesting, and thanks for sharing. I tried several of the same statistics as you, e.g., total bids, mean, sd, median, min interbid time, etc., plus some other far out and crazy ideas. Regarding the model, I tried a bunch of different models too, but random forests seem to work the best for this problem.
\nBTW, what is meant by entropy of the number of bids per user, or of URL/IP? Also, how did you know how long 1 day is? Or one week? Or if it's a weekday?
", "rawMarkdown": "", "votes": null }, { "id": "81309", "postDate": "06/09/2015 02:53:54", "content": "1) Kitchensink-approach
\n2) Thinking for the model-model
\n3) Bad IP Model
\n4) Simple stats
\nMin, Max, Average, Unique:
\n5) Final submission
\nWeighted rank average of models.
\n---
\nFor modeling 1-4 I mostly used trees and stacked ensembling of feature sets.
\nModel 4, though with a modest feature set, did best on LB.
", "rawMarkdown": "", "votes": null }, { "id": "81310", "postDate": "06/09/2015 03:01:46", "content": "Thanks for sharing your approach so fast!
\nQuick question: how do you average your five RF models, just arithmetric average or average the rank? I didn't got the chance to try different averaging approach, a ranking average of four single models :one from xgboost, three from sklearn(randomforest, gradient boost, extra tree) boost my score by 0.002, is there better ways to ensemble?
\nI was working with entropy all the time in my research, never thought of using it to generate features for this competition...shame on me...
\nFor me, besides the obvious counting features and their ratios(number of bids, number of auctions, number of ip, number of country...),
\nI also calculated the changing frequency of ip, country, auction, the density of bidding time in all auctions and in a single auctions(calculate the mean and std variance)
\nFor bidding pattern in an auction, I count how often people make continuous bids; the portions of bids in first 10%,30%50%, last 10% 30% 50% of an auction; the maximum number of bids made in an auction(average over all auctions by the bidder and calculate the difference between maximum and mean) ...
\nI also split the ip, address and account into two parts and take the first part as feature, but I don't think it help the performance.
\nOne thing bothers me is that there are 5 instances in the training data marked as bots but only have one bids, so I though their account, address or ip are in some kind of blacklist, so I build a blacklist of account, address and ip, but it doesn't help at all.
\nAbout validation and LB, no matter how I add or reduce features(pca, extra tree importance), even though sometimes my validation score gets higher, my LB score remains the same or get even worse, it's really frustrating, which discouraged me from spending more time to generate new features, guess I didn't find the features that make the public and private LB consistent.
", "rawMarkdown": "", "votes": null }, { "id": "81316", "postDate": "06/09/2015 03:57:33", "content": "Thanks for the sharing.. My first time to know using entropy to generate features(instead of var).
", "rawMarkdown": "", "votes": null }, { "id": "81317", "postDate": "06/09/2015 04:33:31", "content": "\nScored 0.93939, 7th place on private leaderboard
\nI just combined 8 similar models to prevent overfitting.
\nI used average of XGBoost logistic regression, Vowpal Wabbit , GraphLab boosted trees, sk-l Randomforest, sk-l GradientBoosting, sk-l RandomForest+AdaBoost, sk-l RandomForest + CalibratedClassifier, sk-l RandomForest + BaggingClassifier.
", "rawMarkdown": "", "votes": null }, { "id": "81318", "postDate": "06/09/2015 04:33:53", "content": "Congrats smallyellowduck, and awesome writeup!
\n\nI also used the 'everything but the kitchen sink' approach lol.
\n7862 total features were used. (Most of these are just useless device counts)
\nSimple stats (mean, std, min, max) were computed over the following features:
\n* Diversity
* # unique counties, devices, ips, (countries, devices), (countries, ips), (devices, ips) per auction
* Streaks
* longest streak of the same device, ip, (device, ip) per auction
* Peak
* for overlapping time windows:
* # of bids in a given window
* # of auctions in a given window
* # unique auctions in the argmax window
* Count-Vectors
* count vectors for each unique device, country, merch
* First Last
* # instances a bidder submitted the first, last bid of an auction
* Consecutive
* # instances that a bidder has 0, 1, 2, ..., 10 bids in between their bids
* Wavelets
* 3-level db4 wavelet decomposition for 3 overlapping windowed histograms of # of bids
* |mean|, std of coefficients
* differences in |mean|, std of coefficients between windows
* Time Difference
* time since previous bid
* Position
* # of bids in windows relative to auction life
Model
=====
Simple average of extra-trees, gradient boosted trees, neural net. For the neural net, I used the top 1000 features as prioritized by the extra-trees classifier.
Other
=====
I always try semi-supervised learning... and it never helps my CV or LB score. =[
Score
=====
* CV: ~0.92, 0.93, 0.91 (for the respective models)
* Public: 0.91503 (138)
* Private: 0.93649 (20)
Congrats to all the winners.! It was great knowing the different approaches. Here goes mine.
\nFeatures :
1. Overall Count and no of changes (on all features)
2. Auction wise count and No of changes (on all features including mean, min and max)
3. Percentile values of time difference between bids in an auction and overall
4. Number of consecutive bids in an auction
5. Time difference between given bidders bid and the previous bid. (mean, min and max)
6. Count and No of changes of two way interactions (ip-country, ip-device and so on)
I tried using the payment account and address info but with no success.
Modeling Algorithm:
My final model is an ensemble of
1. Random Forest Classifier - scikit
2. Extra Trees Classifier - scikit
3. Gradient Boosting Classifier - scikit
4. XGBoost
I tried Neural Nets as well but it fell well short of 0.9 mark and I was not able to tune it further. Would be interested to know if anyone had success with NN in this competition.
\nI used repeated 5-fold cross validation for model tuning. Mostly, my CV was in sync with public LB. One thing I noticed was when I tuned the models to get best parameters, it performed a little worse than the model with relatively good parameters (based on local CV). I think this may be due to the fact that some bots may be labelled as non-bots in the training set and so the best models tried to classify them correctly as well thereby causing over-fitting.
", "rawMarkdown": "", "votes": null }, { "id": "81320", "postDate": "06/09/2015 04:48:19", "content": "Congratulations to all the winners.
\nMy approach is outlined in this Jupyter notebook.
\nI used somewhat of a text classification approach. I concatenated the bid entries into one long space-delimited string, and then performed tfidf-vectorization on these text features. The idea was that bots would exhibit certain sequential patterns in their behavior that could be discovered with this type of a text mining approach.
\nIn addition to the text features, I also computed some descriptive statistics (median, max, min, range, deciles) for the time differences between bids, along with counts of unique entries for the columns.
\nIn CV, the numeric features stood for the majority of the auc-gain, but the text-features definitely helped a bit.
", "rawMarkdown": "", "votes": null }, { "id": "81321", "postDate": "06/09/2015 05:02:25", "content": "Congrats to the winners! This was a pretty fun competition; figuring out whether my CV scores were lying or not was interesting to say the least.
\nI used the following features:
\nI realized my best scores with XGBoost and 10 fold stratified CV 10 times. My best single model score came as a result of using about half the features listed above. I then used the unused features with the first half to create an ensemble which put me near the top 10% on the public leader board. After which I hit a wall and couldn't really see any ways of improving my score. I am very happy to see that my ensemble moved me closer to the top 5% on the private leader board however!
\nI just want to add one interesting note, K nearest neighbours (using 30 neighbours, and a distance based weight per neighbour) worked surprisingly well with only 5 features. Using the log transformed number of bids, devices, countries, auctions and urls per bidder, I was able to get a score of 0.885 on the public leader board (0.90559 on the private). Not bad for about 15 minutes of effort :)
", "rawMarkdown": "", "votes": null }, { "id": "81331", "postDate": "06/09/2015 08:43:56", "content": "Well, that was an unexpected surprise this morning. I was just hoping to stay in the top 10%.
Congratulations to small yellow duck and the other top scorers.
My approach seems similar to a number of the ones posted above. I used scikit-learn GradientBoostingClassifier (thanks to the authors!).
\nEarly on I saw that CV was going to be relatively inaccurate so I ended up choosing 500 resamples of different 2/3 – 1/3 splits. This took the standard error on my CV AUC calculation down to about 0.0007 so that I could reasonably have an idea of whether each feature I tested was making a positive or negative difference. The inaccuracy on a single CV fold and the helpful post by T. Scharf, https://www.kaggle.com/c/facebook-recruiting-iv-human-or-bot/forums/t/14394/a-basic-look-at-lb-scores
suggested that public LB scores were not going to be particularly useful so I only made 3 submissions to avoid any temptation to overfit the public LB - I find this a very difficult thing to do…
Some of my most helpful features were:
- The proportion of each bidder’s bids that were in each day ( I believe there were 9 days of bid data).
- The proportion of each bidder’s bids that were in each of 8 3-hour time slots.
- The mean time of day for the bids of each bidder.
- The total number of bids for each bidder.
- The frequency of each final part of the IP address averaged for each bidder – the first three parts had no positive effect.
- The average proportion of bots associated with each device code. NB – (difficult to put into CV without data leakage – a slightly different data set for each fold!)
- Same as above for each country code.
- The total number of bids made in each auction averaged for each bidder. (i.e. did the bidder tend to bid in popular auctions or less popular ones)
- The maximum number of auctions a bidder has participated in, in one hour.
Hopefully, this gives a flavour of what I was trying. Some of the features are quite hard to put into words!
\nOne final trick that may have made the difference is that I looked at the histogram of the number of bids made by each bot in the training set. There were 5 bots that made just one bid, which looked anomalous to my eye. I wondered if this might not help to predict on a different data set so I submitted an entry which was just my main model but trained on the training set without these five bots. I think this one was the winner, but I’ll confirm when I can see all my submission private scores.
\nI’ll post in greater detail when I have written my model up. Apologies, as was not expecting to win!
", "rawMarkdown": "", "votes": null }, { "id": "81338", "postDate": "06/09/2015 12:13:52", "content": "Two ideas:
\n1- removing the last 5 characters from both address and payment_account
\n2- grouping time to 3 categories (First, Middle, Last)
\n\nthen, i guess this is not good, i trained a Robot Bid detector (instead of a Robot detector).
\nand then i used a aggregation function to detect the Robots based on their RobotsBid probabilities; i tried many ones, but MAX is the better one i found (MAX of probability of all RoboticBids that the bidder made; Median or Mean performs poorly).
\n\nfor this i tried AzureML with some light code of R
\ni used CV of a SVM (single parameter)
\n\ni didn't have enough time to try more:
\nidea 2 was not a good one, i guess
\ni liked the ideas of Small Yellow Duck
\n\n\n\nthanks for sharing your ideas
", "rawMarkdown": "", "votes": null }, { "id": "81343", "postDate": "06/09/2015 12:29:49", "content": "[quote=fakeplastictrees;81331]
\nOne final trick that may have made the difference is that I looked at the histogram of the number of bids made by each bot in the training set. There were 5 bots that made just one bid, which looked anomalous to my eye. I wondered if this might not help to predict on a different data set so I submitted an entry which was just my main model but trained on the training set without these five bots. I think this one was the winner, but I’ll confirm when I can see all my submission private scores.
\n[/quote]
\nEven though my final score is dreadful due to starting the competition really late, I can add to this.
Removing the 5 bots that made just one bid increased my CV auc by ~0.03, however adding another step at postprocessing increased the public leaderboard score by about 0.02 aswell.
I searched for the "signatures" of the bids of the five bots that were banned just after one bid in the bid dataset, and it appeared that each one of them had the same signature as a detected bot with a number of bids > 1.
The signatures were the auction, referral site, time, device, ip and (obviously) country of the bids. If there were two bids that had equal information, and one bid belonged to a bot bidder, the other bid was also flagged as a bot bidder.
So I added a post-processing step which searched for all the matches (test set bid - train set flagged bot bid) after classification, and flagged them as a highly probable bot.
Unfortunately, since I had only three submissions to use, I couldn't experiment with searching for matches in the test set as well (same signature as a highly probable bot from the test set), and then copying the probabilities to the max of the two.
", "rawMarkdown": "", "votes": null }, { "id": "81352", "postDate": "06/09/2015 14:39:26", "content": "Fun contest! Congrats to everyone who participated.
\nHere's my model and description.
\nI used sklearn's RandomForestClassifier and LogisticRegression models (an 80-20 mix of the two). Most of my features were the same as what others have mentioned (total # bids, total # auctions, average response time, etc.) One interesting feature that may have helped me out slightly was the mean number of known bots in each auction of participation (not including the bidder we're trying to classify).
\nThe CV was difficult for me too - I averaged my performance over 500 trials with a random 80-20 split of the training data. This showed around a 0.93 performance on the CV data (closer to what I got on the private leaderboard) but on the public leaderboard the best I could get was 0.915.
", "rawMarkdown": "", "votes": null }, { "id": "81396", "postDate": "06/09/2015 19:11:59", "content": "Congrats to all the winners!
I used scikit learn RandomForestClassifier algorithm.
List of decent features according to cross-validation feature selection method I used:
1. absolute number of bids per user.
2. num uniq devices, auctions, ips and urls
3. avg and first-non-zero time diffs between user bids
4. percentage of bids made with favorite(most common) device
5. percentage of bids made from favorite country
6. max number of bids in a single auction
7. number of simultaneous bids
8. number of device switches, numer of auctions switches, number of ips switches, number of countries switches
e.g. this sequence corresponds to 3 'switches' (with 2 devices): [dev1, dev1, dev2, dev2, dev2, dev2, dev1]
9. percentage of user bids made in first/last 5% of auction observed timeline
(observed timeline doesn't correspond to real auction timeline, but in my opinion still helps to understand if user prefers to bid at the beginning or end of an auction)
10. average number of transaction in the same country in 1-hour window with various normalizations
(in my opinion this feature roughly reflects user favorite bidding time of the day)
11. two categ->response transformations features. average response for country and device.
the same idea as discussed here: https://www.kaggle.com/forums/f/15/kaggle-forum/t/8005/replacing-categorical-variables-with-historic-response-rate
there were also a few other trivial features
\nFor cross validation I calculated average AUC for 10 different but fixed 10-fold splits, so totaly it's 10*10 algorithm trains for a single feature check.
", "rawMarkdown": "", "votes": null }, { "id": "81397", "postDate": "06/09/2015 19:48:38", "content": "[quote=mechatroner;81396]11. two categ->response transformations features. average response for country and device.
the same idea as discussed here: https://www.kaggle.com/forums/f/15/kaggle-forum/t/8005/replacing-categorical-variables-with-historic-response-rate
[/quote]
\nI tried this (also for merchandise) and, while it incremented my local CV score by about 0.1 - 0.2 (only the country historical information), it did not reflect well on the leaderboard. So I think this introduces more leakage / circularity problems than anything else.
", "rawMarkdown": "", "votes": null }, { "id": "81398", "postDate": "06/09/2015 20:04:20", "content": "Congratulations to the winners, this was a really fun contest!
\nFor classification I used a two-stage meta-learner approach with RF, XGBoost and NN as base learners and a combination of those three as meta learners.
\nThe final features I used are as follows:
\n1. Fraction of auctions won by member (out of total number of auctions member participated in).
\n2. Mean fraction of bids made by member (in auctions member participated in).
\n3. Mean fraction of bids made by member within first 10% of bids (in auctions member participated in).
\n4. Mean fraction of bids made by member within last 25% of bids (in auctions member participated in).
\n5. Mean fraction of bids made by member (in auctions member won).
\n6. Mean fraction of bids made by member within first 10% of bids (in auctions member won).
\n7. Mean fraction of bids made by member within last 25% of bids (in auctions member won).
\n8. , 9. , 10. Max, Mean and Standard Deviation of number of auctions member is active in per minute.
\n11., 12., 13. Max, Mean and Standard Deviation of number of auctions member is active in per hour.
\n14.,15.,16. Probability of bot given Country, Merchandise search category, and device
\n17., 18., 19. Max, Mean and Standard Deviation of number of ips used per minute.
\n20., 21., 22. Max, Mean and Standard Deviation of number of ips used per hour.
\n23. Probabilty of being referred by url : 'vasstdc27m7nks3'
\n24. Probability of being a Bot given you belong to a cluster of Payment Accounts (Clustered using Hamming Distance, with a Ward Linkage)
\n25. Probability of being a Bot given you belong to a cluster of Addresses (Clustered using Hamming Distance, with a Ward Linkage)
\n26., 27., 28. Mean, Standard Deviation and Sarle's bimodality coefficient of auction bid value (assuming first bid has a value of 10 cents and each subsequent bid increases it by one cent)
\n29., 30. Mean and Standard Deviation of length of continuous bid streaks by member.
\n31., 32. Mean and Standard Deviation of time between Member's bid streaks and previous bidders' bid.
\nI used a Random Forest classifier with Recursive Feature Elimination + Cross Validation to select these features, my 5-fold cross-validation score was similar to the public LB score with -0.003 difference, and it did not change much on the private LB either. :)
", "rawMarkdown": "", "votes": null }, { "id": "81424", "postDate": "06/09/2015 23:29:49", "content": "Hi all, congrats to everyone. I've seen people doing incredible work here!
\nI was ranked 32nd on private LB (but could have been 17th if I had sent my submission using the most-frequent-URL-of-bidder feature :( ). My work on feature and modeling is actually pretty simple. You can find my blog post which is basically a dump of my IPython notebook (so code + comments) here: http://www.thomas-robert.fr/en/kaggles-human-or-robot-competition-feedback/
", "rawMarkdown": "", "votes": null }, { "id": "81425", "postDate": "06/09/2015 23:36:39", "content": "Please find my aftermath blog post here http://podlipensky.com/2015/06/kaggle-competition-aftermath/
\nHere is my best feature set:
\n['ip_rank', 'auction_rank', 'auction/increments', 'avg_unique_ips',
'ip/increments', 'std_inc_per_auction', 'product/won_auctions',
'payment_type=0875307e1731af94b3b64725ad0deb7d', 'device/increments',
'country_rank', 'avg_inc_per_auction', 'ip/ref', 'avg_ips_per_increment',
'true_bids_count', 'increments/sim_auctions', 'increments/won_auctions',
'country/increments', 'ip_change', 'ip/sim_auctions', 'price_mean',
'sim_auctions', 'frequent_ip=8.116', 'std_ips_per_auction', 'device/ip',
'product/increments', 'device/ref', 'product/ip', 'product/sim_auctions',
'referral_change', 'product/ref', 'ref/auction', 'device/auction',
'avg_referrals_per_auction', 'ref/sim_auctions', 'avg_ips_per_auction',
'median_inc_per_auction', 'ref/increments', 'price_avg', 'auctions_count',
'unique_ips', 'country/ip', 'won_auctions/sim_auctions',
'unique_countries_count', 'robot_price_rmse', 'avg_devices_per_auction',
'std_referrals_per_auction', 'std_devices_per_auction',
'device/sim_auctions', 'ip/auction', 'ref/won_auctions']
(it is kind of cryptic - please find better description of those in my blog post)
", "rawMarkdown": "", "votes": null }, { "id": "81442", "postDate": "06/10/2015 03:53:28", "content": "The code fbHomeBag35.R obtains: 0.90607 0.93256
\nThe code fbHomeBag16.R obtains: 0.90811 0.93419
\nI didn't select either.
\nTo overfit is only human.
", "rawMarkdown": "", "votes": null }, { "id": "81448", "postDate": "06/10/2015 05:52:12", "content": "[quote=small yellow duck;81297]
\nHere's a little blurb about how I approached feature generation and cross-validation for this problem:
\nhttp://small-yellow-duck.github.io/auction.html
\ncode: https://github.com/small-yellow-duck/facebook_auction
\n[/quote]
\nHow you decoded time variable is really amazing. What I've done about that variable was assuming it as the count of milliseconds since 1.1.1970 and converting it to a date after ignoring last 4 numbers of the integer value. Thus, I assume time/10000 as the time in milliseconds, which is apparently not correct according to your analysis.
\nSuch as: 9704553947368421 -> 970455394736 -> 02 October 2000 Monday 02:56:34 GMT
\nhttp://www.fileformat.info/tip/java/date2millis.htm
\nCongrats..
", "rawMarkdown": "", "votes": null }, { "id": "81454", "postDate": "06/10/2015 07:19:37", "content": "I have a long way to go with feature engineering.In this competition.I just pick out 6 properties:AUCTION,DEVICE,TIME,COUNTRY,IP and URL.Then I calculate the pearson correlation coefficient between any two of the 6 properties and 21 features are generated.
\nI believe my method is more generalized as the feature engineering nees domain knowledge and the feature-extraction will differ in various cases.However,my method is not able to give an accuracy more than 77%.The truth is that the feature engineering is one of the most powerful tools to seek truth
", "rawMarkdown": "", "votes": null }, { "id": "81480", "postDate": "06/10/2015 14:03:16", "content": "[quote=Sushant Tripathy;81398]
\n14.,15.,16. Probability of bot given Country, Merchandise search category, and device
\n[/quote]
\n\nCongratulations to the winners, and thanks to Kaggle/facebook for hosting/organizing the competition. I had a lot of fun participating it. I just wish I had more time to work on it.
\nSushant Tripathy, Impressive work on feature engineering!
\nHope you don't mind just a clarification on how you calculated feature 14, 15 and 16.
\nFor example let's consider feature 15.
\nIn my understanding you may have calculated it as follows:
\n(# of merchandise type per class bot)/(size of the fold)
\nso for example (just random figures here), if your merchandise 'jewelry' has 100 class 1 (which means bot), and your fold size is 1200 then your feature 15 would be 100/1200= 0.083
\nThen you would build a feature that has 0.083 when that bidder has 'jewelry' as merchandise.
\nis this explanation correct?
\nHave you maybe tried dummifying merchandise? did you see any performance change?
\nI tried both and I actually did notice an increase in overfitting. I explained that to myself, considering "cheating" including any information about the output class in your train dataset. Any comments about this?
", "rawMarkdown": "", "votes": null }, { "id": "81481", "postDate": "06/10/2015 14:08:21", "content": "[quote=Dario Bernardo;81480]I tried both and I actually did notice an increase in overfitting. I explained that to myself, considering "cheating" including any information about the output class in your train dataset. Any comments about this?
\n[/quote]
\nSame in my experience. I did not try "dummyfying" the variables, but I noticed the overfitting effect.
", "rawMarkdown": "", "votes": null }, { "id": "81482", "postDate": "06/10/2015 14:23:19", "content": "[quote=Dario Bernardo;81480]
\nSushant Tripathy, Impressive work on feature engineering!
\nHope you don't mind just a clarification on how you calculated feature 14, 15 and 16.
\nFor example let's consider feature 15.
\nIn my understanding you may have calculated it as follows:
\n(# of merchandise type per class bot)/(size of the fold)
\nso for example (just random figures here), if your merchandise 'jewelry' has 100 class 1 (which means bot), and your fold size is 1200 then your feature 15 would be 100/1200= 0.083
\nThen you would build a feature that has 0.083 when that bidder has 'jewelry' as merchandise.
\nis this explanation correct?
\nHave you maybe tried dummifying merchandise? did you see any performance change?
\nI tried both and I actually did notice an increase in overfitting. I explained that to myself, considering "cheating" including any information about the output class in your train dataset. Any comments about this?
\n[/quote]
\nHi Dario, thanks for the kind words. My approach for building these features relied on using values for samples with known outcomes.
\nFor example if merchandise search category was 'jewellery', and my stratified training set (66% partition) contained 5 known Bots and 10 known humans (using the search for 'jewellery') then I would assign a P(Bot|Merch='jewellery') as 5/15 = 0.33.
\nIn case I don't have any bots or I don't have any humans or there aren't enough labelled members using that search category (thresholded at >= 50), I would assign the overall probability of encountering bots for that run of Stratified CV. So, P(Bot|Merch='jewellery') = 50/1050 = 0.0476 (Assuming my training set partition has 50 bots and 1000 humans).
\nNow say I am constructing features for member X, who has 12 searches for 'jewellery' and 10 searches for 'sports goods'. Her/his P(Bot|Merch) = (12*P(Bot|Merch='jewellery') + 10*P(Bot|Merch='sports goods'))/(12 + 10)
\nAlso, I restricted my probabilistic feature construction to only the samples within my training partition, the CV scores on the validation partition were pretty spot on.
\n[quote=Dario Bernardo;81480]
\nHave you maybe tried dummifying merchandise? did you see any performance change?
\nI tried both and I actually did notice an increase in overfitting. I explained that to myself, considering "cheating" including any information about the output class in your train dataset. Any comments about this?
\n[/quote]
\nRegarding dummification, the assignment of the overall probability is a step in that direction although not a full-blown one. Here, in my opinion, as long as one doesn't use the validation partition to construct the probabilistic features (thus avoiding data leakage/circularity) the CV score would be an accurate representation of out of sample performance. Would love to hear your thoughts about it. :)
\nAs an after-thought, it was really early morning here when I started writing this comment, I am really sleepy and I probably made some mistakes. :)
", "rawMarkdown": "", "votes": null }, { "id": "81596", "postDate": "06/11/2015 11:30:18", "content": "I detailed my feature engineering in this blog post (120 + features, hope this helps). I tested a lot of things after that, mostly blend of GBT and random forests by I ended up overfitting my training data even though my private and public leaderboard score are pretty close.
", "rawMarkdown": "", "votes": null }, { "id": "81692", "postDate": "06/12/2015 13:06:10", "content": "Sushant,
\nIf I correctly understood your procedure, bidders with same "search category" (or merchandise) will end up having the same feature 15 score, so effectively that feature will not help distinguish bidders coming from the same search category, because will have the same score, but eventually will help to distinguish bidders coming from different search categories. Therefore my point was that in my opinion there shouldn't be much difference with a dummification approach of the "merchandise".
\nPlease note that I am not trying to say that you did a mistake or that one or the other approach is better. I am just making some general considerations. In order to make such conclusion (assuming that i correctly got your point) I should run lot's of experiments first. After this competition there is so much to revise and re-analyse. I don't want to flood the discussion with further general consideration so I will stop here. I will conclude just thanking you for your kind response and thanking you and the others for posting your solutions. I have learned so much!
", "rawMarkdown": "", "votes": null }, { "id": "81763", "postDate": "06/12/2015 22:51:25", "content": "Hi Dario,
\nThe feature calculation consists of two parts one is to determine what is the probability of bots using the "search category" and then to determine what is the weighted probability of a member being a bot based on the number of bids of that member that originated with certain "search categories".
\nTo make it a little clear, I have provided an example below:
\nUser ID Bot Search Category Bids
u1 Yes s1 5
u1 Yes s2 4
u1 Yes s3 5
u1 Yes s4 1
u2 No s1 10
u2 No s2 8
u2 No s3 9
u2 No s4 7
u3 No s3 11
u3 No s4 12
Now Step 1:
\nSearch Category Humans Using It Bots Using It p(Bot Use|Search Category)
s1 1 1 0.5
s2 1 1 0.5
s3 2 1 0.33
s4 2 1 0.33
Now Step 2 (calculation of bid weighted probabiliies):
\nUser ID Bot Search Category Bids p(Bot Use| Search Category ) Bid Weighted Probabilities
u1 Yes s1 5 0.5 2.5
u1 Yes s2 4 0.5 2
u1 Yes s3 5 0.33 1.65
u1 Yes s4 1 0.33 0.33
u2 No s1 10 0.5 5
u2 No s2 8 0.5 4
u2 No s3 9 0.33 2.97
u2 No s4 7 0.33 2.31
u3 No s3 11 0.33 3.63
u3 No s4 12 0.33 3.96
Now finally to calculate P(Bot|Search Usage) (Bids weighted average of P(Bot usage|Search Categories):
\nFor u1 = (2.5 + 2 + 1.65 + 0.33)/(5 + 4 + 5 + 1) = 0.432
\nFor u2 = (5 + 4 + 2.97 + 2.31)/(10 + 8 + 9 + 7) = 0.42
\nFor u3 = (3.63 + 3.96)/(11+12) = 0.33
\nNow these values are Feature 15. Hopefully, I could explain the calculations a bit more clearly this time around! :) Some members do have similar values, but never the same, there is always some difference (unless of course two members had the exact same number of bids within the exact same search categories and used the same search categories [rarely happens]). That being said, I did not try any extensive dummification approaches, and have no idea how dummification would have fared. :) I would love to hear about it, when you have had time to compare the approaches.
\nDiscussions are always welcome on Kaggle forums (in my opinion :D ), and of course I would be happy to discuss feature construction with you further. I have a lot to learn as well. :)
", "rawMarkdown": "", "votes": null }, { "id": "81791", "postDate": "06/13/2015 07:00:13", "content": "Hey guys
\nI did a bit of a write-up on my approach that got me to first place on the public leaderboard, but 17th on the private leaderboard. Given I'm still fairly new to all this, I'd be interested to hear people's thoughts on how this might have happened. I suppose I just hill-climbed on the public leaderboard a bit...
", "rawMarkdown": "", "votes": null }, { "id": "82369", "postDate": "06/19/2015 01:34:51", "content": "Very useful to see feedback. This was my first competition (I'm a hobbyist), and it was fun even though I am nowhere to be seen in the leader-boards.
", "rawMarkdown": "", "votes": null }, { "id": "82428", "postDate": "06/19/2015 18:12:56", "content": "small yellow duck's approach, background, and interesting ideas on Kaggle and ML in medicine can now be found in this competition write-up on no free hunch.
\nThanks for again for putting together such a great post!
", "rawMarkdown": "", "votes": null }, { "id": "107844", "postDate": "02/13/2016 02:13:57", "content": "[quote=Artem;81317]
\n\nScored 0.93939, 7th place on private leaderboard
\nI just combined 8 similar models to prevent overfitting.
\nI used average of XGBoost logistic regression, Vowpal Wabbit , GraphLab boosted trees, sk-l Randomforest, sk-l GradientBoosting, sk-l RandomForest+AdaBoost, sk-l RandomForest + CalibratedClassifier, sk-l RandomForest + BaggingClassifier.
\n[/quote]
\nHi,
\nDid you posted the approach and solution on any blog? I am beginner and I want to learn. Thanks
", "rawMarkdown": "", "votes": null }, { "id": "172305", "postDate": "04/03/2017 06:24:17", "content": "Link doesn't seem to work.
", "rawMarkdown": "Link doesn't seem to work.", "votes": null }, { "id": "501998", "postDate": "03/28/2019 03:30:50", "content": "Can you share me the link of your code? Your blog looks very impressive. I am a beginner and I want to learn.
", "rawMarkdown": "Can you share me the link of your code? Your blog looks very impressive. I am a beginner and I want to learn.", "votes": null }, { "id": "736050", "postDate": "02/03/2020 18:48:34", "content": "Can you please share the github link of the notebook. Thank you
", "rawMarkdown": "Can you please share the github link of the notebook. Thank you", "votes": null } ], "comments": [ { "id": 81300, "author_name": "mconway", "author_url": "", "post_date": "06/09/2015 02:03:53", "content": "Nice feature engineering... well done. Pushed the random forest and deep learning models as far as possible but could not get above 0.90, as I principally relied on ratios among the various counts per bidder. Thanks for sharing!
", "votes": null, "replies": [] }, { "id": 81302, "author_name": "rinzewind", "author_url": "", "post_date": "06/09/2015 02:08:41", "content": "Beat me by 3 minutes :D
\nWell, here is my code and here is a brief blog entry I wrote about this competition.
", "votes": null, "replies": [] }, { "id": 81303, "author_name": "ivanko", "author_url": "", "post_date": "06/09/2015 02:13:33", "content": "Thanks for sharing, this is great! Quick question: how were you able to parse the time into 20-minute intervals and also days of the week? Was that intuition or did I miss an explanation somewhere?
\n\nThanks again, this is extremely helpful!
", "votes": null, "replies": [] }, { "id": 81304, "author_name": "leoyoung8", "author_url": "", "post_date": "06/09/2015 02:15:00", "content": "thanks for sharing, and nice job! I had 48 features, thought I had enough features, but turns out you have some features that I don't have which might explain our difference in score.
\n\n[quote=small yellow duck;81297]
\nHere's a little blurb about how I approached feature generation and cross-validation for this problem:
\nhttp://small-yellow-duck.github.io/auction.html
\nI used sklearn's RandomForestClassifier. The most useful features I identified were:
\n- the median time between a user's bid and that user's previous bid
\n- the mean number of bids a user made per auction
\n- the entropy for how many bids a user placed on each day of the week
\n- the means of the per-auction URL entropy and IP entropy for each user
\n- the maximum number of bids in a 20 min span
\n- the total number of bids placed by the user
\n- the average number of bids a user placed per referring URL
\n- the number of bids placed by the user on each of the three weekdays in the data
\n- the minimum and median times between a user's bid and the previous bid by another user in the same auction.
\nCross-validation was a bit tricky: I did CV 100 times on different 80-20 splits. My mean CV score was almost the same (+/- 0.003) as my final leaderboard score.
\n[/quote]
", "votes": null, "replies": [] }, { "id": 81306, "author_name": "lchiaying", "author_url": "", "post_date": "06/09/2015 02:23:15", "content": "Interesting, and thanks for sharing. I tried several of the same statistics as you, e.g., total bids, mean, sd, median, min interbid time, etc., plus some other far out and crazy ideas. Regarding the model, I tried a bunch of different models too, but random forests seem to work the best for this problem.
\nBTW, what is meant by entropy of the number of bids per user, or of URL/IP? Also, how did you know how long 1 day is? Or one week? Or if it's a weekday?
", "votes": null, "replies": [] }, { "id": 81309, "author_name": "triskelion", "author_url": "", "post_date": "06/09/2015 02:53:54", "content": "1) Kitchensink-approach
\n2) Thinking for the model-model
\n3) Bad IP Model
\n4) Simple stats
\nMin, Max, Average, Unique:
\n5) Final submission
\nWeighted rank average of models.
\n---
\nFor modeling 1-4 I mostly used trees and stacked ensembling of feature sets.
\nModel 4, though with a modest feature set, did best on LB.
", "votes": null, "replies": [] }, { "id": 81310, "author_name": "skylibrary", "author_url": "", "post_date": "06/09/2015 03:01:46", "content": "Thanks for sharing your approach so fast!
\nQuick question: how do you average your five RF models, just arithmetric average or average the rank? I didn't got the chance to try different averaging approach, a ranking average of four single models :one from xgboost, three from sklearn(randomforest, gradient boost, extra tree) boost my score by 0.002, is there better ways to ensemble?
\nI was working with entropy all the time in my research, never thought of using it to generate features for this competition...shame on me...
\nFor me, besides the obvious counting features and their ratios(number of bids, number of auctions, number of ip, number of country...),
\nI also calculated the changing frequency of ip, country, auction, the density of bidding time in all auctions and in a single auctions(calculate the mean and std variance)
\nFor bidding pattern in an auction, I count how often people make continuous bids; the portions of bids in first 10%,30%50%, last 10% 30% 50% of an auction; the maximum number of bids made in an auction(average over all auctions by the bidder and calculate the difference between maximum and mean) ...
\nI also split the ip, address and account into two parts and take the first part as feature, but I don't think it help the performance.
\nOne thing bothers me is that there are 5 instances in the training data marked as bots but only have one bids, so I though their account, address or ip are in some kind of blacklist, so I build a blacklist of account, address and ip, but it doesn't help at all.
\nAbout validation and LB, no matter how I add or reduce features(pca, extra tree importance), even though sometimes my validation score gets higher, my LB score remains the same or get even worse, it's really frustrating, which discouraged me from spending more time to generate new features, guess I didn't find the features that make the public and private LB consistent.
", "votes": null, "replies": [] }, { "id": 81316, "author_name": "lijunjie", "author_url": "", "post_date": "06/09/2015 03:57:33", "content": "Thanks for the sharing.. My first time to know using entropy to generate features(instead of var).
", "votes": null, "replies": [] }, { "id": 81317, "author_name": "rushter", "author_url": "", "post_date": "06/09/2015 04:33:31", "content": "\nScored 0.93939, 7th place on private leaderboard
\nI just combined 8 similar models to prevent overfitting.
\nI used average of XGBoost logistic regression, Vowpal Wabbit , GraphLab boosted trees, sk-l Randomforest, sk-l GradientBoosting, sk-l RandomForest+AdaBoost, sk-l RandomForest + CalibratedClassifier, sk-l RandomForest + BaggingClassifier.
", "votes": null, "replies": [] }, { "id": 81318, "author_name": "jasontam", "author_url": "", "post_date": "06/09/2015 04:33:53", "content": "Congrats smallyellowduck, and awesome writeup!
\n\nI also used the 'everything but the kitchen sink' approach lol.
\n7862 total features were used. (Most of these are just useless device counts)
\nSimple stats (mean, std, min, max) were computed over the following features:
\n* Diversity
* # unique counties, devices, ips, (countries, devices), (countries, ips), (devices, ips) per auction
* Streaks
* longest streak of the same device, ip, (device, ip) per auction
* Peak
* for overlapping time windows:
* # of bids in a given window
* # of auctions in a given window
* # unique auctions in the argmax window
* Count-Vectors
* count vectors for each unique device, country, merch
* First Last
* # instances a bidder submitted the first, last bid of an auction
* Consecutive
* # instances that a bidder has 0, 1, 2, ..., 10 bids in between their bids
* Wavelets
* 3-level db4 wavelet decomposition for 3 overlapping windowed histograms of # of bids
* |mean|, std of coefficients
* differences in |mean|, std of coefficients between windows
* Time Difference
* time since previous bid
* Position
* # of bids in windows relative to auction life
Model
=====
Simple average of extra-trees, gradient boosted trees, neural net. For the neural net, I used the top 1000 features as prioritized by the extra-trees classifier.
Other
=====
I always try semi-supervised learning... and it never helps my CV or LB score. =[
Score
=====
* CV: ~0.92, 0.93, 0.91 (for the respective models)
* Public: 0.91503 (138)
* Private: 0.93649 (20)
Congrats to all the winners.! It was great knowing the different approaches. Here goes mine.
\nFeatures :
1. Overall Count and no of changes (on all features)
2. Auction wise count and No of changes (on all features including mean, min and max)
3. Percentile values of time difference between bids in an auction and overall
4. Number of consecutive bids in an auction
5. Time difference between given bidders bid and the previous bid. (mean, min and max)
6. Count and No of changes of two way interactions (ip-country, ip-device and so on)
I tried using the payment account and address info but with no success.
Modeling Algorithm:
My final model is an ensemble of
1. Random Forest Classifier - scikit
2. Extra Trees Classifier - scikit
3. Gradient Boosting Classifier - scikit
4. XGBoost
I tried Neural Nets as well but it fell well short of 0.9 mark and I was not able to tune it further. Would be interested to know if anyone had success with NN in this competition.
\nI used repeated 5-fold cross validation for model tuning. Mostly, my CV was in sync with public LB. One thing I noticed was when I tuned the models to get best parameters, it performed a little worse than the model with relatively good parameters (based on local CV). I think this may be due to the fact that some bots may be labelled as non-bots in the training set and so the best models tried to classify them correctly as well thereby causing over-fitting.
", "votes": null, "replies": [ { "id": 736050, "author_name": "mojtylor", "author_url": "", "post_date": "02/03/2020 18:48:34", "content": "Can you please share the github link of the notebook. Thank you
", "votes": null, "replies": [] } ] }, { "id": 81320, "author_name": "wallinm1", "author_url": "", "post_date": "06/09/2015 04:48:19", "content": "Congratulations to all the winners.
\nMy approach is outlined in this Jupyter notebook.
\nI used somewhat of a text classification approach. I concatenated the bid entries into one long space-delimited string, and then performed tfidf-vectorization on these text features. The idea was that bots would exhibit certain sequential patterns in their behavior that could be discovered with this type of a text mining approach.
\nIn addition to the text features, I also computed some descriptive statistics (median, max, min, range, deciles) for the time differences between bids, along with counts of unique entries for the columns.
\nIn CV, the numeric features stood for the majority of the auc-gain, but the text-features definitely helped a bit.
", "votes": null, "replies": [] }, { "id": 81321, "author_name": "abhishekbarnwal", "author_url": "", "post_date": "06/09/2015 05:02:25", "content": "Congrats to the winners! This was a pretty fun competition; figuring out whether my CV scores were lying or not was interesting to say the least.
\nI used the following features:
\nI realized my best scores with XGBoost and 10 fold stratified CV 10 times. My best single model score came as a result of using about half the features listed above. I then used the unused features with the first half to create an ensemble which put me near the top 10% on the public leader board. After which I hit a wall and couldn't really see any ways of improving my score. I am very happy to see that my ensemble moved me closer to the top 5% on the private leader board however!
\nI just want to add one interesting note, K nearest neighbours (using 30 neighbours, and a distance based weight per neighbour) worked surprisingly well with only 5 features. Using the log transformed number of bids, devices, countries, auctions and urls per bidder, I was able to get a score of 0.885 on the public leader board (0.90559 on the private). Not bad for about 15 minutes of effort :)
", "votes": null, "replies": [] }, { "id": 81331, "author_name": "fakeplastictrees", "author_url": "", "post_date": "06/09/2015 08:43:56", "content": "Well, that was an unexpected surprise this morning. I was just hoping to stay in the top 10%.
Congratulations to small yellow duck and the other top scorers.
My approach seems similar to a number of the ones posted above. I used scikit-learn GradientBoostingClassifier (thanks to the authors!).
\nEarly on I saw that CV was going to be relatively inaccurate so I ended up choosing 500 resamples of different 2/3 – 1/3 splits. This took the standard error on my CV AUC calculation down to about 0.0007 so that I could reasonably have an idea of whether each feature I tested was making a positive or negative difference. The inaccuracy on a single CV fold and the helpful post by T. Scharf, https://www.kaggle.com/c/facebook-recruiting-iv-human-or-bot/forums/t/14394/a-basic-look-at-lb-scores
suggested that public LB scores were not going to be particularly useful so I only made 3 submissions to avoid any temptation to overfit the public LB - I find this a very difficult thing to do…
Some of my most helpful features were:
- The proportion of each bidder’s bids that were in each day ( I believe there were 9 days of bid data).
- The proportion of each bidder’s bids that were in each of 8 3-hour time slots.
- The mean time of day for the bids of each bidder.
- The total number of bids for each bidder.
- The frequency of each final part of the IP address averaged for each bidder – the first three parts had no positive effect.
- The average proportion of bots associated with each device code. NB – (difficult to put into CV without data leakage – a slightly different data set for each fold!)
- Same as above for each country code.
- The total number of bids made in each auction averaged for each bidder. (i.e. did the bidder tend to bid in popular auctions or less popular ones)
- The maximum number of auctions a bidder has participated in, in one hour.
Hopefully, this gives a flavour of what I was trying. Some of the features are quite hard to put into words!
\nOne final trick that may have made the difference is that I looked at the histogram of the number of bids made by each bot in the training set. There were 5 bots that made just one bid, which looked anomalous to my eye. I wondered if this might not help to predict on a different data set so I submitted an entry which was just my main model but trained on the training set without these five bots. I think this one was the winner, but I’ll confirm when I can see all my submission private scores.
\nI’ll post in greater detail when I have written my model up. Apologies, as was not expecting to win!
", "votes": null, "replies": [] }, { "id": 81338, "author_name": "hasbou", "author_url": "", "post_date": "06/09/2015 12:13:52", "content": "Two ideas:
\n1- removing the last 5 characters from both address and payment_account
\n2- grouping time to 3 categories (First, Middle, Last)
\n\nthen, i guess this is not good, i trained a Robot Bid detector (instead of a Robot detector).
\nand then i used a aggregation function to detect the Robots based on their RobotsBid probabilities; i tried many ones, but MAX is the better one i found (MAX of probability of all RoboticBids that the bidder made; Median or Mean performs poorly).
\n\nfor this i tried AzureML with some light code of R
\ni used CV of a SVM (single parameter)
\n\ni didn't have enough time to try more:
\nidea 2 was not a good one, i guess
\ni liked the ideas of Small Yellow Duck
\n\n\n\nthanks for sharing your ideas
", "votes": null, "replies": [] }, { "id": 81343, "author_name": "mtutek", "author_url": "", "post_date": "06/09/2015 12:29:49", "content": "[quote=fakeplastictrees;81331]
\nOne final trick that may have made the difference is that I looked at the histogram of the number of bids made by each bot in the training set. There were 5 bots that made just one bid, which looked anomalous to my eye. I wondered if this might not help to predict on a different data set so I submitted an entry which was just my main model but trained on the training set without these five bots. I think this one was the winner, but I’ll confirm when I can see all my submission private scores.
\n[/quote]
\nEven though my final score is dreadful due to starting the competition really late, I can add to this.
Removing the 5 bots that made just one bid increased my CV auc by ~0.03, however adding another step at postprocessing increased the public leaderboard score by about 0.02 aswell.
I searched for the "signatures" of the bids of the five bots that were banned just after one bid in the bid dataset, and it appeared that each one of them had the same signature as a detected bot with a number of bids > 1.
The signatures were the auction, referral site, time, device, ip and (obviously) country of the bids. If there were two bids that had equal information, and one bid belonged to a bot bidder, the other bid was also flagged as a bot bidder.
So I added a post-processing step which searched for all the matches (test set bid - train set flagged bot bid) after classification, and flagged them as a highly probable bot.
Unfortunately, since I had only three submissions to use, I couldn't experiment with searching for matches in the test set as well (same signature as a highly probable bot from the test set), and then copying the probabilities to the max of the two.
", "votes": null, "replies": [] }, { "id": 81352, "author_name": "mikeyg263", "author_url": "", "post_date": "06/09/2015 14:39:26", "content": "Fun contest! Congrats to everyone who participated.
\nHere's my model and description.
\nI used sklearn's RandomForestClassifier and LogisticRegression models (an 80-20 mix of the two). Most of my features were the same as what others have mentioned (total # bids, total # auctions, average response time, etc.) One interesting feature that may have helped me out slightly was the mean number of known bots in each auction of participation (not including the bidder we're trying to classify).
\nThe CV was difficult for me too - I averaged my performance over 500 trials with a random 80-20 split of the training data. This showed around a 0.93 performance on the CV data (closer to what I got on the private leaderboard) but on the public leaderboard the best I could get was 0.915.
", "votes": null, "replies": [] }, { "id": 81396, "author_name": "mechatroner", "author_url": "", "post_date": "06/09/2015 19:11:59", "content": "Congrats to all the winners!
I used scikit learn RandomForestClassifier algorithm.
List of decent features according to cross-validation feature selection method I used:
1. absolute number of bids per user.
2. num uniq devices, auctions, ips and urls
3. avg and first-non-zero time diffs between user bids
4. percentage of bids made with favorite(most common) device
5. percentage of bids made from favorite country
6. max number of bids in a single auction
7. number of simultaneous bids
8. number of device switches, numer of auctions switches, number of ips switches, number of countries switches
e.g. this sequence corresponds to 3 'switches' (with 2 devices): [dev1, dev1, dev2, dev2, dev2, dev2, dev1]
9. percentage of user bids made in first/last 5% of auction observed timeline
(observed timeline doesn't correspond to real auction timeline, but in my opinion still helps to understand if user prefers to bid at the beginning or end of an auction)
10. average number of transaction in the same country in 1-hour window with various normalizations
(in my opinion this feature roughly reflects user favorite bidding time of the day)
11. two categ->response transformations features. average response for country and device.
the same idea as discussed here: https://www.kaggle.com/forums/f/15/kaggle-forum/t/8005/replacing-categorical-variables-with-historic-response-rate
there were also a few other trivial features
\nFor cross validation I calculated average AUC for 10 different but fixed 10-fold splits, so totaly it's 10*10 algorithm trains for a single feature check.
", "votes": null, "replies": [] }, { "id": 81397, "author_name": "rinzewind", "author_url": "", "post_date": "06/09/2015 19:48:38", "content": "[quote=mechatroner;81396]11. two categ->response transformations features. average response for country and device.
the same idea as discussed here: https://www.kaggle.com/forums/f/15/kaggle-forum/t/8005/replacing-categorical-variables-with-historic-response-rate
[/quote]
\nI tried this (also for merchandise) and, while it incremented my local CV score by about 0.1 - 0.2 (only the country historical information), it did not reflect well on the leaderboard. So I think this introduces more leakage / circularity problems than anything else.
", "votes": null, "replies": [] }, { "id": 81398, "author_name": "sushanttripathy", "author_url": "", "post_date": "06/09/2015 20:04:20", "content": "Congratulations to the winners, this was a really fun contest!
\nFor classification I used a two-stage meta-learner approach with RF, XGBoost and NN as base learners and a combination of those three as meta learners.
\nThe final features I used are as follows:
\n1. Fraction of auctions won by member (out of total number of auctions member participated in).
\n2. Mean fraction of bids made by member (in auctions member participated in).
\n3. Mean fraction of bids made by member within first 10% of bids (in auctions member participated in).
\n4. Mean fraction of bids made by member within last 25% of bids (in auctions member participated in).
\n5. Mean fraction of bids made by member (in auctions member won).
\n6. Mean fraction of bids made by member within first 10% of bids (in auctions member won).
\n7. Mean fraction of bids made by member within last 25% of bids (in auctions member won).
\n8. , 9. , 10. Max, Mean and Standard Deviation of number of auctions member is active in per minute.
\n11., 12., 13. Max, Mean and Standard Deviation of number of auctions member is active in per hour.
\n14.,15.,16. Probability of bot given Country, Merchandise search category, and device
\n17., 18., 19. Max, Mean and Standard Deviation of number of ips used per minute.
\n20., 21., 22. Max, Mean and Standard Deviation of number of ips used per hour.
\n23. Probabilty of being referred by url : 'vasstdc27m7nks3'
\n24. Probability of being a Bot given you belong to a cluster of Payment Accounts (Clustered using Hamming Distance, with a Ward Linkage)
\n25. Probability of being a Bot given you belong to a cluster of Addresses (Clustered using Hamming Distance, with a Ward Linkage)
\n26., 27., 28. Mean, Standard Deviation and Sarle's bimodality coefficient of auction bid value (assuming first bid has a value of 10 cents and each subsequent bid increases it by one cent)
\n29., 30. Mean and Standard Deviation of length of continuous bid streaks by member.
\n31., 32. Mean and Standard Deviation of time between Member's bid streaks and previous bidders' bid.
\nI used a Random Forest classifier with Recursive Feature Elimination + Cross Validation to select these features, my 5-fold cross-validation score was similar to the public LB score with -0.003 difference, and it did not change much on the private LB either. :)
", "votes": null, "replies": [] }, { "id": 81424, "author_name": "thomasrobertfr", "author_url": "", "post_date": "06/09/2015 23:29:49", "content": "Hi all, congrats to everyone. I've seen people doing incredible work here!
\nI was ranked 32nd on private LB (but could have been 17th if I had sent my submission using the most-frequent-URL-of-bidder feature :( ). My work on feature and modeling is actually pretty simple. You can find my blog post which is basically a dump of my IPython notebook (so code + comments) here: http://www.thomas-robert.fr/en/kaggles-human-or-robot-competition-feedback/
", "votes": null, "replies": [] }, { "id": 81425, "author_name": "podlipensky", "author_url": "", "post_date": "06/09/2015 23:36:39", "content": "Please find my aftermath blog post here http://podlipensky.com/2015/06/kaggle-competition-aftermath/
\nHere is my best feature set:
\n['ip_rank', 'auction_rank', 'auction/increments', 'avg_unique_ips',
'ip/increments', 'std_inc_per_auction', 'product/won_auctions',
'payment_type=0875307e1731af94b3b64725ad0deb7d', 'device/increments',
'country_rank', 'avg_inc_per_auction', 'ip/ref', 'avg_ips_per_increment',
'true_bids_count', 'increments/sim_auctions', 'increments/won_auctions',
'country/increments', 'ip_change', 'ip/sim_auctions', 'price_mean',
'sim_auctions', 'frequent_ip=8.116', 'std_ips_per_auction', 'device/ip',
'product/increments', 'device/ref', 'product/ip', 'product/sim_auctions',
'referral_change', 'product/ref', 'ref/auction', 'device/auction',
'avg_referrals_per_auction', 'ref/sim_auctions', 'avg_ips_per_auction',
'median_inc_per_auction', 'ref/increments', 'price_avg', 'auctions_count',
'unique_ips', 'country/ip', 'won_auctions/sim_auctions',
'unique_countries_count', 'robot_price_rmse', 'avg_devices_per_auction',
'std_referrals_per_auction', 'std_devices_per_auction',
'device/sim_auctions', 'ip/auction', 'ref/won_auctions']
(it is kind of cryptic - please find better description of those in my blog post)
", "votes": null, "replies": [] }, { "id": 81442, "author_name": "mikeskim", "author_url": "", "post_date": "06/10/2015 03:53:28", "content": "The code fbHomeBag35.R obtains: 0.90607 0.93256
\nThe code fbHomeBag16.R obtains: 0.90811 0.93419
\nI didn't select either.
\nTo overfit is only human.
", "votes": null, "replies": [] }, { "id": 81448, "author_name": "yusufengintetik", "author_url": "", "post_date": "06/10/2015 05:52:12", "content": "[quote=small yellow duck;81297]
\nHere's a little blurb about how I approached feature generation and cross-validation for this problem:
\nhttp://small-yellow-duck.github.io/auction.html
\ncode: https://github.com/small-yellow-duck/facebook_auction
\n[/quote]
\nHow you decoded time variable is really amazing. What I've done about that variable was assuming it as the count of milliseconds since 1.1.1970 and converting it to a date after ignoring last 4 numbers of the integer value. Thus, I assume time/10000 as the time in milliseconds, which is apparently not correct according to your analysis.
\nSuch as: 9704553947368421 -> 970455394736 -> 02 October 2000 Monday 02:56:34 GMT
\nhttp://www.fileformat.info/tip/java/date2millis.htm
\nCongrats..
", "votes": null, "replies": [] }, { "id": 81454, "author_name": "ab19026", "author_url": "", "post_date": "06/10/2015 07:19:37", "content": "I have a long way to go with feature engineering.In this competition.I just pick out 6 properties:AUCTION,DEVICE,TIME,COUNTRY,IP and URL.Then I calculate the pearson correlation coefficient between any two of the 6 properties and 21 features are generated.
\nI believe my method is more generalized as the feature engineering nees domain knowledge and the feature-extraction will differ in various cases.However,my method is not able to give an accuracy more than 77%.The truth is that the feature engineering is one of the most powerful tools to seek truth
", "votes": null, "replies": [] }, { "id": 81480, "author_name": "darioo", "author_url": "", "post_date": "06/10/2015 14:03:16", "content": "[quote=Sushant Tripathy;81398]
\n14.,15.,16. Probability of bot given Country, Merchandise search category, and device
\n[/quote]
\n\nCongratulations to the winners, and thanks to Kaggle/facebook for hosting/organizing the competition. I had a lot of fun participating it. I just wish I had more time to work on it.
\nSushant Tripathy, Impressive work on feature engineering!
\nHope you don't mind just a clarification on how you calculated feature 14, 15 and 16.
\nFor example let's consider feature 15.
\nIn my understanding you may have calculated it as follows:
\n(# of merchandise type per class bot)/(size of the fold)
\nso for example (just random figures here), if your merchandise 'jewelry' has 100 class 1 (which means bot), and your fold size is 1200 then your feature 15 would be 100/1200= 0.083
\nThen you would build a feature that has 0.083 when that bidder has 'jewelry' as merchandise.
\nis this explanation correct?
\nHave you maybe tried dummifying merchandise? did you see any performance change?
\nI tried both and I actually did notice an increase in overfitting. I explained that to myself, considering "cheating" including any information about the output class in your train dataset. Any comments about this?
", "votes": null, "replies": [] }, { "id": 81481, "author_name": "rinzewind", "author_url": "", "post_date": "06/10/2015 14:08:21", "content": "[quote=Dario Bernardo;81480]I tried both and I actually did notice an increase in overfitting. I explained that to myself, considering "cheating" including any information about the output class in your train dataset. Any comments about this?
\n[/quote]
\nSame in my experience. I did not try "dummyfying" the variables, but I noticed the overfitting effect.
", "votes": null, "replies": [] }, { "id": 81482, "author_name": "sushanttripathy", "author_url": "", "post_date": "06/10/2015 14:23:19", "content": "[quote=Dario Bernardo;81480]
\nSushant Tripathy, Impressive work on feature engineering!
\nHope you don't mind just a clarification on how you calculated feature 14, 15 and 16.
\nFor example let's consider feature 15.
\nIn my understanding you may have calculated it as follows:
\n(# of merchandise type per class bot)/(size of the fold)
\nso for example (just random figures here), if your merchandise 'jewelry' has 100 class 1 (which means bot), and your fold size is 1200 then your feature 15 would be 100/1200= 0.083
\nThen you would build a feature that has 0.083 when that bidder has 'jewelry' as merchandise.
\nis this explanation correct?
\nHave you maybe tried dummifying merchandise? did you see any performance change?
\nI tried both and I actually did notice an increase in overfitting. I explained that to myself, considering "cheating" including any information about the output class in your train dataset. Any comments about this?
\n[/quote]
\nHi Dario, thanks for the kind words. My approach for building these features relied on using values for samples with known outcomes.
\nFor example if merchandise search category was 'jewellery', and my stratified training set (66% partition) contained 5 known Bots and 10 known humans (using the search for 'jewellery') then I would assign a P(Bot|Merch='jewellery') as 5/15 = 0.33.
\nIn case I don't have any bots or I don't have any humans or there aren't enough labelled members using that search category (thresholded at >= 50), I would assign the overall probability of encountering bots for that run of Stratified CV. So, P(Bot|Merch='jewellery') = 50/1050 = 0.0476 (Assuming my training set partition has 50 bots and 1000 humans).
\nNow say I am constructing features for member X, who has 12 searches for 'jewellery' and 10 searches for 'sports goods'. Her/his P(Bot|Merch) = (12*P(Bot|Merch='jewellery') + 10*P(Bot|Merch='sports goods'))/(12 + 10)
\nAlso, I restricted my probabilistic feature construction to only the samples within my training partition, the CV scores on the validation partition were pretty spot on.
\n[quote=Dario Bernardo;81480]
\nHave you maybe tried dummifying merchandise? did you see any performance change?
\nI tried both and I actually did notice an increase in overfitting. I explained that to myself, considering "cheating" including any information about the output class in your train dataset. Any comments about this?
\n[/quote]
\nRegarding dummification, the assignment of the overall probability is a step in that direction although not a full-blown one. Here, in my opinion, as long as one doesn't use the validation partition to construct the probabilistic features (thus avoiding data leakage/circularity) the CV score would be an accurate representation of out of sample performance. Would love to hear your thoughts about it. :)
\nAs an after-thought, it was really early morning here when I started writing this comment, I am really sleepy and I probably made some mistakes. :)
", "votes": null, "replies": [] }, { "id": 81596, "author_name": "pierregutierrez", "author_url": "", "post_date": "06/11/2015 11:30:18", "content": "I detailed my feature engineering in this blog post (120 + features, hope this helps). I tested a lot of things after that, mostly blend of GBT and random forests by I ended up overfitting my training data even though my private and public leaderboard score are pretty close.
", "votes": null, "replies": [ { "id": 501998, "author_name": "ranjithav", "author_url": "", "post_date": "03/28/2019 03:30:50", "content": "Can you share me the link of your code? Your blog looks very impressive. I am a beginner and I want to learn.
", "votes": null, "replies": [] } ] }, { "id": 81692, "author_name": "darioo", "author_url": "", "post_date": "06/12/2015 13:06:10", "content": "Sushant,
\nIf I correctly understood your procedure, bidders with same "search category" (or merchandise) will end up having the same feature 15 score, so effectively that feature will not help distinguish bidders coming from the same search category, because will have the same score, but eventually will help to distinguish bidders coming from different search categories. Therefore my point was that in my opinion there shouldn't be much difference with a dummification approach of the "merchandise".
\nPlease note that I am not trying to say that you did a mistake or that one or the other approach is better. I am just making some general considerations. In order to make such conclusion (assuming that i correctly got your point) I should run lot's of experiments first. After this competition there is so much to revise and re-analyse. I don't want to flood the discussion with further general consideration so I will stop here. I will conclude just thanking you for your kind response and thanking you and the others for posting your solutions. I have learned so much!
", "votes": null, "replies": [] }, { "id": 81763, "author_name": "sushanttripathy", "author_url": "", "post_date": "06/12/2015 22:51:25", "content": "Hi Dario,
\nThe feature calculation consists of two parts one is to determine what is the probability of bots using the "search category" and then to determine what is the weighted probability of a member being a bot based on the number of bids of that member that originated with certain "search categories".
\nTo make it a little clear, I have provided an example below:
\nUser ID Bot Search Category Bids
u1 Yes s1 5
u1 Yes s2 4
u1 Yes s3 5
u1 Yes s4 1
u2 No s1 10
u2 No s2 8
u2 No s3 9
u2 No s4 7
u3 No s3 11
u3 No s4 12
Now Step 1:
\nSearch Category Humans Using It Bots Using It p(Bot Use|Search Category)
s1 1 1 0.5
s2 1 1 0.5
s3 2 1 0.33
s4 2 1 0.33
Now Step 2 (calculation of bid weighted probabiliies):
\nUser ID Bot Search Category Bids p(Bot Use| Search Category ) Bid Weighted Probabilities
u1 Yes s1 5 0.5 2.5
u1 Yes s2 4 0.5 2
u1 Yes s3 5 0.33 1.65
u1 Yes s4 1 0.33 0.33
u2 No s1 10 0.5 5
u2 No s2 8 0.5 4
u2 No s3 9 0.33 2.97
u2 No s4 7 0.33 2.31
u3 No s3 11 0.33 3.63
u3 No s4 12 0.33 3.96
Now finally to calculate P(Bot|Search Usage) (Bids weighted average of P(Bot usage|Search Categories):
\nFor u1 = (2.5 + 2 + 1.65 + 0.33)/(5 + 4 + 5 + 1) = 0.432
\nFor u2 = (5 + 4 + 2.97 + 2.31)/(10 + 8 + 9 + 7) = 0.42
\nFor u3 = (3.63 + 3.96)/(11+12) = 0.33
\nNow these values are Feature 15. Hopefully, I could explain the calculations a bit more clearly this time around! :) Some members do have similar values, but never the same, there is always some difference (unless of course two members had the exact same number of bids within the exact same search categories and used the same search categories [rarely happens]). That being said, I did not try any extensive dummification approaches, and have no idea how dummification would have fared. :) I would love to hear about it, when you have had time to compare the approaches.
\nDiscussions are always welcome on Kaggle forums (in my opinion :D ), and of course I would be happy to discuss feature construction with you further. I have a lot to learn as well. :)
", "votes": null, "replies": [] }, { "id": 81791, "author_name": "liamsch", "author_url": "", "post_date": "06/13/2015 07:00:13", "content": "Hey guys
\nI did a bit of a write-up on my approach that got me to first place on the public leaderboard, but 17th on the private leaderboard. Given I'm still fairly new to all this, I'd be interested to hear people's thoughts on how this might have happened. I suppose I just hill-climbed on the public leaderboard a bit...
", "votes": null, "replies": [ { "id": 172305, "author_name": "mihir11", "author_url": "", "post_date": "04/03/2017 06:24:17", "content": "Link doesn't seem to work.
", "votes": null, "replies": [] } ] }, { "id": 82369, "author_name": "rawalc", "author_url": "", "post_date": "06/19/2015 01:34:51", "content": "Very useful to see feedback. This was my first competition (I'm a hobbyist), and it was fun even though I am nowhere to be seen in the leader-boards.
", "votes": null, "replies": [] }, { "id": 82428, "author_name": "annavictoria", "author_url": "", "post_date": "06/19/2015 18:12:56", "content": "small yellow duck's approach, background, and interesting ideas on Kaggle and ML in medicine can now be found in this competition write-up on no free hunch.
\nThanks for again for putting together such a great post!
", "votes": null, "replies": [] }, { "id": 107844, "author_name": "", "author_url": "", "post_date": "02/13/2016 02:13:57", "content": "[quote=Artem;81317]
\n\nScored 0.93939, 7th place on private leaderboard
\nI just combined 8 similar models to prevent overfitting.
\nI used average of XGBoost logistic regression, Vowpal Wabbit , GraphLab boosted trees, sk-l Randomforest, sk-l GradientBoosting, sk-l RandomForest+AdaBoost, sk-l RandomForest + CalibratedClassifier, sk-l RandomForest + BaggingClassifier.
\n[/quote]
\nHi,
\nDid you posted the approach and solution on any blog? I am beginner and I want to learn. Thanks
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "81297": "", "81300": "", "81302": "", "81303": "", "81304": "", "81306": "", "81309": "", "81310": "", "81316": "", "81317": "", "81318": "", "81319": "", "81320": "", "81321": "", "81331": "", "81338": "", "81343": "", "81352": "", "81396": "", "81397": "", "81398": "", "81424": "", "81425": "", "81442": "", "81448": "", "81454": "", "81480": "", "81481": "", "81482": "", "81596": "", "81692": "", "81763": "", "81791": "", "82369": "", "82428": "", "107844": "", "172305": "Link doesn't seem to work.", "501998": "Can you share me the link of your code? Your blog looks very impressive. I am a beginner and I want to learn.", "736050": "Can you please share the github link of the notebook. Thank you" }, "source": "existing_discussion", "complete": true, "competition_id": 4294, "competition_slug": "facebook-recruiting-iv-human-or-bot", "team_id": 183920, "solution_url": "/competitions/facebook-recruiting-iv-human-or-bot/writeups/small-yellow-duck-share-your-secret-sauce" }