Congrats to everyone for a great competition, in particular to the guys at the top. Congrats especially to tvdwiele and Markus who were significantly ahead of the field yet neck-and-neck for a long time. @tvdwiele also thanks for the shiny app, that was probably the most useful data visualizations I've seen on Kaggle (https://www.kaggle.com/c/facebook-v-predicting-check-ins/forums/t/21365/interactive-exploratory-analysis-using-a-shiny-app).
\n\nMy solution (place 20) was the combination of an xgb (0.592 on public lb) and knn (0.585), which somehow combined to get 0.603. I combined them by averaging the top 6 probabilities predicted by each model. Ensembling is surprisingly powerful. I used a 40 x 50 grid for xgb and a 20 x 25 grid for knn, with the extension technique from the scripts.
\n\nMy xgb model took 4 days to run on my dual-core macbook pro. I used the basic time features (hour, weekday, year etc), did not have any real success with advanced feature engineering this competition. I also used the xgb weight option, so more recent times were weighted more heavily while training.
\n\nMy knn model I tried to get around the hour cyclic problem by shifting the hour feature and averaging the predictions from different models. Otherwise it was pretty similar to the early knn scripts.
\n\nOne interesting thing I did was to use the knn model to predict how common a place id was in the test set. This is peaking into the future, so it isn't all that useful in the real world but it provided a boost of about 0.004 on the public lb. I basically used knn to get the average predicted probability, then applied the ratio of the predicted probability in test over the probability in train to weight the probabilities of the next level models.
\n\nMy code needs to be cleaned up, but if you don't mind looking at messy code it is here:\nhttps://github.com/jturkewitz/SideProjects/tree/master/Kaggle/Facebook_Checkin
", "messages": [ { "id": "126191", "postDate": "07/07/2016 01:07:21", "content": "Congrats to everyone for a great competition, in particular to the guys at the top. Congrats especially to tvdwiele and Markus who were significantly ahead of the field yet neck-and-neck for a long time. @tvdwiele also thanks for the shiny app, that was probably the most useful data visualizations I've seen on Kaggle (https://www.kaggle.com/c/facebook-v-predicting-check-ins/forums/t/21365/interactive-exploratory-analysis-using-a-shiny-app).
\n\nMy solution (place 20) was the combination of an xgb (0.592 on public lb) and knn (0.585), which somehow combined to get 0.603. I combined them by averaging the top 6 probabilities predicted by each model. Ensembling is surprisingly powerful. I used a 40 x 50 grid for xgb and a 20 x 25 grid for knn, with the extension technique from the scripts.
\n\nMy xgb model took 4 days to run on my dual-core macbook pro. I used the basic time features (hour, weekday, year etc), did not have any real success with advanced feature engineering this competition. I also used the xgb weight option, so more recent times were weighted more heavily while training.
\n\nMy knn model I tried to get around the hour cyclic problem by shifting the hour feature and averaging the predictions from different models. Otherwise it was pretty similar to the early knn scripts.
\n\nOne interesting thing I did was to use the knn model to predict how common a place id was in the test set. This is peaking into the future, so it isn't all that useful in the real world but it provided a boost of about 0.004 on the public lb. I basically used knn to get the average predicted probability, then applied the ratio of the predicted probability in test over the probability in train to weight the probabilities of the next level models.
\n\nMy code needs to be cleaned up, but if you don't mind looking at messy code it is here:\nhttps://github.com/jturkewitz/SideProjects/tree/master/Kaggle/Facebook_Checkin
", "rawMarkdown": "Congrats to everyone for a great competition, in particular to the guys at the top. Congrats especially to tvdwiele and Markus who were significantly ahead of the field yet neck-and-neck for a long time. @tvdwiele also thanks for the shiny app, that was probably the most useful data visualizations I've seen on Kaggle (https://www.kaggle.com/c/facebook-v-predicting-check-ins/forums/t/21365/interactive-exploratory-analysis-using-a-shiny-app).\r\n\r\nMy solution (place 20) was the combination of an xgb (0.592 on public lb) and knn (0.585), which somehow combined to get 0.603. I combined them by averaging the top 6 probabilities predicted by each model. Ensembling is surprisingly powerful. I used a 40 x 50 grid for xgb and a 20 x 25 grid for knn, with the extension technique from the scripts.\r\n\r\nMy xgb model took 4 days to run on my dual-core macbook pro. I used the basic time features (hour, weekday, year etc), did not have any real success with advanced feature engineering this competition. I also used the xgb weight option, so more recent times were weighted more heavily while training.\r\n\r\nMy knn model I tried to get around the hour cyclic problem by shifting the hour feature and averaging the predictions from different models. Otherwise it was pretty similar to the early knn scripts.\r\n\r\nOne interesting thing I did was to use the knn model to predict how common a place id was in the test set. This is peaking into the future, so it isn't all that useful in the real world but it provided a boost of about 0.004 on the public lb. I basically used knn to get the average predicted probability, then applied the ratio of the predicted probability in test over the probability in train to weight the probabilities of the next level models.\r\n\r\nMy code needs to be cleaned up, but if you don't mind looking at messy code it is here:\r\nhttps://github.com/jturkewitz/SideProjects/tree/master/Kaggle/Facebook_Checkin", "votes": null }, { "id": "126195", "postDate": "07/07/2016 01:24:29", "content": "@jturkewitz Thanks for sharing the "peek into the future" idea. Clever thinking!
\n\nMy best submission (place 7) is an ensemble of three models.
\n\n1. Regression Model
\n\nThis model performs logistic regression using XGBoost to determine the likelihood of a pairing of a checkin and a place. For each observation in the training set, 50 observations are derived. The actual place serves as the positive observation, and the 49 closest places serve as negative observations. Training observations thus grow by a factor of 50, and only 20% of the original training data was used to build this model. Gains in precission became diminishing after that.
\n\nThe features of this model are based on a set of pre-computed per-place data, such as as mean and standard deviation of location and accuracy, visitor frequencies, and neighborhood sparsity. Outlier removal using Turkey's test was used for some of these statistics.
\n\nIn particular, the model features include:
\n\nIndividual MAP@3 on the public leaderboard: 0.58452
\n\nWhile this is not the best performing individual model, a strong point of this model is its high degree of generalization. Adding predictive support for a new place only requires calculating the per-place statistics for that place. This is different from the other two models which need to be partially re-trained whenever the set of places changes.
\n\n2. Classification Model
\n\nThis is a grid-based classification model using XGBoost to directly classify an observation to a place. 25 X cuts, and 100 Y cuts are used. Instead of heavy border augmentation, four instances of the model are built, each one shifted by a quarter of a grid.
\n\nFeatures include:
\n\nIndividual MAP@3 on the public leaderboard: 0.59420
\n\n3. Nearest Neighbor Model
\n\nThis is pretty much the awesome and extremely efficient model developed Sandro, Alex, Michael Hartmann, and many others. As with the classification model, four instances of the model are built, each one shifted by a quarter of a grid.
\n\n4. Ensemble
\n\nFor each model instance and test observation, the top 6 places and related probabilities are stored.
\n\nEnsembling then consists of two phases. In the first phase, the model instances are aggregated per main model to derive the average probability of a place by that model; in the second phase, a weighted average of the probabilities from the three main models is calculated. The submission then consists of the three places with the highest average probability.
\n\nThanks to Facebook for coming up with an awesome problem, based on a simple data set consisting of only 5 attributes, but providing significant depth. And thanks to the Kaggle community! The discussions on the forums and the activity in the scripts have really been a highlight of this competition.
", "rawMarkdown": "jturkewitz Thanks for sharing the \"peek into the future\" idea. Clever thinking!\r\n\r\nMy best submission (place 7) is an ensemble of three models.\r\n\r\n**1. Regression Model**\r\n\r\nThis model performs logistic regression using XGBoost to determine the likelihood of a pairing of a checkin and a place. For each observation in the training set, 50 observations are derived. The actual place serves as the positive observation, and the 49 closest places serve as negative observations. Training observations thus grow by a factor of 50, and only 20% of the original training data was used to build this model. Gains in precission became diminishing after that.\r\n\r\nThe features of this model are based on a set of pre-computed per-place data, such as as mean and standard deviation of location and accuracy, visitor frequencies, and neighborhood sparsity. Outlier removal using Turkey's test was used for some of these statistics.\r\n\r\nIn particular, the model features include:\r\n\r\n- Basic: x, y, accuracy\r\n- Location: L1 and L2 distances with/without normalization, uni- and multi-variate normal PDFs of location and accuracy (all calculated on the observation data with regard to the pre-calculated per-place data of the place under consideration)\r\n- Frequencies: time of final visit, mean visitors on the particular day/weekday/hours, using a shifted weekday concept (+ 12 h) for places peaking across midnight\r\n- Neighborhood: L1 and L2 sparsity of the place as determined by mean distance to the next 25 neighbors\r\n- Comparisons: min/max/delta on location and various visitor frequencies of the place and its nearest neighbors.\r\n\r\nIndividual MAP@3 on the public leaderboard: 0.58452\r\n\r\nWhile this is not the best performing individual model, a strong point of this model is its high degree of generalization. Adding predictive support for a new place only requires calculating the per-place statistics for that place. This is different from the other two models which need to be partially re-trained whenever the set of places changes.\r\n\r\n\r\n**2. Classification Model**\r\n\r\nThis is a grid-based classification model using XGBoost to directly classify an observation to a place. 25 X cuts, and 100 Y cuts are used. Instead of heavy border augmentation, four instances of the model are built, each one shifted by a quarter of a grid.\r\n\r\nFeatures include:\r\n\r\n- Basic: x, y, accuracy, time\r\n- Periodic time: day time (% 1440), weekday time (% 7 * 1440)\r\n- Time factors: weekday, shifted weekday, hour of day, hour of week, month of year\r\n\r\nIndividual MAP@3 on the public leaderboard: 0.59420\r\n\r\n\r\n**3. Nearest Neighbor Model**\r\n\r\nThis is pretty much the awesome and extremely efficient model developed Sandro, Alex, Michael Hartmann, and many others. As with the classification model, four instances of the model are built, each one shifted by a quarter of a grid.\r\n\r\n\r\n**4. Ensemble**\r\n\r\nFor each model instance and test observation, the top 6 places and related probabilities are stored.\r\n\r\nEnsembling then consists of two phases. In the first phase, the model instances are aggregated per main model to derive the average probability of a place by that model; in the second phase, a weighted average of the probabilities from the three main models is calculated. The submission then consists of the three places with the highest average probability.\r\n\r\n\r\nThanks to Facebook for coming up with an awesome problem, based on a simple data set consisting of only 5 attributes, but providing significant depth. And thanks to the Kaggle community! The discussions on the forums and the activity in the scripts have really been a highlight of this competition.", "votes": null }, { "id": "126198", "postDate": "07/07/2016 01:30:45", "content": "Nice solution @Jared - if I remember correctly you were the first person to mention using a classifier(xgboost) on a x-y grid early on and that seems to be the best approach by far. Thanks for the inspiration.
\n\nI got lazy and just combined the probabilities from a knn and a random forest on the same grid - 40 by 40 - which was sub optimal. Could you elaborate a bit how you averaged the top 6 probabilities from the two different models?
", "rawMarkdown": "Nice solution @Jared - if I remember correctly you were the first person to mention using a classifier(xgboost) on a x-y grid early on and that seems to be the best approach by far. Thanks for the inspiration.\r\n\r\nI got lazy and just combined the probabilities from a knn and a random forest on the same grid - 40 by 40 - which was sub optimal. Could you elaborate a bit how you averaged the top 6 probabilities from the two different models?", "votes": null }, { "id": "126200", "postDate": "07/07/2016 01:34:23", "content": "Post mortem: I was wondering how others did their ensembling. I tried to build model stacking using XGBoost, and failed horribly (the result was worse than individual submissions, and I couldn't figure out why). I ended up implementing a weighted voting classifier with some simple guesses for weights and a huge 24 member ensemble
\n\n3 Naive XGboost runs (only time based features) -- In range (0.580 - 0.586)\n10 XGBoost with Nearest Neighbor features -- In range (0.580 - 0.59)\n5 kNN (16, 19, 24, 29, 31) with Bayesian Gaussian Process optimized NN weights -- (0.575 - 0.582)\n1 Abhi's fantastic public script (0.5865)\n1 Michael Hartmann's fantastic public script (0.58?)\n2 Sklearn Random Forest with only time features -- (0.575)\n1 ExtraTrees with nearest neighbor features - (0.583)\n1 Sklearn Random Forest with nearest neighbor features (0.589)\nI tried and excluded:
\n\n- Keras based Convolutional Neural Network solution (MAP3 = 0.4)\n- T-SNE based clustering (tried this multiple ways) (MAP3 = 0.3)\n- Stacked RBM based ideas (vanishing gradient problem)\n- Naive Bayes (well this one is obvious, but I thought I'd give it a shot) \n- Sklearn ExtraTrees classifier (MAP3 = 0.52)\n- Simple first principles Kernel Density estimation and likelihood ratios (MAP3 = 0.52)\n- Kernel SVM (MAP3 = 0.42)\nVoting classifier weights depended on validation MAP3 score (higher score, higher weight) and a similarity metric for predictions I quickly thought up last night (more similar to other runs, lower weight).
", "rawMarkdown": "Post mortem: I was wondering how others did their ensembling. I tried to build model stacking using XGBoost, and failed horribly (the result was worse than individual submissions, and I couldn't figure out why). I ended up implementing a weighted voting classifier with some simple guesses for weights and a huge 24 member ensemble\r\n\r\n 3 Naive XGboost runs (only time based features) -- In range (0.580 - 0.586)\r\n 10 XGBoost with Nearest Neighbor features -- In range (0.580 - 0.59)\r\n 5 kNN (16, 19, 24, 29, 31) with Bayesian Gaussian Process optimized NN weights -- (0.575 - 0.582)\r\n 1 Abhi's fantastic public script (0.5865)\r\n 1 Michael Hartmann's fantastic public script (0.58?)\r\n 2 Sklearn Random Forest with only time features -- (0.575)\r\n 1 ExtraTrees with nearest neighbor features - (0.583)\r\n 1 Sklearn Random Forest with nearest neighbor features (0.589)\r\n\r\nI tried and excluded:\r\n\r\n - Keras based Convolutional Neural Network solution (MAP3 = 0.4)\r\n - T-SNE based clustering (tried this multiple ways) (MAP3 = 0.3)\r\n - Stacked RBM based ideas (vanishing gradient problem)\r\n - Naive Bayes (well this one is obvious, but I thought I'd give it a shot) \r\n - Sklearn ExtraTrees classifier (MAP3 = 0.52)\r\n - Simple first principles Kernel Density estimation and likelihood ratios (MAP3 = 0.52)\r\n - Kernel SVM (MAP3 = 0.42)\r\n\r\nVoting classifier weights depended on validation MAP3 score (higher score, higher weight) and a similarity metric for predictions I quickly thought up last night (more similar to other runs, lower weight).", "votes": null }, { "id": "126207", "postDate": "07/07/2016 02:05:19", "content": "I had an ensemble of one RF and one XGB.
\n\nMy only working trick was to multiply the predicted probability of a place_id like this:
\n\npredicted_probab_place_id*(0.5+first_check_in_normalized)*(0.2+last_check_in_normalized)
\n\nso if the first check in for that place_it was at timestamp 10000 and the last one at 200000:
\n\nfirst_check_in_normalized = 10000/786240 = 0.012
\n\nlast_check_in_normalized = 200000/786240 = 0.254
\n\nso the new_predict_prob = predicted_probab_place_id*(0.5+0.012)*(0.2+0.254) = 0.232*predicted_probab_place_id
\n\nthen out of the 5, return most_common(k)
\n\nIn other words I penalize latecomers and early goers (place_ids in the dataset)
\n\nEdit: I had a second working trick. I counted the number neighboring check ins 4 hours before and after and made a new feature, this is peeking into the future.
\n\nThe other methods I used were public knowledge before the competition was over.
", "rawMarkdown": "I had an ensemble of one RF and one XGB.\r\n\r\nMy only working trick was to multiply the predicted probability of a place_id like this:\r\n\r\npredicted_probab_place_id*(0.5+first_check_in_normalized)*(0.2+last_check_in_normalized)\r\n\r\nso if the first check in for that place_it was at timestamp 10000 and the last one at 200000:\r\n\r\nfirst_check_in_normalized = 10000/786240 = 0.012\r\n\r\nlast_check_in_normalized = 200000/786240 = 0.254\r\n\r\nso the new_predict_prob = predicted_probab_place_id*(0.5+0.012)*(0.2+0.254) = 0.232*predicted_probab_place_id\r\n\r\nthen out of the 5, return most_common(k)\r\n\r\n\r\nIn other words I penalize latecomers and early goers (place_ids in the dataset)\r\n\r\n\r\nEdit: I had a second working trick. I counted the number neighboring check ins 4 hours before and after and made a new feature, this is peeking into the future.\r\n\r\nThe other methods I used were public knowledge before the competition was over.", "votes": null }, { "id": "126211", "postDate": "07/07/2016 02:13:46", "content": "[quote=Alexandru Papiu;126198]
\n\nNice solution @Jared - if I remember correctly you were the first person to mention using a classifier(xgboost) on a x-y grid early on and that seems to be the best approach by far. Thanks for the inspiration.
\n\nI got lazy and just combined the probabilities from a knn and a random forest on the same grid - 40 by 40 - which was sub optimal. Could you elaborate a bit how you averaged the top 6 probabilities from the two different models?
\n\n[/quote]
\n\nAs a particle physics grad student I regularly deal with large data that can be treated independently so the grid idea came to me pretty naturally. In physics every collision is independent, and we collect data from tons of collisions so you have to analyze it with many computers in parallel.
\n\nTo average the probabilities I first stored the top 6 place_ids with their probabilities from each model. Then if the same place_id was in both models, I just took the sum from the xgb prob and the knn prob (the knn prob was weighted at 0.9). If an xgb place_id was not in the top 6 knn place_ids, then I would just divide that probability by 2. If a knn place_id was not in the top 6 xgb place ids, I would divide that probability by 3. Then I reordered the place_ids based on the new top 3 probs.
", "rawMarkdown": "[quote=Alexandru Papiu;126198]\r\n\r\nNice solution @Jared - if I remember correctly you were the first person to mention using a classifier(xgboost) on a x-y grid early on and that seems to be the best approach by far. Thanks for the inspiration.\r\n\r\nI got lazy and just combined the probabilities from a knn and a random forest on the same grid - 40 by 40 - which was sub optimal. Could you elaborate a bit how you averaged the top 6 probabilities from the two different models?\r\n\r\n[/quote]\r\n\r\nAs a particle physics grad student I regularly deal with large data that can be treated independently so the grid idea came to me pretty naturally. In physics every collision is independent, and we collect data from tons of collisions so you have to analyze it with many computers in parallel.\r\n\r\nTo average the probabilities I first stored the top 6 place_ids with their probabilities from each model. Then if the same place_id was in both models, I just took the sum from the xgb prob and the knn prob (the knn prob was weighted at 0.9). If an xgb place_id was not in the top 6 knn place_ids, then I would just divide that probability by 2. If a knn place_id was not in the top 6 xgb place ids, I would divide that probability by 3. Then I reordered the place_ids based on the new top 3 probs.", "votes": null }, { "id": "126212", "postDate": "07/07/2016 02:23:47", "content": "I didn't use XGBoost because it seemed to take a lot longer than ExtraTrees/RandomForest. My entire model would run in less than a day or even half a day. So I focused on feature engineering into ET/RF. I think this was a mistake since it's probable an ensemble of varying seed XGs could probably beat my best hand crafted model of ET/RF. I guess XG is really too strong.
\n\nBut I had a lot of "interesting" albeit not so strong ideas. Potentially any of these features could improve an XG model, but I don't know for certain. I've attached one feature that seemed to add quite a bit, but there are many others.
", "rawMarkdown": "I didn't use XGBoost because it seemed to take a lot longer than ExtraTrees/RandomForest. My entire model would run in less than a day or even half a day. So I focused on feature engineering into ET/RF. I think this was a mistake since it's probable an ensemble of varying seed XGs could probably beat my best hand crafted model of ET/RF. I guess XG is really too strong.\r\n\r\nBut I had a lot of \"interesting\" albeit not so strong ideas. Potentially any of these features could improve an XG model, but I don't know for certain. I've attached one feature that seemed to add quite a bit, but there are many others.", "votes": null }, { "id": "126222", "postDate": "07/07/2016 03:09:32", "content": "Thank @Jared for creating this topic, and thank everyone here for sharing your method.
\n\nMy laptop had a hardware failure in the last week and I lost two sets of predictions that I could have use for my ensemble model. But anyway, here is my model of my best submission:
\n\nBase models
\n\nxgboost model 1 on 25*80 grids
\n\nfeatures: x, y, accuracy, hour of a day, day of a week, week of a year, year
\n\nexponentially decaying sample weights
\n\nexpanded grid border in training (or border augmentation)
\n\n3 grid offsets
\n\nbase model public LB score: 0.597 with 1 grid offset, 0.601 with 3 grid offsets combined
xgboost model 2
\n\nall the same as model 1, except that it used "sin", "cos" transforms on the day of week feature
\n\n3 grid offsets
\n\nbase model public LB score: 0.598 with 1 grid offset, unknown with 3 grid offsets as I initially planned to have 5 grid offsets if my laptop didn't fail me.
KNN from the kaggle script
\n\nbase model public LB score: 0.583
Ensemble
\n\nSoft voting of top 10 predictions from each model.
\n\nFinal score: 0.607
\n\nI think the diversity between xgb model and KNN model played an important role in improving the score of the final ensemble model. I would also consider adding random forest, extra trees, and even neural networks to the final ensemble because they worked as good as KNN in my early testing of various models.
\n\nAlthough with more careful parameter tuning my model could still get a better score, I don't think it would get anywhere close to 0.620, so I'm really looking forward to @tvdwiele and @Markus sharing their awesome methods.
", "rawMarkdown": "Thank @Jared for creating this topic, and thank everyone here for sharing your method.\r\n\r\nMy laptop had a hardware failure in the last week and I lost two sets of predictions that I could have use for my ensemble model. But anyway, here is my model of my best submission:\r\n\r\n**Base models**\r\n\r\n1. xgboost model 1 on 25*80 grids\r\n\r\n features: x, y, accuracy, hour of a day, day of a week, week of a year, year\r\n\r\n exponentially decaying sample weights\r\n\r\n expanded grid border in training (or border augmentation)\r\n\r\n 3 grid offsets\r\n\r\n base model public LB score: 0.597 with 1 grid offset, 0.601 with 3 grid offsets combined\r\n\r\n\r\n2. xgboost model 2\r\n \r\n all the same as model 1, except that it used \"sin\", \"cos\" transforms on the day of week feature\r\n\r\n 3 grid offsets\r\n\r\n base model public LB score: 0.598 with 1 grid offset, unknown with 3 grid offsets as I initially planned to have 5 grid offsets if my laptop didn't fail me.\r\n\r\n\r\n3. KNN from the [kaggle script](https://www.kaggle.com/lscoelho/facebook-v-predicting-check-ins/modified-grid-knn/run/288176)\r\n\r\n base model public LB score: 0.583\r\n\r\n**Ensemble**\r\n\r\nSoft voting of top 10 predictions from each model.\r\n\r\nFinal score: 0.607\r\n\r\nI think the diversity between xgb model and KNN model played an important role in improving the score of the final ensemble model. I would also consider adding random forest, extra trees, and even neural networks to the final ensemble because they worked as good as KNN in my early testing of various models.\r\n\r\nAlthough with more careful parameter tuning my model could still get a better score, I don't think it would get anywhere close to 0.620, so I'm really looking forward to @tvdwiele and @Markus sharing their awesome methods.", "votes": null }, { "id": "126229", "postDate": "07/07/2016 04:17:57", "content": "https://github.com/syllogismos/facebook-v-predicting-check-ins/blob/master/summary.ipynb
\n\nI wrote up the summary of my submission here.
\n\nWow I didn't know ensembling can improve score like that.
\n\nMy best submission is a single xgboost model, a 40x200 gird size with border augmentation.. that came up with 0.58685 on pub lb. I initially saved last two days for ensembling models, but decided against it for god knows what reason.. and instead worked on doing something else. I genuinely thought I was missing something..
\n\nAlong with using nthread = 4 I also trained 8 grid rows at a time parallely.. which took around 3.5 hours on c4.8xlarge spot machine to generate a submission..
\n\nI did have KNN and xgboost of several grid sizes and top 10 predictions and I didn't really make use of any of them..
\n\nbut I didn't even think I would beat the lb scores of people from public scripts.. but with 6 days to go, I came in top 100 which was pretty cool :P\nno matter what I did in the end, my score wasn't improving at all.. all my submissions are single models only. I'm pretty bummed about it now. I will try submitting my ensembles and check and weep if they are better..
\n\nI was trying things like creating more features from the top 10 predictions I got from a run, features like the place cardinality, its median, its mode of time intervals and all that, but none of them gave me any gains.
", "rawMarkdown": "https://github.com/syllogismos/facebook-v-predicting-check-ins/blob/master/summary.ipynb\r\n\r\nI wrote up the summary of my submission here. \r\n\r\nWow I didn't know ensembling can improve score like that.\r\n\r\nMy best submission is a single xgboost model, a 40x200 gird size with border augmentation.. that came up with 0.58685 on pub lb. I initially saved last two days for ensembling models, but decided against it for god knows what reason.. and instead worked on doing something else. I genuinely thought I was missing something..\r\n\r\nAlong with using nthread = 4 I also trained 8 grid rows at a time parallely.. which took around 3.5 hours on c4.8xlarge spot machine to generate a submission..\r\n\r\n\r\nI did have KNN and xgboost of several grid sizes and top 10 predictions and I didn't really make use of any of them..\r\n\r\nbut I didn't even think I would beat the lb scores of people from public scripts.. but with 6 days to go, I came in top 100 which was pretty cool :P\r\nno matter what I did in the end, my score wasn't improving at all.. all my submissions are single models only. I'm pretty bummed about it now. I will try submitting my ensembles and check and weep if they are better..\r\n\r\n\r\nI was trying things like creating more features from the top 10 predictions I got from a run, features like the place cardinality, its median, its mode of time intervals and all that, but none of them gave me any gains.", "votes": null }, { "id": "126235", "postDate": "07/07/2016 04:42:58", "content": "Here are the main ideas of my approach. I'll clean up the code a little and post it on Github within the next few days.
\n\nOverall structure
\n\nThe overall idea was to compute for each checkin and some candidate set of place_id's a probability proportional to
\n\nP(x, y | place_id) P(day of week | place_id) P(time of day | place_id) P(accuracy | place_id) P(place_id) .
\n\nChallenges were:
\n\nLearning the conditional distributions like P(x, y | place_id) efficiently without losing too much information.
The relative popularity of places, P(place_id), varied considerably over time. (So really it should be P(place_id, time).) How do we learn the relative popularities for the times in the test data?
1. Learning the conditional distributions efficiently
\n\nMy first attempts to learn the conditional distributions were far too slow given the size of the data.
\n\nBut early in the competition @ZFTurbo's "Mad Scripts Battle" script did pretty well (and efficiently so!) by essentially just using a histogram to learn the joint distribution P(x, y, quarter time of day). Inspired by this, I decided to try to learn the conditional distributions using histograms. I also switched from Python to C++ to speed things up even more.
\n\nTo make the histograms more accurate, I made them periodic for time of day and day of week, and I used various kernels (triangular, Gaussian, exponential) to both save and read values from the histogram - kind of like KDE but applied to a histogram of a fixed number of bins rather than all data points. This remained reasonably efficient while giving results almost as good as KDE.
\n\nFor later submissions, I averaged the spatial histogram with another approach: I fitted GMM's with up to 15 components on the train data (the exact number for each place_id selected by AIC). This took pretty long and I just computed the GMM components once and loaded the results for all future runs.
\n\nAlso, as observed in the forums, P(accuracy | place_id) and P(x, y | place_id) seemed to vary over time. Giving more weight to later data helped.
\n\n2. Learning from the future
\n\nUsing just P(place_id) based on the train data, the predictions were already very good: About half of the time the first prediction was correct. So we could think of the top predictions as actually labeling the test data - just very noisily - and train on the test data using the top 10-20 or so predicted place_id's as labels with their probabilities as weights.
\n\nThis idea led to an approach of iterating between making predictions for the test data and learning from those predictions. The final submission used 20 iterations. (Actually, I ended up learning not only P(place_id, time) from these iterations but also kept learning the other histograms based on the test data. This helped the score and made the convergence much more stable.)
\n\n(The 40-minute single-pass solution I mentioned elsewhere would just line-by-line make a prediction and then learn from it before making the next prediction.)
\n\nPossible alternative? My original thought was actually to try to learn P(place_id, time) using likelihood maximization in some form. Conceptually, I still think this would be a nicer approach than what I did, but what I had was working well and I never got around to this. It also seemed harder to integrate learning the other histograms into this approach. I may have needed to alternate it with predict/learn iterations in any case.
", "rawMarkdown": "Here are the main ideas of my approach. I'll clean up the code a little and post it on Github within the next few days.\r\n\r\n**Overall structure**\r\n\r\nThe overall idea was to compute for each checkin and some candidate set of place_id's a probability proportional to\r\n\r\nP(x, y | place_id) P(day of week | place_id) P(time of day | place_id) P(accuracy | place_id) P(place_id) .\r\n\r\nChallenges were:\r\n\r\n1. Learning the conditional distributions like P(x, y | place_id) efficiently without losing too much information.\r\n\r\n2. The relative popularity of places, P(place_id), varied considerably over time. (So really it should be P(place_id, time).) How do we learn the relative popularities for the times in the test data?\r\n\r\n**1. Learning the conditional distributions efficiently**\r\n\r\nMy first attempts to learn the conditional distributions were far too slow given the size of the data.\r\n\r\nBut early in the competition @ZFTurbo's \"Mad Scripts Battle\" script did pretty well (and efficiently so!) by essentially just using a histogram to learn the joint distribution P(x, y, quarter time of day). Inspired by this, I decided to try to learn the conditional distributions using histograms. I also switched from Python to C++ to speed things up even more.\r\n\r\nTo make the histograms more accurate, I made them periodic for time of day and day of week, and I used various kernels (triangular, Gaussian, exponential) to both save and read values from the histogram - kind of like KDE but applied to a histogram of a fixed number of bins rather than all data points. This remained reasonably efficient while giving results almost as good as KDE.\r\n\r\nFor later submissions, I averaged the spatial histogram with another approach: I fitted GMM's with up to 15 components on the train data (the exact number for each place_id selected by AIC). This took pretty long and I just computed the GMM components once and loaded the results for all future runs.\r\n\r\nAlso, as observed in the forums, P(accuracy | place_id) and P(x, y | place_id) seemed to vary over time. Giving more weight to later data helped.\r\n\r\n**2. Learning from the future**\r\n\r\nUsing just P(place_id) based on the train data, the predictions were already very good: About half of the time the first prediction was correct. So we could think of the top predictions as actually labeling the test data - just very noisily - and train on the test data using the top 10-20 or so predicted place_id's as labels with their probabilities as weights.\r\n\r\nThis idea led to an approach of iterating between making predictions for the test data and learning from those predictions. The final submission used 20 iterations. (Actually, I ended up learning not only P(place_id, time) from these iterations but also kept learning the other histograms based on the test data. This helped the score and made the convergence much more stable.)\r\n\r\n(The 40-minute single-pass solution I mentioned elsewhere would just line-by-line make a prediction and then learn from it before making the next prediction.)\r\n\r\n*Possible alternative?* My original thought was actually to try to learn P(place_id, time) using likelihood maximization in some form. Conceptually, I still think this would be a nicer approach than what I did, but what I had was working well and I never got around to this. It also seemed harder to integrate learning the other histograms into this approach. I may have needed to alternate it with predict/learn iterations in any case.", "votes": null }, { "id": "126237", "postDate": "07/07/2016 04:45:54", "content": "[quote=Alexandru Papiu;126198]
\n\nI got lazy and just combined the probabilities from a knn and a random forest on the same grid - 40 by 40 - which was sub optimal. Could you elaborate a bit how you averaged the top 6 probabilities from the two different models?
\n\n[/quote]
\n\n@Alexandru Papiu, I did the same with XGB and KNN probabilities on the same grid 20x40, in which the XGB is too long so I could only finish 693 cells out of 800.
\n\nSurprised for the power of the voting ensemble. @all, thanks a lot for sharing, I appreciate it.
", "rawMarkdown": "[quote=Alexandru Papiu;126198]\r\n\r\nI got lazy and just combined the probabilities from a knn and a random forest on the same grid - 40 by 40 - which was sub optimal. Could you elaborate a bit how you averaged the top 6 probabilities from the two different models?\r\n\r\n[/quote]\r\n\r\n@Alexandru Papiu, I did the same with XGB and KNN probabilities on the same grid 20x40, in which the XGB is too long so I could only finish 693 cells out of 800. \r\n\r\nSurprised for the power of the voting ensemble. @all, thanks a lot for sharing, I appreciate it.", "votes": null }, { "id": "126242", "postDate": "07/07/2016 05:11:17", "content": "Congratulations everyone, and specially to the guys at the top. I just tried to use KNN model implementation publicly available in this competition and played around the feature engg and variable importance. The idea was to see how much it can score in environment similar to kaggle sandbox. I used aws m4.large and all of my model took sub-30 minutes to run. Lastly I tried ensembling of my top models which increased my score by .00020. Maybe ensembling of different types of model will result in far better result as evident from top results.
\n\nTop results just confirm that XG is one of the powerful models to use in these kind of competitions. Maybe I will try ensemble this with KNN model to see how far I can go.
", "rawMarkdown": "Congratulations everyone, and specially to the guys at the top. I just tried to use KNN model implementation publicly available in this competition and played around the feature engg and variable importance. The idea was to see how much it can score in environment similar to kaggle sandbox. I used aws m4.large and all of my model took sub-30 minutes to run. Lastly I tried ensembling of my top models which increased my score by .00020. Maybe ensembling of different types of model will result in far better result as evident from top results.\r\n\r\nTop results just confirm that XG is one of the powerful models to use in these kind of competitions. Maybe I will try ensemble this with KNN model to see how far I can go.", "votes": null }, { "id": "126244", "postDate": "07/07/2016 05:23:32", "content": "@Markus, Hi Markus, congratulations and thanks for sharing your model. It is interesting to see a good solution without using any of common tools of kagglers.
\n\nI just have some questions about what you did with learning the conditional probabilities.
\n\nWhen you train GMM models, are you training over (x,y), week of day, time of day, accuracy separately and did you use any special strategies to deal with the periodic nature of the week of day, time of day feature ?
How did you eventually combine these conditional probability to produce the probability for each check in, was it like a weighted sum of log probability or something else ? If it is, did you find the weights varies with location, time, etc.
@Markus
\n\nWow, it's amazing! Using predictions on test data as pseudo-labels for training and iterating between them is indeed a very interesting idea.
", "rawMarkdown": "Markus\r\n\r\nWow, it's amazing! Using predictions on test data as pseudo-labels for training and iterating between them is indeed a very interesting idea.", "votes": null }, { "id": "126252", "postDate": "07/07/2016 05:46:01", "content": "Like many other kagglers, my final model is an ensemble model of various different models.
\n\nMy single best model was an xgb model with the usual features seen in other xgb models on the forum combined with knn probability for selected place_id. judging from the local validation, it would achieve about 0.591 on LB trained over a 63 * 159 grid. For ensembling, I also trained another 3 versions of the same model with grid shifted in x, y and (x,y) by half the grid size, which would boost to about 0.597 on LB, again judging from the local validation.
", "rawMarkdown": "Like many other kagglers, my final model is an ensemble model of various different models.\r\n\r\nMy single best model was an xgb model with the usual features seen in other xgb models on the forum combined with knn probability for selected place_id. judging from the local validation, it would achieve about 0.591 on LB trained over a 63 * 159 grid. For ensembling, I also trained another 3 versions of the same model with grid shifted in x, y and (x,y) by half the grid size, which would boost to about 0.597 on LB, again judging from the local validation.", "votes": null }, { "id": "126260", "postDate": "07/07/2016 06:36:30", "content": "Hi @Markus,
\n\nI am curious how was the performance of only P(x, y | place_id) P(place_id) ?\nI gave it a shot using sklearn's GMM, 6 component, and training it on the whole training set but it didn't give me good result at all (even worse than Gaussian Naive Base on only x and y.)
", "rawMarkdown": "Hi @Markus,\r\n\r\nI am curious how was the performance of only P(x, y | place_id) P(place_id) ?\r\nI gave it a shot using sklearn's GMM, 6 component, and training it on the whole training set but it didn't give me good result at all (even worse than Gaussian Naive Base on only x and y.)", "votes": null }, { "id": "126262", "postDate": "07/07/2016 07:14:35", "content": "Thanks to Facebook and all the competition participants. Very impressive lead by the leader board top seats, looking forward to look into their solutions.
\n\nEnsemble
\n\nMy solution is a simple ensemble of xgboost and kNN. LB results:
\n\nXgboost model details
\n\nSegmented the data into grid with 200x200m squares. The boundaries of the train squares was extended by an additional 15m.
\n\n9-11 features based on:
\n\nMy best submission was ensemble of 7 XGBoost grid based solutions and 5 best KNNs from public scripts.
\n\nCode to ensemble was pretty straightforward (applied for each ID):
\n\ndef process_places(arr):\n res = dict()\n for i in range(len(arr)):\n item = arr[i][0]\n if item in res:\n res[item] += 3.02 - i*0.0001\n else:\n res[item] = 3.02 - i*0.0001\n item = arr[i][1]\n if item in res:\n res[item] += 2.01 - i*0.0001\n else:\n res[item] = 2.01 - i*0.0001\n item = arr[i][2]\n if item in res:\n res[item] += 1 - i*0.0001\n else:\n res[item] = 1 - i*0.0001\n topitems = nlargest(3, sorted(res.items()), key=itemgetter(1))\n return topitems\n@Markus Awesome approach, and congratulations on your strong finish!
\n\nCan you share some insight on what makes your iterative approach converge, as opposed to going more and more off-track?
", "rawMarkdown": "Markus Awesome approach, and congratulations on your strong finish!\r\n\r\nCan you share some insight on what makes your iterative approach converge, as opposed to going more and more off-track?", "votes": null }, { "id": "126285", "postDate": "07/07/2016 11:05:08", "content": "My solution was an ensemble of 4 methods: xgboost, knn, random forests, and kernel density estimation. My xgboost score was slightly worse than Jared's, my knn score was ~0.587, my rf score was ~0.584, and my kde score was by far the best at ~0.604. I'm very happy to see that both Tom and Markus had used very different methods from what is traditionally used in Kaggle competitions as we now have new things to learn.
", "rawMarkdown": "My solution was an ensemble of 4 methods: xgboost, knn, random forests, and kernel density estimation. My xgboost score was slightly worse than Jared's, my knn score was ~0.587, my rf score was ~0.584, and my kde score was by far the best at ~0.604. I'm very happy to see that both Tom and Markus had used very different methods from what is traditionally used in Kaggle competitions as we now have new things to learn.", "votes": null }, { "id": "126290", "postDate": "07/07/2016 12:35:09", "content": "The idea of my solution is similar to that of Markus.
\n\nProblem definition
\n\nFind top 3 places of following probability for candidates of each row.
\n\nPr(place_id | x, y, time, accuracy)
\n\n∼ Pr(x, y, time, accuracy | place_id) * Pr(place_id)
\n\n≈ Pr(x | place_id) * Pr(y | place_id) * Pr(time of day | place_id) * Pr(week of day | place_id) * Pr(accuracy | place_id) * Pr(place_id)
\n\nSteps of my solution
\n\nI spent most of time to estimate distributions, and I never tried ensemble.
\n\nMy approach took about 2 hours by my 8GB RAM PC.
\n\nFinally, congratulations to tvdwiele and Markus, and everyone!
", "rawMarkdown": "The idea of my solution is similar to that of Markus.\r\n\r\n**Problem definition**\r\n\r\nFind top 3 places of following probability for candidates of each row.\r\n\r\nPr(place_id | x, y, time, accuracy)\r\n\r\n ∼ Pr(x, y, time, accuracy | place_id) * Pr(place_id)\r\n\r\n ≈ Pr(x | place_id) * Pr(y | place_id) * Pr(time of day | place_id) * Pr(week of day | place_id) * Pr(accuracy | place_id) * Pr(place_id)\r\n\r\n**Steps of my solution**\r\n\r\n1. Define the candidate set of place_ids for each cell of rough grid (100*200), which appeared at least two check-ins in the cell.\r\n2. Predict the amount of future occurrence Pr(place_id) by XGBOOST.\r\n3. Estimate marginal probability distributions of x, y, time of day, day of week, and accuracy.\r\n4. Calculate the product of 2 and 3 for all candidates.\r\n5. Elect top 3 place_ids and make submission.\r\n\r\nI spent most of time to estimate distributions, and I never tried ensemble. \r\n\r\nMy approach took about 2 hours by my 8GB RAM PC.\r\n\r\nFinally, congratulations to tvdwiele and Markus, and everyone!", "votes": null }, { "id": "126297", "postDate": "07/07/2016 14:34:07", "content": "@Jack (Japan), smart idea~ i am quite interested in this method. Could you share the code detail on github or somewhere else if u do not mind?
\n\nthx a lot!
", "rawMarkdown": "Jack (Japan), smart idea~ i am quite interested in this method. Could you share the code detail on github or somewhere else if u do not mind?\r\n\r\nthx a lot!", "votes": null }, { "id": "126298", "postDate": "07/07/2016 14:45:18", "content": "I'm very impressed by the simplicity and elegance of a few of the solutions presented here, in particular @Jack. While my results aren't nearly as clean, and I scored 40th, it was a fantastic learning experience, and I'm really grateful to be able to learn from such bright people. I thought I would share my code and techniques.
\n\nStep 0:\nMy first submissions were based on the random forest 0.55 starter script. Didn't end up using it because it didn't perform well enough.
\n\nStep 1:\nUse a variation on the KNN script posted by @Sandro and @David, that I modified to allow for Bayesian optimization of hyperparameters. This is available on Github here. I ran 8 parallel copies of this script in Bayes mode on an 8 core machine for a couple of days. I then picked a few of the well performing sets of parameters for different kNN numbers. I ended up getting good weights for 16, 19, 24, 29, 31 neighbors, which I felt spanned the k space "well enough".
\n\nStep 2: \nI ran XGBoost with three different sets of hyperparameters, nothing too special. This included border augmentation and different grid sizes. Here's the script.
\n\nI also went down the many failed approaches avenue here. I explored the space of possible classifiers here, trying out most Sklearn classifiers (after all I was supposed to be learning things). Tried Keras based parametric T-SNE to get useful clustering, which turned out to not work very well. Oh well, live and learn.
\n\nI ended up finding that Sklearn Random Forest performed comparably to XGB, and kept a couple of the predictions.
\n\nStep 3:\nI added nearest neighbor features to XGBoost and other classifiers. An example script is here. At first I was unhappy because this didn't seem to perform better than plain XGBoost without nearest neighbor features. However, I found that ensembling this with other predictions improved the score substantially. Despite the individual classifier not performing better, it was actually substantially decorrelated from other classifiers, making a superior ensemble.
\n\nStep 4:\nI tried a blending classifier with XGB but it didn't work and I couldn't figure out why, so I abandoned it.
\n\nI was swamped with other obligations towards the end of the competition (like writing my PhD dissertation), so when Abhi's and Michael Hartmann's kNN improvements were released, I didn't really have time to work them into my solution and rerun. So I added their public scripts to my ensemble, and used a weighted voting classifier where weights depended on score and similarity to other classifiers. This voting classifier forms my final result, with 0.5995 on the public leaderboard, and 0.5993 on the private leaderboard.
", "rawMarkdown": "I'm very impressed by the simplicity and elegance of a few of the solutions presented here, in particular @Jack. While my results aren't nearly as clean, and I scored 40th, it was a fantastic learning experience, and I'm really grateful to be able to learn from such bright people. I thought I would share my code and techniques.\r\n\r\n**Step 0:**\r\nMy first submissions were based on the random forest 0.55 starter script. Didn't end up using it because it didn't perform well enough. \r\n\r\n**Step 1:**\r\nUse a variation on the KNN script posted by @Sandro and @David, that I modified to allow for Bayesian optimization of hyperparameters. This is available [on Github here][1]. I ran 8 parallel copies of this script in Bayes mode on an 8 core machine for a couple of days. I then picked a few of the well performing sets of parameters for different kNN numbers. I ended up getting good weights for 16, 19, 24, 29, 31 neighbors, which I felt spanned the k space \"well enough\".\r\n\r\n**Step 2:** \r\nI ran XGBoost with three different sets of hyperparameters, nothing too special. This included border augmentation and different grid sizes. [Here's the script.][2]\r\n\r\nI also went down the many failed approaches avenue here. I explored the space of possible classifiers here, trying out most Sklearn classifiers (after all I was supposed to be learning things). Tried [Keras based parametric T-SNE][3] to get useful clustering, which turned out to not work very well. Oh well, live and learn.\r\n\r\nI ended up finding that Sklearn Random Forest performed comparably to XGB, and kept a couple of the predictions. \r\n\r\n**Step 3:**\r\nI added nearest neighbor features to XGBoost and other classifiers. An example script is [here][4]. At first I was unhappy because this didn't seem to perform better than plain XGBoost without nearest neighbor features. However, I found that ensembling this with other predictions improved the score substantially. Despite the individual classifier not performing better, it was actually substantially decorrelated from other classifiers, making a superior ensemble.\r\n\r\n**Step 4:**\r\nI tried a blending classifier with XGB but it didn't work and I couldn't figure out why, so I abandoned it. \r\n\r\nI was swamped with other obligations towards the end of the competition (like writing my PhD dissertation), so when Abhi's and Michael Hartmann's kNN improvements were released, I didn't really have time to work them into my solution and rerun. So I added their public scripts to my ensemble, and used a [weighted voting classifier][5] where weights depended on score and similarity to other classifiers. This voting classifier forms my final result, with 0.5995 on the public leaderboard, and 0.5993 on the private leaderboard. \r\n\r\n [1]: https://github.com/r-shekhar/Kaggle-Facebook-V/blob/master/L01_kNN_bayes.py\r\n [2]: https://github.com/r-shekhar/Kaggle-Facebook-V/blob/master/L10_naive_xgboost.py\r\n [3]: https://github.com/r-shekhar/Kaggle-Facebook-V/blob/master/failed_ideas/train_keras_tsne.py\r\n [4]: https://github.com/r-shekhar/Kaggle-Facebook-V/blob/master/L15_xgb_with_kNN_features.py\r\n [5]: https://github.com/r-shekhar/Kaggle-Facebook-V/blob/master/L20_voting_clf.py", "votes": null }, { "id": "126299", "postDate": "07/07/2016 14:47:47", "content": "I'm not sure if I created a feature similar to Jack's 2), but here's the R code of one of my features.
", "rawMarkdown": "I'm not sure if I created a feature similar to Jack's 2), but here's the R code of one of my features.", "votes": null }, { "id": "126301", "postDate": "07/07/2016 15:02:06", "content": "Jack's solution is simple and great. It looks like a Naive Bayes classifier to me, but I think he did a good job to customize the distribution and density estimation.
\n\nI also tried Naive Bayes with prior class probability predicted by time models, and it scores ~0.593 which is more than my best XGBoost model that was tuned and ran for a week. I only used the standard Naive Bayes function so this makes a big difference.
", "rawMarkdown": "Jack's solution is simple and great. It looks like a Naive Bayes classifier to me, but I think he did a good job to customize the distribution and density estimation.\r\n\r\nI also tried Naive Bayes with prior class probability predicted by time models, and it scores ~0.593 which is more than my best XGBoost model that was tuned and ran for a week. I only used the standard Naive Bayes function so this makes a big difference.", "votes": null }, { "id": "126309", "postDate": "07/07/2016 16:14:59", "content": "[quote=zrt;126244]
\n\n@Markus, Hi Markus, congratulations and thanks for sharing your model. It is interesting to see a good solution without using any of common tools of kagglers.
\n\nI just have some questions about what you did with learning the conditional probabilities.
\n\nWhen you train GMM models, are you training over (x,y), week of day, time of day, accuracy separately and did you use any special strategies to deal with the periodic nature of the week of day, time of day feature ?
How did you eventually combine these conditional probability to produce the probability for each check in, was it like a weighted sum of log probability or something else ? If it is, did you find the weights varies with location, time, etc.
[/quote]
\n\nI only trained GMM for P(x, y | place_id). I tried briefly for time of day, but it did not work well - perhaps because it did not consider periodicity, as you point out.
When I had two methods for predicting the same probability, I just combined them using a geometric mean (i.e., arithmetic mean of the log probabilities). I just used equal weights - I think tuning weights, esp. by location, time, etc., would have taken too much effort and could be hard to do without overfitting - but maybe it would have helped.
[quote=Andre Naef;126284]
\n\n@Markus Awesome approach, and congratulations on your strong finish!
\n\nCan you share some insight on what makes your iterative approach converge, as opposed to going more and more off-track?
\n\n[/quote]
\n\nThe analogy to tuning (learning rate, number of epochs) when using SGD is pretty good for the issues I had. If you go for too many iterations, the score starts to decrease. When this happens could be predicted reasonably well using validation.
\n\nAside from using the equivalent of a conservative low "learning rate" in the end, these items helped stabilize the convergence:
\n\nI predicted only on the top 25 candidates from the first pass in the iterations. (This is maybe similar to the benefit many people saw of using KNN to limit the number of candidates considered.)
I learned relative popularity / conditional probabilities only from the top 15 predictions. The weights for learning from these predictions were the geometric average of
\n\na. the predicted probabilities
\n\nb. the empirical probability of positions 1-15 being correct based on validation
\n\nThis average stabilized the convergence a fair amount compared to using either of the two individually.
I cumulatively learned the conditional probabilities rather than learning them from scratch during each iteration. This stabilized convergence, but at presumably increased cost of getting stuck in some local optimum.
Initialization: Starting from a less confident initial guess like P(place_id)^{0.3} (where P(place_id) is relative popularity based on train data) helped combat the cost of #3 to some extent.
I also tried learning from future confident predictions but the score deteriorated. I used random forest and selected those samples for which the random forest was outputting a probability of nore than 0.8 . I then combined those test samples to train samples and applied knn to predict the leftover samples. But i think i was doing something wrong or maybe i was doing it half-heartedly.
", "rawMarkdown": "I also tried learning from future confident predictions but the score deteriorated. I used random forest and selected those samples for which the random forest was outputting a probability of nore than 0.8 . I then combined those test samples to train samples and applied knn to predict the leftover samples. But i think i was doing something wrong or maybe i was doing it half-heartedly.", "votes": null }, { "id": "126332", "postDate": "07/07/2016 19:09:55", "content": "Didn't get a chance to try weighting. I wasted a lot of time trying various approaches and features thinking there must be some clever way to do this other than ensembling (since @markus was able to get 0.59 in 40 mins). All of them ended up giving more or less the same score. In retrospect, I should have ensembled them all.
", "rawMarkdown": "1. Grouped data by place_id and computed mean_x, mean_y, place kde (x, y, accuracy) and hour_kde. This was done with joblib in parallel with each place_group_df as a chunk.\r\n2. Built a KDTree with mean x, y computed in (1) to query top 100 place_id indexes for test_df. This already represented 78% accuracy amongst those filtered places.\r\n3. Divided test_df into n_jobs chunks. For each chunk computed place_kde, hour_kde values. Selected top 3 using (place_kde + hour_kde) * prob(dow) * prob(place_id)\r\n\r\nDidn't get a chance to try weighting. I wasted a lot of time trying various approaches and features thinking there must be some clever way to do this other than ensembling (since @markus was able to get 0.59 in 40 mins). All of them ended up giving more or less the same score. In retrospect, I should have ensembled them all.", "votes": null }, { "id": "126338", "postDate": "07/07/2016 19:35:41", "content": "@Jack(Japan) Congratulations! Your solution is very elegent and effective. Could you briefly expain how you estimate the distribution? KDE or GMM or other good tools?
\n\nAlso, I am curious about how to use XGB to predict the amount of occurence in future. You must have deep insights in this problem.
", "rawMarkdown": "Jack(Japan) Congratulations! Your solution is very elegent and effective. Could you briefly expain how you estimate the distribution? KDE or GMM or other good tools?\r\n\r\nAlso, I am curious about how to use XGB to predict the amount of occurence in future. You must have deep insights in this problem.", "votes": null }, { "id": "126360", "postDate": "07/07/2016 21:52:41", "content": "@Jack (Japan)
\n\n2.Predict the amount of future occurrence Pr(place_id) by XGBOOST.
\n\nElegant solution indeed.\nWould you please elaborate on the step 2 above? Which features did you use to predict? Training dataset alone does not provide good estimate of Pr(place_id)?
", "rawMarkdown": "Jack (Japan)\r\n\r\n2.Predict the amount of future occurrence Pr(place_id) by XGBOOST.\r\n\r\nElegant solution indeed.\r\nWould you please elaborate on the step 2 above? Which features did you use to predict? Training dataset alone does not provide good estimate of Pr(place_id)?", "votes": null }, { "id": "126398", "postDate": "07/08/2016 06:05:33", "content": "@Jack (Japan)\nExcellent solution. I am also looking forward to hear from the detail of your solution:
\n\nHow do you estimate the distribution? (@Rongcheng Lin)
How to predict the future occurence? (@data_js)
First I would like to congratulate everyone and especially @Tom Van de Wiele and @Markus Kliegl - their results are truly impressive and I hope to learn a lot from their ideas and solutions.
\n\nMy approach was somewhat similar to @Jack (Japan) and at the same time it was quite mainstream:
\n\nThe sad part of all this is that my full prediction cycle took around 60 hours with 6 threads and 20-30GB of RAM needed - so I'm extremely impressed with @Jack (Japan) 2 hours performance
", "rawMarkdown": "First I would like to congratulate everyone and especially @Tom Van de Wiele and @Markus Kliegl - their results are truly impressive and I hope to learn a lot from their ideas and solutions.\r\n\r\nMy approach was somewhat similar to @Jack (Japan) and at the same time it was quite mainstream:\r\n\r\n - kde models were build from full data to predict Pr(place_id | x), Pr(place_id | hour), etc. \r\n - a 80x120 grid, samples with place_id with less then 10 occurrences was dropped out\r\n - for each cell, an xgboost model was trained on features ['x', 'y', 'x_d_y', 'x_t_y', 'hour', 'weekday', 'month', 'year', 'accuracy', 'accuracy_diff'] (where 'accuracy_diff' is difference between accuracy in current sample and rolling median accuracy in train set) to predict Pr(place_id)\r\n - a product of all these probabilities have been taken, then based on it a top3 places was selected\r\n - finally, I had a script for ensembling my submission files, based on idea that was somewhere here on forum: we take entry one by one from each file, then we summarize scores for each place_id based on its place in each prediction entry : 3 for first place, 2 for second and 1 for third. Then we re-order them according to score, save this new prediction and move to the next one. This gave me ~0.0025 comparing to my best single prediction (0.6060 LB), after combining with other 4 similar submissions. For my older models, I was able to gain ~0.007 by combining more then 10 submission files.\r\n\r\nThe sad part of all this is that my full prediction cycle took around 60 hours with 6 threads and 20-30GB of RAM needed - so I'm extremely impressed with @Jack (Japan) 2 hours performance", "votes": null }, { "id": "126536", "postDate": "07/09/2016 13:46:45", "content": "Thank you for all your reply to me!
\n\nIn fact, I didn't use special technique to estimate the distribution, but for x, y and accuracy, latest check-ins were weighted more.
\n\nI uploaded the script of my approach here (but a little simple version), and I hope this will help you!
", "rawMarkdown": "Thank you for all your reply to me!\r\n\r\nIn fact, I didn't use special technique to estimate the distribution, but for x, y and accuracy, latest check-ins were weighted more.\r\n\r\nI uploaded the script of my approach [here][1] (but a little simple version), and I hope this will help you!\r\n\r\n\r\n [1]: https://www.kaggle.com/rsakata/facebook-v-predicting-check-ins/3rd-place-solution-simple-version", "votes": null }, { "id": "126601", "postDate": "07/10/2016 15:53:14", "content": "I'm still working on cleaning up the many-iteration/GMM version that got to second place, but in the meanwhile here is code for a simplified single-pass solution that gets to 0.60456 (14th place) in around 90 minutes on my laptop: https://github.com/mkliegl/kaggle-Facebook-V
", "rawMarkdown": "I'm still working on cleaning up the many-iteration/GMM version that got to second place, but in the meanwhile here is code for a simplified single-pass solution that gets to 0.60456 (14th place) in around 90 minutes on my laptop: https://github.com/mkliegl/kaggle-Facebook-V", "votes": null }, { "id": "127067", "postDate": "07/13/2016 08:02:24", "content": "Thank you all for your warm feedback so far! This has really been an extraordinarily fun competition.
\n\nI've finally updated the repository https://github.com/mkliegl/kaggle-Facebook-V with two more versions.
\n\nOne version scores 0.61047 without using any information about the future. (This is just the previous single-pass version but with P(x, y | place_id) averaged with probabilities from Gaussian Mixture Models).
\n\nThe other version scores 0.62036 by doing 20 iterations of learning from predictions. (This is a slightly simplified and cleaned up version of my actual 2nd place submission.)
\n\nThere is still a lot more documenting, code refactoring, etc. that could be done, but I think I will need to leave it at this for the moment. Please do feel free to ask questions, post suggestions or feedback, etc. here or by messaging me.
", "rawMarkdown": "Thank you all for your warm feedback so far! This has really been an extraordinarily fun competition.\r\n\r\nI've finally updated the repository https://github.com/mkliegl/kaggle-Facebook-V with two more versions.\r\n\r\nOne version scores 0.61047 without using any information about the future. (This is just the previous single-pass version but with P(x, y | place_id) averaged with probabilities from Gaussian Mixture Models).\r\n\r\nThe other version scores 0.62036 by doing 20 iterations of learning from predictions. (This is a slightly simplified and cleaned up version of my actual 2nd place submission.)\r\n\r\nThere is still a lot more documenting, code refactoring, etc. that could be done, but I think I will need to leave it at this for the moment. Please do feel free to ask questions, post suggestions or feedback, etc. here or by messaging me.", "votes": null }, { "id": "128236", "postDate": "07/18/2016 16:38:17", "content": "@Markus,
\n\nThanks very much for sharing your code. I really appreciate your write ups! :-) I am learning a lot.
\n\nJust to see what would happen, I took an ensemble of your 3 models, combined them with the 2 KDE models from Qinchen, and my own models.
\n\nThe result is a score of: 0.62287 which, surprising to me, beats Tom's amazing score. See my screenshot in the attachment.
\n\nWhat's interesting to me is that these weights are not optimized (I'm in the process now of integrating your models with my code to find the strongest soft-voting weights for an ensemble -- and I am planning to release the code in gitub soon).
\n\nHere are the weights that I used:
\n\nscore("sub_20.csv",32); # multi pass model\nscore("sub_singlepass2.csv",10); # single pass/GMM model\nscore("sub_singlepass.csv",8); # single pass model\nscore("submit_kernel.csv",8); # Qinchen's kde best model\nscore("submit_kernel2.csv",8); # Qinchen's kde alternative model\nscore("xgb_feature_set_1.csv",8); # My xgb feature set 1 model\nscore("knn_feature_set_2.csv",12); # My knn feature set 2 model \nscore("rfe_feature_set_2.csv",8); # My rfe feature set 2 model\nscore("xgb_feature_set_2.csv",8); # My xgb feature set 2 model\nI've added the screenshot as an attachment.
", "rawMarkdown": "Markus,\r\n\r\nThanks very much for sharing your code. I really appreciate your write ups! :-) I am learning a lot.\r\n\r\nJust to see what would happen, I took an ensemble of your 3 models, combined them with the [2 KDE models][1] from Qinchen, and [my own models][2].\r\n\r\nThe result is a score of: 0.62287 which, surprising to me, beats Tom's amazing score. See my screenshot in the attachment. \r\n\r\nWhat's interesting to me is that these weights are not optimized (I'm in the process now of integrating your models with my code to find the strongest soft-voting weights for an ensemble -- and I am planning to release the code in gitub soon).\r\n\r\nHere are the weights that I used:\r\n\r\n score(\"sub_20.csv\",32); # multi pass model\r\n score(\"sub_singlepass2.csv\",10); # single pass/GMM model\r\n score(\"sub_singlepass.csv\",8); # single pass model\r\n score(\"submit_kernel.csv\",8); # Qinchen's kde best model\r\n score(\"submit_kernel2.csv\",8); # Qinchen's kde alternative model\r\n score(\"xgb_feature_set_1.csv\",8); # My xgb feature set 1 model\r\n score(\"knn_feature_set_2.csv\",12); # My knn feature set 2 model \r\n score(\"rfe_feature_set_2.csv\",8); # My rfe feature set 2 model\r\n score(\"xgb_feature_set_2.csv\",8); # My xgb feature set 2 model\r\n\r\nI've added the screenshot as an attachment.\r\n\r\n\r\n\r\n\r\n\r\n [1]: https://www.kaggle.com/c/facebook-v-predicting-check-ins/forums/t/22123/6th-place-kernel-density-estimation\r\n [2]: https://www.kaggle.com/c/facebook-v-predicting-check-ins/forums/t/22086/solution-sharing-i-got-to-23-with-xgb-rf-knn", "votes": null }, { "id": "128239", "postDate": "07/18/2016 17:01:19", "content": "Thank you for your interesting research Larry! I have added the zipped winning submission to the main folder of my GitHub page of the competition, it would be great if you could incorporate that into your ensembling analysis!
", "rawMarkdown": "Thank you for your interesting research Larry! I have added the zipped winning submission to the main folder of my [GitHub][1] page of the competition, it would be great if you could incorporate that into your ensembling analysis!\r\n\r\n\r\n [1]: https://github.com/ttvand/Facebook-V", "votes": null }, { "id": "128240", "postDate": "07/18/2016 17:08:35", "content": "Thanks, @Tom. This is great. It will be fascinating to see how high the score goes! :-)
", "rawMarkdown": "Thanks, @Tom. This is great. It will be fascinating to see how high the score goes! :-)", "votes": null }, { "id": "128287", "postDate": "07/19/2016 00:13:38", "content": "@Markus Kliegl
\n\nThanks for sharing the code. It is very informative.
\n\nI have a question regarding how you calculate histogram. In a few places (for example the lines below), you take a sample, compute its bin number, then do a rounded version of that bin number ("get_scaled_time"). You distribute the value between either (n, n+1) or (n-1, n). What's rationale behind it? For a straightforward histogram, if a sample falls in a bin, all its contribution is on that bin. Are you assuming some simple kernel here?
\n\nhttps://github.com/mkliegl/kaggle-Facebook-V/blob/master/multipass.cc#L101-L110
", "rawMarkdown": "Markus Kliegl\r\n\r\nThanks for sharing the code. It is very informative. \r\n\r\nI have a question regarding how you calculate histogram. In a few places (for example the lines below), you take a sample, compute its bin number, then do a rounded version of that bin number (\"get_scaled_time\"). You distribute the value between either (n, n+1) or (n-1, n). What's rationale behind it? For a straightforward histogram, if a sample falls in a bin, all its contribution is on that bin. Are you assuming some simple kernel here?\r\n\r\nhttps://github.com/mkliegl/kaggle-Facebook-V/blob/master/multipass.cc#L101-L110", "votes": null }, { "id": "128305", "postDate": "07/19/2016 03:26:39", "content": "@Larry Freeman - Interesting work! Since Tom's approach was quite different, I think there is potential for another big jump if you add his submission to the ensemble. I'd be curious to see if that really works out :-)
\n\n@data_js - Good question. The code started as an ordinary "histogram" but grew into something rather different so the terminology is not really appropriate anymore. (The intuition was to smooth out the histograms to make them a little more like KDE.)
\n\nReally what is happening in these "histograms" is something like this:
\n\nLearning from data:
\n\nGetting a prediction for data:
\n\n(Does somebody know a good technical term for this? I think these sorts of ideas are common in sampling theory, computer graphics, etc.)
\n\nIn most cases, I used just a triangular filter of width equal to twice the distance between bins for filter A. (That gives rise to distributing the value between (n, n+1) or (n-1, n) as you observe.) This filter can maybe be thought of as representing the uncertainty of which "bin" to assign a given data point to.
\n\nFor the spatial histogram, the filters could more naturally be thought of as representing the uncertainty in the recorded (x, y) values themselves and using Gaussian filters worked well.
\n\n(Aside: In my actual submission, I had included a little accuracy dependence in the x standard deviation of the filters - but the signal was not very strong, so I took this out for the simplified versions on Github. @Tom Van de Wiele did a more careful analysis of accuracy vs x uncertainty in his writeup.)
", "rawMarkdown": "Larry Freeman - Interesting work! Since Tom's approach was quite different, I think there is potential for another big jump if you add his submission to the ensemble. I'd be curious to see if that really works out :-)\r\n\r\n@data_js - Good question. The code started as an ordinary \"histogram\" but grew into something rather different so the terminology is not really appropriate anymore. (The intuition was to smooth out the histograms to make them a little more like KDE.)\r\n\r\nReally what is happening in these \"histograms\" is something like this:\r\n\r\n- Learning from data:\r\n\r\n 1. convolve point with filter A\r\n 2. sample the result at discrete points (what were originally histogram \"bins\")\r\n\r\n- Getting a prediction for data:\r\n\r\n 1. convolve point with filter B\r\n 2. sample at the \"bins\" accordingly\r\n\r\n(Does somebody know a good technical term for this? I think these sorts of ideas are common in sampling theory, computer graphics, etc.)\r\n\r\nIn most cases, I used just a triangular filter of width equal to twice the distance between bins for filter A. (That gives rise to distributing the value between (n, n+1) or (n-1, n) as you observe.) This filter can maybe be thought of as representing the uncertainty of which \"bin\" to assign a given data point to.\r\n\r\nFor the spatial histogram, the filters could more naturally be thought of as representing the uncertainty in the recorded (x, y) values themselves and using Gaussian filters worked well.\r\n\r\n(Aside: In my actual submission, I had included a little accuracy dependence in the x standard deviation of the filters - but the signal was not very strong, so I took this out for the simplified versions on Github. @Tom Van de Wiele did a more careful analysis of accuracy vs x uncertainty in [his writeup](https://ttvand.github.io/Winning-approach-of-the-Facebook-V-Kaggle-competition/).)", "votes": null }, { "id": "128405", "postDate": "07/19/2016 22:43:06", "content": "@Markus Kliegl
\n\nYou interpolate the samples with triangle functions and evaluate the values at the center of the bins. This is how you distribute the values between (n-1, n) or (n, n+1), depending where the sample lands. Thanks for your detailed explanation. Your code is very clean and well worth reading.
", "rawMarkdown": "Markus Kliegl\r\n\r\nYou interpolate the samples with triangle functions and evaluate the values at the center of the bins. This is how you distribute the values between (n-1, n) or (n, n+1), depending where the sample lands. Thanks for your detailed explanation. Your code is very clean and well worth reading.", "votes": null }, { "id": "128862", "postDate": "07/24/2016 22:46:24", "content": "Hi Everyone,
\n\nWhen I ensemble Tom's winning csv, with Markus's csv, and a few other models, the result scores 0.62865.
\n\nIt should be possible to get a higher score as we include further models. Here is the code that I used to ensemble the models.
\n\nAttached is the screenshot of the score. I am still finishing my code to iterate through the possible weighted combinations. I will add it to the repository once it is completed.
\n\nCheers,
\n\n-Larry
", "rawMarkdown": "Hi Everyone,\r\n\r\nWhen I ensemble Tom's winning csv, with Markus's csv, and a few other models, the result scores 0.62865.\r\n\r\nIt should be possible to get a higher score as we include further models. Here is the [code][1] that I used to ensemble the models.\r\n\r\nAttached is the screenshot of the score. I am still finishing my code to iterate through the possible weighted combinations. I will add it to the repository once it is completed.\r\n\r\nCheers,\r\n\r\n-Larry\r\n\r\n\r\n [1]: https://github.com/larry94555/kaggle-Facebook-V", "votes": null }, { "id": "128969", "postDate": "07/25/2016 15:35:37", "content": "Thanks for your great analysis Larry! I guess we could end up with a score close to 0.63 if the models would contain more than 3 predicted places.
", "rawMarkdown": "Thanks for your great analysis Larry! I guess we could end up with a score close to 0.63 if the models would contain more than 3 predicted places.", "votes": null } ], "comments": [ { "id": 126195, "author_name": "andrenaef", "author_url": "", "post_date": "07/07/2016 01:24:29", "content": "@jturkewitz Thanks for sharing the "peek into the future" idea. Clever thinking!
\n\nMy best submission (place 7) is an ensemble of three models.
\n\n1. Regression Model
\n\nThis model performs logistic regression using XGBoost to determine the likelihood of a pairing of a checkin and a place. For each observation in the training set, 50 observations are derived. The actual place serves as the positive observation, and the 49 closest places serve as negative observations. Training observations thus grow by a factor of 50, and only 20% of the original training data was used to build this model. Gains in precission became diminishing after that.
\n\nThe features of this model are based on a set of pre-computed per-place data, such as as mean and standard deviation of location and accuracy, visitor frequencies, and neighborhood sparsity. Outlier removal using Turkey's test was used for some of these statistics.
\n\nIn particular, the model features include:
\n\nIndividual MAP@3 on the public leaderboard: 0.58452
\n\nWhile this is not the best performing individual model, a strong point of this model is its high degree of generalization. Adding predictive support for a new place only requires calculating the per-place statistics for that place. This is different from the other two models which need to be partially re-trained whenever the set of places changes.
\n\n2. Classification Model
\n\nThis is a grid-based classification model using XGBoost to directly classify an observation to a place. 25 X cuts, and 100 Y cuts are used. Instead of heavy border augmentation, four instances of the model are built, each one shifted by a quarter of a grid.
\n\nFeatures include:
\n\nIndividual MAP@3 on the public leaderboard: 0.59420
\n\n3. Nearest Neighbor Model
\n\nThis is pretty much the awesome and extremely efficient model developed Sandro, Alex, Michael Hartmann, and many others. As with the classification model, four instances of the model are built, each one shifted by a quarter of a grid.
\n\n4. Ensemble
\n\nFor each model instance and test observation, the top 6 places and related probabilities are stored.
\n\nEnsembling then consists of two phases. In the first phase, the model instances are aggregated per main model to derive the average probability of a place by that model; in the second phase, a weighted average of the probabilities from the three main models is calculated. The submission then consists of the three places with the highest average probability.
\n\nThanks to Facebook for coming up with an awesome problem, based on a simple data set consisting of only 5 attributes, but providing significant depth. And thanks to the Kaggle community! The discussions on the forums and the activity in the scripts have really been a highlight of this competition.
", "votes": null, "replies": [] }, { "id": 126198, "author_name": "apapiu", "author_url": "", "post_date": "07/07/2016 01:30:45", "content": "Nice solution @Jared - if I remember correctly you were the first person to mention using a classifier(xgboost) on a x-y grid early on and that seems to be the best approach by far. Thanks for the inspiration.
\n\nI got lazy and just combined the probabilities from a knn and a random forest on the same grid - 40 by 40 - which was sub optimal. Could you elaborate a bit how you averaged the top 6 probabilities from the two different models?
", "votes": null, "replies": [] }, { "id": 126200, "author_name": "rshekhar2", "author_url": "", "post_date": "07/07/2016 01:34:23", "content": "Post mortem: I was wondering how others did their ensembling. I tried to build model stacking using XGBoost, and failed horribly (the result was worse than individual submissions, and I couldn't figure out why). I ended up implementing a weighted voting classifier with some simple guesses for weights and a huge 24 member ensemble
\n\n3 Naive XGboost runs (only time based features) -- In range (0.580 - 0.586)\n10 XGBoost with Nearest Neighbor features -- In range (0.580 - 0.59)\n5 kNN (16, 19, 24, 29, 31) with Bayesian Gaussian Process optimized NN weights -- (0.575 - 0.582)\n1 Abhi's fantastic public script (0.5865)\n1 Michael Hartmann's fantastic public script (0.58?)\n2 Sklearn Random Forest with only time features -- (0.575)\n1 ExtraTrees with nearest neighbor features - (0.583)\n1 Sklearn Random Forest with nearest neighbor features (0.589)\nI tried and excluded:
\n\n- Keras based Convolutional Neural Network solution (MAP3 = 0.4)\n- T-SNE based clustering (tried this multiple ways) (MAP3 = 0.3)\n- Stacked RBM based ideas (vanishing gradient problem)\n- Naive Bayes (well this one is obvious, but I thought I'd give it a shot) \n- Sklearn ExtraTrees classifier (MAP3 = 0.52)\n- Simple first principles Kernel Density estimation and likelihood ratios (MAP3 = 0.52)\n- Kernel SVM (MAP3 = 0.42)\nVoting classifier weights depended on validation MAP3 score (higher score, higher weight) and a similarity metric for predictions I quickly thought up last night (more similar to other runs, lower weight).
", "votes": null, "replies": [] }, { "id": 126207, "author_name": "radustoicescu", "author_url": "", "post_date": "07/07/2016 02:05:19", "content": "I had an ensemble of one RF and one XGB.
\n\nMy only working trick was to multiply the predicted probability of a place_id like this:
\n\npredicted_probab_place_id*(0.5+first_check_in_normalized)*(0.2+last_check_in_normalized)
\n\nso if the first check in for that place_it was at timestamp 10000 and the last one at 200000:
\n\nfirst_check_in_normalized = 10000/786240 = 0.012
\n\nlast_check_in_normalized = 200000/786240 = 0.254
\n\nso the new_predict_prob = predicted_probab_place_id*(0.5+0.012)*(0.2+0.254) = 0.232*predicted_probab_place_id
\n\nthen out of the 5, return most_common(k)
\n\nIn other words I penalize latecomers and early goers (place_ids in the dataset)
\n\nEdit: I had a second working trick. I counted the number neighboring check ins 4 hours before and after and made a new feature, this is peeking into the future.
\n\nThe other methods I used were public knowledge before the competition was over.
", "votes": null, "replies": [] }, { "id": 126211, "author_name": "jturkewitz", "author_url": "", "post_date": "07/07/2016 02:13:46", "content": "[quote=Alexandru Papiu;126198]
\n\nNice solution @Jared - if I remember correctly you were the first person to mention using a classifier(xgboost) on a x-y grid early on and that seems to be the best approach by far. Thanks for the inspiration.
\n\nI got lazy and just combined the probabilities from a knn and a random forest on the same grid - 40 by 40 - which was sub optimal. Could you elaborate a bit how you averaged the top 6 probabilities from the two different models?
\n\n[/quote]
\n\nAs a particle physics grad student I regularly deal with large data that can be treated independently so the grid idea came to me pretty naturally. In physics every collision is independent, and we collect data from tons of collisions so you have to analyze it with many computers in parallel.
\n\nTo average the probabilities I first stored the top 6 place_ids with their probabilities from each model. Then if the same place_id was in both models, I just took the sum from the xgb prob and the knn prob (the knn prob was weighted at 0.9). If an xgb place_id was not in the top 6 knn place_ids, then I would just divide that probability by 2. If a knn place_id was not in the top 6 xgb place ids, I would divide that probability by 3. Then I reordered the place_ids based on the new top 3 probs.
", "votes": null, "replies": [] }, { "id": 126212, "author_name": "mikeskim", "author_url": "", "post_date": "07/07/2016 02:23:47", "content": "I didn't use XGBoost because it seemed to take a lot longer than ExtraTrees/RandomForest. My entire model would run in less than a day or even half a day. So I focused on feature engineering into ET/RF. I think this was a mistake since it's probable an ensemble of varying seed XGs could probably beat my best hand crafted model of ET/RF. I guess XG is really too strong.
\n\nBut I had a lot of "interesting" albeit not so strong ideas. Potentially any of these features could improve an XG model, but I don't know for certain. I've attached one feature that seemed to add quite a bit, but there are many others.
", "votes": null, "replies": [] }, { "id": 126222, "author_name": "leonlu", "author_url": "", "post_date": "07/07/2016 03:09:32", "content": "Thank @Jared for creating this topic, and thank everyone here for sharing your method.
\n\nMy laptop had a hardware failure in the last week and I lost two sets of predictions that I could have use for my ensemble model. But anyway, here is my model of my best submission:
\n\nBase models
\n\nxgboost model 1 on 25*80 grids
\n\nfeatures: x, y, accuracy, hour of a day, day of a week, week of a year, year
\n\nexponentially decaying sample weights
\n\nexpanded grid border in training (or border augmentation)
\n\n3 grid offsets
\n\nbase model public LB score: 0.597 with 1 grid offset, 0.601 with 3 grid offsets combined
xgboost model 2
\n\nall the same as model 1, except that it used "sin", "cos" transforms on the day of week feature
\n\n3 grid offsets
\n\nbase model public LB score: 0.598 with 1 grid offset, unknown with 3 grid offsets as I initially planned to have 5 grid offsets if my laptop didn't fail me.
KNN from the kaggle script
\n\nbase model public LB score: 0.583
Ensemble
\n\nSoft voting of top 10 predictions from each model.
\n\nFinal score: 0.607
\n\nI think the diversity between xgb model and KNN model played an important role in improving the score of the final ensemble model. I would also consider adding random forest, extra trees, and even neural networks to the final ensemble because they worked as good as KNN in my early testing of various models.
\n\nAlthough with more careful parameter tuning my model could still get a better score, I don't think it would get anywhere close to 0.620, so I'm really looking forward to @tvdwiele and @Markus sharing their awesome methods.
", "votes": null, "replies": [] }, { "id": 126229, "author_name": "syllogismos", "author_url": "", "post_date": "07/07/2016 04:17:57", "content": "https://github.com/syllogismos/facebook-v-predicting-check-ins/blob/master/summary.ipynb
\n\nI wrote up the summary of my submission here.
\n\nWow I didn't know ensembling can improve score like that.
\n\nMy best submission is a single xgboost model, a 40x200 gird size with border augmentation.. that came up with 0.58685 on pub lb. I initially saved last two days for ensembling models, but decided against it for god knows what reason.. and instead worked on doing something else. I genuinely thought I was missing something..
\n\nAlong with using nthread = 4 I also trained 8 grid rows at a time parallely.. which took around 3.5 hours on c4.8xlarge spot machine to generate a submission..
\n\nI did have KNN and xgboost of several grid sizes and top 10 predictions and I didn't really make use of any of them..
\n\nbut I didn't even think I would beat the lb scores of people from public scripts.. but with 6 days to go, I came in top 100 which was pretty cool :P\nno matter what I did in the end, my score wasn't improving at all.. all my submissions are single models only. I'm pretty bummed about it now. I will try submitting my ensembles and check and weep if they are better..
\n\nI was trying things like creating more features from the top 10 predictions I got from a run, features like the place cardinality, its median, its mode of time intervals and all that, but none of them gave me any gains.
", "votes": null, "replies": [] }, { "id": 126235, "author_name": "mkliegl", "author_url": "", "post_date": "07/07/2016 04:42:58", "content": "Here are the main ideas of my approach. I'll clean up the code a little and post it on Github within the next few days.
\n\nOverall structure
\n\nThe overall idea was to compute for each checkin and some candidate set of place_id's a probability proportional to
\n\nP(x, y | place_id) P(day of week | place_id) P(time of day | place_id) P(accuracy | place_id) P(place_id) .
\n\nChallenges were:
\n\nLearning the conditional distributions like P(x, y | place_id) efficiently without losing too much information.
The relative popularity of places, P(place_id), varied considerably over time. (So really it should be P(place_id, time).) How do we learn the relative popularities for the times in the test data?
1. Learning the conditional distributions efficiently
\n\nMy first attempts to learn the conditional distributions were far too slow given the size of the data.
\n\nBut early in the competition @ZFTurbo's "Mad Scripts Battle" script did pretty well (and efficiently so!) by essentially just using a histogram to learn the joint distribution P(x, y, quarter time of day). Inspired by this, I decided to try to learn the conditional distributions using histograms. I also switched from Python to C++ to speed things up even more.
\n\nTo make the histograms more accurate, I made them periodic for time of day and day of week, and I used various kernels (triangular, Gaussian, exponential) to both save and read values from the histogram - kind of like KDE but applied to a histogram of a fixed number of bins rather than all data points. This remained reasonably efficient while giving results almost as good as KDE.
\n\nFor later submissions, I averaged the spatial histogram with another approach: I fitted GMM's with up to 15 components on the train data (the exact number for each place_id selected by AIC). This took pretty long and I just computed the GMM components once and loaded the results for all future runs.
\n\nAlso, as observed in the forums, P(accuracy | place_id) and P(x, y | place_id) seemed to vary over time. Giving more weight to later data helped.
\n\n2. Learning from the future
\n\nUsing just P(place_id) based on the train data, the predictions were already very good: About half of the time the first prediction was correct. So we could think of the top predictions as actually labeling the test data - just very noisily - and train on the test data using the top 10-20 or so predicted place_id's as labels with their probabilities as weights.
\n\nThis idea led to an approach of iterating between making predictions for the test data and learning from those predictions. The final submission used 20 iterations. (Actually, I ended up learning not only P(place_id, time) from these iterations but also kept learning the other histograms based on the test data. This helped the score and made the convergence much more stable.)
\n\n(The 40-minute single-pass solution I mentioned elsewhere would just line-by-line make a prediction and then learn from it before making the next prediction.)
\n\nPossible alternative? My original thought was actually to try to learn P(place_id, time) using likelihood maximization in some form. Conceptually, I still think this would be a nicer approach than what I did, but what I had was working well and I never got around to this. It also seemed harder to integrate learning the other histograms into this approach. I may have needed to alternate it with predict/learn iterations in any case.
", "votes": null, "replies": [] }, { "id": 126237, "author_name": "aldente", "author_url": "", "post_date": "07/07/2016 04:45:54", "content": "[quote=Alexandru Papiu;126198]
\n\nI got lazy and just combined the probabilities from a knn and a random forest on the same grid - 40 by 40 - which was sub optimal. Could you elaborate a bit how you averaged the top 6 probabilities from the two different models?
\n\n[/quote]
\n\n@Alexandru Papiu, I did the same with XGB and KNN probabilities on the same grid 20x40, in which the XGB is too long so I could only finish 693 cells out of 800.
\n\nSurprised for the power of the voting ensemble. @all, thanks a lot for sharing, I appreciate it.
", "votes": null, "replies": [] }, { "id": 126242, "author_name": "gopalsharma2001", "author_url": "", "post_date": "07/07/2016 05:11:17", "content": "Congratulations everyone, and specially to the guys at the top. I just tried to use KNN model implementation publicly available in this competition and played around the feature engg and variable importance. The idea was to see how much it can score in environment similar to kaggle sandbox. I used aws m4.large and all of my model took sub-30 minutes to run. Lastly I tried ensembling of my top models which increased my score by .00020. Maybe ensembling of different types of model will result in far better result as evident from top results.
\n\nTop results just confirm that XG is one of the powerful models to use in these kind of competitions. Maybe I will try ensemble this with KNN model to see how far I can go.
", "votes": null, "replies": [] }, { "id": 126244, "author_name": "tonyzhangrt", "author_url": "", "post_date": "07/07/2016 05:23:32", "content": "@Markus, Hi Markus, congratulations and thanks for sharing your model. It is interesting to see a good solution without using any of common tools of kagglers.
\n\nI just have some questions about what you did with learning the conditional probabilities.
\n\nWhen you train GMM models, are you training over (x,y), week of day, time of day, accuracy separately and did you use any special strategies to deal with the periodic nature of the week of day, time of day feature ?
How did you eventually combine these conditional probability to produce the probability for each check in, was it like a weighted sum of log probability or something else ? If it is, did you find the weights varies with location, time, etc.
@Markus
\n\nWow, it's amazing! Using predictions on test data as pseudo-labels for training and iterating between them is indeed a very interesting idea.
", "votes": null, "replies": [] }, { "id": 126252, "author_name": "tonyzhangrt", "author_url": "", "post_date": "07/07/2016 05:46:01", "content": "Like many other kagglers, my final model is an ensemble model of various different models.
\n\nMy single best model was an xgb model with the usual features seen in other xgb models on the forum combined with knn probability for selected place_id. judging from the local validation, it would achieve about 0.591 on LB trained over a 63 * 159 grid. For ensembling, I also trained another 3 versions of the same model with grid shifted in x, y and (x,y) by half the grid size, which would boost to about 0.597 on LB, again judging from the local validation.
", "votes": null, "replies": [] }, { "id": 126260, "author_name": "pinocchio", "author_url": "", "post_date": "07/07/2016 06:36:30", "content": "Hi @Markus,
\n\nI am curious how was the performance of only P(x, y | place_id) P(place_id) ?\nI gave it a shot using sklearn's GMM, 6 component, and training it on the whole training set but it didn't give me good result at all (even worse than Gaussian Naive Base on only x and y.)
", "votes": null, "replies": [] }, { "id": 126262, "author_name": "frederik", "author_url": "", "post_date": "07/07/2016 07:14:35", "content": "Thanks to Facebook and all the competition participants. Very impressive lead by the leader board top seats, looking forward to look into their solutions.
\n\nEnsemble
\n\nMy solution is a simple ensemble of xgboost and kNN. LB results:
\n\nXgboost model details
\n\nSegmented the data into grid with 200x200m squares. The boundaries of the train squares was extended by an additional 15m.
\n\n9-11 features based on:
\n\nMy best submission was ensemble of 7 XGBoost grid based solutions and 5 best KNNs from public scripts.
\n\nCode to ensemble was pretty straightforward (applied for each ID):
\n\ndef process_places(arr):\n res = dict()\n for i in range(len(arr)):\n item = arr[i][0]\n if item in res:\n res[item] += 3.02 - i*0.0001\n else:\n res[item] = 3.02 - i*0.0001\n item = arr[i][1]\n if item in res:\n res[item] += 2.01 - i*0.0001\n else:\n res[item] = 2.01 - i*0.0001\n item = arr[i][2]\n if item in res:\n res[item] += 1 - i*0.0001\n else:\n res[item] = 1 - i*0.0001\n topitems = nlargest(3, sorted(res.items()), key=itemgetter(1))\n return topitems\n@Markus Awesome approach, and congratulations on your strong finish!
\n\nCan you share some insight on what makes your iterative approach converge, as opposed to going more and more off-track?
", "votes": null, "replies": [] }, { "id": 126285, "author_name": "qwang88", "author_url": "", "post_date": "07/07/2016 11:05:08", "content": "My solution was an ensemble of 4 methods: xgboost, knn, random forests, and kernel density estimation. My xgboost score was slightly worse than Jared's, my knn score was ~0.587, my rf score was ~0.584, and my kde score was by far the best at ~0.604. I'm very happy to see that both Tom and Markus had used very different methods from what is traditionally used in Kaggle competitions as we now have new things to learn.
", "votes": null, "replies": [] }, { "id": 126290, "author_name": "rsakata", "author_url": "", "post_date": "07/07/2016 12:35:09", "content": "The idea of my solution is similar to that of Markus.
\n\nProblem definition
\n\nFind top 3 places of following probability for candidates of each row.
\n\nPr(place_id | x, y, time, accuracy)
\n\n∼ Pr(x, y, time, accuracy | place_id) * Pr(place_id)
\n\n≈ Pr(x | place_id) * Pr(y | place_id) * Pr(time of day | place_id) * Pr(week of day | place_id) * Pr(accuracy | place_id) * Pr(place_id)
\n\nSteps of my solution
\n\nI spent most of time to estimate distributions, and I never tried ensemble.
\n\nMy approach took about 2 hours by my 8GB RAM PC.
\n\nFinally, congratulations to tvdwiele and Markus, and everyone!
", "votes": null, "replies": [] }, { "id": 126297, "author_name": "gradient88", "author_url": "", "post_date": "07/07/2016 14:34:07", "content": "@Jack (Japan), smart idea~ i am quite interested in this method. Could you share the code detail on github or somewhere else if u do not mind?
\n\nthx a lot!
", "votes": null, "replies": [] }, { "id": 126298, "author_name": "rshekhar2", "author_url": "", "post_date": "07/07/2016 14:45:18", "content": "I'm very impressed by the simplicity and elegance of a few of the solutions presented here, in particular @Jack. While my results aren't nearly as clean, and I scored 40th, it was a fantastic learning experience, and I'm really grateful to be able to learn from such bright people. I thought I would share my code and techniques.
\n\nStep 0:\nMy first submissions were based on the random forest 0.55 starter script. Didn't end up using it because it didn't perform well enough.
\n\nStep 1:\nUse a variation on the KNN script posted by @Sandro and @David, that I modified to allow for Bayesian optimization of hyperparameters. This is available on Github here. I ran 8 parallel copies of this script in Bayes mode on an 8 core machine for a couple of days. I then picked a few of the well performing sets of parameters for different kNN numbers. I ended up getting good weights for 16, 19, 24, 29, 31 neighbors, which I felt spanned the k space "well enough".
\n\nStep 2: \nI ran XGBoost with three different sets of hyperparameters, nothing too special. This included border augmentation and different grid sizes. Here's the script.
\n\nI also went down the many failed approaches avenue here. I explored the space of possible classifiers here, trying out most Sklearn classifiers (after all I was supposed to be learning things). Tried Keras based parametric T-SNE to get useful clustering, which turned out to not work very well. Oh well, live and learn.
\n\nI ended up finding that Sklearn Random Forest performed comparably to XGB, and kept a couple of the predictions.
\n\nStep 3:\nI added nearest neighbor features to XGBoost and other classifiers. An example script is here. At first I was unhappy because this didn't seem to perform better than plain XGBoost without nearest neighbor features. However, I found that ensembling this with other predictions improved the score substantially. Despite the individual classifier not performing better, it was actually substantially decorrelated from other classifiers, making a superior ensemble.
\n\nStep 4:\nI tried a blending classifier with XGB but it didn't work and I couldn't figure out why, so I abandoned it.
\n\nI was swamped with other obligations towards the end of the competition (like writing my PhD dissertation), so when Abhi's and Michael Hartmann's kNN improvements were released, I didn't really have time to work them into my solution and rerun. So I added their public scripts to my ensemble, and used a weighted voting classifier where weights depended on score and similarity to other classifiers. This voting classifier forms my final result, with 0.5995 on the public leaderboard, and 0.5993 on the private leaderboard.
", "votes": null, "replies": [] }, { "id": 126299, "author_name": "mikeskim", "author_url": "", "post_date": "07/07/2016 14:47:47", "content": "I'm not sure if I created a feature similar to Jack's 2), but here's the R code of one of my features.
", "votes": null, "replies": [] }, { "id": 126301, "author_name": "citynight", "author_url": "", "post_date": "07/07/2016 15:02:06", "content": "Jack's solution is simple and great. It looks like a Naive Bayes classifier to me, but I think he did a good job to customize the distribution and density estimation.
\n\nI also tried Naive Bayes with prior class probability predicted by time models, and it scores ~0.593 which is more than my best XGBoost model that was tuned and ran for a week. I only used the standard Naive Bayes function so this makes a big difference.
", "votes": null, "replies": [] }, { "id": 126309, "author_name": "mkliegl", "author_url": "", "post_date": "07/07/2016 16:14:59", "content": "[quote=zrt;126244]
\n\n@Markus, Hi Markus, congratulations and thanks for sharing your model. It is interesting to see a good solution without using any of common tools of kagglers.
\n\nI just have some questions about what you did with learning the conditional probabilities.
\n\nWhen you train GMM models, are you training over (x,y), week of day, time of day, accuracy separately and did you use any special strategies to deal with the periodic nature of the week of day, time of day feature ?
How did you eventually combine these conditional probability to produce the probability for each check in, was it like a weighted sum of log probability or something else ? If it is, did you find the weights varies with location, time, etc.
[/quote]
\n\nI only trained GMM for P(x, y | place_id). I tried briefly for time of day, but it did not work well - perhaps because it did not consider periodicity, as you point out.
When I had two methods for predicting the same probability, I just combined them using a geometric mean (i.e., arithmetic mean of the log probabilities). I just used equal weights - I think tuning weights, esp. by location, time, etc., would have taken too much effort and could be hard to do without overfitting - but maybe it would have helped.
[quote=Andre Naef;126284]
\n\n@Markus Awesome approach, and congratulations on your strong finish!
\n\nCan you share some insight on what makes your iterative approach converge, as opposed to going more and more off-track?
\n\n[/quote]
\n\nThe analogy to tuning (learning rate, number of epochs) when using SGD is pretty good for the issues I had. If you go for too many iterations, the score starts to decrease. When this happens could be predicted reasonably well using validation.
\n\nAside from using the equivalent of a conservative low "learning rate" in the end, these items helped stabilize the convergence:
\n\nI predicted only on the top 25 candidates from the first pass in the iterations. (This is maybe similar to the benefit many people saw of using KNN to limit the number of candidates considered.)
I learned relative popularity / conditional probabilities only from the top 15 predictions. The weights for learning from these predictions were the geometric average of
\n\na. the predicted probabilities
\n\nb. the empirical probability of positions 1-15 being correct based on validation
\n\nThis average stabilized the convergence a fair amount compared to using either of the two individually.
I cumulatively learned the conditional probabilities rather than learning them from scratch during each iteration. This stabilized convergence, but at presumably increased cost of getting stuck in some local optimum.
Initialization: Starting from a less confident initial guess like P(place_id)^{0.3} (where P(place_id) is relative popularity based on train data) helped combat the cost of #3 to some extent.
I also tried learning from future confident predictions but the score deteriorated. I used random forest and selected those samples for which the random forest was outputting a probability of nore than 0.8 . I then combined those test samples to train samples and applied knn to predict the leftover samples. But i think i was doing something wrong or maybe i was doing it half-heartedly.
", "votes": null, "replies": [] }, { "id": 126332, "author_name": "raghakot", "author_url": "", "post_date": "07/07/2016 19:09:55", "content": "Didn't get a chance to try weighting. I wasted a lot of time trying various approaches and features thinking there must be some clever way to do this other than ensembling (since @markus was able to get 0.59 in 40 mins). All of them ended up giving more or less the same score. In retrospect, I should have ensembled them all.
", "votes": null, "replies": [] }, { "id": 126338, "author_name": "phoenixl", "author_url": "", "post_date": "07/07/2016 19:35:41", "content": "@Jack(Japan) Congratulations! Your solution is very elegent and effective. Could you briefly expain how you estimate the distribution? KDE or GMM or other good tools?
\n\nAlso, I am curious about how to use XGB to predict the amount of occurence in future. You must have deep insights in this problem.
", "votes": null, "replies": [] }, { "id": 126360, "author_name": "junshi", "author_url": "", "post_date": "07/07/2016 21:52:41", "content": "@Jack (Japan)
\n\n2.Predict the amount of future occurrence Pr(place_id) by XGBOOST.
\n\nElegant solution indeed.\nWould you please elaborate on the step 2 above? Which features did you use to predict? Training dataset alone does not provide good estimate of Pr(place_id)?
", "votes": null, "replies": [] }, { "id": 126398, "author_name": "aikinogard", "author_url": "", "post_date": "07/08/2016 06:05:33", "content": "@Jack (Japan)\nExcellent solution. I am also looking forward to hear from the detail of your solution:
\n\nHow do you estimate the distribution? (@Rongcheng Lin)
How to predict the future occurence? (@data_js)
First I would like to congratulate everyone and especially @Tom Van de Wiele and @Markus Kliegl - their results are truly impressive and I hope to learn a lot from their ideas and solutions.
\n\nMy approach was somewhat similar to @Jack (Japan) and at the same time it was quite mainstream:
\n\nThe sad part of all this is that my full prediction cycle took around 60 hours with 6 threads and 20-30GB of RAM needed - so I'm extremely impressed with @Jack (Japan) 2 hours performance
", "votes": null, "replies": [] }, { "id": 126536, "author_name": "rsakata", "author_url": "", "post_date": "07/09/2016 13:46:45", "content": "Thank you for all your reply to me!
\n\nIn fact, I didn't use special technique to estimate the distribution, but for x, y and accuracy, latest check-ins were weighted more.
\n\nI uploaded the script of my approach here (but a little simple version), and I hope this will help you!
", "votes": null, "replies": [] }, { "id": 126601, "author_name": "mkliegl", "author_url": "", "post_date": "07/10/2016 15:53:14", "content": "I'm still working on cleaning up the many-iteration/GMM version that got to second place, but in the meanwhile here is code for a simplified single-pass solution that gets to 0.60456 (14th place) in around 90 minutes on my laptop: https://github.com/mkliegl/kaggle-Facebook-V
", "votes": null, "replies": [] }, { "id": 127067, "author_name": "mkliegl", "author_url": "", "post_date": "07/13/2016 08:02:24", "content": "Thank you all for your warm feedback so far! This has really been an extraordinarily fun competition.
\n\nI've finally updated the repository https://github.com/mkliegl/kaggle-Facebook-V with two more versions.
\n\nOne version scores 0.61047 without using any information about the future. (This is just the previous single-pass version but with P(x, y | place_id) averaged with probabilities from Gaussian Mixture Models).
\n\nThe other version scores 0.62036 by doing 20 iterations of learning from predictions. (This is a slightly simplified and cleaned up version of my actual 2nd place submission.)
\n\nThere is still a lot more documenting, code refactoring, etc. that could be done, but I think I will need to leave it at this for the moment. Please do feel free to ask questions, post suggestions or feedback, etc. here or by messaging me.
", "votes": null, "replies": [] }, { "id": 128236, "author_name": "larryfreeman", "author_url": "", "post_date": "07/18/2016 16:38:17", "content": "@Markus,
\n\nThanks very much for sharing your code. I really appreciate your write ups! :-) I am learning a lot.
\n\nJust to see what would happen, I took an ensemble of your 3 models, combined them with the 2 KDE models from Qinchen, and my own models.
\n\nThe result is a score of: 0.62287 which, surprising to me, beats Tom's amazing score. See my screenshot in the attachment.
\n\nWhat's interesting to me is that these weights are not optimized (I'm in the process now of integrating your models with my code to find the strongest soft-voting weights for an ensemble -- and I am planning to release the code in gitub soon).
\n\nHere are the weights that I used:
\n\nscore("sub_20.csv",32); # multi pass model\nscore("sub_singlepass2.csv",10); # single pass/GMM model\nscore("sub_singlepass.csv",8); # single pass model\nscore("submit_kernel.csv",8); # Qinchen's kde best model\nscore("submit_kernel2.csv",8); # Qinchen's kde alternative model\nscore("xgb_feature_set_1.csv",8); # My xgb feature set 1 model\nscore("knn_feature_set_2.csv",12); # My knn feature set 2 model \nscore("rfe_feature_set_2.csv",8); # My rfe feature set 2 model\nscore("xgb_feature_set_2.csv",8); # My xgb feature set 2 model\nI've added the screenshot as an attachment.
", "votes": null, "replies": [] }, { "id": 128239, "author_name": "tvdwiele", "author_url": "", "post_date": "07/18/2016 17:01:19", "content": "Thank you for your interesting research Larry! I have added the zipped winning submission to the main folder of my GitHub page of the competition, it would be great if you could incorporate that into your ensembling analysis!
", "votes": null, "replies": [] }, { "id": 128240, "author_name": "larryfreeman", "author_url": "", "post_date": "07/18/2016 17:08:35", "content": "Thanks, @Tom. This is great. It will be fascinating to see how high the score goes! :-)
", "votes": null, "replies": [] }, { "id": 128287, "author_name": "junshi", "author_url": "", "post_date": "07/19/2016 00:13:38", "content": "@Markus Kliegl
\n\nThanks for sharing the code. It is very informative.
\n\nI have a question regarding how you calculate histogram. In a few places (for example the lines below), you take a sample, compute its bin number, then do a rounded version of that bin number ("get_scaled_time"). You distribute the value between either (n, n+1) or (n-1, n). What's rationale behind it? For a straightforward histogram, if a sample falls in a bin, all its contribution is on that bin. Are you assuming some simple kernel here?
\n\nhttps://github.com/mkliegl/kaggle-Facebook-V/blob/master/multipass.cc#L101-L110
", "votes": null, "replies": [] }, { "id": 128305, "author_name": "mkliegl", "author_url": "", "post_date": "07/19/2016 03:26:39", "content": "@Larry Freeman - Interesting work! Since Tom's approach was quite different, I think there is potential for another big jump if you add his submission to the ensemble. I'd be curious to see if that really works out :-)
\n\n@data_js - Good question. The code started as an ordinary "histogram" but grew into something rather different so the terminology is not really appropriate anymore. (The intuition was to smooth out the histograms to make them a little more like KDE.)
\n\nReally what is happening in these "histograms" is something like this:
\n\nLearning from data:
\n\nGetting a prediction for data:
\n\n(Does somebody know a good technical term for this? I think these sorts of ideas are common in sampling theory, computer graphics, etc.)
\n\nIn most cases, I used just a triangular filter of width equal to twice the distance between bins for filter A. (That gives rise to distributing the value between (n, n+1) or (n-1, n) as you observe.) This filter can maybe be thought of as representing the uncertainty of which "bin" to assign a given data point to.
\n\nFor the spatial histogram, the filters could more naturally be thought of as representing the uncertainty in the recorded (x, y) values themselves and using Gaussian filters worked well.
\n\n(Aside: In my actual submission, I had included a little accuracy dependence in the x standard deviation of the filters - but the signal was not very strong, so I took this out for the simplified versions on Github. @Tom Van de Wiele did a more careful analysis of accuracy vs x uncertainty in his writeup.)
", "votes": null, "replies": [] }, { "id": 128405, "author_name": "junshi", "author_url": "", "post_date": "07/19/2016 22:43:06", "content": "@Markus Kliegl
\n\nYou interpolate the samples with triangle functions and evaluate the values at the center of the bins. This is how you distribute the values between (n-1, n) or (n, n+1), depending where the sample lands. Thanks for your detailed explanation. Your code is very clean and well worth reading.
", "votes": null, "replies": [] }, { "id": 128862, "author_name": "larryfreeman", "author_url": "", "post_date": "07/24/2016 22:46:24", "content": "Hi Everyone,
\n\nWhen I ensemble Tom's winning csv, with Markus's csv, and a few other models, the result scores 0.62865.
\n\nIt should be possible to get a higher score as we include further models. Here is the code that I used to ensemble the models.
\n\nAttached is the screenshot of the score. I am still finishing my code to iterate through the possible weighted combinations. I will add it to the repository once it is completed.
\n\nCheers,
\n\n-Larry
", "votes": null, "replies": [] }, { "id": 128969, "author_name": "tvdwiele", "author_url": "", "post_date": "07/25/2016 15:35:37", "content": "Thanks for your great analysis Larry! I guess we could end up with a score close to 0.63 if the models would contain more than 3 predicted places.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "126191": "Congrats to everyone for a great competition, in particular to the guys at the top. Congrats especially to tvdwiele and Markus who were significantly ahead of the field yet neck-and-neck for a long time. @tvdwiele also thanks for the shiny app, that was probably the most useful data visualizations I've seen on Kaggle (https://www.kaggle.com/c/facebook-v-predicting-check-ins/forums/t/21365/interactive-exploratory-analysis-using-a-shiny-app).\r\n\r\nMy solution (place 20) was the combination of an xgb (0.592 on public lb) and knn (0.585), which somehow combined to get 0.603. I combined them by averaging the top 6 probabilities predicted by each model. Ensembling is surprisingly powerful. I used a 40 x 50 grid for xgb and a 20 x 25 grid for knn, with the extension technique from the scripts.\r\n\r\nMy xgb model took 4 days to run on my dual-core macbook pro. I used the basic time features (hour, weekday, year etc), did not have any real success with advanced feature engineering this competition. I also used the xgb weight option, so more recent times were weighted more heavily while training.\r\n\r\nMy knn model I tried to get around the hour cyclic problem by shifting the hour feature and averaging the predictions from different models. Otherwise it was pretty similar to the early knn scripts.\r\n\r\nOne interesting thing I did was to use the knn model to predict how common a place id was in the test set. This is peaking into the future, so it isn't all that useful in the real world but it provided a boost of about 0.004 on the public lb. I basically used knn to get the average predicted probability, then applied the ratio of the predicted probability in test over the probability in train to weight the probabilities of the next level models.\r\n\r\nMy code needs to be cleaned up, but if you don't mind looking at messy code it is here:\r\nhttps://github.com/jturkewitz/SideProjects/tree/master/Kaggle/Facebook_Checkin", "126195": "jturkewitz Thanks for sharing the \"peek into the future\" idea. Clever thinking!\r\n\r\nMy best submission (place 7) is an ensemble of three models.\r\n\r\n**1. Regression Model**\r\n\r\nThis model performs logistic regression using XGBoost to determine the likelihood of a pairing of a checkin and a place. For each observation in the training set, 50 observations are derived. The actual place serves as the positive observation, and the 49 closest places serve as negative observations. Training observations thus grow by a factor of 50, and only 20% of the original training data was used to build this model. Gains in precission became diminishing after that.\r\n\r\nThe features of this model are based on a set of pre-computed per-place data, such as as mean and standard deviation of location and accuracy, visitor frequencies, and neighborhood sparsity. Outlier removal using Turkey's test was used for some of these statistics.\r\n\r\nIn particular, the model features include:\r\n\r\n- Basic: x, y, accuracy\r\n- Location: L1 and L2 distances with/without normalization, uni- and multi-variate normal PDFs of location and accuracy (all calculated on the observation data with regard to the pre-calculated per-place data of the place under consideration)\r\n- Frequencies: time of final visit, mean visitors on the particular day/weekday/hours, using a shifted weekday concept (+ 12 h) for places peaking across midnight\r\n- Neighborhood: L1 and L2 sparsity of the place as determined by mean distance to the next 25 neighbors\r\n- Comparisons: min/max/delta on location and various visitor frequencies of the place and its nearest neighbors.\r\n\r\nIndividual MAP@3 on the public leaderboard: 0.58452\r\n\r\nWhile this is not the best performing individual model, a strong point of this model is its high degree of generalization. Adding predictive support for a new place only requires calculating the per-place statistics for that place. This is different from the other two models which need to be partially re-trained whenever the set of places changes.\r\n\r\n\r\n**2. Classification Model**\r\n\r\nThis is a grid-based classification model using XGBoost to directly classify an observation to a place. 25 X cuts, and 100 Y cuts are used. Instead of heavy border augmentation, four instances of the model are built, each one shifted by a quarter of a grid.\r\n\r\nFeatures include:\r\n\r\n- Basic: x, y, accuracy, time\r\n- Periodic time: day time (% 1440), weekday time (% 7 * 1440)\r\n- Time factors: weekday, shifted weekday, hour of day, hour of week, month of year\r\n\r\nIndividual MAP@3 on the public leaderboard: 0.59420\r\n\r\n\r\n**3. Nearest Neighbor Model**\r\n\r\nThis is pretty much the awesome and extremely efficient model developed Sandro, Alex, Michael Hartmann, and many others. As with the classification model, four instances of the model are built, each one shifted by a quarter of a grid.\r\n\r\n\r\n**4. Ensemble**\r\n\r\nFor each model instance and test observation, the top 6 places and related probabilities are stored.\r\n\r\nEnsembling then consists of two phases. In the first phase, the model instances are aggregated per main model to derive the average probability of a place by that model; in the second phase, a weighted average of the probabilities from the three main models is calculated. The submission then consists of the three places with the highest average probability.\r\n\r\n\r\nThanks to Facebook for coming up with an awesome problem, based on a simple data set consisting of only 5 attributes, but providing significant depth. And thanks to the Kaggle community! The discussions on the forums and the activity in the scripts have really been a highlight of this competition.", "126198": "Nice solution @Jared - if I remember correctly you were the first person to mention using a classifier(xgboost) on a x-y grid early on and that seems to be the best approach by far. Thanks for the inspiration.\r\n\r\nI got lazy and just combined the probabilities from a knn and a random forest on the same grid - 40 by 40 - which was sub optimal. Could you elaborate a bit how you averaged the top 6 probabilities from the two different models?", "126200": "Post mortem: I was wondering how others did their ensembling. I tried to build model stacking using XGBoost, and failed horribly (the result was worse than individual submissions, and I couldn't figure out why). I ended up implementing a weighted voting classifier with some simple guesses for weights and a huge 24 member ensemble\r\n\r\n 3 Naive XGboost runs (only time based features) -- In range (0.580 - 0.586)\r\n 10 XGBoost with Nearest Neighbor features -- In range (0.580 - 0.59)\r\n 5 kNN (16, 19, 24, 29, 31) with Bayesian Gaussian Process optimized NN weights -- (0.575 - 0.582)\r\n 1 Abhi's fantastic public script (0.5865)\r\n 1 Michael Hartmann's fantastic public script (0.58?)\r\n 2 Sklearn Random Forest with only time features -- (0.575)\r\n 1 ExtraTrees with nearest neighbor features - (0.583)\r\n 1 Sklearn Random Forest with nearest neighbor features (0.589)\r\n\r\nI tried and excluded:\r\n\r\n - Keras based Convolutional Neural Network solution (MAP3 = 0.4)\r\n - T-SNE based clustering (tried this multiple ways) (MAP3 = 0.3)\r\n - Stacked RBM based ideas (vanishing gradient problem)\r\n - Naive Bayes (well this one is obvious, but I thought I'd give it a shot) \r\n - Sklearn ExtraTrees classifier (MAP3 = 0.52)\r\n - Simple first principles Kernel Density estimation and likelihood ratios (MAP3 = 0.52)\r\n - Kernel SVM (MAP3 = 0.42)\r\n\r\nVoting classifier weights depended on validation MAP3 score (higher score, higher weight) and a similarity metric for predictions I quickly thought up last night (more similar to other runs, lower weight).", "126207": "I had an ensemble of one RF and one XGB.\r\n\r\nMy only working trick was to multiply the predicted probability of a place_id like this:\r\n\r\npredicted_probab_place_id*(0.5+first_check_in_normalized)*(0.2+last_check_in_normalized)\r\n\r\nso if the first check in for that place_it was at timestamp 10000 and the last one at 200000:\r\n\r\nfirst_check_in_normalized = 10000/786240 = 0.012\r\n\r\nlast_check_in_normalized = 200000/786240 = 0.254\r\n\r\nso the new_predict_prob = predicted_probab_place_id*(0.5+0.012)*(0.2+0.254) = 0.232*predicted_probab_place_id\r\n\r\nthen out of the 5, return most_common(k)\r\n\r\n\r\nIn other words I penalize latecomers and early goers (place_ids in the dataset)\r\n\r\n\r\nEdit: I had a second working trick. I counted the number neighboring check ins 4 hours before and after and made a new feature, this is peeking into the future.\r\n\r\nThe other methods I used were public knowledge before the competition was over.", "126211": "[quote=Alexandru Papiu;126198]\r\n\r\nNice solution @Jared - if I remember correctly you were the first person to mention using a classifier(xgboost) on a x-y grid early on and that seems to be the best approach by far. Thanks for the inspiration.\r\n\r\nI got lazy and just combined the probabilities from a knn and a random forest on the same grid - 40 by 40 - which was sub optimal. Could you elaborate a bit how you averaged the top 6 probabilities from the two different models?\r\n\r\n[/quote]\r\n\r\nAs a particle physics grad student I regularly deal with large data that can be treated independently so the grid idea came to me pretty naturally. In physics every collision is independent, and we collect data from tons of collisions so you have to analyze it with many computers in parallel.\r\n\r\nTo average the probabilities I first stored the top 6 place_ids with their probabilities from each model. Then if the same place_id was in both models, I just took the sum from the xgb prob and the knn prob (the knn prob was weighted at 0.9). If an xgb place_id was not in the top 6 knn place_ids, then I would just divide that probability by 2. If a knn place_id was not in the top 6 xgb place ids, I would divide that probability by 3. Then I reordered the place_ids based on the new top 3 probs.", "126212": "I didn't use XGBoost because it seemed to take a lot longer than ExtraTrees/RandomForest. My entire model would run in less than a day or even half a day. So I focused on feature engineering into ET/RF. I think this was a mistake since it's probable an ensemble of varying seed XGs could probably beat my best hand crafted model of ET/RF. I guess XG is really too strong.\r\n\r\nBut I had a lot of \"interesting\" albeit not so strong ideas. Potentially any of these features could improve an XG model, but I don't know for certain. I've attached one feature that seemed to add quite a bit, but there are many others.", "126222": "Thank @Jared for creating this topic, and thank everyone here for sharing your method.\r\n\r\nMy laptop had a hardware failure in the last week and I lost two sets of predictions that I could have use for my ensemble model. But anyway, here is my model of my best submission:\r\n\r\n**Base models**\r\n\r\n1. xgboost model 1 on 25*80 grids\r\n\r\n features: x, y, accuracy, hour of a day, day of a week, week of a year, year\r\n\r\n exponentially decaying sample weights\r\n\r\n expanded grid border in training (or border augmentation)\r\n\r\n 3 grid offsets\r\n\r\n base model public LB score: 0.597 with 1 grid offset, 0.601 with 3 grid offsets combined\r\n\r\n\r\n2. xgboost model 2\r\n \r\n all the same as model 1, except that it used \"sin\", \"cos\" transforms on the day of week feature\r\n\r\n 3 grid offsets\r\n\r\n base model public LB score: 0.598 with 1 grid offset, unknown with 3 grid offsets as I initially planned to have 5 grid offsets if my laptop didn't fail me.\r\n\r\n\r\n3. KNN from the [kaggle script](https://www.kaggle.com/lscoelho/facebook-v-predicting-check-ins/modified-grid-knn/run/288176)\r\n\r\n base model public LB score: 0.583\r\n\r\n**Ensemble**\r\n\r\nSoft voting of top 10 predictions from each model.\r\n\r\nFinal score: 0.607\r\n\r\nI think the diversity between xgb model and KNN model played an important role in improving the score of the final ensemble model. I would also consider adding random forest, extra trees, and even neural networks to the final ensemble because they worked as good as KNN in my early testing of various models.\r\n\r\nAlthough with more careful parameter tuning my model could still get a better score, I don't think it would get anywhere close to 0.620, so I'm really looking forward to @tvdwiele and @Markus sharing their awesome methods.", "126229": "https://github.com/syllogismos/facebook-v-predicting-check-ins/blob/master/summary.ipynb\r\n\r\nI wrote up the summary of my submission here. \r\n\r\nWow I didn't know ensembling can improve score like that.\r\n\r\nMy best submission is a single xgboost model, a 40x200 gird size with border augmentation.. that came up with 0.58685 on pub lb. I initially saved last two days for ensembling models, but decided against it for god knows what reason.. and instead worked on doing something else. I genuinely thought I was missing something..\r\n\r\nAlong with using nthread = 4 I also trained 8 grid rows at a time parallely.. which took around 3.5 hours on c4.8xlarge spot machine to generate a submission..\r\n\r\n\r\nI did have KNN and xgboost of several grid sizes and top 10 predictions and I didn't really make use of any of them..\r\n\r\nbut I didn't even think I would beat the lb scores of people from public scripts.. but with 6 days to go, I came in top 100 which was pretty cool :P\r\nno matter what I did in the end, my score wasn't improving at all.. all my submissions are single models only. I'm pretty bummed about it now. I will try submitting my ensembles and check and weep if they are better..\r\n\r\n\r\nI was trying things like creating more features from the top 10 predictions I got from a run, features like the place cardinality, its median, its mode of time intervals and all that, but none of them gave me any gains.", "126235": "Here are the main ideas of my approach. I'll clean up the code a little and post it on Github within the next few days.\r\n\r\n**Overall structure**\r\n\r\nThe overall idea was to compute for each checkin and some candidate set of place_id's a probability proportional to\r\n\r\nP(x, y | place_id) P(day of week | place_id) P(time of day | place_id) P(accuracy | place_id) P(place_id) .\r\n\r\nChallenges were:\r\n\r\n1. Learning the conditional distributions like P(x, y | place_id) efficiently without losing too much information.\r\n\r\n2. The relative popularity of places, P(place_id), varied considerably over time. (So really it should be P(place_id, time).) How do we learn the relative popularities for the times in the test data?\r\n\r\n**1. Learning the conditional distributions efficiently**\r\n\r\nMy first attempts to learn the conditional distributions were far too slow given the size of the data.\r\n\r\nBut early in the competition @ZFTurbo's \"Mad Scripts Battle\" script did pretty well (and efficiently so!) by essentially just using a histogram to learn the joint distribution P(x, y, quarter time of day). Inspired by this, I decided to try to learn the conditional distributions using histograms. I also switched from Python to C++ to speed things up even more.\r\n\r\nTo make the histograms more accurate, I made them periodic for time of day and day of week, and I used various kernels (triangular, Gaussian, exponential) to both save and read values from the histogram - kind of like KDE but applied to a histogram of a fixed number of bins rather than all data points. This remained reasonably efficient while giving results almost as good as KDE.\r\n\r\nFor later submissions, I averaged the spatial histogram with another approach: I fitted GMM's with up to 15 components on the train data (the exact number for each place_id selected by AIC). This took pretty long and I just computed the GMM components once and loaded the results for all future runs.\r\n\r\nAlso, as observed in the forums, P(accuracy | place_id) and P(x, y | place_id) seemed to vary over time. Giving more weight to later data helped.\r\n\r\n**2. Learning from the future**\r\n\r\nUsing just P(place_id) based on the train data, the predictions were already very good: About half of the time the first prediction was correct. So we could think of the top predictions as actually labeling the test data - just very noisily - and train on the test data using the top 10-20 or so predicted place_id's as labels with their probabilities as weights.\r\n\r\nThis idea led to an approach of iterating between making predictions for the test data and learning from those predictions. The final submission used 20 iterations. (Actually, I ended up learning not only P(place_id, time) from these iterations but also kept learning the other histograms based on the test data. This helped the score and made the convergence much more stable.)\r\n\r\n(The 40-minute single-pass solution I mentioned elsewhere would just line-by-line make a prediction and then learn from it before making the next prediction.)\r\n\r\n*Possible alternative?* My original thought was actually to try to learn P(place_id, time) using likelihood maximization in some form. Conceptually, I still think this would be a nicer approach than what I did, but what I had was working well and I never got around to this. It also seemed harder to integrate learning the other histograms into this approach. I may have needed to alternate it with predict/learn iterations in any case.", "126237": "[quote=Alexandru Papiu;126198]\r\n\r\nI got lazy and just combined the probabilities from a knn and a random forest on the same grid - 40 by 40 - which was sub optimal. Could you elaborate a bit how you averaged the top 6 probabilities from the two different models?\r\n\r\n[/quote]\r\n\r\n@Alexandru Papiu, I did the same with XGB and KNN probabilities on the same grid 20x40, in which the XGB is too long so I could only finish 693 cells out of 800. \r\n\r\nSurprised for the power of the voting ensemble. @all, thanks a lot for sharing, I appreciate it.", "126242": "Congratulations everyone, and specially to the guys at the top. I just tried to use KNN model implementation publicly available in this competition and played around the feature engg and variable importance. The idea was to see how much it can score in environment similar to kaggle sandbox. I used aws m4.large and all of my model took sub-30 minutes to run. Lastly I tried ensembling of my top models which increased my score by .00020. Maybe ensembling of different types of model will result in far better result as evident from top results.\r\n\r\nTop results just confirm that XG is one of the powerful models to use in these kind of competitions. Maybe I will try ensemble this with KNN model to see how far I can go.", "126244": "Markus, Hi Markus, congratulations and thanks for sharing your model. It is interesting to see a good solution without using any of common tools of kagglers. \r\n\r\nI just have some questions about what you did with learning the conditional probabilities. \r\n\r\n1. When you train GMM models, are you training over (x,y), week of day, time of day, accuracy separately and did you use any special strategies to deal with the periodic nature of the week of day, time of day feature ?\r\n\r\n2. How did you eventually combine these conditional probability to produce the probability for each check in, was it like a weighted sum of log probability or something else ? If it is, did you find the weights varies with location, time, etc.", "126247": "Markus\r\n\r\nWow, it's amazing! Using predictions on test data as pseudo-labels for training and iterating between them is indeed a very interesting idea.", "126252": "Like many other kagglers, my final model is an ensemble model of various different models.\r\n\r\nMy single best model was an xgb model with the usual features seen in other xgb models on the forum combined with knn probability for selected place_id. judging from the local validation, it would achieve about 0.591 on LB trained over a 63 * 159 grid. For ensembling, I also trained another 3 versions of the same model with grid shifted in x, y and (x,y) by half the grid size, which would boost to about 0.597 on LB, again judging from the local validation.", "126260": "Hi @Markus,\r\n\r\nI am curious how was the performance of only P(x, y | place_id) P(place_id) ?\r\nI gave it a shot using sklearn's GMM, 6 component, and training it on the whole training set but it didn't give me good result at all (even worse than Gaussian Naive Base on only x and y.)", "126262": "Thanks to Facebook and all the competition participants. Very impressive lead by the leader board top seats, looking forward to look into their solutions.\r\n\r\n**Ensemble**\r\n\r\nMy solution is a simple ensemble of xgboost and kNN. LB results:\r\n\r\n- Single best xgb: 0.5933\r\n- 4 ensembled xgb: 0.5964\r\n- Single best kNN: 0.5838 - forked from scripts on day 0, thanks ;-)\r\n- 4 xgb + 1 kNN: 0.6017\r\n\r\n**Xgboost model details**\r\n\r\nSegmented the data into grid with 200x200m squares. The boundaries of the train squares was extended by an additional 15m. \r\n\r\n9-11 features based on:\r\n\r\n- x,y coordinates bucketing and rotation of 45 degrees\r\n- Accuracy bucketed\r\n- Time converted to year, month etc.", "126268": "My best submission was ensemble of 7 XGBoost grid based solutions and 5 best KNNs from public scripts.\r\n\r\n- XGBoost grid based solutions was calculated on different grid\r\n sizes (from 40x40 up to 100x150). They individual score was\r\n ~0.56-0.57. After ensemble gave good growth ~0.588 \r\n- Ensemble of\r\n KNNs doesn't improve score much, because solutions was too similar.\r\n- Using overall ensemble was very good because of low correlation\r\n between XGBoost and KNN solutions. It gives 0.59740 on LB\r\n\r\nCode to ensemble was pretty straightforward (applied for each ID):\r\n\r\n def process_places(arr):\r\n res = dict()\r\n for i in range(len(arr)):\r\n item = arr[i][0]\r\n if item in res:\r\n res[item] += 3.02 - i*0.0001\r\n else:\r\n res[item] = 3.02 - i*0.0001\r\n item = arr[i][1]\r\n if item in res:\r\n res[item] += 2.01 - i*0.0001\r\n else:\r\n res[item] = 2.01 - i*0.0001\r\n item = arr[i][2]\r\n if item in res:\r\n res[item] += 1 - i*0.0001\r\n else:\r\n res[item] = 1 - i*0.0001\r\n topitems = nlargest(3, sorted(res.items()), key=itemgetter(1))\r\n return topitems", "126284": "Markus Awesome approach, and congratulations on your strong finish!\r\n\r\nCan you share some insight on what makes your iterative approach converge, as opposed to going more and more off-track?", "126285": "My solution was an ensemble of 4 methods: xgboost, knn, random forests, and kernel density estimation. My xgboost score was slightly worse than Jared's, my knn score was ~0.587, my rf score was ~0.584, and my kde score was by far the best at ~0.604. I'm very happy to see that both Tom and Markus had used very different methods from what is traditionally used in Kaggle competitions as we now have new things to learn.", "126290": "The idea of my solution is similar to that of Markus.\r\n\r\n**Problem definition**\r\n\r\nFind top 3 places of following probability for candidates of each row.\r\n\r\nPr(place_id | x, y, time, accuracy)\r\n\r\n ∼ Pr(x, y, time, accuracy | place_id) * Pr(place_id)\r\n\r\n ≈ Pr(x | place_id) * Pr(y | place_id) * Pr(time of day | place_id) * Pr(week of day | place_id) * Pr(accuracy | place_id) * Pr(place_id)\r\n\r\n**Steps of my solution**\r\n\r\n1. Define the candidate set of place_ids for each cell of rough grid (100*200), which appeared at least two check-ins in the cell.\r\n2. Predict the amount of future occurrence Pr(place_id) by XGBOOST.\r\n3. Estimate marginal probability distributions of x, y, time of day, day of week, and accuracy.\r\n4. Calculate the product of 2 and 3 for all candidates.\r\n5. Elect top 3 place_ids and make submission.\r\n\r\nI spent most of time to estimate distributions, and I never tried ensemble. \r\n\r\nMy approach took about 2 hours by my 8GB RAM PC.\r\n\r\nFinally, congratulations to tvdwiele and Markus, and everyone!", "126297": "Jack (Japan), smart idea~ i am quite interested in this method. Could you share the code detail on github or somewhere else if u do not mind?\r\n\r\nthx a lot!", "126298": "I'm very impressed by the simplicity and elegance of a few of the solutions presented here, in particular @Jack. While my results aren't nearly as clean, and I scored 40th, it was a fantastic learning experience, and I'm really grateful to be able to learn from such bright people. I thought I would share my code and techniques.\r\n\r\n**Step 0:**\r\nMy first submissions were based on the random forest 0.55 starter script. Didn't end up using it because it didn't perform well enough. \r\n\r\n**Step 1:**\r\nUse a variation on the KNN script posted by @Sandro and @David, that I modified to allow for Bayesian optimization of hyperparameters. This is available [on Github here][1]. I ran 8 parallel copies of this script in Bayes mode on an 8 core machine for a couple of days. I then picked a few of the well performing sets of parameters for different kNN numbers. I ended up getting good weights for 16, 19, 24, 29, 31 neighbors, which I felt spanned the k space \"well enough\".\r\n\r\n**Step 2:** \r\nI ran XGBoost with three different sets of hyperparameters, nothing too special. This included border augmentation and different grid sizes. [Here's the script.][2]\r\n\r\nI also went down the many failed approaches avenue here. I explored the space of possible classifiers here, trying out most Sklearn classifiers (after all I was supposed to be learning things). Tried [Keras based parametric T-SNE][3] to get useful clustering, which turned out to not work very well. Oh well, live and learn.\r\n\r\nI ended up finding that Sklearn Random Forest performed comparably to XGB, and kept a couple of the predictions. \r\n\r\n**Step 3:**\r\nI added nearest neighbor features to XGBoost and other classifiers. An example script is [here][4]. At first I was unhappy because this didn't seem to perform better than plain XGBoost without nearest neighbor features. However, I found that ensembling this with other predictions improved the score substantially. Despite the individual classifier not performing better, it was actually substantially decorrelated from other classifiers, making a superior ensemble.\r\n\r\n**Step 4:**\r\nI tried a blending classifier with XGB but it didn't work and I couldn't figure out why, so I abandoned it. \r\n\r\nI was swamped with other obligations towards the end of the competition (like writing my PhD dissertation), so when Abhi's and Michael Hartmann's kNN improvements were released, I didn't really have time to work them into my solution and rerun. So I added their public scripts to my ensemble, and used a [weighted voting classifier][5] where weights depended on score and similarity to other classifiers. This voting classifier forms my final result, with 0.5995 on the public leaderboard, and 0.5993 on the private leaderboard. \r\n\r\n [1]: https://github.com/r-shekhar/Kaggle-Facebook-V/blob/master/L01_kNN_bayes.py\r\n [2]: https://github.com/r-shekhar/Kaggle-Facebook-V/blob/master/L10_naive_xgboost.py\r\n [3]: https://github.com/r-shekhar/Kaggle-Facebook-V/blob/master/failed_ideas/train_keras_tsne.py\r\n [4]: https://github.com/r-shekhar/Kaggle-Facebook-V/blob/master/L15_xgb_with_kNN_features.py\r\n [5]: https://github.com/r-shekhar/Kaggle-Facebook-V/blob/master/L20_voting_clf.py", "126299": "I'm not sure if I created a feature similar to Jack's 2), but here's the R code of one of my features.", "126301": "Jack's solution is simple and great. It looks like a Naive Bayes classifier to me, but I think he did a good job to customize the distribution and density estimation.\r\n\r\nI also tried Naive Bayes with prior class probability predicted by time models, and it scores ~0.593 which is more than my best XGBoost model that was tuned and ran for a week. I only used the standard Naive Bayes function so this makes a big difference.", "126309": "[quote=zrt;126244]\r\n\r\n@Markus, Hi Markus, congratulations and thanks for sharing your model. It is interesting to see a good solution without using any of common tools of kagglers. \r\n\r\nI just have some questions about what you did with learning the conditional probabilities. \r\n\r\n1. When you train GMM models, are you training over (x,y), week of day, time of day, accuracy separately and did you use any special strategies to deal with the periodic nature of the week of day, time of day feature ?\r\n\r\n2. How did you eventually combine these conditional probability to produce the probability for each check in, was it like a weighted sum of log probability or something else ? If it is, did you find the weights varies with location, time, etc.\r\n\r\n\r\n[/quote]\r\n\r\n1. I only trained GMM for P(x, y | place_id). I tried briefly for time of day, but it did not work well - perhaps because it did not consider periodicity, as you point out.\r\n\r\n2. When I had two methods for predicting the same probability, I just combined them using a geometric mean (i.e., arithmetic mean of the log probabilities). I just used equal weights - I think tuning weights, esp. by location, time, etc., would have taken too much effort and could be hard to do without overfitting - but maybe it would have helped.", "126316": "[quote=Andre Naef;126284]\r\n\r\n@Markus Awesome approach, and congratulations on your strong finish!\r\n\r\nCan you share some insight on what makes your iterative approach converge, as opposed to going more and more off-track?\r\n\r\n[/quote]\r\n\r\nThe analogy to tuning (learning rate, number of epochs) when using SGD is pretty good for the issues I had. If you go for too many iterations, the score starts to decrease. When this happens could be predicted reasonably well using validation.\r\n\r\nAside from using the equivalent of a conservative low \"learning rate\" in the end, these items helped stabilize the convergence:\r\n\r\n1. I predicted only on the top 25 candidates from the first pass in the iterations. (This is maybe similar to the benefit many people saw of using KNN to limit the number of candidates considered.)\r\n\r\n2. I learned relative popularity / conditional probabilities only from the top 15 predictions. The weights for learning from these predictions were the geometric average of\r\n\r\n a. the predicted probabilities\r\n\r\n b. the empirical probability of positions 1-15 being correct based on validation\r\n\r\n This average stabilized the convergence a fair amount compared to using either of the two individually.\r\n\r\n3. I cumulatively learned the conditional probabilities rather than learning them from scratch during each iteration. This stabilized convergence, but at presumably increased cost of getting stuck in some local optimum.\r\n\r\n4. Initialization: Starting from a less confident initial guess like P(place_id)^{0.3} (where P(place_id) is relative popularity based on train data) helped combat the cost of #3 to some extent.", "126317": "I also tried learning from future confident predictions but the score deteriorated. I used random forest and selected those samples for which the random forest was outputting a probability of nore than 0.8 . I then combined those test samples to train samples and applied knn to predict the leftover samples. But i think i was doing something wrong or maybe i was doing it half-heartedly.", "126332": "1. Grouped data by place_id and computed mean_x, mean_y, place kde (x, y, accuracy) and hour_kde. This was done with joblib in parallel with each place_group_df as a chunk.\r\n2. Built a KDTree with mean x, y computed in (1) to query top 100 place_id indexes for test_df. This already represented 78% accuracy amongst those filtered places.\r\n3. Divided test_df into n_jobs chunks. For each chunk computed place_kde, hour_kde values. Selected top 3 using (place_kde + hour_kde) * prob(dow) * prob(place_id)\r\n\r\nDidn't get a chance to try weighting. I wasted a lot of time trying various approaches and features thinking there must be some clever way to do this other than ensembling (since @markus was able to get 0.59 in 40 mins). All of them ended up giving more or less the same score. In retrospect, I should have ensembled them all.", "126338": "Jack(Japan) Congratulations! Your solution is very elegent and effective. Could you briefly expain how you estimate the distribution? KDE or GMM or other good tools?\r\n\r\nAlso, I am curious about how to use XGB to predict the amount of occurence in future. You must have deep insights in this problem.", "126360": "Jack (Japan)\r\n\r\n2.Predict the amount of future occurrence Pr(place_id) by XGBOOST.\r\n\r\nElegant solution indeed.\r\nWould you please elaborate on the step 2 above? Which features did you use to predict? Training dataset alone does not provide good estimate of Pr(place_id)?", "126398": "Jack (Japan)\r\nExcellent solution. I am also looking forward to hear from the detail of your solution:\r\n\r\n* How do you estimate the distribution? (@Rongcheng Lin)\r\n\r\n* How to predict the future occurence? (@data_js)", "126439": "First I would like to congratulate everyone and especially @Tom Van de Wiele and @Markus Kliegl - their results are truly impressive and I hope to learn a lot from their ideas and solutions.\r\n\r\nMy approach was somewhat similar to @Jack (Japan) and at the same time it was quite mainstream:\r\n\r\n - kde models were build from full data to predict Pr(place_id | x), Pr(place_id | hour), etc. \r\n - a 80x120 grid, samples with place_id with less then 10 occurrences was dropped out\r\n - for each cell, an xgboost model was trained on features ['x', 'y', 'x_d_y', 'x_t_y', 'hour', 'weekday', 'month', 'year', 'accuracy', 'accuracy_diff'] (where 'accuracy_diff' is difference between accuracy in current sample and rolling median accuracy in train set) to predict Pr(place_id)\r\n - a product of all these probabilities have been taken, then based on it a top3 places was selected\r\n - finally, I had a script for ensembling my submission files, based on idea that was somewhere here on forum: we take entry one by one from each file, then we summarize scores for each place_id based on its place in each prediction entry : 3 for first place, 2 for second and 1 for third. Then we re-order them according to score, save this new prediction and move to the next one. This gave me ~0.0025 comparing to my best single prediction (0.6060 LB), after combining with other 4 similar submissions. For my older models, I was able to gain ~0.007 by combining more then 10 submission files.\r\n\r\nThe sad part of all this is that my full prediction cycle took around 60 hours with 6 threads and 20-30GB of RAM needed - so I'm extremely impressed with @Jack (Japan) 2 hours performance", "126536": "Thank you for all your reply to me!\r\n\r\nIn fact, I didn't use special technique to estimate the distribution, but for x, y and accuracy, latest check-ins were weighted more.\r\n\r\nI uploaded the script of my approach [here][1] (but a little simple version), and I hope this will help you!\r\n\r\n\r\n [1]: https://www.kaggle.com/rsakata/facebook-v-predicting-check-ins/3rd-place-solution-simple-version", "126601": "I'm still working on cleaning up the many-iteration/GMM version that got to second place, but in the meanwhile here is code for a simplified single-pass solution that gets to 0.60456 (14th place) in around 90 minutes on my laptop: https://github.com/mkliegl/kaggle-Facebook-V", "127067": "Thank you all for your warm feedback so far! This has really been an extraordinarily fun competition.\r\n\r\nI've finally updated the repository https://github.com/mkliegl/kaggle-Facebook-V with two more versions.\r\n\r\nOne version scores 0.61047 without using any information about the future. (This is just the previous single-pass version but with P(x, y | place_id) averaged with probabilities from Gaussian Mixture Models).\r\n\r\nThe other version scores 0.62036 by doing 20 iterations of learning from predictions. (This is a slightly simplified and cleaned up version of my actual 2nd place submission.)\r\n\r\nThere is still a lot more documenting, code refactoring, etc. that could be done, but I think I will need to leave it at this for the moment. Please do feel free to ask questions, post suggestions or feedback, etc. here or by messaging me.", "128236": "Markus,\r\n\r\nThanks very much for sharing your code. I really appreciate your write ups! :-) I am learning a lot.\r\n\r\nJust to see what would happen, I took an ensemble of your 3 models, combined them with the [2 KDE models][1] from Qinchen, and [my own models][2].\r\n\r\nThe result is a score of: 0.62287 which, surprising to me, beats Tom's amazing score. See my screenshot in the attachment. \r\n\r\nWhat's interesting to me is that these weights are not optimized (I'm in the process now of integrating your models with my code to find the strongest soft-voting weights for an ensemble -- and I am planning to release the code in gitub soon).\r\n\r\nHere are the weights that I used:\r\n\r\n score(\"sub_20.csv\",32); # multi pass model\r\n score(\"sub_singlepass2.csv\",10); # single pass/GMM model\r\n score(\"sub_singlepass.csv\",8); # single pass model\r\n score(\"submit_kernel.csv\",8); # Qinchen's kde best model\r\n score(\"submit_kernel2.csv\",8); # Qinchen's kde alternative model\r\n score(\"xgb_feature_set_1.csv\",8); # My xgb feature set 1 model\r\n score(\"knn_feature_set_2.csv\",12); # My knn feature set 2 model \r\n score(\"rfe_feature_set_2.csv\",8); # My rfe feature set 2 model\r\n score(\"xgb_feature_set_2.csv\",8); # My xgb feature set 2 model\r\n\r\nI've added the screenshot as an attachment.\r\n\r\n\r\n\r\n\r\n\r\n [1]: https://www.kaggle.com/c/facebook-v-predicting-check-ins/forums/t/22123/6th-place-kernel-density-estimation\r\n [2]: https://www.kaggle.com/c/facebook-v-predicting-check-ins/forums/t/22086/solution-sharing-i-got-to-23-with-xgb-rf-knn", "128239": "Thank you for your interesting research Larry! I have added the zipped winning submission to the main folder of my [GitHub][1] page of the competition, it would be great if you could incorporate that into your ensembling analysis!\r\n\r\n\r\n [1]: https://github.com/ttvand/Facebook-V", "128240": "Thanks, @Tom. This is great. It will be fascinating to see how high the score goes! :-)", "128287": "Markus Kliegl\r\n\r\nThanks for sharing the code. It is very informative. \r\n\r\nI have a question regarding how you calculate histogram. In a few places (for example the lines below), you take a sample, compute its bin number, then do a rounded version of that bin number (\"get_scaled_time\"). You distribute the value between either (n, n+1) or (n-1, n). What's rationale behind it? For a straightforward histogram, if a sample falls in a bin, all its contribution is on that bin. Are you assuming some simple kernel here?\r\n\r\nhttps://github.com/mkliegl/kaggle-Facebook-V/blob/master/multipass.cc#L101-L110", "128305": "Larry Freeman - Interesting work! Since Tom's approach was quite different, I think there is potential for another big jump if you add his submission to the ensemble. I'd be curious to see if that really works out :-)\r\n\r\n@data_js - Good question. The code started as an ordinary \"histogram\" but grew into something rather different so the terminology is not really appropriate anymore. (The intuition was to smooth out the histograms to make them a little more like KDE.)\r\n\r\nReally what is happening in these \"histograms\" is something like this:\r\n\r\n- Learning from data:\r\n\r\n 1. convolve point with filter A\r\n 2. sample the result at discrete points (what were originally histogram \"bins\")\r\n\r\n- Getting a prediction for data:\r\n\r\n 1. convolve point with filter B\r\n 2. sample at the \"bins\" accordingly\r\n\r\n(Does somebody know a good technical term for this? I think these sorts of ideas are common in sampling theory, computer graphics, etc.)\r\n\r\nIn most cases, I used just a triangular filter of width equal to twice the distance between bins for filter A. (That gives rise to distributing the value between (n, n+1) or (n-1, n) as you observe.) This filter can maybe be thought of as representing the uncertainty of which \"bin\" to assign a given data point to.\r\n\r\nFor the spatial histogram, the filters could more naturally be thought of as representing the uncertainty in the recorded (x, y) values themselves and using Gaussian filters worked well.\r\n\r\n(Aside: In my actual submission, I had included a little accuracy dependence in the x standard deviation of the filters - but the signal was not very strong, so I took this out for the simplified versions on Github. @Tom Van de Wiele did a more careful analysis of accuracy vs x uncertainty in [his writeup](https://ttvand.github.io/Winning-approach-of-the-Facebook-V-Kaggle-competition/).)", "128405": "Markus Kliegl\r\n\r\nYou interpolate the samples with triangle functions and evaluate the values at the center of the bins. This is how you distribute the values between (n-1, n) or (n, n+1), depending where the sample lands. Thanks for your detailed explanation. Your code is very clean and well worth reading.", "128862": "Hi Everyone,\r\n\r\nWhen I ensemble Tom's winning csv, with Markus's csv, and a few other models, the result scores 0.62865.\r\n\r\nIt should be possible to get a higher score as we include further models. Here is the [code][1] that I used to ensemble the models.\r\n\r\nAttached is the screenshot of the score. I am still finishing my code to iterate through the possible weighted combinations. I will add it to the repository once it is completed.\r\n\r\nCheers,\r\n\r\n-Larry\r\n\r\n\r\n [1]: https://github.com/larry94555/kaggle-Facebook-V", "128969": "Thanks for your great analysis Larry! I guess we could end up with a score close to 0.63 if the models would contain more than 3 predicted places." }, "source": "meta" }