![\"three](\"https://s31.postimg.org/8v9pzp43f/2_pic_hd.jpg\")
So my method is an ensemble of three models: KNN(0.586x), RF(0.583x), XGB(0.588x). Final result is 0.600x in pulic lb and 0.5999x in private lb.
\n\nAfter getting the top 5 checkin place_id for these three models, like this:\n
Then use the following method:\n![\"enter](\"https://s31.postimg.org/czbrzbjhn/5_pic.jpg\")
The above code is basically to vote for the best 3 place_id. The 1st rank weights for 5, and the 5th rank weights only for 1. I've post similar code before at here but this is a much faster method.
\n\nThe final ensembling will be like this:\n![\"enter](\"https://s31.postimg.org/g0irgswej/6_pic_hd.jpg\")
Some other tricks I find about ensembling:
\n\nusing np.array (.values) could be much faster(4 times?), the Cython also makes it slightly faster (and simple).
I have created several xgb, rf and knn method. The one I used for each model is actually an ensembling of several same model. For example,
\n\nThe final xgb would be an ensemble of all the five models, and weight xgb_1 as 4 and the rest as 1, 0.99, 0.98, 0.97. If all weighted the same, the ensemble result will be worse.
I didn't do much tricks to deal with the parameter tuning. If any would be tuning the xgb learning_rate, and n_estimators (grid search and testing on sub set of examples). I guess there are much much better solutions out there.
\n\nI assume my method is easy enough to reproduce for those who didn't try ensemble, but have more than 3 models, so I paste it here. In fact, the final knn method is an ensembling of several different public available results. I am also more than happy for kagglers to comment on my solution and see how should I improve it.
", "messages": [ { "id": "126209", "postDate": "07/07/2016 02:09:32", "content": "So my method is an ensemble of three models: KNN(0.586x), RF(0.583x), XGB(0.588x). Final result is 0.600x in pulic lb and 0.5999x in private lb.
\n\nAfter getting the top 5 checkin place_id for these three models, like this:\n
Then use the following method:\n
The above code is basically to vote for the best 3 place_id. The 1st rank weights for 5, and the 5th rank weights only for 1. I've post similar code before at here but this is a much faster method.
\n\nThe final ensembling will be like this:\n
Some other tricks I find about ensembling:
\n\nusing np.array (.values) could be much faster(4 times?), the Cython also makes it slightly faster (and simple).
I have created several xgb, rf and knn method. The one I used for each model is actually an ensembling of several same model. For example,
\n\nThe final xgb would be an ensemble of all the five models, and weight xgb_1 as 4 and the rest as 1, 0.99, 0.98, 0.97. If all weighted the same, the ensemble result will be worse.
I didn't do much tricks to deal with the parameter tuning. If any would be tuning the xgb learning_rate, and n_estimators (grid search and testing on sub set of examples). I guess there are much much better solutions out there.
\n\nI assume my method is easy enough to reproduce for those who didn't try ensemble, but have more than 3 models, so I paste it here. In fact, the final knn method is an ensembling of several different public available results. I am also more than happy for kagglers to comment on my solution and see how should I improve it.
", "rawMarkdown": "So my method is an ensemble of three models: **KNN**(0.586x), **RF**(0.583x), **XGB**(0.588x). Final result is 0.600x in pulic lb and 0.5999x in private lb.\r\n\r\nAfter getting the top 5 checkin place_id for these three models, like this:\r\n![three predictions of xgb, rf and knn][1]\r\n\r\nThen use the following method:\r\n![enter image description here][2]\r\n\r\nThe above code is basically to vote for the best 3 place_id. The 1st rank weights for 5, and the 5th rank weights only for 1. I've post similar code before at [here][3] but this is a much faster method.\r\n\r\nThe final ensembling will be like this:\r\n![enter image description here][4]\r\n\r\nSome other tricks I find about ensembling:\r\n\r\n - using np.array (.values) could be much faster(4 times?), the Cython also makes it slightly faster (and simple).\r\n\r\n - I have created several xgb, rf and knn method. The one I used for each model is actually an ensembling of several same model. For example, \r\n - xgb_1.map3 == 0.580\r\n - xgb_2.map3 == 0.543\r\n - xgb_3.map3 == 0.542\r\n - xgb_4.map3 == 0.541\r\n - xgb_5.map3 == 0.540\r\n \r\n The final xgb would be an ensemble of all the five models, and weight xgb_1 as 4 and the rest as 1, 0.99, 0.98, 0.97. If all weighted the same, the ensemble result will be worse.\r\n\r\n\r\nI didn't do much tricks to deal with the parameter tuning. If any would be tuning the xgb learning_rate, and n_estimators (grid search and testing on sub set of examples). I guess there are much much better solutions out there.\r\n\r\nI assume my method is easy enough to reproduce for those who didn't try ensemble, but have more than 3 models, so I paste it here. In fact, the final knn method is an ensembling of several different public available results. **I am also more than happy for kagglers to comment on my solution and see how should I improve it.**\r\n\r\n\r\n [1]: https://s31.postimg.org/8v9pzp43f/2_pic_hd.jpg\r\n [2]: https://s31.postimg.org/czbrzbjhn/5_pic.jpg\r\n [3]: https://www.kaggle.com/c/facebook-v-predicting-check-ins/forums/t/21265/creating-ensemble-of-2-submissions\r\n [4]: https://s31.postimg.org/g0irgswej/6_pic_hd.jpg", "votes": null }, { "id": "126217", "postDate": "07/07/2016 02:52:14", "content": "what i did was to just apply 0.2,0.2,0.6 weights to rf,knn, xgb prediction outcome and take the top 3 out of it. i am almost sure that if i re-run rf, and xgb with different seed and ensemble them, my result will improve. however, i do not have time.
\n\ni think the ideal situation is to left out maybe 10% data in each grid to train a logistic or xgb model on top of these model result and let the model decide the weights instead of manually decide. again, this will require a lot of computing time.
\n\nthis is my two cents
", "rawMarkdown": "what i did was to just apply 0.2,0.2,0.6 weights to rf,knn, xgb prediction outcome and take the top 3 out of it. i am almost sure that if i re-run rf, and xgb with different seed and ensemble them, my result will improve. however, i do not have time.\r\n\r\ni think the ideal situation is to left out maybe 10% data in each grid to train a logistic or xgb model on top of these model result and let the model decide the weights instead of manually decide. again, this will require a lot of computing time.\r\n\r\nthis is my two cents", "votes": null }, { "id": "126224", "postDate": "07/07/2016 03:30:21", "content": "@Shubin, Thanks for commenting!
\n\nCorrect me if I am wrong. So you use the predict_proba() function, and generate a dictionary to save the probability. (I wouldn't assume you create new place_id_x columns to save each probability, because it will be a huge table.)
\n\nThe reason I do the ensembling in a voting way is because I don't know how to save the probability for later use / the second layer. Just curious about how you solve the problem.
", "rawMarkdown": "Shubin, Thanks for commenting!\r\n\r\nCorrect me if I am wrong. So you use the **predict_proba()** function, and generate a dictionary to save the probability. (I wouldn't assume you create new place_id_x columns to save each probability, because it will be a huge table.)\r\n\r\nThe reason I do the ensembling in a voting way is because I don't know how to save the probability for later use / the second layer. Just curious about how you solve the problem.", "votes": null }, { "id": "126232", "postDate": "07/07/2016 04:29:27", "content": "@YijunMa, @Shubin, thanks a lot for sharing, I appreciate it.
\n\nI was doing the same thing as Shubin, only without RF because of limited RAM (6 GB). So I could only do XGB+KNN with weights = 0.51,0.49. I wanted to play with weights but having time problem. Also until last hour I could only finish 693 cells of XGB out of 800 (20x40 grid).
\n\nMy code didn't save the probabilities (huge table), instead for each cell (x,y) it calculated:
\n\nprob(ensemble) = prob(XGB) * wXGB + prob(KNN) * wKNN \nThen, it extracted the top-3 from prob(ensemble) immediately for each cell. So, no saving for the probabilities.
\n\nBtw, what's the size of your grid?
", "rawMarkdown": "YijunMa, @Shubin, thanks a lot for sharing, I appreciate it. \r\n\r\nI was doing the same thing as Shubin, only without RF because of limited RAM (6 GB). So I could only do XGB+KNN with weights = 0.51,0.49. I wanted to play with weights but having time problem. Also until last hour I could only finish 693 cells of XGB out of 800 (20x40 grid). \r\n\r\nMy code didn't save the probabilities (huge table), instead for each cell (x,y) it calculated: \r\n\r\n prob(ensemble) = prob(XGB) * wXGB + prob(KNN) * wKNN \r\n\r\nThen, it extracted the top-3 from prob(ensemble) immediately for each cell. So, no saving for the probabilities. \r\n\r\nBtw, what's the size of your grid?", "votes": null }, { "id": "126267", "postDate": "07/07/2016 07:56:20", "content": "I was doing something similar too! My ensemble contained 7 models (3 x XGB, 3 x KNN, 1 x RF); however, at the end I found that in my case just using the same weights worked better than anything else. Looking at the Private LB, it seems that this assumption was good :)
\n\nRegarding grid sizes, I used several across the models (the smallest one was 100 x 200).
\n\nAnd before anyone asks: Yes all of this was done in a laptop (16GB). The most time consuming models took about two days for training, but it is still doable. No need for AWS or similar, in my opinion.
", "rawMarkdown": "I was doing something similar too! My ensemble contained 7 models (3 x XGB, 3 x KNN, 1 x RF); however, at the end I found that in my case just using the same weights worked better than anything else. Looking at the Private LB, it seems that this assumption was good :)\r\n\r\nRegarding grid sizes, I used several across the models (the smallest one was 100 x 200).\r\n\r\nAnd before anyone asks: Yes all of this was done in a laptop (16GB). The most time consuming models took about two days for training, but it is still doable. No need for AWS or similar, in my opinion.", "votes": null }, { "id": "126387", "postDate": "07/08/2016 03:58:27", "content": "@aldente, I find Li Li's solution is a better way for the probability huge table problem. Just use a table with row_id, place_id, and probability to save the top10 predictions is enough. Might spend some time on coding, but shouldn't be a hard work. What's good is that it could be saved for later use.
\n\nI tried the same as what you did, but what's weird is that the result is worse. I would assume it's because I didn't use sum(log(prob)).
\n\nI didn't remember the exact size, should be (100 * 100 ~ 50 * 100). My exploration told me the size influence the running time more than the final result after extending the border. If anything could improve more would be shifting the grid to a quarter place.
\n\n@ CarrDelling. Interesting that same weights works better, maybe it's because we are using different predictions. What are the map@3 for each of your model? If they are close to each other, same weights make sense then.
", "rawMarkdown": "aldente, I find [Li Li][1]'s solution is a better way for the probability huge table problem. Just use a table with row_id, place_id, and probability to save the top10 predictions is enough. Might spend some time on coding, but shouldn't be a hard work. What's good is that it could be saved for later use.\r\n\r\nI tried the same as what you did, but what's weird is that the result is worse. I would assume it's because I didn't use **sum(log(prob))**.\r\n\r\nI didn't remember the exact size, should be (100 * 100 ~ 50 * 100). My exploration told me the size influence the running time more than the final result after extending the border. If anything could improve more would be shifting the grid to a quarter place.\r\n\r\n@ CarrDelling. Interesting that same weights works better, maybe it's because we are using different predictions. What are the map@3 for each of your model? If they are close to each other, same weights make sense then.\r\n\r\n [1]: https://www.kaggle.com/c/facebook-v-predicting-check-ins/forums/t/22112/5th-place-solution", "votes": null }, { "id": "126390", "postDate": "07/08/2016 05:00:17", "content": "@YijunMa, I am glad you like my post. The idea to store (row_id, place_id, proba) is just like to store non zero elements in a sparse matrix. Just for your preparation of Facebook interview, how to calculate the inner product of two sparse vector? :-)
", "rawMarkdown": "YijunMa, I am glad you like my post. The idea to store (row_id, place_id, proba) is just like to store non zero elements in a sparse matrix. Just for your preparation of Facebook interview, how to calculate the inner product of two sparse vector? :-)", "votes": null }, { "id": "126395", "postDate": "07/08/2016 05:45:18", "content": "@LiLi, ToT... Sadly, I have to google...
", "rawMarkdown": "LiLi, ToT... Sadly, I have to google...", "votes": null }, { "id": "126409", "postDate": "07/08/2016 08:08:21", "content": "[quote=YijunMa;126387]\n@ CarrDelling. Interesting that same weights works better, maybe it's because we are using different predictions. What are the map@3 for each of your model? If they are close to each other, same weights make sense then.\n[/quote]
\n\nWell, you can always say that I didn't follow an exhaustive method , and maybe there are other combinations that would had improved my score further. But the sad truth is that nothing of what I tried improved the score of just using the same weights.
\n\nLB map@3 are:
\n\nXGB1: 0.58925
\n\nXGB2: 0.58840
\n\nXGB3: 0.58592
\n\nKNN1: 0.58313
\n\nKNN2: 0.58515
\n\nKNN3: 0.58401
\n\nRF1: 0.57938
\n\nAll-methods: 0.60343
\n\nPS: Just in case, by same weights I mean that I used the weights 0.5, 0.3, 0.2, 0.15, 0.1 (I was computing the top 5 places, not top 3) for all the models. I also discounted a very small delta for the KNNs and RFs, in case of ties.
", "rawMarkdown": "[quote=YijunMa;126387]\r\n@ CarrDelling. Interesting that same weights works better, maybe it's because we are using different predictions. What are the map@3 for each of your model? If they are close to each other, same weights make sense then.\r\n[/quote]\r\n\r\nWell, you can always say that I didn't follow an exhaustive method , *and maybe there are other combinations that would had improved my score further*. But the sad truth is that nothing of what I tried improved the score of just using the same weights.\r\n\r\nLB map@3 are:\r\n\r\nXGB1: 0.58925\r\n\r\nXGB2: 0.58840\r\n\r\nXGB3: 0.58592\r\n\r\nKNN1: 0.58313\r\n\r\nKNN2: 0.58515\r\n\r\nKNN3: 0.58401\r\n\r\nRF1: 0.57938\r\n\r\nAll-methods: 0.60343\r\n\r\nPS: Just in case, by same weights I mean that I used the weights 0.5, 0.3, 0.2, 0.15, 0.1 (I was computing the top 5 places, not top 3) for all the models. I also discounted a very small delta for the KNNs and RFs, in case of ties.", "votes": null }, { "id": "126423", "postDate": "07/08/2016 09:22:18", "content": "@YijunMa, you're right about Li Li's solution, I did underestimate the power of voting/soft-voting ensemble, I didn't realize it until the end. In previous (smaller) competitions I never got it to work. Lesson learned ;)
\n\nThanks again to all for sharing, you're all very kind, +1 to all, good luck with the job interview :)
", "rawMarkdown": "YijunMa, you're right about Li Li's solution, I did underestimate the power of voting/soft-voting ensemble, I didn't realize it until the end. In previous (smaller) competitions I never got it to work. Lesson learned ;) \r\n\r\nThanks again to all for sharing, you're all very kind, +1 to all, good luck with the job interview :)", "votes": null }, { "id": "126466", "postDate": "07/08/2016 16:25:37", "content": "@CarrDelling, Your map@3 score for each model is really close compare to mine. Mine is:
\n\nXGB1: 0.54x
\n\nXGB2: 0.54x
\n\nXGB3: 0.54x
\n\nXGB4: 0.54x (all the above cost ~12 hours to run)
\n\nXGB5: 0.58x (2 days to run)
\n\nWeighting them the same makes the result even worse. However, if I set the best one weight to be 4, and all the rest to be 1~0.97. This means only if all the weak XGB1~4 commonly agree on a answer down voted by XGB5, the ensembling could change the order.
\n\nIn retrospect, I should have train more robust XGBs and give up all those weak ones. But anyway, I still happy to have this kind of experience to explore ensembling.
", "rawMarkdown": "CarrDelling, Your map@3 score for each model is really close compare to mine. Mine is:\r\n\r\nXGB1: 0.54x\r\n\r\nXGB2: 0.54x\r\n\r\nXGB3: 0.54x\r\n\r\nXGB4: 0.54x (all the above cost ~12 hours to run)\r\n\r\nXGB5: 0.58x (2 days to run)\r\n\r\nWeighting them the same makes the result even worse. However, if I set the best one weight to be 4, and all the rest to be 1~0.97. This means only if all the weak XGB1~4 commonly agree on a answer down voted by XGB5, the ensembling could change the order.\r\n\r\nIn retrospect, I should have train more robust XGBs and give up all those weak ones. But anyway, I still happy to have this kind of experience to explore ensembling.", "votes": null } ], "comments": [ { "id": 126217, "author_name": "lishubin", "author_url": "", "post_date": "07/07/2016 02:52:14", "content": "what i did was to just apply 0.2,0.2,0.6 weights to rf,knn, xgb prediction outcome and take the top 3 out of it. i am almost sure that if i re-run rf, and xgb with different seed and ensemble them, my result will improve. however, i do not have time.
\n\ni think the ideal situation is to left out maybe 10% data in each grid to train a logistic or xgb model on top of these model result and let the model decide the weights instead of manually decide. again, this will require a lot of computing time.
\n\nthis is my two cents
", "votes": null, "replies": [] }, { "id": 126224, "author_name": "ma350365879", "author_url": "", "post_date": "07/07/2016 03:30:21", "content": "@Shubin, Thanks for commenting!
\n\nCorrect me if I am wrong. So you use the predict_proba() function, and generate a dictionary to save the probability. (I wouldn't assume you create new place_id_x columns to save each probability, because it will be a huge table.)
\n\nThe reason I do the ensembling in a voting way is because I don't know how to save the probability for later use / the second layer. Just curious about how you solve the problem.
", "votes": null, "replies": [] }, { "id": 126232, "author_name": "aldente", "author_url": "", "post_date": "07/07/2016 04:29:27", "content": "@YijunMa, @Shubin, thanks a lot for sharing, I appreciate it.
\n\nI was doing the same thing as Shubin, only without RF because of limited RAM (6 GB). So I could only do XGB+KNN with weights = 0.51,0.49. I wanted to play with weights but having time problem. Also until last hour I could only finish 693 cells of XGB out of 800 (20x40 grid).
\n\nMy code didn't save the probabilities (huge table), instead for each cell (x,y) it calculated:
\n\nprob(ensemble) = prob(XGB) * wXGB + prob(KNN) * wKNN \nThen, it extracted the top-3 from prob(ensemble) immediately for each cell. So, no saving for the probabilities.
\n\nBtw, what's the size of your grid?
", "votes": null, "replies": [] }, { "id": 126267, "author_name": "carrdelling", "author_url": "", "post_date": "07/07/2016 07:56:20", "content": "I was doing something similar too! My ensemble contained 7 models (3 x XGB, 3 x KNN, 1 x RF); however, at the end I found that in my case just using the same weights worked better than anything else. Looking at the Private LB, it seems that this assumption was good :)
\n\nRegarding grid sizes, I used several across the models (the smallest one was 100 x 200).
\n\nAnd before anyone asks: Yes all of this was done in a laptop (16GB). The most time consuming models took about two days for training, but it is still doable. No need for AWS or similar, in my opinion.
", "votes": null, "replies": [] }, { "id": 126387, "author_name": "ma350365879", "author_url": "", "post_date": "07/08/2016 03:58:27", "content": "@aldente, I find Li Li's solution is a better way for the probability huge table problem. Just use a table with row_id, place_id, and probability to save the top10 predictions is enough. Might spend some time on coding, but shouldn't be a hard work. What's good is that it could be saved for later use.
\n\nI tried the same as what you did, but what's weird is that the result is worse. I would assume it's because I didn't use sum(log(prob)).
\n\nI didn't remember the exact size, should be (100 * 100 ~ 50 * 100). My exploration told me the size influence the running time more than the final result after extending the border. If anything could improve more would be shifting the grid to a quarter place.
\n\n@ CarrDelling. Interesting that same weights works better, maybe it's because we are using different predictions. What are the map@3 for each of your model? If they are close to each other, same weights make sense then.
", "votes": null, "replies": [] }, { "id": 126390, "author_name": "aikinogard", "author_url": "", "post_date": "07/08/2016 05:00:17", "content": "@YijunMa, I am glad you like my post. The idea to store (row_id, place_id, proba) is just like to store non zero elements in a sparse matrix. Just for your preparation of Facebook interview, how to calculate the inner product of two sparse vector? :-)
", "votes": null, "replies": [] }, { "id": 126395, "author_name": "ma350365879", "author_url": "", "post_date": "07/08/2016 05:45:18", "content": "@LiLi, ToT... Sadly, I have to google...
", "votes": null, "replies": [] }, { "id": 126409, "author_name": "carrdelling", "author_url": "", "post_date": "07/08/2016 08:08:21", "content": "[quote=YijunMa;126387]\n@ CarrDelling. Interesting that same weights works better, maybe it's because we are using different predictions. What are the map@3 for each of your model? If they are close to each other, same weights make sense then.\n[/quote]
\n\nWell, you can always say that I didn't follow an exhaustive method , and maybe there are other combinations that would had improved my score further. But the sad truth is that nothing of what I tried improved the score of just using the same weights.
\n\nLB map@3 are:
\n\nXGB1: 0.58925
\n\nXGB2: 0.58840
\n\nXGB3: 0.58592
\n\nKNN1: 0.58313
\n\nKNN2: 0.58515
\n\nKNN3: 0.58401
\n\nRF1: 0.57938
\n\nAll-methods: 0.60343
\n\nPS: Just in case, by same weights I mean that I used the weights 0.5, 0.3, 0.2, 0.15, 0.1 (I was computing the top 5 places, not top 3) for all the models. I also discounted a very small delta for the KNNs and RFs, in case of ties.
", "votes": null, "replies": [] }, { "id": 126423, "author_name": "aldente", "author_url": "", "post_date": "07/08/2016 09:22:18", "content": "@YijunMa, you're right about Li Li's solution, I did underestimate the power of voting/soft-voting ensemble, I didn't realize it until the end. In previous (smaller) competitions I never got it to work. Lesson learned ;)
\n\nThanks again to all for sharing, you're all very kind, +1 to all, good luck with the job interview :)
", "votes": null, "replies": [] }, { "id": 126466, "author_name": "ma350365879", "author_url": "", "post_date": "07/08/2016 16:25:37", "content": "@CarrDelling, Your map@3 score for each model is really close compare to mine. Mine is:
\n\nXGB1: 0.54x
\n\nXGB2: 0.54x
\n\nXGB3: 0.54x
\n\nXGB4: 0.54x (all the above cost ~12 hours to run)
\n\nXGB5: 0.58x (2 days to run)
\n\nWeighting them the same makes the result even worse. However, if I set the best one weight to be 4, and all the rest to be 1~0.97. This means only if all the weak XGB1~4 commonly agree on a answer down voted by XGB5, the ensembling could change the order.
\n\nIn retrospect, I should have train more robust XGBs and give up all those weak ones. But anyway, I still happy to have this kind of experience to explore ensembling.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "126209": "So my method is an ensemble of three models: **KNN**(0.586x), **RF**(0.583x), **XGB**(0.588x). Final result is 0.600x in pulic lb and 0.5999x in private lb.\r\n\r\nAfter getting the top 5 checkin place_id for these three models, like this:\r\n![three predictions of xgb, rf and knn][1]\r\n\r\nThen use the following method:\r\n![enter image description here][2]\r\n\r\nThe above code is basically to vote for the best 3 place_id. The 1st rank weights for 5, and the 5th rank weights only for 1. I've post similar code before at [here][3] but this is a much faster method.\r\n\r\nThe final ensembling will be like this:\r\n![enter image description here][4]\r\n\r\nSome other tricks I find about ensembling:\r\n\r\n - using np.array (.values) could be much faster(4 times?), the Cython also makes it slightly faster (and simple).\r\n\r\n - I have created several xgb, rf and knn method. The one I used for each model is actually an ensembling of several same model. For example, \r\n - xgb_1.map3 == 0.580\r\n - xgb_2.map3 == 0.543\r\n - xgb_3.map3 == 0.542\r\n - xgb_4.map3 == 0.541\r\n - xgb_5.map3 == 0.540\r\n \r\n The final xgb would be an ensemble of all the five models, and weight xgb_1 as 4 and the rest as 1, 0.99, 0.98, 0.97. If all weighted the same, the ensemble result will be worse.\r\n\r\n\r\nI didn't do much tricks to deal with the parameter tuning. If any would be tuning the xgb learning_rate, and n_estimators (grid search and testing on sub set of examples). I guess there are much much better solutions out there.\r\n\r\nI assume my method is easy enough to reproduce for those who didn't try ensemble, but have more than 3 models, so I paste it here. In fact, the final knn method is an ensembling of several different public available results. **I am also more than happy for kagglers to comment on my solution and see how should I improve it.**\r\n\r\n\r\n [1]: https://s31.postimg.org/8v9pzp43f/2_pic_hd.jpg\r\n [2]: https://s31.postimg.org/czbrzbjhn/5_pic.jpg\r\n [3]: https://www.kaggle.com/c/facebook-v-predicting-check-ins/forums/t/21265/creating-ensemble-of-2-submissions\r\n [4]: https://s31.postimg.org/g0irgswej/6_pic_hd.jpg", "126217": "what i did was to just apply 0.2,0.2,0.6 weights to rf,knn, xgb prediction outcome and take the top 3 out of it. i am almost sure that if i re-run rf, and xgb with different seed and ensemble them, my result will improve. however, i do not have time.\r\n\r\ni think the ideal situation is to left out maybe 10% data in each grid to train a logistic or xgb model on top of these model result and let the model decide the weights instead of manually decide. again, this will require a lot of computing time.\r\n\r\nthis is my two cents", "126224": "Shubin, Thanks for commenting!\r\n\r\nCorrect me if I am wrong. So you use the **predict_proba()** function, and generate a dictionary to save the probability. (I wouldn't assume you create new place_id_x columns to save each probability, because it will be a huge table.)\r\n\r\nThe reason I do the ensembling in a voting way is because I don't know how to save the probability for later use / the second layer. Just curious about how you solve the problem.", "126232": "YijunMa, @Shubin, thanks a lot for sharing, I appreciate it. \r\n\r\nI was doing the same thing as Shubin, only without RF because of limited RAM (6 GB). So I could only do XGB+KNN with weights = 0.51,0.49. I wanted to play with weights but having time problem. Also until last hour I could only finish 693 cells of XGB out of 800 (20x40 grid). \r\n\r\nMy code didn't save the probabilities (huge table), instead for each cell (x,y) it calculated: \r\n\r\n prob(ensemble) = prob(XGB) * wXGB + prob(KNN) * wKNN \r\n\r\nThen, it extracted the top-3 from prob(ensemble) immediately for each cell. So, no saving for the probabilities. \r\n\r\nBtw, what's the size of your grid?", "126267": "I was doing something similar too! My ensemble contained 7 models (3 x XGB, 3 x KNN, 1 x RF); however, at the end I found that in my case just using the same weights worked better than anything else. Looking at the Private LB, it seems that this assumption was good :)\r\n\r\nRegarding grid sizes, I used several across the models (the smallest one was 100 x 200).\r\n\r\nAnd before anyone asks: Yes all of this was done in a laptop (16GB). The most time consuming models took about two days for training, but it is still doable. No need for AWS or similar, in my opinion.", "126387": "aldente, I find [Li Li][1]'s solution is a better way for the probability huge table problem. Just use a table with row_id, place_id, and probability to save the top10 predictions is enough. Might spend some time on coding, but shouldn't be a hard work. What's good is that it could be saved for later use.\r\n\r\nI tried the same as what you did, but what's weird is that the result is worse. I would assume it's because I didn't use **sum(log(prob))**.\r\n\r\nI didn't remember the exact size, should be (100 * 100 ~ 50 * 100). My exploration told me the size influence the running time more than the final result after extending the border. If anything could improve more would be shifting the grid to a quarter place.\r\n\r\n@ CarrDelling. Interesting that same weights works better, maybe it's because we are using different predictions. What are the map@3 for each of your model? If they are close to each other, same weights make sense then.\r\n\r\n [1]: https://www.kaggle.com/c/facebook-v-predicting-check-ins/forums/t/22112/5th-place-solution", "126390": "YijunMa, I am glad you like my post. The idea to store (row_id, place_id, proba) is just like to store non zero elements in a sparse matrix. Just for your preparation of Facebook interview, how to calculate the inner product of two sparse vector? :-)", "126395": "LiLi, ToT... Sadly, I have to google...", "126409": "[quote=YijunMa;126387]\r\n@ CarrDelling. Interesting that same weights works better, maybe it's because we are using different predictions. What are the map@3 for each of your model? If they are close to each other, same weights make sense then.\r\n[/quote]\r\n\r\nWell, you can always say that I didn't follow an exhaustive method , *and maybe there are other combinations that would had improved my score further*. But the sad truth is that nothing of what I tried improved the score of just using the same weights.\r\n\r\nLB map@3 are:\r\n\r\nXGB1: 0.58925\r\n\r\nXGB2: 0.58840\r\n\r\nXGB3: 0.58592\r\n\r\nKNN1: 0.58313\r\n\r\nKNN2: 0.58515\r\n\r\nKNN3: 0.58401\r\n\r\nRF1: 0.57938\r\n\r\nAll-methods: 0.60343\r\n\r\nPS: Just in case, by same weights I mean that I used the weights 0.5, 0.3, 0.2, 0.15, 0.1 (I was computing the top 5 places, not top 3) for all the models. I also discounted a very small delta for the KNNs and RFs, in case of ties.", "126423": "YijunMa, you're right about Li Li's solution, I did underestimate the power of voting/soft-voting ensemble, I didn't realize it until the end. In previous (smaller) competitions I never got it to work. Lesson learned ;) \r\n\r\nThanks again to all for sharing, you're all very kind, +1 to all, good luck with the job interview :)", "126466": "CarrDelling, Your map@3 score for each model is really close compare to mine. Mine is:\r\n\r\nXGB1: 0.54x\r\n\r\nXGB2: 0.54x\r\n\r\nXGB3: 0.54x\r\n\r\nXGB4: 0.54x (all the above cost ~12 hours to run)\r\n\r\nXGB5: 0.58x (2 days to run)\r\n\r\nWeighting them the same makes the result even worse. However, if I set the best one weight to be 4, and all the rest to be 1~0.97. This means only if all the weak XGB1~4 commonly agree on a answer down voted by XGB5, the ensembling could change the order.\r\n\r\nIn retrospect, I should have train more robust XGBs and give up all those weak ones. But anyway, I still happy to have this kind of experience to explore ensembling." }, "source": "meta" }