The attached code takes two submissions and creates a third submission that is basically a weighted average of the first two. There probably was a more efficient and elegant way to do this, but this is what I could think of quickly.
\n\nTo use it, you will of course need to change the directory name and input submission file names. The weights of the two submissions and of the three places are also configurable. If you see any bugs in the code, please let me know.
\n\nAlso, if you have any methods for creating ensembles of submissions, please feel free to share.
\n\nI tried combining my best solution with the best public script solution, and did not see a score improvement, but maybe you will have more luck!
", "messages": [ { "id": "121562", "postDate": "05/27/2016 12:01:14", "content": "The attached code takes two submissions and creates a third submission that is basically a weighted average of the first two. There probably was a more efficient and elegant way to do this, but this is what I could think of quickly.
\n\nTo use it, you will of course need to change the directory name and input submission file names. The weights of the two submissions and of the three places are also configurable. If you see any bugs in the code, please let me know.
\n\nAlso, if you have any methods for creating ensembles of submissions, please feel free to share.
\n\nI tried combining my best solution with the best public script solution, and did not see a score improvement, but maybe you will have more luck!
", "rawMarkdown": "The attached code takes two submissions and creates a third submission that is basically a weighted average of the first two. There probably was a more efficient and elegant way to do this, but this is what I could think of quickly. \r\n\r\nTo use it, you will of course need to change the directory name and input submission file names. The weights of the two submissions and of the three places are also configurable. If you see any bugs in the code, please let me know. \r\n\r\nAlso, if you have any methods for creating ensembles of submissions, please feel free to share. \r\n\r\nI tried combining my best solution with the best public script solution, and did not see a score improvement, but maybe you will have more luck!", "votes": null }, { "id": "121570", "postDate": "05/27/2016 14:29:14", "content": "ensemble works for me! add from 0.55 to about 0.57.
\n\n\n", "rawMarkdown": "ensemble works for me! add from 0.55 to about 0.57.\r\n\r\n> \r\n> def ensemble(x):\r\n> # each place_id is a 5 top place_id guess\r\n> # yes, I am using map@5 instead of map@3.\r\n> place_id_1 = x['place_id_1'].split()\r\n> place_id_2 = x['place_id_2'].split()\r\n> place_id_3 = x['place_id_3'].split()\r\n> place_id_4 = x['place_id_4'].split()\r\n> place_id_5 = x['place_id_5'].split()\r\n\r\n> # place_id_1 has the best prediction, place_id_5 has the worst prediction\r\n> output = [(place_id_1[i]+\" \")*(6-i) for i in range(len(place_id_1))] \r\n> output += [(place_id_2[i]+\" \")*(5-i) for i in range(len(place_id_2))]\r\n> output += [(place_id_3[i]+\" \")*(5-i) for i in range(len(place_id_3))]\r\n> output += [(place_id_4[i]+\" \")*(5-i) for i in range(len(place_id_4))]\r\n> output += [(place_id_5[i]+\" \")*(4-i) for i in range(len(place_id_5))]\r\n> output = \"\".join(output).split()\r\n\r\n> m = Counter(output) # this Counter method is to count word frequency.\r\n> return [x[0] for x in m.most_common(3)]\r\n\r\n> submission['place_id'] = submission.apply(lambda x: ensemble(x), axis=1)", "votes": null }, { "id": "121574", "postDate": "05/27/2016 14:52:18", "content": "\ndef ensemble(x):\n # each place_id is a 5 top place_id guess\n # yes, I am using map@5 instead of map@3.\n place_id_1 = x['place_id_1'].split()\n place_id_2 = x['place_id_2'].split()\n place_id_3 = x['place_id_3'].split()\n place_id_4 = x['place_id_4'].split()\n place_id_5 = x['place_id_5'].split()\n\n # place_id_1 has the best prediction, place_id_5 has the worst prediction\n output = [(place_id_1[i]+" ")*(6-i) for i in range(len(place_id_1))] \n output += [(place_id_2[i]+" ")*(5-i) for i in range(len(place_id_2))]\n output += [(place_id_3[i]+" ")*(5-i) for i in range(len(place_id_3))]\n output += [(place_id_4[i]+" ")*(5-i) for i in range(len(place_id_4))]\n output += [(place_id_5[i]+" ")*(4-i) for i in range(len(place_id_5))]\n output = "".join(output).split()\n\n m = Counter(output) # this Counter method is to count word frequency.\n return [x[0] for x in m.most_common(3)]\n\n submission['place_id'] = submission.apply(lambda x: ensemble(x), axis=1)\n
@YijunMa, do you have 5 different models? If so, just curious, how long does it take to make a full prediction?
", "rawMarkdown": "YijunMa, do you have 5 different models? If so, just curious, how long does it take to make a full prediction?", "votes": null }, { "id": "121575", "postDate": "05/27/2016 15:00:20", "content": "@Mikhail Pershin, all these are random forest run by small chunks (around 100*100). Roughly about 18 hours or more for each model.
", "rawMarkdown": "Mikhail Pershin, all these are random forest run by small chunks (around 100*100). Roughly about 18 hours or more for each model.", "votes": null }, { "id": "121577", "postDate": "05/27/2016 15:10:19", "content": "I was thinking of an ensemble method, but instead of taking 2 submission files as an input, like so:
\n\n1, A B C\n2, E F G\n3, H I J\nIt will take 2 files with the probability information kept, and maybe with more answers per entry, like so:
\n\n1, (A,0.6) (B,0.1),(C,0.05),(D,0.04),(E,0.02)\n2, (D,0.5) (E,0.2),(F,0.09),(G,0.06),(H,0.01)\n3, (H,0.7) (I,0.1),(J,0.07),(K,0.02),(L,0.01)\nI am not sure how to implement it, or if it will give any improvements over a simple ensemble of 3 ranked answers.
", "rawMarkdown": "I was thinking of an ensemble method, but instead of taking 2 submission files as an input, like so:\r\n\r\n 1, A B C\r\n 2, E F G\r\n 3, H I J\r\n\r\nIt will take 2 files with the probability information kept, and maybe with more answers per entry, like so:\r\n\r\n 1, (A,0.6) (B,0.1),(C,0.05),(D,0.04),(E,0.02)\r\n 2, (D,0.5) (E,0.2),(F,0.09),(G,0.06),(H,0.01)\r\n 3, (H,0.7) (I,0.1),(J,0.07),(K,0.02),(L,0.01)\r\n\r\n\r\nI am not sure how to implement it, or if it will give any improvements over a simple ensemble of 3 ranked answers.", "votes": null }, { "id": "121580", "postDate": "05/27/2016 15:20:15", "content": "@YijunMa, thereby its more than 90 hours in total! You are really patient! Though improvement from 0.55 to 0.57 is quite significant, so I guess it’s worth it :)
", "rawMarkdown": "YijunMa, thereby its more than 90 hours in total! You are really patient! Though improvement from 0.55 to 0.57 is quite significant, so I guess it’s worth it :)", "votes": null }, { "id": "121581", "postDate": "05/27/2016 15:31:33", "content": "@Radu Stoicescu, probability information could have different scale and thereby a different meaning if you are aimed to use estimators of different types (for example knn vs random forest) - so be careful with it ;)
", "rawMarkdown": "Radu Stoicescu, probability information could have different scale and thereby a different meaning if you are aimed to use estimators of different types (for example knn vs random forest) - so be careful with it ;)", "votes": null }, { "id": "121582", "postDate": "05/27/2016 15:38:30", "content": "@Mikhail Pershin, thx! I am almost sure it should works before I run it. And there are small tricks to makes it 2~5 times faster, depends on the memory available on my computer.
\n\n@Radu Stoicescu. This should work even better than my method in theory.
", "rawMarkdown": "Mikhail Pershin, thx! I am almost sure it should works before I run it. And there are small tricks to makes it 2~5 times faster, depends on the memory available on my computer.\r\n\r\n@Radu Stoicescu. This should work even better than my method in theory.", "votes": null }, { "id": "121600", "postDate": "05/27/2016 18:07:38", "content": "I've tried with three different methods to combine submissions:
\n\nFor example if a given row in two submissions was:
\n\n1, 123 345 567\nand
\n\n1, 987 766 523\nI would combine this into something like:
\n\n1, 123 987 345.\nUnfortunately none of the above produced any improvement for me, but I have only tried it on my local cross validation test.
", "rawMarkdown": "I've tried with three different methods to combine submissions:\r\n\r\n - Voting with equal weights,\r\n - Voting with confidence based weights (similarly to what was proposed by Radu Stoicescu),\r\n - Taking the first answer of the \"better\" method and using the first answer of the second best method as the second proposal.\r\n\r\nFor example if a given row in two submissions was:\r\n\r\n 1, 123 345 567\r\n\r\nand\r\n\r\n 1, 987 766 523\r\n\r\nI would combine this into something like:\r\n\r\n 1, 123 987 345.\r\n\r\nUnfortunately none of the above produced any improvement for me, but I have only tried it on my local cross validation test.", "votes": null }, { "id": "125172", "postDate": "06/27/2016 05:07:08", "content": "@YijunMa That's a great improvement. Are those two different kind of models or simply the same model with different grids?
", "rawMarkdown": "YijunMa That's a great improvement. Are those two different kind of models or simply the same model with different grids?", "votes": null }, { "id": "125201", "postDate": "06/27/2016 15:23:38", "content": "@Juahuan He, my most "insane" ensembling is to combine 10 different models' predictions, so both situation occurs. Have to mention that this code is computationally inefficient, will be super slow if used for ensembling 10 models' result. Better to adjust the code little bit.
", "rawMarkdown": "Juahuan He, my most \"insane\" ensembling is to combine 10 different models' predictions, so both situation occurs. Have to mention that this code is computationally inefficient, will be super slow if used for ensembling 10 models' result. Better to adjust the code little bit.", "votes": null }, { "id": "125218", "postDate": "06/27/2016 17:46:31", "content": "", "rawMarkdown": "", "votes": null }, { "id": "125249", "postDate": "06/28/2016 04:08:58", "content": "[quote=YijunMa;125201]
\n\n@Juahuan He, my most "insane" ensembling is to combine 10 different models' predictions, so both situation occurs. Have to mention that this code is computationally inefficient, will be super slow if used for ensembling 10 models' result. Better to adjust the code little bit.
\n\n[/quote]\nI am curious about the ensemble that you mentioned in the previous post which brought you from 0.55 to 0.57 on LB. Are those different models(like one knn and one XGB), or just different grids?
", "rawMarkdown": "[quote=YijunMa;125201]\r\n\r\n@Juahuan He, my most \"insane\" ensembling is to combine 10 different models' predictions, so both situation occurs. Have to mention that this code is computationally inefficient, will be super slow if used for ensembling 10 models' result. Better to adjust the code little bit.\r\n\r\n[/quote]\r\nI am curious about the ensemble that you mentioned in the previous post which brought you from 0.55 to 0.57 on LB. Are those different models(like one knn and one XGB), or just different grids?", "votes": null }, { "id": "125290", "postDate": "06/28/2016 13:12:23", "content": "@Jahuan He, both. T0 be honest, this is my first time to try ensembling, so I don't have any theory to guide me.
", "rawMarkdown": "Jahuan He, both. T0 be honest, this is my first time to try ensembling, so I don't have any theory to guide me.", "votes": null } ], "comments": [ { "id": 121570, "author_name": "ma350365879", "author_url": "", "post_date": "05/27/2016 14:29:14", "content": "ensemble works for me! add from 0.55 to about 0.57.
\n\n\n", "votes": null, "replies": [] }, { "id": 121574, "author_name": "pershinmr", "author_url": "", "post_date": "05/27/2016 14:52:18", "content": "\ndef ensemble(x):\n # each place_id is a 5 top place_id guess\n # yes, I am using map@5 instead of map@3.\n place_id_1 = x['place_id_1'].split()\n place_id_2 = x['place_id_2'].split()\n place_id_3 = x['place_id_3'].split()\n place_id_4 = x['place_id_4'].split()\n place_id_5 = x['place_id_5'].split()\n\n # place_id_1 has the best prediction, place_id_5 has the worst prediction\n output = [(place_id_1[i]+" ")*(6-i) for i in range(len(place_id_1))] \n output += [(place_id_2[i]+" ")*(5-i) for i in range(len(place_id_2))]\n output += [(place_id_3[i]+" ")*(5-i) for i in range(len(place_id_3))]\n output += [(place_id_4[i]+" ")*(5-i) for i in range(len(place_id_4))]\n output += [(place_id_5[i]+" ")*(4-i) for i in range(len(place_id_5))]\n output = "".join(output).split()\n\n m = Counter(output) # this Counter method is to count word frequency.\n return [x[0] for x in m.most_common(3)]\n\n submission['place_id'] = submission.apply(lambda x: ensemble(x), axis=1)\n
@YijunMa, do you have 5 different models? If so, just curious, how long does it take to make a full prediction?
", "votes": null, "replies": [] }, { "id": 121575, "author_name": "ma350365879", "author_url": "", "post_date": "05/27/2016 15:00:20", "content": "@Mikhail Pershin, all these are random forest run by small chunks (around 100*100). Roughly about 18 hours or more for each model.
", "votes": null, "replies": [] }, { "id": 121577, "author_name": "radustoicescu", "author_url": "", "post_date": "05/27/2016 15:10:19", "content": "I was thinking of an ensemble method, but instead of taking 2 submission files as an input, like so:
\n\n1, A B C\n2, E F G\n3, H I J\nIt will take 2 files with the probability information kept, and maybe with more answers per entry, like so:
\n\n1, (A,0.6) (B,0.1),(C,0.05),(D,0.04),(E,0.02)\n2, (D,0.5) (E,0.2),(F,0.09),(G,0.06),(H,0.01)\n3, (H,0.7) (I,0.1),(J,0.07),(K,0.02),(L,0.01)\nI am not sure how to implement it, or if it will give any improvements over a simple ensemble of 3 ranked answers.
", "votes": null, "replies": [] }, { "id": 121580, "author_name": "pershinmr", "author_url": "", "post_date": "05/27/2016 15:20:15", "content": "@YijunMa, thereby its more than 90 hours in total! You are really patient! Though improvement from 0.55 to 0.57 is quite significant, so I guess it’s worth it :)
", "votes": null, "replies": [] }, { "id": 121581, "author_name": "pershinmr", "author_url": "", "post_date": "05/27/2016 15:31:33", "content": "@Radu Stoicescu, probability information could have different scale and thereby a different meaning if you are aimed to use estimators of different types (for example knn vs random forest) - so be careful with it ;)
", "votes": null, "replies": [] }, { "id": 121582, "author_name": "ma350365879", "author_url": "", "post_date": "05/27/2016 15:38:30", "content": "@Mikhail Pershin, thx! I am almost sure it should works before I run it. And there are small tricks to makes it 2~5 times faster, depends on the memory available on my computer.
\n\n@Radu Stoicescu. This should work even better than my method in theory.
", "votes": null, "replies": [] }, { "id": 121600, "author_name": "marekkowalski", "author_url": "", "post_date": "05/27/2016 18:07:38", "content": "I've tried with three different methods to combine submissions:
\n\nFor example if a given row in two submissions was:
\n\n1, 123 345 567\nand
\n\n1, 987 766 523\nI would combine this into something like:
\n\n1, 123 987 345.\nUnfortunately none of the above produced any improvement for me, but I have only tried it on my local cross validation test.
", "votes": null, "replies": [] }, { "id": 125172, "author_name": "gospursgo", "author_url": "", "post_date": "06/27/2016 05:07:08", "content": "@YijunMa That's a great improvement. Are those two different kind of models or simply the same model with different grids?
", "votes": null, "replies": [] }, { "id": 125201, "author_name": "ma350365879", "author_url": "", "post_date": "06/27/2016 15:23:38", "content": "@Juahuan He, my most "insane" ensembling is to combine 10 different models' predictions, so both situation occurs. Have to mention that this code is computationally inefficient, will be super slow if used for ensembling 10 models' result. Better to adjust the code little bit.
", "votes": null, "replies": [] }, { "id": 125218, "author_name": "beedata", "author_url": "", "post_date": "06/27/2016 17:46:31", "content": "", "votes": null, "replies": [] }, { "id": 125249, "author_name": "gospursgo", "author_url": "", "post_date": "06/28/2016 04:08:58", "content": "[quote=YijunMa;125201]
\n\n@Juahuan He, my most "insane" ensembling is to combine 10 different models' predictions, so both situation occurs. Have to mention that this code is computationally inefficient, will be super slow if used for ensembling 10 models' result. Better to adjust the code little bit.
\n\n[/quote]\nI am curious about the ensemble that you mentioned in the previous post which brought you from 0.55 to 0.57 on LB. Are those different models(like one knn and one XGB), or just different grids?
", "votes": null, "replies": [] }, { "id": 125290, "author_name": "ma350365879", "author_url": "", "post_date": "06/28/2016 13:12:23", "content": "@Jahuan He, both. T0 be honest, this is my first time to try ensembling, so I don't have any theory to guide me.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "121562": "The attached code takes two submissions and creates a third submission that is basically a weighted average of the first two. There probably was a more efficient and elegant way to do this, but this is what I could think of quickly. \r\n\r\nTo use it, you will of course need to change the directory name and input submission file names. The weights of the two submissions and of the three places are also configurable. If you see any bugs in the code, please let me know. \r\n\r\nAlso, if you have any methods for creating ensembles of submissions, please feel free to share. \r\n\r\nI tried combining my best solution with the best public script solution, and did not see a score improvement, but maybe you will have more luck!", "121570": "ensemble works for me! add from 0.55 to about 0.57.\r\n\r\n> \r\n> def ensemble(x):\r\n> # each place_id is a 5 top place_id guess\r\n> # yes, I am using map@5 instead of map@3.\r\n> place_id_1 = x['place_id_1'].split()\r\n> place_id_2 = x['place_id_2'].split()\r\n> place_id_3 = x['place_id_3'].split()\r\n> place_id_4 = x['place_id_4'].split()\r\n> place_id_5 = x['place_id_5'].split()\r\n\r\n> # place_id_1 has the best prediction, place_id_5 has the worst prediction\r\n> output = [(place_id_1[i]+\" \")*(6-i) for i in range(len(place_id_1))] \r\n> output += [(place_id_2[i]+\" \")*(5-i) for i in range(len(place_id_2))]\r\n> output += [(place_id_3[i]+\" \")*(5-i) for i in range(len(place_id_3))]\r\n> output += [(place_id_4[i]+\" \")*(5-i) for i in range(len(place_id_4))]\r\n> output += [(place_id_5[i]+\" \")*(4-i) for i in range(len(place_id_5))]\r\n> output = \"\".join(output).split()\r\n\r\n> m = Counter(output) # this Counter method is to count word frequency.\r\n> return [x[0] for x in m.most_common(3)]\r\n\r\n> submission['place_id'] = submission.apply(lambda x: ensemble(x), axis=1)", "121574": "YijunMa, do you have 5 different models? If so, just curious, how long does it take to make a full prediction?", "121575": "Mikhail Pershin, all these are random forest run by small chunks (around 100*100). Roughly about 18 hours or more for each model.", "121577": "I was thinking of an ensemble method, but instead of taking 2 submission files as an input, like so:\r\n\r\n 1, A B C\r\n 2, E F G\r\n 3, H I J\r\n\r\nIt will take 2 files with the probability information kept, and maybe with more answers per entry, like so:\r\n\r\n 1, (A,0.6) (B,0.1),(C,0.05),(D,0.04),(E,0.02)\r\n 2, (D,0.5) (E,0.2),(F,0.09),(G,0.06),(H,0.01)\r\n 3, (H,0.7) (I,0.1),(J,0.07),(K,0.02),(L,0.01)\r\n\r\n\r\nI am not sure how to implement it, or if it will give any improvements over a simple ensemble of 3 ranked answers.", "121580": "YijunMa, thereby its more than 90 hours in total! You are really patient! Though improvement from 0.55 to 0.57 is quite significant, so I guess it’s worth it :)", "121581": "Radu Stoicescu, probability information could have different scale and thereby a different meaning if you are aimed to use estimators of different types (for example knn vs random forest) - so be careful with it ;)", "121582": "Mikhail Pershin, thx! I am almost sure it should works before I run it. And there are small tricks to makes it 2~5 times faster, depends on the memory available on my computer.\r\n\r\n@Radu Stoicescu. This should work even better than my method in theory.", "121600": "I've tried with three different methods to combine submissions:\r\n\r\n - Voting with equal weights,\r\n - Voting with confidence based weights (similarly to what was proposed by Radu Stoicescu),\r\n - Taking the first answer of the \"better\" method and using the first answer of the second best method as the second proposal.\r\n\r\nFor example if a given row in two submissions was:\r\n\r\n 1, 123 345 567\r\n\r\nand\r\n\r\n 1, 987 766 523\r\n\r\nI would combine this into something like:\r\n\r\n 1, 123 987 345.\r\n\r\nUnfortunately none of the above produced any improvement for me, but I have only tried it on my local cross validation test.", "125172": "YijunMa That's a great improvement. Are those two different kind of models or simply the same model with different grids?", "125201": "Juahuan He, my most \"insane\" ensembling is to combine 10 different models' predictions, so both situation occurs. Have to mention that this code is computationally inefficient, will be super slow if used for ensembling 10 models' result. Better to adjust the code little bit.", "125218": "", "125249": "[quote=YijunMa;125201]\r\n\r\n@Juahuan He, my most \"insane\" ensembling is to combine 10 different models' predictions, so both situation occurs. Have to mention that this code is computationally inefficient, will be super slow if used for ensembling 10 models' result. Better to adjust the code little bit.\r\n\r\n[/quote]\r\nI am curious about the ensemble that you mentioned in the previous post which brought you from 0.55 to 0.57 on LB. Are those different models(like one knn and one XGB), or just different grids?", "125290": "Jahuan He, both. T0 be honest, this is my first time to try ensembling, so I don't have any theory to guide me." }, "source": "meta" }