I used two types of feature sets:
\n\nFeature Set 1: x,y,logaccuracy,hour,year,weekday,month
\n\nFor example, see the code in this post by Jiahuan He.
\n\nFeature Set 2: x,y,logaccuracy,minute_sin,minute_cos, day_of_year_sin,day_of_year_cos, weekday_sin,weekday_cos,month,year
\n\nDetails for feature set 2 came from this post by Michael Hartman
\n\nFor each feature set, i built an knn model (near neighbors = 37), random forest, and xgb classifier.
\n\nI hand-tuned weights for knn (starting with weights posted in the forums). For rf and xgb, I used weights on x,,y, and logaccuracy only. I took a look at this algorithm from AnujChopra but did not end up using the results from it.
\n\nFor each knn model, i added probabilities that were mentioned in this post by Abhi. i found that this improved my knn model in the LB by 0.005.
\n\nThe individual models themselves got me to a 0.585 at best which was my knn/feature set 2 with probabiilities based on weekday/hour/four hour. My best xgb and rf were around 0.580.
\n\nI got to #18 early on (0.582 on LB at the time) by ensembling the list of models. I wrote a simple application that takes as input a list of tuples of (csv file,weight) and then it tallies predictions for each test row id based on these tuples. The tally is based on weight*1 for the first prediction, weight*(1/2) for second prediction, and weight*(1/3) for third. The ensemble consists of the tallying all the weighted votes and then using the top 3 place ids by votes in order of votes.
\n\nThe way I got to #23 was that I wrote a tool to make it easier to figure out the best combination of models to ensemble.
\n\nThe tool divides the training set into 70% training data and 30% cross-validation data based on time >= 587158. It then goes cell-by-cell in a 100x100 grid, to generate predictions based on weights models.
\n\nFor example, here is the input to my tool that led me to my second best ensemble which got me to #27:
\n\nweighted_models = [("xgb_featureset1",24),\n ("rf_featureset2",10),\n ("knn_featureset2",16),\n ("rf_featureset1",11),\n ("xgb_featureset2",10)]\nI got to #23 by adding my original ensemble that I mentioned before.
\n\nI am planning to share all my code which was written in python.
\n\nHere is my method for combining multiple dataframes by weight so that I could cross-validate and evalute the results of the ensemble where weights is a list of weights and names is a list of model names, and df_list is the predictions made by each model for the given cell.
\n\ntopn=3\n\n# combine all models into a single dataframe\ndf = pd.concat([d.set_index(d.index) for d in df_list],\n keys=names)\nweights_df = pd.DataFrame(weights,names)\n\n# 1 for first, 1/2 for second, 1/3 for third.\nweights_pr = pd.DataFrame([1./float(i+1) for i in range(topn)],['predict'+str(i+1) for i in range(topn)]).T\nweights = weights_df.dot(weights_pr)\nweights_by_group = df.groupby(level='row_id').apply(lambda x:weights).unstack(0).stack()\nbdf = pd.concat([df[['predict'+str(i+1) for i in range(topn)]],weights_by_group],axis=1,keys=['choices','weights'])\nbdf1 = bdf.stack().set_index('choices',append=True)\n\n# sum up the votes for each model\nbdf2 = bdf1.groupby(level=[1,3]).sum().unstack(0)\nsort = lambda x: x.sort_values(ascending=False).index\nreturn pd.DataFrame(bdf2.apply(sort).reset_index(drop=True).head(topn).values.T)\nI used two types of feature sets:
\n\nFeature Set 1: x,y,logaccuracy,hour,year,weekday,month
\n\nFor example, see the code in this post by Jiahuan He.
\n\nFeature Set 2: x,y,logaccuracy,minute_sin,minute_cos, day_of_year_sin,day_of_year_cos, weekday_sin,weekday_cos,month,year
\n\nDetails for feature set 2 came from this post by Michael Hartman
\n\nFor each feature set, i built an knn model (near neighbors = 37), random forest, and xgb classifier.
\n\nI hand-tuned weights for knn (starting with weights posted in the forums). For rf and xgb, I used weights on x,,y, and logaccuracy only. I took a look at this algorithm from AnujChopra but did not end up using the results from it.
\n\nFor each knn model, i added probabilities that were mentioned in this post by Abhi. i found that this improved my knn model in the LB by 0.005.
\n\nThe individual models themselves got me to a 0.585 at best which was my knn/feature set 2 with probabiilities based on weekday/hour/four hour. My best xgb and rf were around 0.580.
\n\nI got to #18 early on (0.582 on LB at the time) by ensembling the list of models. I wrote a simple application that takes as input a list of tuples of (csv file,weight) and then it tallies predictions for each test row id based on these tuples. The tally is based on weight*1 for the first prediction, weight*(1/2) for second prediction, and weight*(1/3) for third. The ensemble consists of the tallying all the weighted votes and then using the top 3 place ids by votes in order of votes.
\n\nThe way I got to #23 was that I wrote a tool to make it easier to figure out the best combination of models to ensemble.
\n\nThe tool divides the training set into 70% training data and 30% cross-validation data based on time >= 587158. It then goes cell-by-cell in a 100x100 grid, to generate predictions based on weights models.
\n\nFor example, here is the input to my tool that led me to my second best ensemble which got me to #27:
\n\nweighted_models = [("xgb_featureset1",24),\n ("rf_featureset2",10),\n ("knn_featureset2",16),\n ("rf_featureset1",11),\n ("xgb_featureset2",10)]\nI got to #23 by adding my original ensemble that I mentioned before.
\n\nI am planning to share all my code which was written in python.
\n\nHere is my method for combining multiple dataframes by weight so that I could cross-validate and evalute the results of the ensemble where weights is a list of weights and names is a list of model names, and df_list is the predictions made by each model for the given cell.
\n\ntopn=3\n\n# combine all models into a single dataframe\ndf = pd.concat([d.set_index(d.index) for d in df_list],\n keys=names)\nweights_df = pd.DataFrame(weights,names)\n\n# 1 for first, 1/2 for second, 1/3 for third.\nweights_pr = pd.DataFrame([1./float(i+1) for i in range(topn)],['predict'+str(i+1) for i in range(topn)]).T\nweights = weights_df.dot(weights_pr)\nweights_by_group = df.groupby(level='row_id').apply(lambda x:weights).unstack(0).stack()\nbdf = pd.concat([df[['predict'+str(i+1) for i in range(topn)]],weights_by_group],axis=1,keys=['choices','weights'])\nbdf1 = bdf.stack().set_index('choices',append=True)\n\n# sum up the votes for each model\nbdf2 = bdf1.groupby(level=[1,3]).sum().unstack(0)\nsort = lambda x: x.sort_values(ascending=False).index\nreturn pd.DataFrame(bdf2.apply(sort).reset_index(drop=True).head(topn).values.T)\nthank you for sharing your recipe. I was also working on ensembling xgb knn and rf. However my rf tuning is not very successful. Your methods is cool!
", "rawMarkdown": "thank you for sharing your recipe. I was also working on ensembling xgb knn and rf. However my rf tuning is not very successful. Your methods is cool!", "votes": null }, { "id": "126325", "postDate": "07/07/2016 17:55:52", "content": "@Larry, did you try the probabilities with the xgb or rf models?
", "rawMarkdown": "Larry, did you try the probabilities with the xgb or rf models?", "votes": null }, { "id": "126330", "postDate": "07/07/2016 19:04:55", "content": "@abhi, I did not. Have you tried?
", "rawMarkdown": "abhi, I did not. Have you tried?", "votes": null }, { "id": "126339", "postDate": "07/07/2016 19:51:01", "content": "I have not tried it on xgb of rf. I focused mostly on kdes and knn during the competition. Just wondering if the probabilities give a similar benefit for other types of models. Thanks for sharing the ensembling code!
", "rawMarkdown": "I have not tried it on xgb of rf. I focused mostly on kdes and knn during the competition. Just wondering if the probabilities give a similar benefit for other types of models. Thanks for sharing the ensembling code!", "votes": null }, { "id": "126460", "postDate": "07/08/2016 15:42:16", "content": "@Larry Freeman, Nice script, how did you find out the weight for each model? Grid Search?
\n\nweighted_models = [("xgb_featureset1",24),\n ("rf_featureset2",10),\n ("knn_featureset2",16),\n ("rf_featureset1",11),\n ("xgb_featureset2",10)]\n[quote=David;126460]
\n\n@Larry Freeman, Nice script, how did you find out the weight for each model? Grid Search?
\n\nweighted_models = [("xgb_featureset1",24),\n ("rf_featureset2",10),\n ("knn_featureset2",16),\n ("rf_featureset1",11),\n ("xgb_featureset2",10)]\n[/quote]
\n\nI hand-tuned each weight based on the first 30 cells in a 100 x 100 grid. I wrote code to do ensemble of models cell-by-cell so I could see the results in less than 1 minute per model. The testing of all 5 models for the first 30 cells took roughly 5 minutes.
\n\nI chose the first 30 cells with 70% training/30% cross validation because this was consistently (and surprisingly) accurate for predicting which ensemble would have a better LB score.
\n\nFirst, I estimated the CV for each model individually which turned out to be knn_featureset2. Then, I paired with each other model starting with weights (1,1). Once I figured out the best weights for these two, I took the weaker model and found the best pairing. And so on. It was time consuming.
\n\nI had been working on automating this process but ran out of time. I am planning to release my code. If I can get my automation working, I will include this when I release my code.
", "rawMarkdown": "[quote=David;126460]\r\n\r\n@Larry Freeman, Nice script, how did you find out the weight for each model? Grid Search?\r\n\r\n weighted_models = [(\"xgb_featureset1\",24),\r\n (\"rf_featureset2\",10),\r\n (\"knn_featureset2\",16),\r\n (\"rf_featureset1\",11),\r\n (\"xgb_featureset2\",10)]\r\n\r\n[/quote]\r\n\r\nI hand-tuned each weight based on the first 30 cells in a 100 x 100 grid. I wrote code to do ensemble of models cell-by-cell so I could see the results in less than 1 minute per model. The testing of all 5 models for the first 30 cells took roughly 5 minutes.\r\n\r\nI chose the first 30 cells with 70% training/30% cross validation because this was consistently (and surprisingly) accurate for predicting which ensemble would have a better LB score.\r\n\r\nFirst, I estimated the CV for each model individually which turned out to be knn_featureset2. Then, I paired with each other model starting with weights (1,1). Once I figured out the best weights for these two, I took the weaker model and found the best pairing. And so on. It was time consuming.\r\n\r\nI had been working on automating this process but ran out of time. I am planning to release my code. If I can get my automation working, I will include this when I release my code.", "votes": null }, { "id": "126480", "postDate": "07/08/2016 18:53:47", "content": "Thanks for the explanation. I stored the top 5 predicted labels with their probabilities for each model, and then ensemble (vote) them by selecting the top 3 labels from probability of each label from evaluation function: proba_of_label-K = proba_of_label-K_from_model1 * weight1 + proba_of_label-K_from_model2 * weight2 + ... where weight is the map@3 score of each model. But the ensemble of xgb(0.58078), rf(0.57967), and knn(0.58285) only result in 0.59733.
", "rawMarkdown": "Thanks for the explanation. I stored the top 5 predicted labels with their probabilities for each model, and then ensemble (vote) them by selecting the top 3 labels from probability of each label from evaluation function: proba_of_label-K = proba_of_label-K_from_model1 * weight1 + proba_of_label-K_from_model2 * weight2 + ... where weight is the map@3 score of each model. But the ensemble of xgb(0.58078), rf(0.57967), and knn(0.58285) only result in 0.59733.", "votes": null } ], "comments": [ { "id": 126319, "author_name": "chengcen", "author_url": "", "post_date": "07/07/2016 17:09:46", "content": "thank you for sharing your recipe. I was also working on ensembling xgb knn and rf. However my rf tuning is not very successful. Your methods is cool!
", "votes": null, "replies": [] }, { "id": 126325, "author_name": "suryavan", "author_url": "", "post_date": "07/07/2016 17:55:52", "content": "@Larry, did you try the probabilities with the xgb or rf models?
", "votes": null, "replies": [] }, { "id": 126330, "author_name": "larryfreeman", "author_url": "", "post_date": "07/07/2016 19:04:55", "content": "@abhi, I did not. Have you tried?
", "votes": null, "replies": [] }, { "id": 126339, "author_name": "suryavan", "author_url": "", "post_date": "07/07/2016 19:51:01", "content": "I have not tried it on xgb of rf. I focused mostly on kdes and knn during the competition. Just wondering if the probabilities give a similar benefit for other types of models. Thanks for sharing the ensembling code!
", "votes": null, "replies": [] }, { "id": 126460, "author_name": "overfit", "author_url": "", "post_date": "07/08/2016 15:42:16", "content": "@Larry Freeman, Nice script, how did you find out the weight for each model? Grid Search?
\n\nweighted_models = [("xgb_featureset1",24),\n ("rf_featureset2",10),\n ("knn_featureset2",16),\n ("rf_featureset1",11),\n ("xgb_featureset2",10)]\n[quote=David;126460]
\n\n@Larry Freeman, Nice script, how did you find out the weight for each model? Grid Search?
\n\nweighted_models = [("xgb_featureset1",24),\n ("rf_featureset2",10),\n ("knn_featureset2",16),\n ("rf_featureset1",11),\n ("xgb_featureset2",10)]\n[/quote]
\n\nI hand-tuned each weight based on the first 30 cells in a 100 x 100 grid. I wrote code to do ensemble of models cell-by-cell so I could see the results in less than 1 minute per model. The testing of all 5 models for the first 30 cells took roughly 5 minutes.
\n\nI chose the first 30 cells with 70% training/30% cross validation because this was consistently (and surprisingly) accurate for predicting which ensemble would have a better LB score.
\n\nFirst, I estimated the CV for each model individually which turned out to be knn_featureset2. Then, I paired with each other model starting with weights (1,1). Once I figured out the best weights for these two, I took the weaker model and found the best pairing. And so on. It was time consuming.
\n\nI had been working on automating this process but ran out of time. I am planning to release my code. If I can get my automation working, I will include this when I release my code.
", "votes": null, "replies": [] }, { "id": 126480, "author_name": "overfit", "author_url": "", "post_date": "07/08/2016 18:53:47", "content": "Thanks for the explanation. I stored the top 5 predicted labels with their probabilities for each model, and then ensemble (vote) them by selecting the top 3 labels from probability of each label from evaluation function: proba_of_label-K = proba_of_label-K_from_model1 * weight1 + proba_of_label-K_from_model2 * weight2 + ... where weight is the map@3 score of each model. But the ensemble of xgb(0.58078), rf(0.57967), and knn(0.58285) only result in 0.59733.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "126278": "I used two types of feature sets:\r\n\r\n**Feature Set 1:** x,y,logaccuracy,hour,year,weekday,month\r\n\r\nFor example, see the code in this [post][1] by Jiahuan He.\r\n\r\n**Feature Set 2:** x,y,logaccuracy,minute_sin,minute_cos, day_of_year_sin,day_of_year_cos, weekday_sin,weekday_cos,month,year\r\n\r\nDetails for feature set 2 came from this [post][2] by Michael Hartman\r\n\r\nFor each feature set, i built an knn model (near neighbors = 37), random forest, and xgb classifier.\r\n\r\nI hand-tuned weights for knn (starting with weights posted in the forums). For rf and xgb, I used weights on x,,y, and logaccuracy only. I took a look at this [algorithm][3] from AnujChopra but did not end up using the results from it.\r\n\r\nFor each knn model, i added probabilities that were mentioned in this [post][4] by Abhi. i found that this improved my knn model in the LB by 0.005.\r\n\r\nThe individual models themselves got me to a 0.585 at best which was my knn/feature set 2 with probabiilities based on weekday/hour/four hour. My best xgb and rf were around 0.580.\r\n\r\nI got to #18 early on (0.582 on LB at the time) by ensembling the list of models. I wrote a simple application that takes as input a list of tuples of (csv file,weight) and then it tallies predictions for each test row id based on these tuples. The tally is based on weight*1 for the first prediction, weight*(1/2) for second prediction, and weight*(1/3) for third. The ensemble consists of the tallying all the weighted votes and then using the top 3 place ids by votes in order of votes.\r\n\r\nThe way I got to #23 was that I wrote a tool to make it easier to figure out the best combination of models to ensemble. \r\n\r\nThe tool divides the training set into 70% training data and 30% cross-validation data based on time >= 587158. It then goes cell-by-cell in a 100x100 grid, to generate predictions based on weights models.\r\n\r\nFor example, here is the input to my tool that led me to my second best ensemble which got me to #27:\r\n\r\n weighted_models = [(\"xgb_featureset1\",24),\r\n (\"rf_featureset2\",10),\r\n (\"knn_featureset2\",16),\r\n (\"rf_featureset1\",11),\r\n (\"xgb_featureset2\",10)]\r\n\r\nI got to #23 by adding my original ensemble that I mentioned before. \r\n \r\nI am planning to share all my code which was written in python. \r\n\r\nHere is my method for combining multiple dataframes by weight so that I could cross-validate and evalute the results of the ensemble where weights is a list of **weights** and **names** is a list of model names, and **df_list** is the predictions made by each model for the given cell.\r\n\r\n topn=3\r\n \r\n # combine all models into a single dataframe\r\n df = pd.concat([d.set_index(d.index) for d in df_list],\r\n keys=names)\r\n weights_df = pd.DataFrame(weights,names)\r\n \r\n # 1 for first, 1/2 for second, 1/3 for third.\r\n weights_pr = pd.DataFrame([1./float(i+1) for i in range(topn)],['predict'+str(i+1) for i in range(topn)]).T\r\n weights = weights_df.dot(weights_pr)\r\n weights_by_group = df.groupby(level='row_id').apply(lambda x:weights).unstack(0).stack()\r\n bdf = pd.concat([df[['predict'+str(i+1) for i in range(topn)]],weights_by_group],axis=1,keys=['choices','weights'])\r\n bdf1 = bdf.stack().set_index('choices',append=True)\r\n\r\n # sum up the votes for each model\r\n bdf2 = bdf1.groupby(level=[1,3]).sum().unstack(0)\r\n sort = lambda x: x.sort_values(ascending=False).index\r\n return pd.DataFrame(bdf2.apply(sort).reset_index(drop=True).head(topn).values.T)\r\n\r\n\r\n [1]: https://www.kaggle.com/gospursgo/facebook-v-predicting-check-ins/modified-grid-knn-without-month-year/discussion\r\n [2]: https://www.kaggle.com/zeroblue/facebook-v-predicting-check-ins/mad-scripts-battle-z\r\n [3]: https://www.kaggle.com/chopra/facebook-v-predicting-check-ins/logistic-regression-to-find-knn-weights/discussion\r\n [4]: https://www.kaggle.com/suryavan/facebook-v-predicting-check-ins/knn-and-probability-lookup-script/discussion", "126319": "thank you for sharing your recipe. I was also working on ensembling xgb knn and rf. However my rf tuning is not very successful. Your methods is cool!", "126325": "Larry, did you try the probabilities with the xgb or rf models?", "126330": "abhi, I did not. Have you tried?", "126339": "I have not tried it on xgb of rf. I focused mostly on kdes and knn during the competition. Just wondering if the probabilities give a similar benefit for other types of models. Thanks for sharing the ensembling code!", "126460": "Larry Freeman, Nice script, how did you find out the weight for each model? Grid Search?\r\n\r\n weighted_models = [(\"xgb_featureset1\",24),\r\n (\"rf_featureset2\",10),\r\n (\"knn_featureset2\",16),\r\n (\"rf_featureset1\",11),\r\n (\"xgb_featureset2\",10)]", "126477": "[quote=David;126460]\r\n\r\n@Larry Freeman, Nice script, how did you find out the weight for each model? Grid Search?\r\n\r\n weighted_models = [(\"xgb_featureset1\",24),\r\n (\"rf_featureset2\",10),\r\n (\"knn_featureset2\",16),\r\n (\"rf_featureset1\",11),\r\n (\"xgb_featureset2\",10)]\r\n\r\n[/quote]\r\n\r\nI hand-tuned each weight based on the first 30 cells in a 100 x 100 grid. I wrote code to do ensemble of models cell-by-cell so I could see the results in less than 1 minute per model. The testing of all 5 models for the first 30 cells took roughly 5 minutes.\r\n\r\nI chose the first 30 cells with 70% training/30% cross validation because this was consistently (and surprisingly) accurate for predicting which ensemble would have a better LB score.\r\n\r\nFirst, I estimated the CV for each model individually which turned out to be knn_featureset2. Then, I paired with each other model starting with weights (1,1). Once I figured out the best weights for these two, I took the weaker model and found the best pairing. And so on. It was time consuming.\r\n\r\nI had been working on automating this process but ran out of time. I am planning to release my code. If I can get my automation working, I will include this when I release my code.", "126480": "Thanks for the explanation. I stored the top 5 predicted labels with their probabilities for each model, and then ensemble (vote) them by selecting the top 3 labels from probability of each label from evaluation function: proba_of_label-K = proba_of_label-K_from_model1 * weight1 + proba_of_label-K_from_model2 * weight2 + ... where weight is the map@3 score of each model. But the ensemble of xgb(0.58078), rf(0.57967), and knn(0.58285) only result in 0.59733." }, "source": "meta" }