This is my first Kaggle competition, and I was wondering if I could tap into the knowledge base here to discuss how to make nearest neighbors a feature for a tree based classifier (specifically XGBoost). I was inspired by @Sandro's KNN script (https://www.kaggle.com/svpons/facebook-v-predicting-check-ins/grid-knn/discussion) which gets roughly 0.56 in my tests. Try as I might, I was unable to improve KNN beyond about 0.57, whereas I have preliminary XGBoost results of 0.59 or so.
\n\nOption 1: Voting classifier
\n\nThis would be straightforward, use a voting classifier with equal (unequal?) weights to combine the KNN classifier with XGBoost. Strictly speaking, this is not making KNN a feature, but an ensemble approach instead.
\n\nOption 2: Neighbor weights per place_id as a feature
\n\nUse something like sklearn kneighbors_graph to find the weights of the nearest k points under some metric (weighted minkowski with p=1 seems to be working well for me). This effectively adds N features per grid cell, where N is the number of categories after all cuts in that grid cell.
\n\nOption 3: Sparse coding of KNeighborsClassifier for XGBoost
\n\nTake the predicted place_ids from KNeighborsClassifier, label encode them and feed them into XGBoost. So for example, for a single row, if I have KNeighbors predict 7995458948, 8393706174, 8643187406 and my label encoder encodes them as categories 0, 2, and 4, respectively I could encode that as
\n\n [ 1, 0, 0.5, 0, 0.333, 0, 0, 0, 0] \nassuming there were 8 predicted labels, and this would generalize to a larger sparse vector. Basically this is One Hot Encoding, but with three allowed categories, with some additional weights.
\n\nI was wondering if the community had any thoughts on these strategies. Have you had great luck with one of these options in the past on some other competition, or with something else I didn't mention?
", "messages": [ { "id": "124078", "postDate": "06/15/2016 11:53:38", "content": "This is my first Kaggle competition, and I was wondering if I could tap into the knowledge base here to discuss how to make nearest neighbors a feature for a tree based classifier (specifically XGBoost). I was inspired by @Sandro's KNN script (https://www.kaggle.com/svpons/facebook-v-predicting-check-ins/grid-knn/discussion) which gets roughly 0.56 in my tests. Try as I might, I was unable to improve KNN beyond about 0.57, whereas I have preliminary XGBoost results of 0.59 or so.
\n\nOption 1: Voting classifier
\n\nThis would be straightforward, use a voting classifier with equal (unequal?) weights to combine the KNN classifier with XGBoost. Strictly speaking, this is not making KNN a feature, but an ensemble approach instead.
\n\nOption 2: Neighbor weights per place_id as a feature
\n\nUse something like sklearn kneighbors_graph to find the weights of the nearest k points under some metric (weighted minkowski with p=1 seems to be working well for me). This effectively adds N features per grid cell, where N is the number of categories after all cuts in that grid cell.
\n\nOption 3: Sparse coding of KNeighborsClassifier for XGBoost
\n\nTake the predicted place_ids from KNeighborsClassifier, label encode them and feed them into XGBoost. So for example, for a single row, if I have KNeighbors predict 7995458948, 8393706174, 8643187406 and my label encoder encodes them as categories 0, 2, and 4, respectively I could encode that as
\n\n [ 1, 0, 0.5, 0, 0.333, 0, 0, 0, 0] \nassuming there were 8 predicted labels, and this would generalize to a larger sparse vector. Basically this is One Hot Encoding, but with three allowed categories, with some additional weights.
\n\nI was wondering if the community had any thoughts on these strategies. Have you had great luck with one of these options in the past on some other competition, or with something else I didn't mention?
", "rawMarkdown": "This is my first Kaggle competition, and I was wondering if I could tap into the knowledge base here to discuss how to make nearest neighbors a feature for a tree based classifier (specifically XGBoost). I was inspired by @Sandro's KNN script (https://www.kaggle.com/svpons/facebook-v-predicting-check-ins/grid-knn/discussion) which gets roughly 0.56 in my tests. Try as I might, I was unable to improve KNN beyond about 0.57, whereas I have preliminary XGBoost results of 0.59 or so. \r\n\r\n**Option 1: Voting classifier**\r\n\r\nThis would be straightforward, use a voting classifier with equal (unequal?) weights to combine the KNN classifier with XGBoost. Strictly speaking, this is not making KNN a feature, but an ensemble approach instead.\r\n\r\n**Option 2: Neighbor weights per place_id as a feature**\r\n\r\nUse something like sklearn kneighbors_graph to find the weights of the nearest k points under some metric (weighted minkowski with p=1 seems to be working well for me). This effectively adds N features per grid cell, where N is the number of categories after all cuts in that grid cell. \r\n\r\n**Option 3: Sparse coding of KNeighborsClassifier for XGBoost**\r\n\r\nTake the predicted place_ids from KNeighborsClassifier, label encode them and feed them into XGBoost. So for example, for a single row, if I have KNeighbors predict 7995458948, 8393706174, 8643187406 and my label encoder encodes them as categories 0, 2, and 4, respectively I could encode that as \r\n\r\n [ 1, 0, 0.5, 0, 0.333, 0, 0, 0, 0] \r\n\r\nassuming there were 8 predicted labels, and this would generalize to a larger sparse vector. Basically this is One Hot Encoding, but with three allowed categories, with some additional weights. \r\n\r\n\r\n----------\r\n\r\n\r\nI was wondering if the community had any thoughts on these strategies. Have you had great luck with one of these options in the past on some other competition, or with something else I didn't mention?", "votes": null }, { "id": "125055", "postDate": "06/25/2016 05:27:45", "content": "After reading the forums, and trying a few things, I think my strategy going forward will be something like this....
\n\n1. Models: KNN, Random Forest, XGB:\nUse a voting classifier to combine predictions of each of the 3 models, but have 9 variables because each model will generate the top 3 predictions for each row_id. Do weighted voting with the MAP@3 metric in mind, value of 1.0 for 1st prediction, 0.5 for second prediction, 0.333 for third prediction.
\n\n2. Generate models with multiple grid sizes:\nBuild models trained with different grid sizes as grids separate the data arbitrarily. Include these in the final ensemble model.
\n\n3. Use clustering on raw data to build clusters to use instead of grid cells:\nTrain models and include in ensembles.
\n\n4. Use feature importance ratios from Random Forest and XGBOOST models for weights in K Nearest Neighbors: I haven't tried this yet, but it seems like it is probably how other competitors are developing their feature weight vectors.
\n\n5. Transform feature matrix and build new models: Try Principal Components Analysis and Independent Components Analysis on training data to build new models.
\n\nHere's some commentary on your proposed items:
\n\nOPTION 1: \nTotally makes sense to do this. It's better if the solutions are uncorrelated.
\n\nOPTION 2: This is interesting, but I worry it would generate thousands of new columns, and slow down computing time substantially.
\n\nOPTION 3: This makes a lot of sense to me, but it seems like you would run into computation time problem. I'd be interested in seeing what your results are with this strategy.
", "rawMarkdown": "After reading the forums, and trying a few things, I think my strategy going forward will be something like this....\r\n\r\n**1. Models: KNN, Random Forest, XGB:**\r\nUse a voting classifier to combine predictions of each of the 3 models, but have 9 variables because each model will generate the top 3 predictions for each row_id. Do weighted voting with the MAP@3 metric in mind, value of 1.0 for 1st prediction, 0.5 for second prediction, 0.333 for third prediction. \r\n\r\n**2. Generate models with multiple grid sizes:**\r\nBuild models trained with different grid sizes as grids separate the data arbitrarily. Include these in the final ensemble model.\r\n\r\n**3. Use clustering on raw data to build clusters to use instead of grid cells:**\r\nTrain models and include in ensembles. \r\n\r\n**4. Use feature importance ratios from Random Forest and XGBOOST models for weights in K Nearest Neighbors:** I haven't tried this yet, but it seems like it is probably how other competitors are developing their feature weight vectors.\r\n\r\n**5. Transform feature matrix and build new models:** Try Principal Components Analysis and Independent Components Analysis on training data to build new models.\r\n\r\nHere's some commentary on your proposed items: \r\n\r\n**OPTION 1:** \r\nTotally makes sense to do this. It's better if the solutions are uncorrelated.\r\n\r\n**OPTION 2:** This is interesting, but I worry it would generate thousands of new columns, and slow down computing time substantially. \r\n\r\n**OPTION 3:** This makes a lot of sense to me, but it seems like you would run into computation time problem. I'd be interested in seeing what your results are with this strategy.", "votes": null }, { "id": "125084", "postDate": "06/25/2016 14:49:39", "content": "Quite insightful!
", "rawMarkdown": "Quite insightful!", "votes": null }, { "id": "125103", "postDate": "06/26/2016 05:09:55", "content": "@TenaciousTerrier
\n\nI have tried option 3 that using clusters instead of grid. I use only position information(x,y) to train 400 clusters. It cost me about one day to run on AWS EC2 8cpu 32G RAM machine. But I didn't see significant improvement on my CV score.
", "rawMarkdown": "TenaciousTerrier\r\n\r\nI have tried option 3 that using clusters instead of grid. I use only position information(x,y) to train 400 clusters. It cost me about one day to run on AWS EC2 8cpu 32G RAM machine. But I didn't see significant improvement on my CV score.", "votes": null }, { "id": "125129", "postDate": "06/26/2016 15:59:01", "content": "@TenaciousTerrier, I see that we're having a lot of the same ideas. I also used xgboost feature weights as distance weights in kNN, applying a bayesian tuning of the weights afterward. Just as an update
\n\nI implemented a voting classifier, and while it improves results, it's not by as much as I would have hoped. Combining KNN with XGBoost gives me an increase of 0.57 to 0.585 using weighted voting. Adding more models (Random Forest, ExtraTrees, Kernel SVM) doesn't seem to be helping.
\n\nI also tried using a weighted voting classifier with xgboost on 7 grid sizes from 33x33 to 200x200. 0.57 -> 0.58 after voting. Extremely computationally intensive, not particularly helpful.
I did end up adding Nearest Neighbor distances to XGBoost (what I originally suggested as option 2). This adds about 30-50 features per cell for a 0.1 to 0.2 grid cell size, which XGBoost handles quite well. With L1 and L2 regularization in XGBoost, this improves my single classifier LB to 0.583 which is quite respectable.
My next plan is to stack or blend the models from my item 2.
", "rawMarkdown": "TenaciousTerrier, I see that we're having a lot of the same ideas. I also used xgboost feature weights as distance weights in kNN, applying a bayesian tuning of the weights afterward. Just as an update\r\n\r\n1. I implemented a voting classifier, and while it improves results, it's not by as much as I would have hoped. Combining KNN with XGBoost gives me an increase of 0.57 to 0.585 using weighted voting. Adding more models (Random Forest, ExtraTrees, Kernel SVM) doesn't seem to be helping. \r\n\r\n I also tried using a weighted voting classifier with xgboost on 7 grid sizes from 33x33 to 200x200. 0.57 -> 0.58 after voting. Extremely computationally intensive, not particularly helpful.\r\n\r\n2. I did end up adding Nearest Neighbor distances to XGBoost (what I originally suggested as option 2). This adds about 30-50 features per cell for a 0.1 to 0.2 grid cell size, which XGBoost handles quite well. With L1 and L2 regularization in XGBoost, this improves my single classifier LB to 0.583 which is quite respectable.\r\n\r\nMy next plan is to stack or blend the models from my item 2.", "votes": null }, { "id": "125236", "postDate": "06/28/2016 00:48:32", "content": "I have tried to use my best KNN predictions (~0.582 on LB) as features (top-3 or top-6 predicted places ids) along with original features to build a second-level model with XGBoost. I haven't seen any improvement, moreover, a score became slightly less. I can't understand why it happened so. Does anyone try to do the same thing or maybe build a second-level model in other way which has improved the score?
", "rawMarkdown": "I have tried to use my best KNN predictions (~0.582 on LB) as features (top-3 or top-6 predicted places ids) along with original features to build a second-level model with XGBoost. I haven't seen any improvement, moreover, a score became slightly less. I can't understand why it happened so. Does anyone try to do the same thing or maybe build a second-level model in other way which has improved the score?", "votes": null } ], "comments": [ { "id": 125055, "author_name": "tenaciousterrier", "author_url": "", "post_date": "06/25/2016 05:27:45", "content": "After reading the forums, and trying a few things, I think my strategy going forward will be something like this....
\n\n1. Models: KNN, Random Forest, XGB:\nUse a voting classifier to combine predictions of each of the 3 models, but have 9 variables because each model will generate the top 3 predictions for each row_id. Do weighted voting with the MAP@3 metric in mind, value of 1.0 for 1st prediction, 0.5 for second prediction, 0.333 for third prediction.
\n\n2. Generate models with multiple grid sizes:\nBuild models trained with different grid sizes as grids separate the data arbitrarily. Include these in the final ensemble model.
\n\n3. Use clustering on raw data to build clusters to use instead of grid cells:\nTrain models and include in ensembles.
\n\n4. Use feature importance ratios from Random Forest and XGBOOST models for weights in K Nearest Neighbors: I haven't tried this yet, but it seems like it is probably how other competitors are developing their feature weight vectors.
\n\n5. Transform feature matrix and build new models: Try Principal Components Analysis and Independent Components Analysis on training data to build new models.
\n\nHere's some commentary on your proposed items:
\n\nOPTION 1: \nTotally makes sense to do this. It's better if the solutions are uncorrelated.
\n\nOPTION 2: This is interesting, but I worry it would generate thousands of new columns, and slow down computing time substantially.
\n\nOPTION 3: This makes a lot of sense to me, but it seems like you would run into computation time problem. I'd be interested in seeing what your results are with this strategy.
", "votes": null, "replies": [] }, { "id": 125084, "author_name": "allenmc", "author_url": "", "post_date": "06/25/2016 14:49:39", "content": "Quite insightful!
", "votes": null, "replies": [] }, { "id": 125103, "author_name": "kuanchen", "author_url": "", "post_date": "06/26/2016 05:09:55", "content": "@TenaciousTerrier
\n\nI have tried option 3 that using clusters instead of grid. I use only position information(x,y) to train 400 clusters. It cost me about one day to run on AWS EC2 8cpu 32G RAM machine. But I didn't see significant improvement on my CV score.
", "votes": null, "replies": [] }, { "id": 125129, "author_name": "rshekhar2", "author_url": "", "post_date": "06/26/2016 15:59:01", "content": "@TenaciousTerrier, I see that we're having a lot of the same ideas. I also used xgboost feature weights as distance weights in kNN, applying a bayesian tuning of the weights afterward. Just as an update
\n\nI implemented a voting classifier, and while it improves results, it's not by as much as I would have hoped. Combining KNN with XGBoost gives me an increase of 0.57 to 0.585 using weighted voting. Adding more models (Random Forest, ExtraTrees, Kernel SVM) doesn't seem to be helping.
\n\nI also tried using a weighted voting classifier with xgboost on 7 grid sizes from 33x33 to 200x200. 0.57 -> 0.58 after voting. Extremely computationally intensive, not particularly helpful.
I did end up adding Nearest Neighbor distances to XGBoost (what I originally suggested as option 2). This adds about 30-50 features per cell for a 0.1 to 0.2 grid cell size, which XGBoost handles quite well. With L1 and L2 regularization in XGBoost, this improves my single classifier LB to 0.583 which is quite respectable.
My next plan is to stack or blend the models from my item 2.
", "votes": null, "replies": [] }, { "id": 125236, "author_name": "arcady27", "author_url": "", "post_date": "06/28/2016 00:48:32", "content": "I have tried to use my best KNN predictions (~0.582 on LB) as features (top-3 or top-6 predicted places ids) along with original features to build a second-level model with XGBoost. I haven't seen any improvement, moreover, a score became slightly less. I can't understand why it happened so. Does anyone try to do the same thing or maybe build a second-level model in other way which has improved the score?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "124078": "This is my first Kaggle competition, and I was wondering if I could tap into the knowledge base here to discuss how to make nearest neighbors a feature for a tree based classifier (specifically XGBoost). I was inspired by @Sandro's KNN script (https://www.kaggle.com/svpons/facebook-v-predicting-check-ins/grid-knn/discussion) which gets roughly 0.56 in my tests. Try as I might, I was unable to improve KNN beyond about 0.57, whereas I have preliminary XGBoost results of 0.59 or so. \r\n\r\n**Option 1: Voting classifier**\r\n\r\nThis would be straightforward, use a voting classifier with equal (unequal?) weights to combine the KNN classifier with XGBoost. Strictly speaking, this is not making KNN a feature, but an ensemble approach instead.\r\n\r\n**Option 2: Neighbor weights per place_id as a feature**\r\n\r\nUse something like sklearn kneighbors_graph to find the weights of the nearest k points under some metric (weighted minkowski with p=1 seems to be working well for me). This effectively adds N features per grid cell, where N is the number of categories after all cuts in that grid cell. \r\n\r\n**Option 3: Sparse coding of KNeighborsClassifier for XGBoost**\r\n\r\nTake the predicted place_ids from KNeighborsClassifier, label encode them and feed them into XGBoost. So for example, for a single row, if I have KNeighbors predict 7995458948, 8393706174, 8643187406 and my label encoder encodes them as categories 0, 2, and 4, respectively I could encode that as \r\n\r\n [ 1, 0, 0.5, 0, 0.333, 0, 0, 0, 0] \r\n\r\nassuming there were 8 predicted labels, and this would generalize to a larger sparse vector. Basically this is One Hot Encoding, but with three allowed categories, with some additional weights. \r\n\r\n\r\n----------\r\n\r\n\r\nI was wondering if the community had any thoughts on these strategies. Have you had great luck with one of these options in the past on some other competition, or with something else I didn't mention?", "125055": "After reading the forums, and trying a few things, I think my strategy going forward will be something like this....\r\n\r\n**1. Models: KNN, Random Forest, XGB:**\r\nUse a voting classifier to combine predictions of each of the 3 models, but have 9 variables because each model will generate the top 3 predictions for each row_id. Do weighted voting with the MAP@3 metric in mind, value of 1.0 for 1st prediction, 0.5 for second prediction, 0.333 for third prediction. \r\n\r\n**2. Generate models with multiple grid sizes:**\r\nBuild models trained with different grid sizes as grids separate the data arbitrarily. Include these in the final ensemble model.\r\n\r\n**3. Use clustering on raw data to build clusters to use instead of grid cells:**\r\nTrain models and include in ensembles. \r\n\r\n**4. Use feature importance ratios from Random Forest and XGBOOST models for weights in K Nearest Neighbors:** I haven't tried this yet, but it seems like it is probably how other competitors are developing their feature weight vectors.\r\n\r\n**5. Transform feature matrix and build new models:** Try Principal Components Analysis and Independent Components Analysis on training data to build new models.\r\n\r\nHere's some commentary on your proposed items: \r\n\r\n**OPTION 1:** \r\nTotally makes sense to do this. It's better if the solutions are uncorrelated.\r\n\r\n**OPTION 2:** This is interesting, but I worry it would generate thousands of new columns, and slow down computing time substantially. \r\n\r\n**OPTION 3:** This makes a lot of sense to me, but it seems like you would run into computation time problem. I'd be interested in seeing what your results are with this strategy.", "125084": "Quite insightful!", "125103": "TenaciousTerrier\r\n\r\nI have tried option 3 that using clusters instead of grid. I use only position information(x,y) to train 400 clusters. It cost me about one day to run on AWS EC2 8cpu 32G RAM machine. But I didn't see significant improvement on my CV score.", "125129": "TenaciousTerrier, I see that we're having a lot of the same ideas. I also used xgboost feature weights as distance weights in kNN, applying a bayesian tuning of the weights afterward. Just as an update\r\n\r\n1. I implemented a voting classifier, and while it improves results, it's not by as much as I would have hoped. Combining KNN with XGBoost gives me an increase of 0.57 to 0.585 using weighted voting. Adding more models (Random Forest, ExtraTrees, Kernel SVM) doesn't seem to be helping. \r\n\r\n I also tried using a weighted voting classifier with xgboost on 7 grid sizes from 33x33 to 200x200. 0.57 -> 0.58 after voting. Extremely computationally intensive, not particularly helpful.\r\n\r\n2. I did end up adding Nearest Neighbor distances to XGBoost (what I originally suggested as option 2). This adds about 30-50 features per cell for a 0.1 to 0.2 grid cell size, which XGBoost handles quite well. With L1 and L2 regularization in XGBoost, this improves my single classifier LB to 0.583 which is quite respectable.\r\n\r\nMy next plan is to stack or blend the models from my item 2.", "125236": "I have tried to use my best KNN predictions (~0.582 on LB) as features (top-3 or top-6 predicted places ids) along with original features to build a second-level model with XGBoost. I haven't seen any improvement, moreover, a score became slightly less. I can't understand why it happened so. Does anyone try to do the same thing or maybe build a second-level model in other way which has improved the score?" }, "source": "meta" }