I am now stuck because I don't know how to improve the XGB and RF results. I know this and this are good tutors for tuning parameters for XGB, and it seems there are not much we can do to improve RF except for adding n_estimators. However, the difference for this problem is that we are doing a grid search - different grid might need different parameters.
\n\nFor RF, this could be solved by making n_estimators to a large value. But it might increase the computational time.
\n\nFor XGB, things become complicated. Max_depth, n_estimators, learning_rate are the three most important parameters that would influence the result. Different from RF, the prediction of XGB will drop if n_estimators become too big. While "n_estimators=50" could be the best choice for grid 1, "n_estimators=250" could be the one for grid 99.
\n\nAlex's code inspired me if it's possible to find the best n_estimators (and other parameters) based on the sample size, number of place_id of each grid. I've did a rapid test on it but didn't find a trend yet (and it's super time consuming).
\n\nI also think of doing warm start - a way to continue the training base on previous model (correct me if I am wrong) - but this require leave out data to check whether the prediction decrease, which makes it computationally slower . And, I didn't find many sample code doing this.
\n\nI faced similar problem when I participate in the Expedia competition. Yet, I didn't find any one talking about this - tuning parameters for each grid. I assume this could be a common problem in different scenario.
\n\nSo, in short, my question is how did you choose the parameter for XGB and RF in this situation?
", "messages": [ { "id": "124713", "postDate": "06/21/2016 15:13:36", "content": "I am now stuck because I don't know how to improve the XGB and RF results. I know this and this are good tutors for tuning parameters for XGB, and it seems there are not much we can do to improve RF except for adding n_estimators. However, the difference for this problem is that we are doing a grid search - different grid might need different parameters.
\n\nFor RF, this could be solved by making n_estimators to a large value. But it might increase the computational time.
\n\nFor XGB, things become complicated. Max_depth, n_estimators, learning_rate are the three most important parameters that would influence the result. Different from RF, the prediction of XGB will drop if n_estimators become too big. While "n_estimators=50" could be the best choice for grid 1, "n_estimators=250" could be the one for grid 99.
\n\nAlex's code inspired me if it's possible to find the best n_estimators (and other parameters) based on the sample size, number of place_id of each grid. I've did a rapid test on it but didn't find a trend yet (and it's super time consuming).
\n\nI also think of doing warm start - a way to continue the training base on previous model (correct me if I am wrong) - but this require leave out data to check whether the prediction decrease, which makes it computationally slower . And, I didn't find many sample code doing this.
\n\nI faced similar problem when I participate in the Expedia competition. Yet, I didn't find any one talking about this - tuning parameters for each grid. I assume this could be a common problem in different scenario.
\n\nSo, in short, my question is how did you choose the parameter for XGB and RF in this situation?
", "rawMarkdown": "I am now stuck because I don't know how to improve the XGB and RF results. I know [this][1] and [this][2] are good tutors for tuning parameters for XGB, and it seems there are not much we can do to improve RF except for adding n_estimators. However, the difference for this problem is that we are doing a grid search - different grid might need different parameters.\r\n\r\nFor RF, this could be solved by making n_estimators to a large value. But it might increase the computational time.\r\n\r\nFor XGB, things become complicated. Max_depth, n_estimators, learning_rate are the three most important parameters that would influence the result. Different from RF, the prediction of XGB will drop if n_estimators become too big. While **\"n_estimators=50\"** could be the best choice for grid 1, **\"n_estimators=250\"** could be the one for grid 99.\r\n\r\nAlex's [code][3] inspired me if it's possible to find the best n_estimators (and other parameters) based on the sample size, number of place_id of each grid. I've did a rapid test on it but didn't find a trend yet (and it's super time consuming).\r\n\r\nI also think of doing warm start - a way to continue the training base on previous model (correct me if I am wrong) - but this require leave out data to check whether the prediction decrease, which makes it computationally slower . And, I didn't find many sample code doing this.\r\n\r\nI faced similar problem when I participate in the [Expedia][4] competition. Yet, I didn't find any one talking about this - **tuning parameters for each grid.** I assume this could be a common problem in different scenario.\r\n\r\nSo, in short, my question is how did you choose the parameter for XGB and RF in this situation?\r\n\r\n [1]: http://www.analyticsvidhya.com/blog/2016/03/complete-guide-parameter-tuning-xgboost-with-codes-python/\r\n [2]: http://datascience.stackexchange.com/questions/9364/hypertuning-xgboost-parameters\r\n [3]: https://www.kaggle.com/drarfc/facebook-v-predicting-check-ins/script-competition-facebook-v/run/273337\r\n [4]: https://www.kaggle.com/c/expedia-hotel-recommendations", "votes": null }, { "id": "124716", "postDate": "06/21/2016 15:49:03", "content": "Hi YijunMa,
\n\nI also struggled with the choice of the hyperparameters and it seemed best to me to let the computer make the decision. My solution includes a blend of different model types with various different hyperparameters. The hyperparameters for the separate model types are varied around the best hyperparameters that I established manually.
\n\nA second tier learner is then used to combine the predictions of the base models.
\n\nThe challenge is of course to calculate and evaluate the models efficiently without overfitting to the local grids.
", "rawMarkdown": "Hi YijunMa,\r\n\r\nI also struggled with the choice of the hyperparameters and it seemed best to me to let the computer make the decision. My solution includes a blend of different model types with various different hyperparameters. The hyperparameters for the separate model types are varied around the best hyperparameters that I established manually.\r\n\r\nA second tier learner is then used to combine the predictions of the base models.\r\n\r\nThe challenge is of course to calculate and evaluate the models efficiently without overfitting to the local grids.", "votes": null }, { "id": "124724", "postDate": "06/21/2016 17:50:45", "content": "@Tom Van de Wiele, I guess we are trying the similar method then. This brings other problems to me:
\n\nNo criteria to weight different models, since I didn't create a prediction for my training set due to memory problem. (I guess there are lots of people didn't do this?)
At the same time, the weight for different models could be different for different grids.
A ceiling effect - result stop increasing and sometimes drop a lot when adding new predictions to the first tier. At the same time, not only the hyperparameters, the grid size could also influence the result, there are too many things to try.
Right now, I just stop my work, cause I am not sure what's the correct direction. If anything I can do so far, I guess I would analysis the correlation between my different predictions and add as much as model for ensembling. This is my first time to do ensembling, will be good if there are any tutorials on this, besides this one.
", "rawMarkdown": "Tom Van de Wiele, I guess we are trying the similar method then. This brings other problems to me:\r\n\r\n1. No criteria to weight different models, since I didn't create a prediction for my training set due to memory problem. (I guess there are lots of people didn't do this?) \r\n\r\n2. At the same time, the weight for different models could be different for different grids.\r\n\r\n3. A ceiling effect - result stop increasing and sometimes drop a lot when adding new predictions to the first tier. At the same time, not only the hyperparameters, the grid size could also influence the result, there are too many things to try.\r\n\r\n\r\nRight now, I just stop my work, cause I am not sure what's the correct direction. If anything I can do so far, I guess I would analysis the correlation between my different predictions and add as much as model for ensembling. This is my first time to do ensembling, will be good if there are any tutorials on this, besides [this][1] one.\r\n\r\n\r\n [1]: http://mlwave.com/kaggle-ensembling-guide/", "votes": null }, { "id": "124726", "postDate": "06/21/2016 18:13:41", "content": "It is a complex problem so I would argue that you should consider to simplify everything that is not critical to the overall performance. Also, it seems that you don't have a validation set (disjoint from your training set). Having a validation set speeds up the optimisation strategy of the hyperparameters and can also be used for feature selection on the first and second tier of the ensemble.
\n\nShort answer to your three problems:
\n\nThanks for all the useful insights and help.
\n\n[quote=Tom Van de Wiele;124716]
\n\nA second tier learner is then used to combine the predictions of the base models.
\n\n[/quote]
\n\nJust a quick question, hope you don't mind answering. So far I have only used higher tier models for regression problems and classification problems with very few classes. In all of those cases I had used all of the predictions for all of the classes. In this problem this seems unwieldy, as there seems to be hundreds of different classes. Did you end up manually restricting the predictions to a finite number of classes, or did you use some kind of feature selection? Or something completely different?
\n\nThanks.
", "rawMarkdown": "Thanks for all the useful insights and help. \r\n\r\n[quote=Tom Van de Wiele;124716]\r\n\r\nA second tier learner is then used to combine the predictions of the base models.\r\n\r\n[/quote]\r\n\r\nJust a quick question, hope you don't mind answering. So far I have only used higher tier models for regression problems and classification problems with very few classes. In all of those cases I had used all of the predictions for all of the classes. In this problem this seems unwieldy, as there seems to be hundreds of different classes. Did you end up manually restricting the predictions to a finite number of classes, or did you use some kind of feature selection? Or something completely different?\r\n\r\nThanks.", "votes": null }, { "id": "124734", "postDate": "06/21/2016 20:54:56", "content": "Hi Bojan,
\n\nI hope you understand that I'll be intentionally vague at this stage of the competition. My answer would be a combination of "restricting to a limited number of top classes (top 20 for each observation)" and "something completely different" given the options you offer. My approach uses between 100 and 300 features in the different base learners and I think that all top solutions so far are using a similar main insight given the drop in leaderboard scores.
", "rawMarkdown": "Hi Bojan,\r\n\r\nI hope you understand that I'll be intentionally vague at this stage of the competition. My answer would be a combination of \"restricting to a limited number of top classes (top 20 for each observation)\" and \"something completely different\" given the options you offer. My approach uses between 100 and 300 features in the different base learners and I think that all top solutions so far are using a similar main insight given the drop in leaderboard scores.", "votes": null }, { "id": "124735", "postDate": "06/21/2016 21:00:08", "content": "Thanks Tom, no problem, I totally understand the need for vagueness. Still, your answer gives me a lot to think about. Much appreciated. :)
", "rawMarkdown": "Thanks Tom, no problem, I totally understand the need for vagueness. Still, your answer gives me a lot to think about. Much appreciated. :)", "votes": null }, { "id": "124740", "postDate": "06/21/2016 21:40:21", "content": "Hi Tom,\n100-300 features seems like a lot given the limited number of features that we start with. Are you counting after one hot encoding or did you maybe mean number of estimators?\nCan you maybe elaborate on this without revealing your secrets?
", "rawMarkdown": "Hi Tom,\r\n100-300 features seems like a lot given the limited number of features that we start with. Are you counting after one hot encoding or did you maybe mean number of estimators?\r\nCan you maybe elaborate on this without revealing your secrets?", "votes": null }, { "id": "124741", "postDate": "06/21/2016 21:57:27", "content": "Hi wilam, all I can reveal right now is that all of my selected ~300 features are numeric (so not one-hot encodings of categorical features) and that they all seem to add limited but positive value to the predictions. The top 10% of my features would result in my best submission score - 0.5% so they do most of the work. Talking about the features in detail will be delayed by a week or two but all features seem intuitive to me.
", "rawMarkdown": "Hi wilam, all I can reveal right now is that all of my selected ~300 features are numeric (so not one-hot encodings of categorical features) and that they all seem to add limited but positive value to the predictions. The top 10% of my features would result in my best submission score - 0.5% so they do most of the work. Talking about the features in detail will be delayed by a week or two but all features seem intuitive to me.", "votes": null }, { "id": "124742", "postDate": "06/21/2016 22:00:42", "content": "That is very interesting. Thank you for your quick response.
", "rawMarkdown": "That is very interesting. Thank you for your quick response.", "votes": null }, { "id": "124743", "postDate": "06/21/2016 22:03:19", "content": "I'm happy to help all of you, together we can make the world a better place :)
", "rawMarkdown": "I'm happy to help all of you, together we can make the world a better place :)", "votes": null }, { "id": "124744", "postDate": "06/21/2016 22:10:25", "content": "[quote=Tom Van de Wiele;124743]
\n\nI'm happy to help all of you, together we can make the world a better place :)
\n\n[/quote]\nIt seems very unlikely that everyone above 0.60 is using a similar approach as yours.
", "rawMarkdown": "[quote=Tom Van de Wiele;124743]\r\n\r\nI'm happy to help all of you, together we can make the world a better place :)\r\n\r\n[/quote]\r\nIt seems very unlikely that everyone above 0.60 is using a similar approach as yours.", "votes": null } ], "comments": [ { "id": 124716, "author_name": "tvdwiele", "author_url": "", "post_date": "06/21/2016 15:49:03", "content": "Hi YijunMa,
\n\nI also struggled with the choice of the hyperparameters and it seemed best to me to let the computer make the decision. My solution includes a blend of different model types with various different hyperparameters. The hyperparameters for the separate model types are varied around the best hyperparameters that I established manually.
\n\nA second tier learner is then used to combine the predictions of the base models.
\n\nThe challenge is of course to calculate and evaluate the models efficiently without overfitting to the local grids.
", "votes": null, "replies": [] }, { "id": 124724, "author_name": "ma350365879", "author_url": "", "post_date": "06/21/2016 17:50:45", "content": "@Tom Van de Wiele, I guess we are trying the similar method then. This brings other problems to me:
\n\nNo criteria to weight different models, since I didn't create a prediction for my training set due to memory problem. (I guess there are lots of people didn't do this?)
At the same time, the weight for different models could be different for different grids.
A ceiling effect - result stop increasing and sometimes drop a lot when adding new predictions to the first tier. At the same time, not only the hyperparameters, the grid size could also influence the result, there are too many things to try.
Right now, I just stop my work, cause I am not sure what's the correct direction. If anything I can do so far, I guess I would analysis the correlation between my different predictions and add as much as model for ensembling. This is my first time to do ensembling, will be good if there are any tutorials on this, besides this one.
", "votes": null, "replies": [] }, { "id": 124726, "author_name": "tvdwiele", "author_url": "", "post_date": "06/21/2016 18:13:41", "content": "It is a complex problem so I would argue that you should consider to simplify everything that is not critical to the overall performance. Also, it seems that you don't have a validation set (disjoint from your training set). Having a validation set speeds up the optimisation strategy of the hyperparameters and can also be used for feature selection on the first and second tier of the ensemble.
\n\nShort answer to your three problems:
\n\nThanks for all the useful insights and help.
\n\n[quote=Tom Van de Wiele;124716]
\n\nA second tier learner is then used to combine the predictions of the base models.
\n\n[/quote]
\n\nJust a quick question, hope you don't mind answering. So far I have only used higher tier models for regression problems and classification problems with very few classes. In all of those cases I had used all of the predictions for all of the classes. In this problem this seems unwieldy, as there seems to be hundreds of different classes. Did you end up manually restricting the predictions to a finite number of classes, or did you use some kind of feature selection? Or something completely different?
\n\nThanks.
", "votes": null, "replies": [] }, { "id": 124734, "author_name": "tvdwiele", "author_url": "", "post_date": "06/21/2016 20:54:56", "content": "Hi Bojan,
\n\nI hope you understand that I'll be intentionally vague at this stage of the competition. My answer would be a combination of "restricting to a limited number of top classes (top 20 for each observation)" and "something completely different" given the options you offer. My approach uses between 100 and 300 features in the different base learners and I think that all top solutions so far are using a similar main insight given the drop in leaderboard scores.
", "votes": null, "replies": [] }, { "id": 124735, "author_name": "tunguz", "author_url": "", "post_date": "06/21/2016 21:00:08", "content": "Thanks Tom, no problem, I totally understand the need for vagueness. Still, your answer gives me a lot to think about. Much appreciated. :)
", "votes": null, "replies": [] }, { "id": 124740, "author_name": "mwilam", "author_url": "", "post_date": "06/21/2016 21:40:21", "content": "Hi Tom,\n100-300 features seems like a lot given the limited number of features that we start with. Are you counting after one hot encoding or did you maybe mean number of estimators?\nCan you maybe elaborate on this without revealing your secrets?
", "votes": null, "replies": [] }, { "id": 124741, "author_name": "tvdwiele", "author_url": "", "post_date": "06/21/2016 21:57:27", "content": "Hi wilam, all I can reveal right now is that all of my selected ~300 features are numeric (so not one-hot encodings of categorical features) and that they all seem to add limited but positive value to the predictions. The top 10% of my features would result in my best submission score - 0.5% so they do most of the work. Talking about the features in detail will be delayed by a week or two but all features seem intuitive to me.
", "votes": null, "replies": [] }, { "id": 124742, "author_name": "mwilam", "author_url": "", "post_date": "06/21/2016 22:00:42", "content": "That is very interesting. Thank you for your quick response.
", "votes": null, "replies": [] }, { "id": 124743, "author_name": "tvdwiele", "author_url": "", "post_date": "06/21/2016 22:03:19", "content": "I'm happy to help all of you, together we can make the world a better place :)
", "votes": null, "replies": [] }, { "id": 124744, "author_name": "ascending", "author_url": "", "post_date": "06/21/2016 22:10:25", "content": "[quote=Tom Van de Wiele;124743]
\n\nI'm happy to help all of you, together we can make the world a better place :)
\n\n[/quote]\nIt seems very unlikely that everyone above 0.60 is using a similar approach as yours.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "124713": "I am now stuck because I don't know how to improve the XGB and RF results. I know [this][1] and [this][2] are good tutors for tuning parameters for XGB, and it seems there are not much we can do to improve RF except for adding n_estimators. However, the difference for this problem is that we are doing a grid search - different grid might need different parameters.\r\n\r\nFor RF, this could be solved by making n_estimators to a large value. But it might increase the computational time.\r\n\r\nFor XGB, things become complicated. Max_depth, n_estimators, learning_rate are the three most important parameters that would influence the result. Different from RF, the prediction of XGB will drop if n_estimators become too big. While **\"n_estimators=50\"** could be the best choice for grid 1, **\"n_estimators=250\"** could be the one for grid 99.\r\n\r\nAlex's [code][3] inspired me if it's possible to find the best n_estimators (and other parameters) based on the sample size, number of place_id of each grid. I've did a rapid test on it but didn't find a trend yet (and it's super time consuming).\r\n\r\nI also think of doing warm start - a way to continue the training base on previous model (correct me if I am wrong) - but this require leave out data to check whether the prediction decrease, which makes it computationally slower . And, I didn't find many sample code doing this.\r\n\r\nI faced similar problem when I participate in the [Expedia][4] competition. Yet, I didn't find any one talking about this - **tuning parameters for each grid.** I assume this could be a common problem in different scenario.\r\n\r\nSo, in short, my question is how did you choose the parameter for XGB and RF in this situation?\r\n\r\n [1]: http://www.analyticsvidhya.com/blog/2016/03/complete-guide-parameter-tuning-xgboost-with-codes-python/\r\n [2]: http://datascience.stackexchange.com/questions/9364/hypertuning-xgboost-parameters\r\n [3]: https://www.kaggle.com/drarfc/facebook-v-predicting-check-ins/script-competition-facebook-v/run/273337\r\n [4]: https://www.kaggle.com/c/expedia-hotel-recommendations", "124716": "Hi YijunMa,\r\n\r\nI also struggled with the choice of the hyperparameters and it seemed best to me to let the computer make the decision. My solution includes a blend of different model types with various different hyperparameters. The hyperparameters for the separate model types are varied around the best hyperparameters that I established manually.\r\n\r\nA second tier learner is then used to combine the predictions of the base models.\r\n\r\nThe challenge is of course to calculate and evaluate the models efficiently without overfitting to the local grids.", "124724": "Tom Van de Wiele, I guess we are trying the similar method then. This brings other problems to me:\r\n\r\n1. No criteria to weight different models, since I didn't create a prediction for my training set due to memory problem. (I guess there are lots of people didn't do this?) \r\n\r\n2. At the same time, the weight for different models could be different for different grids.\r\n\r\n3. A ceiling effect - result stop increasing and sometimes drop a lot when adding new predictions to the first tier. At the same time, not only the hyperparameters, the grid size could also influence the result, there are too many things to try.\r\n\r\n\r\nRight now, I just stop my work, cause I am not sure what's the correct direction. If anything I can do so far, I guess I would analysis the correlation between my different predictions and add as much as model for ensembling. This is my first time to do ensembling, will be good if there are any tutorials on this, besides [this][1] one.\r\n\r\n\r\n [1]: http://mlwave.com/kaggle-ensembling-guide/", "124726": "It is a complex problem so I would argue that you should consider to simplify everything that is not critical to the overall performance. Also, it seems that you don't have a validation set (disjoint from your training set). Having a validation set speeds up the optimisation strategy of the hyperparameters and can also be used for feature selection on the first and second tier of the ensemble.\r\n\r\nShort answer to your three problems:\r\n\r\n - Construct a validation set and use it to weigh the different base models.\r\n - Test if the weights are different in different grids. Drop this concern unless there is a clear spatial pattern.\r\n - Use the validation set to perform feature selection of the ensemble models", "124728": "Thanks for all the useful insights and help. \r\n\r\n[quote=Tom Van de Wiele;124716]\r\n\r\nA second tier learner is then used to combine the predictions of the base models.\r\n\r\n[/quote]\r\n\r\nJust a quick question, hope you don't mind answering. So far I have only used higher tier models for regression problems and classification problems with very few classes. In all of those cases I had used all of the predictions for all of the classes. In this problem this seems unwieldy, as there seems to be hundreds of different classes. Did you end up manually restricting the predictions to a finite number of classes, or did you use some kind of feature selection? Or something completely different?\r\n\r\nThanks.", "124734": "Hi Bojan,\r\n\r\nI hope you understand that I'll be intentionally vague at this stage of the competition. My answer would be a combination of \"restricting to a limited number of top classes (top 20 for each observation)\" and \"something completely different\" given the options you offer. My approach uses between 100 and 300 features in the different base learners and I think that all top solutions so far are using a similar main insight given the drop in leaderboard scores.", "124735": "Thanks Tom, no problem, I totally understand the need for vagueness. Still, your answer gives me a lot to think about. Much appreciated. :)", "124740": "Hi Tom,\r\n100-300 features seems like a lot given the limited number of features that we start with. Are you counting after one hot encoding or did you maybe mean number of estimators?\r\nCan you maybe elaborate on this without revealing your secrets?", "124741": "Hi wilam, all I can reveal right now is that all of my selected ~300 features are numeric (so not one-hot encodings of categorical features) and that they all seem to add limited but positive value to the predictions. The top 10% of my features would result in my best submission score - 0.5% so they do most of the work. Talking about the features in detail will be delayed by a week or two but all features seem intuitive to me.", "124742": "That is very interesting. Thank you for your quick response.", "124743": "I'm happy to help all of you, together we can make the world a better place :)", "124744": "[quote=Tom Van de Wiele;124743]\r\n\r\nI'm happy to help all of you, together we can make the world a better place :)\r\n\r\n[/quote]\r\nIt seems very unlikely that everyone above 0.60 is using a similar approach as yours." }, "source": "meta" }