This is my first competition on Kaggle. I can surely say that one Kaggle competition can teach you a lot more than all data science tutorials out there.
\n\nMy last score was 0.82, I was manually tuning XGBoost hyperparameters. \nThen I searched for some hyper-parameter tuning methods, found GridSearchCV, RandomizedSearchCV and Bayesian Optimization.
\n\nUsed GridSearchCV, left my laptop running for a night, no considerable improvement.\nUsed Hyperopt(just 20 iterations), it too takes a lot of time, but it significantly improves the model.\nCurrent Score: 0.598
\n\nIf there is some other better way for tuning, please share :)\nMy goal it to somehow make it to the top 10%. A lot to learn. :)
", "messages": [ { "id": "605067", "postDate": "08/22/2019 02:55:25", "content": "This is my first competition on Kaggle. I can surely say that one Kaggle competition can teach you a lot more than all data science tutorials out there.
\n\nMy last score was 0.82, I was manually tuning XGBoost hyperparameters. \nThen I searched for some hyper-parameter tuning methods, found GridSearchCV, RandomizedSearchCV and Bayesian Optimization.
\n\nUsed GridSearchCV, left my laptop running for a night, no considerable improvement.\nUsed Hyperopt(just 20 iterations), it too takes a lot of time, but it significantly improves the model.\nCurrent Score: 0.598
\n\nIf there is some other better way for tuning, please share :)\nMy goal it to somehow make it to the top 10%. A lot to learn. :)
", "rawMarkdown": "This is my first competition on Kaggle. I can surely say that one Kaggle competition can teach you a lot more than all data science tutorials out there. \n\nMy last score was 0.82, I was manually tuning XGBoost hyperparameters. \nThen I searched for some hyper-parameter tuning methods, found GridSearchCV, RandomizedSearchCV and Bayesian Optimization. \n\nUsed GridSearchCV, left my laptop running for a night, no considerable improvement.\nUsed Hyperopt(just 20 iterations), it too takes a lot of time, but it significantly improves the model.\nCurrent Score: 0.598\n\nIf there is some other better way for tuning, please share :)\nMy goal it to somehow make it to the top 10%. A lot to learn. :)", "votes": null }, { "id": "605425", "postDate": "08/22/2019 11:18:51", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!", "votes": null }, { "id": "605454", "postDate": "08/22/2019 11:38:12", "content": "WOW! @harshitt21 could you share the tuto where you read about Hyperopt? I'm doing another comp, and I'm using Bayesian opt, but it's too slow!!! And I would really appreciate to speed up my hyperparameters search, thanks!
", "rawMarkdown": "WOW! @harshitt21 could you share the tuto where you read about Hyperopt? I'm doing another comp, and I'm using Bayesian opt, but it's too slow!!! And I would really appreciate to speed up my hyperparameters search, thanks!", "votes": null }, { "id": "605459", "postDate": "08/22/2019 11:45:34", "content": "I prefer Optuna over Hyperopt.
", "rawMarkdown": "I prefer [Optuna ](https://github.com/pfnet/optuna) over Hyperopt.", "votes": null }, { "id": "605462", "postDate": "08/22/2019 11:50:15", "content": "I like Osprey myself, since it parallelizes well over multiple machines (but I guess most people don't have access to that).
", "rawMarkdown": "I like [Osprey](https://github.com/msmbuilder/osprey) myself, since it parallelizes well over multiple machines (but I guess most people don't have access to that).", "votes": null }, { "id": "605507", "postDate": "08/22/2019 12:44:54", "content": "I first came to know about Hyperopt in this Coursera course : https://www.coursera.org/learn/competitive-data-science/\nTutorial for implementation : https://towardsdatascience.com/an-example-of-hyperparameter-optimization-on-xgboost-lightgbm-and-catboost-using-hyperopt-12bc41a271e
\n\nI used Hyperopt on Google Colab's GPU, It was slow on my machine too. Just upload the data on colab and use Hyperopt there.
", "rawMarkdown": "I first came to know about Hyperopt in this Coursera course : https://www.coursera.org/learn/competitive-data-science/\nTutorial for implementation : https://towardsdatascience.com/an-example-of-hyperparameter-optimization-on-xgboost-lightgbm-and-catboost-using-hyperopt-12bc41a271e\n\nI used Hyperopt on Google Colab's GPU, It was slow on my machine too. Just upload the data on colab and use Hyperopt there.", "votes": null }, { "id": "605530", "postDate": "08/22/2019 13:11:54", "content": "I do manual tuning...
", "rawMarkdown": "I do manual tuning...", "votes": null }, { "id": "605536", "postDate": "08/22/2019 13:16:58", "content": "I'll try it. Thanks for sharing.
", "rawMarkdown": "I'll try it. Thanks for sharing.", "votes": null }, { "id": "605538", "postDate": "08/22/2019 13:21:02", "content": "I will see about that, once I reach your level of experience.
", "rawMarkdown": "I will see about that, once I reach your level of experience.", "votes": null }, { "id": "605542", "postDate": "08/22/2019 13:22:01", "content": "Yeah. I have a single machine like most people.
", "rawMarkdown": "Yeah. I have a single machine like most people.", "votes": null }, { "id": "605549", "postDate": "08/22/2019 13:33:00", "content": "Issue is you won't get that experience if you rely on automated tools.
", "rawMarkdown": "Issue is you won't get that experience if you rely on automated tools.", "votes": null }, { "id": "605552", "postDate": "08/22/2019 13:38:40", "content": "More generally, I think that people spend way too much time on HPO compared to data exploration, reading what others did, and feature engineering or NN architecture design. Sure HPO buys you some improvement, but nothing comparable to any of the above.
", "rawMarkdown": "More generally, I think that people spend way too much time on HPO compared to data exploration, reading what others did, and feature engineering or NN architecture design. Sure HPO buys you some improvement, but nothing comparable to any of the above.", "votes": null }, { "id": "605580", "postDate": "08/22/2019 14:06:07", "content": "This is correct. I was doing manual tuning at first, I learnt a lot about how changing the values of some parameters changes the fitting of the model. Then, I found Hyperopt.
\n\nI understand now. Thank you for this valuable advise. :)
", "rawMarkdown": "This is correct. I was doing manual tuning at first, I learnt a lot about how changing the values of some parameters changes the fitting of the model. Then, I found Hyperopt.\n\nI understand now. Thank you for this valuable advise. :)", "votes": null }, { "id": "605692", "postDate": "08/22/2019 16:13:35", "content": "You could probably do better with hyperopt then :P If computationally possible, I also like using hyperopt. I don't see much reason to manually tune certain specific xgboost parameters for example, specifically not if you have a solid CV setup. Experience and knowledge helps you to narrow the scope of the search though.
", "rawMarkdown": "You could probably do better with hyperopt then :P If computationally possible, I also like using hyperopt. I don't see much reason to manually tune certain specific xgboost parameters for example, specifically not if you have a solid CV setup. Experience and knowledge helps you to narrow the scope of the search though.", "votes": null }, { "id": "605900", "postDate": "08/22/2019 23:39:43", "content": "My GBDT (like xgboost, lightgbm, catboost) tuning process is
\n\n1.Decide how many iterations.\nHow many iterations you can run? If you use kernel(Notebooks?), running time limit is 9h. So, maybe 4k~6k iterations can be run each type with 3 folds. If you use other, there's no running time limit. Maybe around 5k iteration is enough.
\n\n2.Decide learning rate.\nLearning rate in GBDT is not the tuning parameter like DNN. Smaller learning rate will get better score, but need more iterations (running time). Usually, use 0.1 while making features and searching the importance, and 0.01 to make submission models. But, in this competition, 0.01 is not good because needed too many iterations (maybe 50k or more with some types).\nI decide learning rate 0.2~0.01 to get early stopping in the last 80~90% iterations.
\n\n3.Tuning other params.\nMax of leaves, Max bins, Min data in leaf (sorry these params are lightgbm) are important params.\nLarge max of leaves or max bins lead (over)fitting data.\nLarge min data in leaf lead underfitting(avoid overfitting) data.
", "rawMarkdown": "My GBDT (like xgboost, lightgbm, catboost) tuning process is \n\n1.Decide how many iterations.\nHow many iterations you can run? If you use kernel(Notebooks?), running time limit is 9h. So, maybe 4k~6k iterations can be run each type with 3 folds. If you use other, there's no running time limit. Maybe around 5k iteration is enough.\n\n2.Decide learning rate.\nLearning rate in GBDT is not the tuning parameter like DNN. Smaller learning rate will get better score, but need more iterations (running time). Usually, use 0.1 while making features and searching the importance, and 0.01 to make submission models. But, in this competition, 0.01 is not good because needed too many iterations (maybe 50k or more with some types).\nI decide learning rate 0.2~0.01 to get early stopping in the last 80~90% iterations.\n\n3.Tuning other params.\nMax of leaves, Max bins, Min data in leaf (sorry these params are lightgbm) are important params.\nLarge max of leaves or max bins lead (over)fitting data.\nLarge min data in leaf lead underfitting(avoid overfitting) data.", "votes": null }, { "id": "606030", "postDate": "08/23/2019 05:23:38", "content": "\n\n\nif you have a solid CV setup.
\n
That's the reason why I prefer manual tuning. When you have enough data, and CV is very reliable, then indeed there is no downside to automated HPO as you rightfully state. But in many cases, including many competitions here, we don't have enough data, and automated HPO is the best way I know to overfit.
", "rawMarkdown": "> if you have a solid CV setup. \n\nThat's the reason why I prefer manual tuning. When you have enough data, and CV is very reliable, then indeed there is no downside to automated HPO as you rightfully state. But in many cases, including many competitions here, we don't have enough data, and automated HPO is the best way I know to overfit.", "votes": null }, { "id": "606339", "postDate": "08/23/2019 13:09:49", "content": "Manual tuning has the same issue though.
", "rawMarkdown": "Manual tuning has the same issue though.", "votes": null }, { "id": "606357", "postDate": "08/23/2019 13:32:57", "content": "Same issue than what?
\n\nWith manual tuning I stop earlier than with automated tuning.
", "rawMarkdown": "Same issue than what?\n\nWith manual tuning I stop earlier than with automated tuning.", "votes": null }, { "id": "607177", "postDate": "08/24/2019 18:04:13", "content": "You said in this competition, too many iterations are needed. \nI generally start with few iterations, then play with the parameters to see what works.\nHow to decide the number of iterations based on the size of data ?
", "rawMarkdown": "You said in this competition, too many iterations are needed. \nI generally start with few iterations, then play with the parameters to see what works.\nHow to decide the number of iterations based on the size of data ?", "votes": null }, { "id": "607231", "postDate": "08/24/2019 21:15:54", "content": "I have a kernel for hyperparameters optimization, I run it for eight hours and then get best hyperparameters. I see how many iterations it computed, divide it by number of folds and by how many times more data there are than I used for hyperoptimization. It is like first guess. Then I also let LGBM train for lets say 500 rounds and get time. When you know how long an iteration takes and you know maximal available time, you can compute number of iterations.
", "rawMarkdown": "I have a kernel for hyperparameters optimization, I run it for eight hours and then get best hyperparameters. I see how many iterations it computed, divide it by number of folds and by how many times more data there are than I used for hyperoptimization. It is like first guess. Then I also let LGBM train for lets say 500 rounds and get time. When you know how long an iteration takes and you know maximal available time, you can compute number of iterations.", "votes": null }, { "id": "607320", "postDate": "08/25/2019 02:57:48", "content": "What I do is the same of above's . And adding some details.\nI run 10k iters model on kernel and see the running time (from the version). Kernel can run 9h (=32400sec), so I calculate about how many iters can run in the model.\nTuning hypo model's number of iters are around 250~500. It's good to run models 4^(number of tuning hypo)~5^(num of ...) times to search hypo. Final submission model's number of iteration based on running time. I'm not sure about how to decide the num of iters based on the size of data, because data is very difficult in the tasks.
", "rawMarkdown": "What I do is the same of above's . And adding some details.\nI run 10k iters model on kernel and see the running time (from the version). Kernel can run 9h (=32400sec), so I calculate about how many iters can run in the model.\nTuning hypo model's number of iters are around 250~500. It's good to run models 4^(number of tuning hypo)~5^(num of ...) times to search hypo. Final submission model's number of iteration based on running time. I'm not sure about how to decide the num of iters based on the size of data, because data is very difficult in the tasks.", "votes": null }, { "id": "607636", "postDate": "08/25/2019 16:24:45", "content": "In my opinion, feature engineering is more important than parameters and model. I only used Random Forester. Without feature engineering I got 1.4 at first, and later with feature engineering, I improved to 0.4, and then -0.9 till now -1.3. Parameters only improve my scores -1.3 to -1.36. I tried XGBoost too, but didn't get as good results as Random Forester. It seems those with high scores use LGB. I am not sure, you can try. But focusing on own improvement is important, at least I learned a lot from the process.
", "rawMarkdown": "In my opinion, feature engineering is more important than parameters and model. I only used Random Forester. Without feature engineering I got 1.4 at first, and later with feature engineering, I improved to 0.4, and then -0.9 till now -1.3. Parameters only improve my scores -1.3 to -1.36. I tried XGBoost too, but didn't get as good results as Random Forester. It seems those with high scores use LGB. I am not sure, you can try. But focusing on own improvement is important, at least I learned a lot from the process.", "votes": null }, { "id": "607778", "postDate": "08/25/2019 21:22:55", "content": "\n\n\nI do manual tuning…
\n
It's called machine learning for a reason. ;-)
", "rawMarkdown": "> I do manual tuning…\n\nIt's called *machine* learning for a reason. ;-)", "votes": null }, { "id": "607789", "postDate": "08/25/2019 21:54:00", "content": "The one learning in Kaggle is actually me, not my algorithms ;)
", "rawMarkdown": "The one learning in Kaggle is actually me, not my algorithms ;)", "votes": null }, { "id": "607864", "postDate": "08/26/2019 02:53:29", "content": "To add to his discussion, an alternative way would be to have from time import time and then do your search iteratively (maybe an infinite while loop) and check runtime at the beginning of each loop until you reach about 32400 seconds (leave some room too) and then break the loop. This way you will get the maximum number of iterations possible. Not to mention that sometimes iterations take longer than what you expect.
@inversion it is called machine learning, but in Kaggle you can see that most of the times the winner teams make a difference because they do a lot of analysis and they understand the problem deeply. Human intelligence still plays a key role.
", "rawMarkdown": "inversion it is called machine learning, but in Kaggle you can see that most of the times the winner teams make a difference because they do a lot of analysis and they understand the problem deeply. Human intelligence still plays a key role.", "votes": null } ], "comments": [ { "id": 605425, "author_name": "suriender", "author_url": "", "post_date": "08/22/2019 11:18:51", "content": "Thanks for sharing!
", "votes": null, "replies": [] }, { "id": 605454, "author_name": "brunhs", "author_url": "", "post_date": "08/22/2019 11:38:12", "content": "WOW! @harshitt21 could you share the tuto where you read about Hyperopt? I'm doing another comp, and I'm using Bayesian opt, but it's too slow!!! And I would really appreciate to speed up my hyperparameters search, thanks!
", "votes": null, "replies": [ { "id": 605507, "author_name": "harshitt21", "author_url": "", "post_date": "08/22/2019 12:44:54", "content": "I first came to know about Hyperopt in this Coursera course : https://www.coursera.org/learn/competitive-data-science/\nTutorial for implementation : https://towardsdatascience.com/an-example-of-hyperparameter-optimization-on-xgboost-lightgbm-and-catboost-using-hyperopt-12bc41a271e
\n\nI used Hyperopt on Google Colab's GPU, It was slow on my machine too. Just upload the data on colab and use Hyperopt there.
", "votes": null, "replies": [] } ] }, { "id": 605459, "author_name": "tayorm", "author_url": "", "post_date": "08/22/2019 11:45:34", "content": "I prefer Optuna over Hyperopt.
", "votes": null, "replies": [ { "id": 605536, "author_name": "harshitt21", "author_url": "", "post_date": "08/22/2019 13:16:58", "content": "I'll try it. Thanks for sharing.
", "votes": null, "replies": [] } ] }, { "id": 605462, "author_name": "larsbratholm", "author_url": "", "post_date": "08/22/2019 11:50:15", "content": "I like Osprey myself, since it parallelizes well over multiple machines (but I guess most people don't have access to that).
", "votes": null, "replies": [ { "id": 605542, "author_name": "harshitt21", "author_url": "", "post_date": "08/22/2019 13:22:01", "content": "Yeah. I have a single machine like most people.
", "votes": null, "replies": [] } ] }, { "id": 605530, "author_name": "cpmpml", "author_url": "", "post_date": "08/22/2019 13:11:54", "content": "I do manual tuning...
", "votes": null, "replies": [ { "id": 605538, "author_name": "harshitt21", "author_url": "", "post_date": "08/22/2019 13:21:02", "content": "I will see about that, once I reach your level of experience.
", "votes": null, "replies": [] }, { "id": 605549, "author_name": "cpmpml", "author_url": "", "post_date": "08/22/2019 13:33:00", "content": "Issue is you won't get that experience if you rely on automated tools.
", "votes": null, "replies": [] }, { "id": 605552, "author_name": "cpmpml", "author_url": "", "post_date": "08/22/2019 13:38:40", "content": "More generally, I think that people spend way too much time on HPO compared to data exploration, reading what others did, and feature engineering or NN architecture design. Sure HPO buys you some improvement, but nothing comparable to any of the above.
", "votes": null, "replies": [] }, { "id": 605580, "author_name": "harshitt21", "author_url": "", "post_date": "08/22/2019 14:06:07", "content": "This is correct. I was doing manual tuning at first, I learnt a lot about how changing the values of some parameters changes the fitting of the model. Then, I found Hyperopt.
\n\nI understand now. Thank you for this valuable advise. :)
", "votes": null, "replies": [] }, { "id": 605692, "author_name": "philippsinger", "author_url": "", "post_date": "08/22/2019 16:13:35", "content": "You could probably do better with hyperopt then :P If computationally possible, I also like using hyperopt. I don't see much reason to manually tune certain specific xgboost parameters for example, specifically not if you have a solid CV setup. Experience and knowledge helps you to narrow the scope of the search though.
", "votes": null, "replies": [] }, { "id": 606030, "author_name": "cpmpml", "author_url": "", "post_date": "08/23/2019 05:23:38", "content": "\n\n\nif you have a solid CV setup.
\n
That's the reason why I prefer manual tuning. When you have enough data, and CV is very reliable, then indeed there is no downside to automated HPO as you rightfully state. But in many cases, including many competitions here, we don't have enough data, and automated HPO is the best way I know to overfit.
", "votes": null, "replies": [] }, { "id": 606339, "author_name": "philippsinger", "author_url": "", "post_date": "08/23/2019 13:09:49", "content": "Manual tuning has the same issue though.
", "votes": null, "replies": [] }, { "id": 606357, "author_name": "cpmpml", "author_url": "", "post_date": "08/23/2019 13:32:57", "content": "Same issue than what?
\n\nWith manual tuning I stop earlier than with automated tuning.
", "votes": null, "replies": [] }, { "id": 607778, "author_name": "inversion", "author_url": "", "post_date": "08/25/2019 21:22:55", "content": "\n\n\nI do manual tuning…
\n
It's called machine learning for a reason. ;-)
", "votes": null, "replies": [] }, { "id": 607789, "author_name": "cpmpml", "author_url": "", "post_date": "08/25/2019 21:54:00", "content": "The one learning in Kaggle is actually me, not my algorithms ;)
", "votes": null, "replies": [] }, { "id": 608286, "author_name": "carlospk", "author_url": "", "post_date": "08/26/2019 15:39:36", "content": "@inversion it is called machine learning, but in Kaggle you can see that most of the times the winner teams make a difference because they do a lot of analysis and they understand the problem deeply. Human intelligence still plays a key role.
", "votes": null, "replies": [] } ] }, { "id": 605900, "author_name": "jpn888", "author_url": "", "post_date": "08/22/2019 23:39:43", "content": "My GBDT (like xgboost, lightgbm, catboost) tuning process is
\n\n1.Decide how many iterations.\nHow many iterations you can run? If you use kernel(Notebooks?), running time limit is 9h. So, maybe 4k~6k iterations can be run each type with 3 folds. If you use other, there's no running time limit. Maybe around 5k iteration is enough.
\n\n2.Decide learning rate.\nLearning rate in GBDT is not the tuning parameter like DNN. Smaller learning rate will get better score, but need more iterations (running time). Usually, use 0.1 while making features and searching the importance, and 0.01 to make submission models. But, in this competition, 0.01 is not good because needed too many iterations (maybe 50k or more with some types).\nI decide learning rate 0.2~0.01 to get early stopping in the last 80~90% iterations.
\n\n3.Tuning other params.\nMax of leaves, Max bins, Min data in leaf (sorry these params are lightgbm) are important params.\nLarge max of leaves or max bins lead (over)fitting data.\nLarge min data in leaf lead underfitting(avoid overfitting) data.
", "votes": null, "replies": [ { "id": 607177, "author_name": "harshitt21", "author_url": "", "post_date": "08/24/2019 18:04:13", "content": "You said in this competition, too many iterations are needed. \nI generally start with few iterations, then play with the parameters to see what works.\nHow to decide the number of iterations based on the size of data ?
", "votes": null, "replies": [] }, { "id": 607231, "author_name": "somnambwl", "author_url": "", "post_date": "08/24/2019 21:15:54", "content": "I have a kernel for hyperparameters optimization, I run it for eight hours and then get best hyperparameters. I see how many iterations it computed, divide it by number of folds and by how many times more data there are than I used for hyperoptimization. It is like first guess. Then I also let LGBM train for lets say 500 rounds and get time. When you know how long an iteration takes and you know maximal available time, you can compute number of iterations.
", "votes": null, "replies": [] }, { "id": 607320, "author_name": "jpn888", "author_url": "", "post_date": "08/25/2019 02:57:48", "content": "What I do is the same of above's . And adding some details.\nI run 10k iters model on kernel and see the running time (from the version). Kernel can run 9h (=32400sec), so I calculate about how many iters can run in the model.\nTuning hypo model's number of iters are around 250~500. It's good to run models 4^(number of tuning hypo)~5^(num of ...) times to search hypo. Final submission model's number of iteration based on running time. I'm not sure about how to decide the num of iters based on the size of data, because data is very difficult in the tasks.
", "votes": null, "replies": [] }, { "id": 607864, "author_name": "mhviraf", "author_url": "", "post_date": "08/26/2019 02:53:29", "content": "To add to his discussion, an alternative way would be to have from time import time and then do your search iteratively (maybe an infinite while loop) and check runtime at the beginning of each loop until you reach about 32400 seconds (leave some room too) and then break the loop. This way you will get the maximum number of iterations possible. Not to mention that sometimes iterations take longer than what you expect.
In my opinion, feature engineering is more important than parameters and model. I only used Random Forester. Without feature engineering I got 1.4 at first, and later with feature engineering, I improved to 0.4, and then -0.9 till now -1.3. Parameters only improve my scores -1.3 to -1.36. I tried XGBoost too, but didn't get as good results as Random Forester. It seems those with high scores use LGB. I am not sure, you can try. But focusing on own improvement is important, at least I learned a lot from the process.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "605067": "This is my first competition on Kaggle. I can surely say that one Kaggle competition can teach you a lot more than all data science tutorials out there. \n\nMy last score was 0.82, I was manually tuning XGBoost hyperparameters. \nThen I searched for some hyper-parameter tuning methods, found GridSearchCV, RandomizedSearchCV and Bayesian Optimization. \n\nUsed GridSearchCV, left my laptop running for a night, no considerable improvement.\nUsed Hyperopt(just 20 iterations), it too takes a lot of time, but it significantly improves the model.\nCurrent Score: 0.598\n\nIf there is some other better way for tuning, please share :)\nMy goal it to somehow make it to the top 10%. A lot to learn. :)", "605425": "Thanks for sharing!", "605454": "WOW! @harshitt21 could you share the tuto where you read about Hyperopt? I'm doing another comp, and I'm using Bayesian opt, but it's too slow!!! And I would really appreciate to speed up my hyperparameters search, thanks!", "605459": "I prefer [Optuna ](https://github.com/pfnet/optuna) over Hyperopt.", "605462": "I like [Osprey](https://github.com/msmbuilder/osprey) myself, since it parallelizes well over multiple machines (but I guess most people don't have access to that).", "605507": "I first came to know about Hyperopt in this Coursera course : https://www.coursera.org/learn/competitive-data-science/\nTutorial for implementation : https://towardsdatascience.com/an-example-of-hyperparameter-optimization-on-xgboost-lightgbm-and-catboost-using-hyperopt-12bc41a271e\n\nI used Hyperopt on Google Colab's GPU, It was slow on my machine too. Just upload the data on colab and use Hyperopt there.", "605530": "I do manual tuning...", "605536": "I'll try it. Thanks for sharing.", "605538": "I will see about that, once I reach your level of experience.", "605542": "Yeah. I have a single machine like most people.", "605549": "Issue is you won't get that experience if you rely on automated tools.", "605552": "More generally, I think that people spend way too much time on HPO compared to data exploration, reading what others did, and feature engineering or NN architecture design. Sure HPO buys you some improvement, but nothing comparable to any of the above.", "605580": "This is correct. I was doing manual tuning at first, I learnt a lot about how changing the values of some parameters changes the fitting of the model. Then, I found Hyperopt.\n\nI understand now. Thank you for this valuable advise. :)", "605692": "You could probably do better with hyperopt then :P If computationally possible, I also like using hyperopt. I don't see much reason to manually tune certain specific xgboost parameters for example, specifically not if you have a solid CV setup. Experience and knowledge helps you to narrow the scope of the search though.", "605900": "My GBDT (like xgboost, lightgbm, catboost) tuning process is \n\n1.Decide how many iterations.\nHow many iterations you can run? If you use kernel(Notebooks?), running time limit is 9h. So, maybe 4k~6k iterations can be run each type with 3 folds. If you use other, there's no running time limit. Maybe around 5k iteration is enough.\n\n2.Decide learning rate.\nLearning rate in GBDT is not the tuning parameter like DNN. Smaller learning rate will get better score, but need more iterations (running time). Usually, use 0.1 while making features and searching the importance, and 0.01 to make submission models. But, in this competition, 0.01 is not good because needed too many iterations (maybe 50k or more with some types).\nI decide learning rate 0.2~0.01 to get early stopping in the last 80~90% iterations.\n\n3.Tuning other params.\nMax of leaves, Max bins, Min data in leaf (sorry these params are lightgbm) are important params.\nLarge max of leaves or max bins lead (over)fitting data.\nLarge min data in leaf lead underfitting(avoid overfitting) data.", "606030": "> if you have a solid CV setup. \n\nThat's the reason why I prefer manual tuning. When you have enough data, and CV is very reliable, then indeed there is no downside to automated HPO as you rightfully state. But in many cases, including many competitions here, we don't have enough data, and automated HPO is the best way I know to overfit.", "606339": "Manual tuning has the same issue though.", "606357": "Same issue than what?\n\nWith manual tuning I stop earlier than with automated tuning.", "607177": "You said in this competition, too many iterations are needed. \nI generally start with few iterations, then play with the parameters to see what works.\nHow to decide the number of iterations based on the size of data ?", "607231": "I have a kernel for hyperparameters optimization, I run it for eight hours and then get best hyperparameters. I see how many iterations it computed, divide it by number of folds and by how many times more data there are than I used for hyperoptimization. It is like first guess. Then I also let LGBM train for lets say 500 rounds and get time. When you know how long an iteration takes and you know maximal available time, you can compute number of iterations.", "607320": "What I do is the same of above's . And adding some details.\nI run 10k iters model on kernel and see the running time (from the version). Kernel can run 9h (=32400sec), so I calculate about how many iters can run in the model.\nTuning hypo model's number of iters are around 250~500. It's good to run models 4^(number of tuning hypo)~5^(num of ...) times to search hypo. Final submission model's number of iteration based on running time. I'm not sure about how to decide the num of iters based on the size of data, because data is very difficult in the tasks.", "607636": "In my opinion, feature engineering is more important than parameters and model. I only used Random Forester. Without feature engineering I got 1.4 at first, and later with feature engineering, I improved to 0.4, and then -0.9 till now -1.3. Parameters only improve my scores -1.3 to -1.36. I tried XGBoost too, but didn't get as good results as Random Forester. It seems those with high scores use LGB. I am not sure, you can try. But focusing on own improvement is important, at least I learned a lot from the process.", "607778": "> I do manual tuning…\n\nIt's called *machine* learning for a reason. ;-)", "607789": "The one learning in Kaggle is actually me, not my algorithms ;)", "607864": "To add to his discussion, an alternative way would be to have `from time import time` and then do your search iteratively (maybe an infinite while loop) and check runtime at the beginning of each loop until you reach about 32400 seconds (leave some room too) and then break the loop. This way you will get the maximum number of iterations possible. Not to mention that sometimes iterations take longer than what you expect.", "608286": "inversion it is called machine learning, but in Kaggle you can see that most of the times the winner teams make a difference because they do a lot of analysis and they understand the problem deeply. Human intelligence still plays a key role." }, "source": "meta" }