There is any reason why all the kernels are using lgbm instead of other boosting algorithms such as XGB, catb etc?
", "messages": [ { "id": "657830", "postDate": "10/25/2019 13:08:38", "content": "There is any reason why all the kernels are using lgbm instead of other boosting algorithms such as XGB, catb etc?
", "rawMarkdown": "There is any reason why all the kernels are using lgbm instead of other boosting algorithms such as XGB, catb etc?", "votes": null }, { "id": "657838", "postDate": "10/25/2019 13:19:37", "content": "The general assumption around using LightGBM vs XGBoost and CatBoost is LGBM usually converges faster as compared to others and it uses all the cores available by default. I have noticed and I guess others may have as well XGBoost and CatBoost are kind of have started to be not so popular these days. For details you can read this topic from a previous competition https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/89909#519241
I tried to use K-Folds with XGBoost, but on the second fold, my kernel fell due to lack of resources.\nLGBM are faster than XGBoost in training, and I think LGBM requires less memory and resources.\nHere is my XGBoost kernel, but I use only one fold:\nhttps://www.kaggle.com/ihelon/simple-xgboost
", "rawMarkdown": "I tried to use K-Folds with XGBoost, but on the second fold, my kernel fell due to lack of resources.\nLGBM are faster than XGBoost in training, and I think LGBM requires less memory and resources.\nHere is my XGBoost kernel, but I use only one fold:\nhttps://www.kaggle.com/ihelon/simple-xgboost", "votes": null }, { "id": "657887", "postDate": "10/25/2019 13:51:25", "content": "I like GPU options for Catboost and XGB - it's way much faster than any CPU LGBM.
\n\nBut I like LGBM more. For me it's more \"white-boxed\" model. Easier to do FE analysis. Less \"params\" dependent.
\n\nFor Catboost you have to do much more runs to confirm that \"score boost\" was because of feature and not of model tuning. Even tuning not very stable, adding features changes model behavior greatly and new tuning step required.
\n\nSo for prototyping my choice is LGBM.
\n\nBut also I know many XGB adepts and Catboost adepts.
", "rawMarkdown": "I like GPU options for Catboost and XGB - it's way much faster than any CPU LGBM.\n\nBut I like LGBM more. For me it's more \"white-boxed\" model. Easier to do FE analysis. Less \"params\" dependent. \n\nFor Catboost you have to do much more runs to confirm that \"score boost\" was because of feature and not of model tuning. Even tuning not very stable, adding features changes model behavior greatly and new tuning step required.\n\nSo for prototyping my choice is LGBM. \n\nBut also I know many XGB adepts and Catboost adepts.", "votes": null }, { "id": "657921", "postDate": "10/25/2019 14:12:33", "content": "The two aspects that jump out to me as reasons why I prioritize lightgbm over the other two options --
\n\nSpeed: lightgbm uses a histogram algorithm for splitting that's significantly faster and more memory efficient than default versions of the alternatives (xgboost does support a histogram mode though). Features are broken up into bins which are then used for making tree decision splits, meaning the data can be represented more efficiently and many fewer possible decision points need to be evaluated. This is even a form of regularization because it forces the model to be less granular in its splitting decisions.
Category handling: lightgbm has native support for categorical features that includes an optimal partitioning algorithm. In my experience this usually works well and prevents the pain of one hot encoding high cardinality categorical features, or figuring out a good numerical representation for those features.
I've admittedly not really used catboost, but my understanding is that it can be significantly slower than lightgbm, and I'm not convinced that its target encoding of categories is a better strategy than partitioning (I tend to be skeptical of target encoding in general).
", "rawMarkdown": "The two aspects that jump out to me as reasons why I prioritize lightgbm over the other two options --\n\n1. **Speed**: lightgbm uses a histogram algorithm for splitting that's significantly faster and more memory efficient than default versions of the alternatives (xgboost does support a histogram mode though). Features are broken up into bins which are then used for making tree decision splits, meaning the data can be represented more efficiently and many fewer possible decision points need to be evaluated. This is even a form of regularization because it forces the model to be less granular in its splitting decisions.\n\n2. **Category handling**: lightgbm has native support for [categorical features ](https://lightgbm.readthedocs.io/en/latest/Advanced-Topics.html) that includes an optimal partitioning algorithm. In my experience this usually works well and prevents the pain of one hot encoding high cardinality categorical features, or figuring out a good numerical representation for those features. \n\nI've admittedly not really used catboost, but my understanding is that it can be significantly slower than lightgbm, and I'm not convinced that its target encoding of categories is a better strategy than partitioning (I tend to be skeptical of target encoding in general).", "votes": null }, { "id": "657996", "postDate": "10/25/2019 15:14:56", "content": "Great explanation @aquatic .
\n\n\n\n\nsignificantly faster and more memory efficient
\n
Any thoughts about python api vs scikit-learn api for lgbm for speed and memory?
", "rawMarkdown": "Great explanation @aquatic . \n\n> significantly faster and more memory efficient\n\nAny thoughts about python api vs scikit-learn api for lgbm for speed and memory?", "votes": null }, { "id": "658000", "postDate": "10/25/2019 15:20:38", "content": "Konstantin have summed it well. I Agree with him.
\n\nThe way I would proceed for classification task is plot feature importance plot from lgbm model as the first thing even before EDA. I feel it is very crucial to know important features asap.
", "rawMarkdown": "Konstantin have summed it well. I Agree with him.\n\nThe way I would proceed for classification task is plot feature importance plot from lgbm model as the first thing even before EDA. I feel it is very crucial to know important features asap.", "votes": null }, { "id": "659027", "postDate": "10/27/2019 00:39:57", "content": "Have used lightgbm over XGB as it's always seemed a bit faster and a bit easier to tune parameters.
\n\nHave fallen in love with Catboost however in this competition. It uses my two GPU to 96% of capacity with only a single line of code needed. Running both Windows and Ubuntu machines and I never could manage to get lightgbm GPU version installed on Win 10.
", "rawMarkdown": "Have used lightgbm over XGB as it's always seemed a bit faster and a bit easier to tune parameters.\n\nHave fallen in love with Catboost however in this competition. It uses my two GPU to 96% of capacity with only a single line of code needed. Running both Windows and Ubuntu machines and I never could manage to get lightgbm GPU version installed on Win 10.", "votes": null } ], "comments": [ { "id": 657838, "author_name": "cyberia", "author_url": "", "post_date": "10/25/2019 13:19:37", "content": "The general assumption around using LightGBM vs XGBoost and CatBoost is LGBM usually converges faster as compared to others and it uses all the cores available by default. I have noticed and I guess others may have as well XGBoost and CatBoost are kind of have started to be not so popular these days. For details you can read this topic from a previous competition https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/89909#519241
I like GPU options for Catboost and XGB - it's way much faster than any CPU LGBM.
\n\nBut I like LGBM more. For me it's more \"white-boxed\" model. Easier to do FE analysis. Less \"params\" dependent.
\n\nFor Catboost you have to do much more runs to confirm that \"score boost\" was because of feature and not of model tuning. Even tuning not very stable, adding features changes model behavior greatly and new tuning step required.
\n\nSo for prototyping my choice is LGBM.
\n\nBut also I know many XGB adepts and Catboost adepts.
", "votes": null, "replies": [] }, { "id": 658000, "author_name": "cyberia", "author_url": "", "post_date": "10/25/2019 15:20:38", "content": "Konstantin have summed it well. I Agree with him.
\n\nThe way I would proceed for classification task is plot feature importance plot from lgbm model as the first thing even before EDA. I feel it is very crucial to know important features asap.
", "votes": null, "replies": [] } ] }, { "id": 657841, "author_name": "ihelon", "author_url": "", "post_date": "10/25/2019 13:21:51", "content": "I tried to use K-Folds with XGBoost, but on the second fold, my kernel fell due to lack of resources.\nLGBM are faster than XGBoost in training, and I think LGBM requires less memory and resources.\nHere is my XGBoost kernel, but I use only one fold:\nhttps://www.kaggle.com/ihelon/simple-xgboost
", "votes": null, "replies": [] }, { "id": 657921, "author_name": "aquatic", "author_url": "", "post_date": "10/25/2019 14:12:33", "content": "The two aspects that jump out to me as reasons why I prioritize lightgbm over the other two options --
\n\nSpeed: lightgbm uses a histogram algorithm for splitting that's significantly faster and more memory efficient than default versions of the alternatives (xgboost does support a histogram mode though). Features are broken up into bins which are then used for making tree decision splits, meaning the data can be represented more efficiently and many fewer possible decision points need to be evaluated. This is even a form of regularization because it forces the model to be less granular in its splitting decisions.
Category handling: lightgbm has native support for categorical features that includes an optimal partitioning algorithm. In my experience this usually works well and prevents the pain of one hot encoding high cardinality categorical features, or figuring out a good numerical representation for those features.
I've admittedly not really used catboost, but my understanding is that it can be significantly slower than lightgbm, and I'm not convinced that its target encoding of categories is a better strategy than partitioning (I tend to be skeptical of target encoding in general).
", "votes": null, "replies": [ { "id": 657996, "author_name": "cyberia", "author_url": "", "post_date": "10/25/2019 15:14:56", "content": "Great explanation @aquatic .
\n\n\n\n\nsignificantly faster and more memory efficient
\n
Any thoughts about python api vs scikit-learn api for lgbm for speed and memory?
", "votes": null, "replies": [] } ] }, { "id": 659027, "author_name": "pcjimmmy", "author_url": "", "post_date": "10/27/2019 00:39:57", "content": "Have used lightgbm over XGB as it's always seemed a bit faster and a bit easier to tune parameters.
\n\nHave fallen in love with Catboost however in this competition. It uses my two GPU to 96% of capacity with only a single line of code needed. Running both Windows and Ubuntu machines and I never could manage to get lightgbm GPU version installed on Win 10.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "657830": "There is any reason why all the kernels are using lgbm instead of other boosting algorithms such as XGB, catb etc?", "657838": "The general assumption around using LightGBM vs XGBoost and CatBoost is `LGBM usually converges faster as compared to others and it uses all the cores available by default`. I have noticed and I guess others may have as well XGBoost and CatBoost are kind of have started to be not so popular these days. For details you can read this topic from a previous competition https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/89909#519241", "657841": "I tried to use K-Folds with XGBoost, but on the second fold, my kernel fell due to lack of resources.\nLGBM are faster than XGBoost in training, and I think LGBM requires less memory and resources.\nHere is my XGBoost kernel, but I use only one fold:\nhttps://www.kaggle.com/ihelon/simple-xgboost", "657887": "I like GPU options for Catboost and XGB - it's way much faster than any CPU LGBM.\n\nBut I like LGBM more. For me it's more \"white-boxed\" model. Easier to do FE analysis. Less \"params\" dependent. \n\nFor Catboost you have to do much more runs to confirm that \"score boost\" was because of feature and not of model tuning. Even tuning not very stable, adding features changes model behavior greatly and new tuning step required.\n\nSo for prototyping my choice is LGBM. \n\nBut also I know many XGB adepts and Catboost adepts.", "657921": "The two aspects that jump out to me as reasons why I prioritize lightgbm over the other two options --\n\n1. **Speed**: lightgbm uses a histogram algorithm for splitting that's significantly faster and more memory efficient than default versions of the alternatives (xgboost does support a histogram mode though). Features are broken up into bins which are then used for making tree decision splits, meaning the data can be represented more efficiently and many fewer possible decision points need to be evaluated. This is even a form of regularization because it forces the model to be less granular in its splitting decisions.\n\n2. **Category handling**: lightgbm has native support for [categorical features ](https://lightgbm.readthedocs.io/en/latest/Advanced-Topics.html) that includes an optimal partitioning algorithm. In my experience this usually works well and prevents the pain of one hot encoding high cardinality categorical features, or figuring out a good numerical representation for those features. \n\nI've admittedly not really used catboost, but my understanding is that it can be significantly slower than lightgbm, and I'm not convinced that its target encoding of categories is a better strategy than partitioning (I tend to be skeptical of target encoding in general).", "657996": "Great explanation @aquatic . \n\n> significantly faster and more memory efficient\n\nAny thoughts about python api vs scikit-learn api for lgbm for speed and memory?", "658000": "Konstantin have summed it well. I Agree with him.\n\nThe way I would proceed for classification task is plot feature importance plot from lgbm model as the first thing even before EDA. I feel it is very crucial to know important features asap.", "659027": "Have used lightgbm over XGB as it's always seemed a bit faster and a bit easier to tune parameters.\n\nHave fallen in love with Catboost however in this competition. It uses my two GPU to 96% of capacity with only a single line of code needed. Running both Windows and Ubuntu machines and I never could manage to get lightgbm GPU version installed on Win 10." }, "source": "meta" }