Hi, there! After viewing most-voted kernels I have noticed that majority tend to use LGBM instead of simple regression models? I am new here and it's my first competition and, thus, can someone clarify why you use LGBM and it's benefits? Also, in these kernels, the baseline is right away based on LGBM without even trying regression. Does that mean that regression is old and not so effective? I have read some papers/works on LGBM and it seems quite vague right now. I would appreciate if you share some tutorial or paper that will open my eyes. Thanks in advance, and sorry if it seems like a 'stupid question from newbie'
", "messages": [ { "id": "388830", "postDate": "09/17/2018 16:58:11", "content": "Hi, there! After viewing most-voted kernels I have noticed that majority tend to use LGBM instead of simple regression models? I am new here and it's my first competition and, thus, can someone clarify why you use LGBM and it's benefits? Also, in these kernels, the baseline is right away based on LGBM without even trying regression. Does that mean that regression is old and not so effective? I have read some papers/works on LGBM and it seems quite vague right now. I would appreciate if you share some tutorial or paper that will open my eyes. Thanks in advance, and sorry if it seems like a 'stupid question from newbie'
", "rawMarkdown": "Hi, there! After viewing most-voted kernels I have noticed that majority tend to use LGBM instead of simple regression models? I am new here and it's my first competition and, thus, can someone clarify why you use LGBM and it's benefits? Also, in these kernels, the baseline is right away based on LGBM without even trying regression. Does that mean that regression is old and not so effective? I have read some papers/works on LGBM and it seems quite vague right now. I would appreciate if you share some tutorial or paper that will open my eyes. Thanks in advance, and sorry if it seems like a 'stupid question from newbie'", "votes": null }, { "id": "388848", "postDate": "09/17/2018 17:30:09", "content": "Not a stupid question at all! Personally, I tend to start with the boosting algorithms right away because they require very little data preparation work. They can handle missing values out of the box, they are very robust to all kinds of distributions among your features, they are very fast to train, and they don't even require one hot encoding (turning categorical features with N different values into N binary columns) which in some cases means your dataset becomes orders of magnitude denser and smaller. Boosting algorithms usually also end up being the best single models for most tabular datasets anyway, so I find that my time is usually best spent getting the best possible results with LightGBM, XGBoost and CatBoost.
", "rawMarkdown": "Not a stupid question at all! Personally, I tend to start with the boosting algorithms right away because they require very little data preparation work. They can handle missing values out of the box, they are very robust to all kinds of distributions among your features, they are very fast to train, and they don't even require one hot encoding (turning categorical features with N different values into N binary columns) which in some cases means your dataset becomes orders of magnitude denser and smaller. Boosting algorithms usually also end up being the best single models for most tabular datasets anyway, so I find that my time is usually best spent getting the best possible results with LightGBM, XGBoost and CatBoost.", "votes": null }, { "id": "388850", "postDate": "09/17/2018 17:35:16", "content": "Do you mean that even after doing good pre-processing, filling missing values and feature generation steps non-boosted regression models tend to lose to boosted ones that even do not require such huge amount of work on 'cleaning'?
", "rawMarkdown": "Do you mean that even after doing good pre-processing, filling missing values and feature generation steps non-boosted regression models tend to lose to boosted ones that even do not require such huge amount of work on 'cleaning'?", "votes": null }, { "id": "388925", "postDate": "09/17/2018 20:38:27", "content": "Gradient boosting is a sort of best of both worlds algorithm, in the sense that models are optimized via gradient descent on generic differentiable loss functions, but also reap the natural benefits of tree-based models like automatic null-handling, automatic feature selection and interaction, scale invariance, and ability to capture (highly) non-linear feature-target relationships. These characteristics make gradient boosted models very robust to messy and heterogenous data (e.g. nullness, outliers, mix of numerics on different scales and categoricals) and also highly expressive, and therefore applicable to a wide range of problems. It gets even better because of how flexible they are -- model complexity can be controlled on a very precise level by tuning a large set of structural (max tree depth, number of leaves) and regularization (learning rate, col/row subsampling, l1/l2 reg, ...) parameters.
\n\nSo in short, gradient boosting models are a pretty all-inclusive package for covering the wide range of complexity settings that can occur in real world datasets, especially heterogenous ones (neural networks are often better when working with homogenous data like images, text, or audio signals). The typical kaggle competition is unlikely to involve features that can be mapped to targets in a simple linear manner, since problems of that form would be much less challenging and interesting. Therefore, a generically powerful and flexible tool like gradient boosting becomes the model of choice in most tabular competitions, at least as a starting point.
\n\nAmong the high-profile gradient boosting libraries (LGB, XGBoost, CatBoost), LGB seems to be both the fastest and most flexible (see things like built-in category handling, I'm not sure if XGB has this). In particular, LGB uses highly optimized histogram binning as a means of speeding up the tree-splitting process. Doing well in kaggle competitions tends to be driven by performing large numbers of diverse experiments (in the sense of feature sets, model hyperparameters, etc.), for which a fast, high-performing library like LGB is outstanding.
", "rawMarkdown": "Gradient boosting is a sort of best of both worlds algorithm, in the sense that models are optimized via gradient descent on generic differentiable loss functions, but also reap the natural benefits of tree-based models like automatic null-handling, automatic feature selection and interaction, scale invariance, and ability to capture (highly) non-linear feature-target relationships. These characteristics make gradient boosted models very robust to messy and heterogenous data (e.g. nullness, outliers, mix of numerics on different scales and categoricals) and also highly expressive, and therefore applicable to a wide range of problems. It gets even better because of how flexible they are -- model complexity can be controlled on a very precise level by tuning a large set of structural (max tree depth, number of leaves) and regularization (learning rate, col/row subsampling, l1/l2 reg, ...) parameters. \n\nSo in short, gradient boosting models are a pretty all-inclusive package for covering the wide range of complexity settings that can occur in real world datasets, especially heterogenous ones (neural networks are often better when working with homogenous data like images, text, or audio signals). The typical kaggle competition is unlikely to involve features that can be mapped to targets in a simple linear manner, since problems of that form would be much less challenging and interesting. Therefore, a generically powerful and flexible tool like gradient boosting becomes the model of choice in most tabular competitions, at least as a starting point. \n\nAmong the high-profile gradient boosting libraries (LGB, XGBoost, CatBoost), LGB seems to be both the fastest and most flexible (see things like built-in category handling, I'm not sure if XGB has this). In particular, LGB uses highly optimized histogram binning as a means of speeding up the tree-splitting process. Doing well in kaggle competitions tends to be driven by performing large numbers of diverse experiments (in the sense of feature sets, model hyperparameters, etc.), for which a fast, high-performing library like LGB is outstanding.", "votes": null }, { "id": "389089", "postDate": "09/18/2018 06:15:25", "content": "Thanks for such detailed and great answer. I will keep in mind this.
", "rawMarkdown": "Thanks for such detailed and great answer. I will keep in mind this.", "votes": null }, { "id": "389193", "postDate": "09/18/2018 10:12:52", "content": "No such thing as a free lunch theorem
\n\nNo one can really answer that question definitively, as it is hard to say for a given problem which model will perform the best.
\n\nHowever, boosted tree models will almost always perform better than a simple regression model - especially if there is reason to suspect there are non-linear relationships in play in the dataset (as there are in this dataset, since browser is a variable).
Note that although boosted tree models need less pre-processing and null filling, they can still be sensitive to feature generation so that would still need doing in both strategies.
", "rawMarkdown": "[No such thing as a free lunch theorem](https://en.wikipedia.org/wiki/No_free_lunch_theorem)\n\nNo one can really answer that question definitively, as it is hard to say for a given problem which model will perform the best.\n\n*However*, boosted tree models will almost always perform better than a simple regression model - especially if there is reason to suspect there are non-linear relationships in play in the dataset (as there are in this dataset, since `browser` is a variable).\n\nNote that although boosted tree models need less pre-processing and null filling, they can still be sensitive to feature generation so that would still need doing in both strategies.", "votes": null }, { "id": "389306", "postDate": "09/18/2018 14:17:02", "content": "Glad if it's helpful! :)
", "rawMarkdown": "Glad if it's helpful! :)", "votes": null }, { "id": "389451", "postDate": "09/18/2018 18:26:52", "content": "Gradient boosting algorithms such as LGBM, XGBoost and Catboost tend to perform very well in Kaggle competitions, primarily in tabular data. That's why you tend to see many people using these techniques. Nevertheless it is good to try out many different models and sometimes a linear regression model works wonders too. Perhaps you could get an edge over others by starting with a linear regression model and finding some hidden gems which could tell you something more about the data by interpreting/validating the model.
", "rawMarkdown": "Gradient boosting algorithms such as LGBM, XGBoost and Catboost tend to perform very well in Kaggle competitions, primarily in tabular data. That's why you tend to see many people using these techniques. Nevertheless it is good to try out many different models and sometimes a linear regression model works wonders too. Perhaps you could get an edge over others by starting with a linear regression model and finding some hidden gems which could tell you something more about the data by interpreting/validating the model.", "votes": null } ], "comments": [ { "id": 388848, "author_name": "keegil", "author_url": "", "post_date": "09/17/2018 17:30:09", "content": "Not a stupid question at all! Personally, I tend to start with the boosting algorithms right away because they require very little data preparation work. They can handle missing values out of the box, they are very robust to all kinds of distributions among your features, they are very fast to train, and they don't even require one hot encoding (turning categorical features with N different values into N binary columns) which in some cases means your dataset becomes orders of magnitude denser and smaller. Boosting algorithms usually also end up being the best single models for most tabular datasets anyway, so I find that my time is usually best spent getting the best possible results with LightGBM, XGBoost and CatBoost.
", "votes": null, "replies": [ { "id": 388850, "author_name": "alpamys", "author_url": "", "post_date": "09/17/2018 17:35:16", "content": "Do you mean that even after doing good pre-processing, filling missing values and feature generation steps non-boosted regression models tend to lose to boosted ones that even do not require such huge amount of work on 'cleaning'?
", "votes": null, "replies": [] }, { "id": 389193, "author_name": "shep312", "author_url": "", "post_date": "09/18/2018 10:12:52", "content": "No such thing as a free lunch theorem
\n\nNo one can really answer that question definitively, as it is hard to say for a given problem which model will perform the best.
\n\nHowever, boosted tree models will almost always perform better than a simple regression model - especially if there is reason to suspect there are non-linear relationships in play in the dataset (as there are in this dataset, since browser is a variable).
Note that although boosted tree models need less pre-processing and null filling, they can still be sensitive to feature generation so that would still need doing in both strategies.
", "votes": null, "replies": [] } ] }, { "id": 388925, "author_name": "aquatic", "author_url": "", "post_date": "09/17/2018 20:38:27", "content": "Gradient boosting is a sort of best of both worlds algorithm, in the sense that models are optimized via gradient descent on generic differentiable loss functions, but also reap the natural benefits of tree-based models like automatic null-handling, automatic feature selection and interaction, scale invariance, and ability to capture (highly) non-linear feature-target relationships. These characteristics make gradient boosted models very robust to messy and heterogenous data (e.g. nullness, outliers, mix of numerics on different scales and categoricals) and also highly expressive, and therefore applicable to a wide range of problems. It gets even better because of how flexible they are -- model complexity can be controlled on a very precise level by tuning a large set of structural (max tree depth, number of leaves) and regularization (learning rate, col/row subsampling, l1/l2 reg, ...) parameters.
\n\nSo in short, gradient boosting models are a pretty all-inclusive package for covering the wide range of complexity settings that can occur in real world datasets, especially heterogenous ones (neural networks are often better when working with homogenous data like images, text, or audio signals). The typical kaggle competition is unlikely to involve features that can be mapped to targets in a simple linear manner, since problems of that form would be much less challenging and interesting. Therefore, a generically powerful and flexible tool like gradient boosting becomes the model of choice in most tabular competitions, at least as a starting point.
\n\nAmong the high-profile gradient boosting libraries (LGB, XGBoost, CatBoost), LGB seems to be both the fastest and most flexible (see things like built-in category handling, I'm not sure if XGB has this). In particular, LGB uses highly optimized histogram binning as a means of speeding up the tree-splitting process. Doing well in kaggle competitions tends to be driven by performing large numbers of diverse experiments (in the sense of feature sets, model hyperparameters, etc.), for which a fast, high-performing library like LGB is outstanding.
", "votes": null, "replies": [ { "id": 389089, "author_name": "alpamys", "author_url": "", "post_date": "09/18/2018 06:15:25", "content": "Thanks for such detailed and great answer. I will keep in mind this.
", "votes": null, "replies": [] }, { "id": 389306, "author_name": "aquatic", "author_url": "", "post_date": "09/18/2018 14:17:02", "content": "Glad if it's helpful! :)
", "votes": null, "replies": [] } ] }, { "id": 389451, "author_name": "jolzer", "author_url": "", "post_date": "09/18/2018 18:26:52", "content": "Gradient boosting algorithms such as LGBM, XGBoost and Catboost tend to perform very well in Kaggle competitions, primarily in tabular data. That's why you tend to see many people using these techniques. Nevertheless it is good to try out many different models and sometimes a linear regression model works wonders too. Perhaps you could get an edge over others by starting with a linear regression model and finding some hidden gems which could tell you something more about the data by interpreting/validating the model.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "388830": "Hi, there! After viewing most-voted kernels I have noticed that majority tend to use LGBM instead of simple regression models? I am new here and it's my first competition and, thus, can someone clarify why you use LGBM and it's benefits? Also, in these kernels, the baseline is right away based on LGBM without even trying regression. Does that mean that regression is old and not so effective? I have read some papers/works on LGBM and it seems quite vague right now. I would appreciate if you share some tutorial or paper that will open my eyes. Thanks in advance, and sorry if it seems like a 'stupid question from newbie'", "388848": "Not a stupid question at all! Personally, I tend to start with the boosting algorithms right away because they require very little data preparation work. They can handle missing values out of the box, they are very robust to all kinds of distributions among your features, they are very fast to train, and they don't even require one hot encoding (turning categorical features with N different values into N binary columns) which in some cases means your dataset becomes orders of magnitude denser and smaller. Boosting algorithms usually also end up being the best single models for most tabular datasets anyway, so I find that my time is usually best spent getting the best possible results with LightGBM, XGBoost and CatBoost.", "388850": "Do you mean that even after doing good pre-processing, filling missing values and feature generation steps non-boosted regression models tend to lose to boosted ones that even do not require such huge amount of work on 'cleaning'?", "388925": "Gradient boosting is a sort of best of both worlds algorithm, in the sense that models are optimized via gradient descent on generic differentiable loss functions, but also reap the natural benefits of tree-based models like automatic null-handling, automatic feature selection and interaction, scale invariance, and ability to capture (highly) non-linear feature-target relationships. These characteristics make gradient boosted models very robust to messy and heterogenous data (e.g. nullness, outliers, mix of numerics on different scales and categoricals) and also highly expressive, and therefore applicable to a wide range of problems. It gets even better because of how flexible they are -- model complexity can be controlled on a very precise level by tuning a large set of structural (max tree depth, number of leaves) and regularization (learning rate, col/row subsampling, l1/l2 reg, ...) parameters. \n\nSo in short, gradient boosting models are a pretty all-inclusive package for covering the wide range of complexity settings that can occur in real world datasets, especially heterogenous ones (neural networks are often better when working with homogenous data like images, text, or audio signals). The typical kaggle competition is unlikely to involve features that can be mapped to targets in a simple linear manner, since problems of that form would be much less challenging and interesting. Therefore, a generically powerful and flexible tool like gradient boosting becomes the model of choice in most tabular competitions, at least as a starting point. \n\nAmong the high-profile gradient boosting libraries (LGB, XGBoost, CatBoost), LGB seems to be both the fastest and most flexible (see things like built-in category handling, I'm not sure if XGB has this). In particular, LGB uses highly optimized histogram binning as a means of speeding up the tree-splitting process. Doing well in kaggle competitions tends to be driven by performing large numbers of diverse experiments (in the sense of feature sets, model hyperparameters, etc.), for which a fast, high-performing library like LGB is outstanding.", "389089": "Thanks for such detailed and great answer. I will keep in mind this.", "389193": "[No such thing as a free lunch theorem](https://en.wikipedia.org/wiki/No_free_lunch_theorem)\n\nNo one can really answer that question definitively, as it is hard to say for a given problem which model will perform the best.\n\n*However*, boosted tree models will almost always perform better than a simple regression model - especially if there is reason to suspect there are non-linear relationships in play in the dataset (as there are in this dataset, since `browser` is a variable).\n\nNote that although boosted tree models need less pre-processing and null filling, they can still be sensitive to feature generation so that would still need doing in both strategies.", "389306": "Glad if it's helpful! :)", "389451": "Gradient boosting algorithms such as LGBM, XGBoost and Catboost tend to perform very well in Kaggle competitions, primarily in tabular data. That's why you tend to see many people using these techniques. Nevertheless it is good to try out many different models and sometimes a linear regression model works wonders too. Perhaps you could get an edge over others by starting with a linear regression model and finding some hidden gems which could tell you something more about the data by interpreting/validating the model." }, "source": "meta" }