{
  "id": 467768,
  "title": "[GBDT vs ANN] Some baseline results",
  "url": "/competitions/bench-tab-v1/discussion/467768",
  "author_name": "",
  "post_date": "2024-01-13T21:28:45.178184300Z",
  "votes": 14,
  "comment_count": 7,
  "views": 0,
  "content": "<p>Some initial results for 'classical' non-parametric ML algorithms (single run, not ensembled) simply using the default hyper-parameters:</p>\n<table>\n<thead>\n<tr>\n<th>Code</th>\n<th>CPU time (s)</th>\n<th>score</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>CatBoost</td>\n<td>2399</td>\n<td>-0.0048</td>\n</tr>\n<tr>\n<td>HistGradientBoosting (scikit-learn)</td>\n<td>202</td>\n<td>-0.0456</td>\n</tr>\n<tr>\n<td>LightGBM</td>\n<td>166</td>\n<td>-0.0556</td>\n</tr>\n<tr>\n<td>GradientBoosting (scikit-learn)</td>\n<td>3054</td>\n<td>-0.0840</td>\n</tr>\n<tr>\n<td><a href=\"https://xgboost.readthedocs.io/en/stable/tutorials/rf.html#standalone-random-forest-with-scikit-learn-like-api\" target=\"_blank\">RandomForest (xgboost)</a></td>\n<td>159</td>\n<td>-0.1436</td>\n</tr>\n<tr>\n<td>XGBoost (version 2.0.2)</td>\n<td>123</td>\n<td>-0.1934</td>\n</tr>\n<tr>\n<td>RandomForest (scikit-learn)</td>\n<td>2216 (<code>n_jobs=-1</code>)</td>\n<td>-0.2765</td>\n</tr>\n<tr>\n<td>Linear/Logistic(*)</td>\n<td>76</td>\n<td>-0.6934</td>\n</tr>\n</tbody>\n</table>\n<p>*  Data pre-processed with <code>RobustScaler</code></p>\n<p><br><br>\nAll the best,<br>\ncarl</p>",
  "messages": [
    {
      "id": "2600787",
      "postDate": "01/13/2024 21:28:45",
      "content": "<p>Some initial results for 'classical' non-parametric ML algorithms (single run, not ensembled) simply using the default hyper-parameters:</p>\n<table>\n<thead>\n<tr>\n<th>Code</th>\n<th>CPU time (s)</th>\n<th>score</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>CatBoost</td>\n<td>2399</td>\n<td>-0.0048</td>\n</tr>\n<tr>\n<td>HistGradientBoosting (scikit-learn)</td>\n<td>202</td>\n<td>-0.0456</td>\n</tr>\n<tr>\n<td>LightGBM</td>\n<td>166</td>\n<td>-0.0556</td>\n</tr>\n<tr>\n<td>GradientBoosting (scikit-learn)</td>\n<td>3054</td>\n<td>-0.0840</td>\n</tr>\n<tr>\n<td><a href=\"https://xgboost.readthedocs.io/en/stable/tutorials/rf.html#standalone-random-forest-with-scikit-learn-like-api\" target=\"_blank\">RandomForest (xgboost)</a></td>\n<td>159</td>\n<td>-0.1436</td>\n</tr>\n<tr>\n<td>XGBoost (version 2.0.2)</td>\n<td>123</td>\n<td>-0.1934</td>\n</tr>\n<tr>\n<td>RandomForest (scikit-learn)</td>\n<td>2216 (<code>n_jobs=-1</code>)</td>\n<td>-0.2765</td>\n</tr>\n<tr>\n<td>Linear/Logistic(*)</td>\n<td>76</td>\n<td>-0.6934</td>\n</tr>\n</tbody>\n</table>\n<p>*  Data pre-processed with <code>RobustScaler</code></p>\n<p><br><br>\nAll the best,<br>\ncarl</p>",
      "rawMarkdown": "Some initial results for 'classical' non-parametric ML algorithms (single run, not ensembled) simply using the default hyper-parameters:\n\n| Code | CPU time (s)  | score |\n| --- | ---: | --- |\n| CatBoost | 2399  | -0.0048 |\n| HistGradientBoosting (scikit-learn) | 202 | -0.0456 |\n| LightGBM | 166 | -0.0556 | \n| GradientBoosting (scikit-learn) | 3054 | -0.0840 |\n| [RandomForest (xgboost)](https://xgboost.readthedocs.io/en/stable/tutorials/rf.html#standalone-random-forest-with-scikit-learn-like-api) | 159  | -0.1436 |\n| XGBoost (version 2.0.2) | 123 | -0.1934 |\n| RandomForest (scikit-learn) | 2216 (`n_jobs=-1`)  | -0.2765 |\n| Linear/Logistic(*) | 76 | -0.6934 |\n\n\\*  Data pre-processed with `RobustScaler`\n\n <br>\nAll the best,\ncarl",
      "votes": null
    },
    {
      "id": "2600853",
      "postDate": "01/14/2024 00:31:14",
      "content": "<p>Thanks. I'll look into the results as they seems a bit weird (should be a bit below zero). But its already interesting (xgb cpu time below lgbm ??).</p>",
      "rawMarkdown": "Thanks. I'll look into the results as they seems a bit weird (should be a bit below zero). But its already interesting (xgb cpu time below lgbm ??).",
      "votes": null
    },
    {
      "id": "2601076",
      "postDate": "01/14/2024 05:43:08",
      "content": "<p>Dear <a href=\"https://www.kaggle.com/lucasmorin\" target=\"_blank\">@lucasmorin</a> </p>\n<blockquote>\n  <p>\"<em>…they seems a bit weird</em>\"</p>\n</blockquote>\n<p>You are right, my bad; I have been using <code>predict</code> and not <code>predict_proba</code> for the classifiers in my version of the <a href=\"https://www.kaggle.com/code/lucasmorin/benchtab-v1-catboost-ensemble\" target=\"_blank\">competition script provided</a>.</p>\n<p>I have re-ran the scripts and updated the table.</p>\n<p>All the best,<br>\ncarl</p>\n<p>BTW: My initial test running CatBoost with <code>task_type=\"GPU\"</code> on GPU T4 x2 led to a time of 1184s, only 2x speedup. This seems wrong to me; will test further (¿maybe GPU is only worthwhile for large datasets?)</p>",
      "rawMarkdown": "Dear @lucasmorin \n\n> \"*...they seems a bit weird*\"\n\nYou are right, my bad; I have been using `predict` and not `predict_proba` for the classifiers in my version of the [competition script provided](https://www.kaggle.com/code/lucasmorin/benchtab-v1-catboost-ensemble).\n\nI have re-ran the scripts and updated the table.\n\nAll the best,\ncarl\n\nBTW: My initial test running CatBoost with `task_type=\"GPU\"` on GPU T4 x2 led to a time of 1184s, only 2x speedup. This seems wrong to me; will test further (¿maybe GPU is only worthwhile for large datasets?)",
      "votes": null
    },
    {
      "id": "2601427",
      "postDate": "01/14/2024 12:20:48",
      "content": "<p>Thanks. Very interesting stuff indeed. Even surprising (xgboost now has a faster run time than lgbm ?). Regarding the baselines, how did you handle categoricals? (Might be what is driving the difference)<br>\nRegarding GPU usage: thanks this is the kind of stuff I want to see and discuss. Yeah It has been my experience too. GPU is slightly underwhelming. </p>",
      "rawMarkdown": "Thanks. Very interesting stuff indeed. Even surprising (xgboost now has a faster run time than lgbm ?). Regarding the baselines, how did you handle categoricals? (Might be what is driving the difference)\nRegarding GPU usage: thanks this is the kind of stuff I want to see and discuss. Yeah It has been my experience too. GPU is slightly underwhelming.",
      "votes": null
    },
    {
      "id": "2601440",
      "postDate": "01/14/2024 12:29:52",
      "content": "<p>Dear <a href=\"https://www.kaggle.com/lucasmorin\" target=\"_blank\">@lucasmorin</a> </p>\n<blockquote>\n  <p>\"<em>…how did you handle categoricals</em>\"</p>\n</blockquote>\n<p>For the moment I just fed the data to the estimators \"as-is\". Will do more testing tomorrow (I have used up all my submissions for today!)</p>\n<blockquote>\n  <p>\"…<em>xgboost now has a faster run time than lgbm</em>\"</p>\n</blockquote>\n<p>Indeed, it was noticed by <a href=\"https://www.kaggle.com/siukeitin\" target=\"_blank\">@siukeitin</a> that the new <a href=\"https://www.kaggle.com/competitions/playground-series-s3e23/discussion/445637\" target=\"_blank\">xgboost 2.0.0 runs significantly faster on CPU</a>.</p>\n<p>All the best,<br>\ncarl</p>",
      "rawMarkdown": "Dear @lucasmorin \n\n> \"*...how did you handle categoricals*\"\n\nFor the moment I just fed the data to the estimators \"as-is\". Will do more testing tomorrow (I have used up all my submissions for today!)\n\n> \"...*xgboost now has a faster run time than lgbm*\"\n\nIndeed, it was noticed by @siukeitin that the new [xgboost 2.0.0 runs significantly faster on CPU](https://www.kaggle.com/competitions/playground-series-s3e23/discussion/445637).\n\nAll the best,\ncarl",
      "votes": null
    },
    {
      "id": "2603488",
      "postDate": "01/15/2024 19:51:43",
      "content": "<p>Ah yes. Getting decent performance might require Encoding / using native support (catboost accept native support, xgboost needs it to be encoded as continuous integers, lgbm I don't know anymore).</p>\n<p>Also, if you know a good way to measure memory usage (peak) I'd like to test wether we can reduce it (reduce dtypes, how models can handle dtypes).</p>",
      "rawMarkdown": "Ah yes. Getting decent performance might require Encoding / using native support (catboost accept native support, xgboost needs it to be encoded as continuous integers, lgbm I don't know anymore).\n\nAlso, if you know a good way to measure memory usage (peak) I'd like to test wether we can reduce it (reduce dtypes, how models can handle dtypes).",
      "votes": null
    },
    {
      "id": "2608038",
      "postDate": "01/18/2024 14:42:11",
      "content": "<p>truly marvelous</p>",
      "rawMarkdown": "truly marvelous",
      "votes": null
    },
    {
      "id": "2609920",
      "postDate": "01/19/2024 19:14:06",
      "content": "<p>Dear <a href=\"https://www.kaggle.com/lucasmorin\" target=\"_blank\">@lucasmorin</a> </p>\n<p>In the just released <a href=\"https://scikit-learn.org/stable/whats_new/v1.4.html\" target=\"_blank\">scikit 1.4.0</a></p>\n<p><strong><a href=\"https://scikit-learn.org/stable/auto_examples/release_highlights/plot_release_highlights_1_4_0.html#histgradientboosting-natively-supports-categorical-dtypes-in-dataframes\" target=\"_blank\">HistGradientBoosting Natively Supports Categorical DTypes in DataFrames</a></strong></p>\n<pre><code>!pip install --upgrade scikit-learn\n...\nHistGradientBoostingRegressor(categorical_features=)\n</code></pre>\n<p>All the best,<br>\ncarl</p>",
      "rawMarkdown": "Dear @lucasmorin \n\nIn the just released [scikit 1.4.0](https://scikit-learn.org/stable/whats_new/v1.4.html)\n\n**[HistGradientBoosting Natively Supports Categorical DTypes in DataFrames](https://scikit-learn.org/stable/auto_examples/release_highlights/plot_release_highlights_1_4_0.html#histgradientboosting-natively-supports-categorical-dtypes-in-dataframes)**\n\n```python\n!pip install --upgrade scikit-learn\n...\nHistGradientBoostingRegressor(categorical_features=\"from_dtype\")\n```\n\nAll the best,\ncarl",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2600853,
      "author_name": "lucasmorin",
      "author_url": "",
      "post_date": "01/14/2024 00:31:14",
      "content": "<p>Thanks. I'll look into the results as they seems a bit weird (should be a bit below zero). But its already interesting (xgb cpu time below lgbm ??).</p>",
      "votes": null,
      "replies": [
        {
          "id": 2601076,
          "author_name": "carlmcbrideellis",
          "author_url": "",
          "post_date": "01/14/2024 05:43:08",
          "content": "<p>Dear <a href=\"https://www.kaggle.com/lucasmorin\" target=\"_blank\">@lucasmorin</a> </p>\n<blockquote>\n  <p>\"<em>…they seems a bit weird</em>\"</p>\n</blockquote>\n<p>You are right, my bad; I have been using <code>predict</code> and not <code>predict_proba</code> for the classifiers in my version of the <a href=\"https://www.kaggle.com/code/lucasmorin/benchtab-v1-catboost-ensemble\" target=\"_blank\">competition script provided</a>.</p>\n<p>I have re-ran the scripts and updated the table.</p>\n<p>All the best,<br>\ncarl</p>\n<p>BTW: My initial test running CatBoost with <code>task_type=\"GPU\"</code> on GPU T4 x2 led to a time of 1184s, only 2x speedup. This seems wrong to me; will test further (¿maybe GPU is only worthwhile for large datasets?)</p>",
          "votes": null,
          "replies": [
            {
              "id": 2601427,
              "author_name": "lucasmorin",
              "author_url": "",
              "post_date": "01/14/2024 12:20:48",
              "content": "<p>Thanks. Very interesting stuff indeed. Even surprising (xgboost now has a faster run time than lgbm ?). Regarding the baselines, how did you handle categoricals? (Might be what is driving the difference)<br>\nRegarding GPU usage: thanks this is the kind of stuff I want to see and discuss. Yeah It has been my experience too. GPU is slightly underwhelming. </p>",
              "votes": null,
              "replies": [
                {
                  "id": 2601440,
                  "author_name": "carlmcbrideellis",
                  "author_url": "",
                  "post_date": "01/14/2024 12:29:52",
                  "content": "<p>Dear <a href=\"https://www.kaggle.com/lucasmorin\" target=\"_blank\">@lucasmorin</a> </p>\n<blockquote>\n  <p>\"<em>…how did you handle categoricals</em>\"</p>\n</blockquote>\n<p>For the moment I just fed the data to the estimators \"as-is\". Will do more testing tomorrow (I have used up all my submissions for today!)</p>\n<blockquote>\n  <p>\"…<em>xgboost now has a faster run time than lgbm</em>\"</p>\n</blockquote>\n<p>Indeed, it was noticed by <a href=\"https://www.kaggle.com/siukeitin\" target=\"_blank\">@siukeitin</a> that the new <a href=\"https://www.kaggle.com/competitions/playground-series-s3e23/discussion/445637\" target=\"_blank\">xgboost 2.0.0 runs significantly faster on CPU</a>.</p>\n<p>All the best,<br>\ncarl</p>",
                  "votes": null,
                  "replies": [
                    {
                      "id": 2603488,
                      "author_name": "lucasmorin",
                      "author_url": "",
                      "post_date": "01/15/2024 19:51:43",
                      "content": "<p>Ah yes. Getting decent performance might require Encoding / using native support (catboost accept native support, xgboost needs it to be encoded as continuous integers, lgbm I don't know anymore).</p>\n<p>Also, if you know a good way to measure memory usage (peak) I'd like to test wether we can reduce it (reduce dtypes, how models can handle dtypes).</p>",
                      "votes": null,
                      "replies": [
                        {
                          "id": 2609920,
                          "author_name": "carlmcbrideellis",
                          "author_url": "",
                          "post_date": "01/19/2024 19:14:06",
                          "content": "<p>Dear <a href=\"https://www.kaggle.com/lucasmorin\" target=\"_blank\">@lucasmorin</a> </p>\n<p>In the just released <a href=\"https://scikit-learn.org/stable/whats_new/v1.4.html\" target=\"_blank\">scikit 1.4.0</a></p>\n<p><strong><a href=\"https://scikit-learn.org/stable/auto_examples/release_highlights/plot_release_highlights_1_4_0.html#histgradientboosting-natively-supports-categorical-dtypes-in-dataframes\" target=\"_blank\">HistGradientBoosting Natively Supports Categorical DTypes in DataFrames</a></strong></p>\n<pre><code>!pip install --upgrade scikit-learn\n...\nHistGradientBoostingRegressor(categorical_features=)\n</code></pre>\n<p>All the best,<br>\ncarl</p>",
                          "votes": null,
                          "replies": []
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "id": 2608038,
      "author_name": "natarajanlalgudi",
      "author_url": "",
      "post_date": "01/18/2024 14:42:11",
      "content": "<p>truly marvelous</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2600787": "Some initial results for 'classical' non-parametric ML algorithms (single run, not ensembled) simply using the default hyper-parameters:\n\n| Code | CPU time (s)  | score |\n| --- | ---: | --- |\n| CatBoost | 2399  | -0.0048 |\n| HistGradientBoosting (scikit-learn) | 202 | -0.0456 |\n| LightGBM | 166 | -0.0556 | \n| GradientBoosting (scikit-learn) | 3054 | -0.0840 |\n| [RandomForest (xgboost)](https://xgboost.readthedocs.io/en/stable/tutorials/rf.html#standalone-random-forest-with-scikit-learn-like-api) | 159  | -0.1436 |\n| XGBoost (version 2.0.2) | 123 | -0.1934 |\n| RandomForest (scikit-learn) | 2216 (`n_jobs=-1`)  | -0.2765 |\n| Linear/Logistic(*) | 76 | -0.6934 |\n\n\\*  Data pre-processed with `RobustScaler`\n\n <br>\nAll the best,\ncarl",
    "2600853": "Thanks. I'll look into the results as they seems a bit weird (should be a bit below zero). But its already interesting (xgb cpu time below lgbm ??).",
    "2601076": "Dear @lucasmorin \n\n> \"*...they seems a bit weird*\"\n\nYou are right, my bad; I have been using `predict` and not `predict_proba` for the classifiers in my version of the [competition script provided](https://www.kaggle.com/code/lucasmorin/benchtab-v1-catboost-ensemble).\n\nI have re-ran the scripts and updated the table.\n\nAll the best,\ncarl\n\nBTW: My initial test running CatBoost with `task_type=\"GPU\"` on GPU T4 x2 led to a time of 1184s, only 2x speedup. This seems wrong to me; will test further (¿maybe GPU is only worthwhile for large datasets?)",
    "2601427": "Thanks. Very interesting stuff indeed. Even surprising (xgboost now has a faster run time than lgbm ?). Regarding the baselines, how did you handle categoricals? (Might be what is driving the difference)\nRegarding GPU usage: thanks this is the kind of stuff I want to see and discuss. Yeah It has been my experience too. GPU is slightly underwhelming.",
    "2601440": "Dear @lucasmorin \n\n> \"*...how did you handle categoricals*\"\n\nFor the moment I just fed the data to the estimators \"as-is\". Will do more testing tomorrow (I have used up all my submissions for today!)\n\n> \"...*xgboost now has a faster run time than lgbm*\"\n\nIndeed, it was noticed by @siukeitin that the new [xgboost 2.0.0 runs significantly faster on CPU](https://www.kaggle.com/competitions/playground-series-s3e23/discussion/445637).\n\nAll the best,\ncarl",
    "2603488": "Ah yes. Getting decent performance might require Encoding / using native support (catboost accept native support, xgboost needs it to be encoded as continuous integers, lgbm I don't know anymore).\n\nAlso, if you know a good way to measure memory usage (peak) I'd like to test wether we can reduce it (reduce dtypes, how models can handle dtypes).",
    "2608038": "truly marvelous",
    "2609920": "Dear @lucasmorin \n\nIn the just released [scikit 1.4.0](https://scikit-learn.org/stable/whats_new/v1.4.html)\n\n**[HistGradientBoosting Natively Supports Categorical DTypes in DataFrames](https://scikit-learn.org/stable/auto_examples/release_highlights/plot_release_highlights_1_4_0.html#histgradientboosting-natively-supports-categorical-dtypes-in-dataframes)**\n\n```python\n!pip install --upgrade scikit-learn\n...\nHistGradientBoostingRegressor(categorical_features=\"from_dtype\")\n```\n\nAll the best,\ncarl"
  },
  "source": "meta"
}