{
  "id": 471197,
  "title": "xgboost performance?",
  "url": "/competitions/bench-tab-v1/discussion/471197",
  "author_name": "",
  "post_date": "2024-01-27T10:28:46.804817200Z",
  "votes": 4,
  "comment_count": 3,
  "views": 0,
  "content": "<p>From the <a href=\"https://www.kaggle.com/carlmcbrideellis\" target=\"_blank\">@carlmcbrideellis</a> trees benchmark I found weird that xgboost was lagging that much behind in terms of performance. I tought it might be due to categorical encoding. Currently xgboost is trialing a new feature for handling categoricals internally. But it has weird requirements (need to be encoded as continuous ints). I am not sure how to do this properly. I have tried OHE (<a href=\"https://www.kaggle.com/code/lucasmorin/benchtab-v1-ohe-xgboost/\" target=\"_blank\">https://www.kaggle.com/code/lucasmorin/benchtab-v1-ohe-xgboost/</a>) and got similar results. </p>\n<p>I tried investigating the performance loss a little bit and funnily the best answer I got was to try CatBoost Encoder (<a href=\"https://datascience.stackexchange.com/questions/126184/handling-categorical-variables-for-xgboost)\" target=\"_blank\">https://datascience.stackexchange.com/questions/126184/handling-categorical-variables-for-xgboost)</a>. And I will investigate default parameters / individual datasets.</p>",
  "messages": [
    {
      "id": "2622204",
      "postDate": "01/27/2024 10:28:46",
      "content": "<p>From the <a href=\"https://www.kaggle.com/carlmcbrideellis\" target=\"_blank\">@carlmcbrideellis</a> trees benchmark I found weird that xgboost was lagging that much behind in terms of performance. I tought it might be due to categorical encoding. Currently xgboost is trialing a new feature for handling categoricals internally. But it has weird requirements (need to be encoded as continuous ints). I am not sure how to do this properly. I have tried OHE (<a href=\"https://www.kaggle.com/code/lucasmorin/benchtab-v1-ohe-xgboost/\" target=\"_blank\">https://www.kaggle.com/code/lucasmorin/benchtab-v1-ohe-xgboost/</a>) and got similar results. </p>\n<p>I tried investigating the performance loss a little bit and funnily the best answer I got was to try CatBoost Encoder (<a href=\"https://datascience.stackexchange.com/questions/126184/handling-categorical-variables-for-xgboost)\" target=\"_blank\">https://datascience.stackexchange.com/questions/126184/handling-categorical-variables-for-xgboost)</a>. And I will investigate default parameters / individual datasets.</p>",
      "rawMarkdown": "From the @carlmcbrideellis trees benchmark I found weird that xgboost was lagging that much behind in terms of performance. I tought it might be due to categorical encoding. Currently xgboost is trialing a new feature for handling categoricals internally. But it has weird requirements (need to be encoded as continuous ints). I am not sure how to do this properly. I have tried OHE (https://www.kaggle.com/code/lucasmorin/benchtab-v1-ohe-xgboost/) and got similar results. \n\nI tried investigating the performance loss a little bit and funnily the best answer I got was to try CatBoost Encoder (https://datascience.stackexchange.com/questions/126184/handling-categorical-variables-for-xgboost). And I will investigate default parameters / individual datasets.",
      "votes": null
    },
    {
      "id": "2628601",
      "postDate": "01/31/2024 11:59:37",
      "content": "<p>I tried tuning hyperparams in my notebook and used target encoding from <code>category_encoders</code></p>\n<p>Surprisingly the performance was not good. I assume two issues:</p>\n<ul>\n<li>To get all datasets through in one run I have limited tuning time for each dataset. However this is pretty naive and too short for bigger datasets. So here Xgboost would be even better with default params.</li>\n<li>Maybe I am wrong, but I read once upon a time that Xgboost is not that good on regression tasks. Potentially it looses many points here. </li>\n</ul>\n<p>When I tried Xgboost's integrated category handling it always performed worse than target encoding. Maybe the feature is too new?</p>\n<p>Out of curiosity: Can you as competition host see all the submission files? After the competition it would be so interesting to not only see overall score, but also score per task (classification vs regression). I would assume that Catboost makes lots of ground on the regression side.</p>",
      "rawMarkdown": "I tried tuning hyperparams in my notebook and used target encoding from `category_encoders`\n\nSurprisingly the performance was not good. I assume two issues:\n* To get all datasets through in one run I have limited tuning time for each dataset. However this is pretty naive and too short for bigger datasets. So here Xgboost would be even better with default params.\n* Maybe I am wrong, but I read once upon a time that Xgboost is not that good on regression tasks. Potentially it looses many points here. \n\nWhen I tried Xgboost's integrated category handling it always performed worse than target encoding. Maybe the feature is too new?\n\nOut of curiosity: Can you as competition host see all the submission files? After the competition it would be so interesting to not only see overall score, but also score per task (classification vs regression). I would assume that Catboost makes lots of ground on the regression side.",
      "votes": null
    },
    {
      "id": "2630275",
      "postDate": "02/01/2024 08:33:24",
      "content": "<p>Do you have a source for this?</p>",
      "rawMarkdown": "Do you have a source for this?",
      "votes": null
    },
    {
      "id": "2776827",
      "postDate": "04/26/2024 11:12:39",
      "content": "<p>for XGBClassifier, XGBRegressor have public LB 0.0255<br>\nthis is using the comparison code from Tabnet without changes<br>\nfor categorical columns (if any) used OHE via pd. dummies -&gt;</p>\n<p><code>X = pd.get_dummies(X, prefix=catcols, columns=catcols, dtype = int)</code></p>\n<p>seemed to work better than sklearn.preprocessing LabelEncoder for xgb or tabnet (still a work in progress)<br>\nXGB is fast.   Time wise and resource wise (now Kaggle only allows 5 cpu sessions) <br>\nSorry only saw about this competition via your Last week post.  Community competitions need better viz/promotion !!</p>",
      "rawMarkdown": "for XGBClassifier, XGBRegressor have public LB 0.0255\nthis is using the comparison code from Tabnet without changes\nfor categorical columns (if any) used OHE via pd. dummies ->\n\n`X = pd.get_dummies(X, prefix=catcols, columns=catcols, dtype = int)`\n\nseemed to work better than sklearn.preprocessing LabelEncoder for xgb or tabnet (still a work in progress)\nXGB is fast.   Time wise and resource wise (now Kaggle only allows 5 cpu sessions) \nSorry only saw about this competition via your Last week post.  Community competitions need better viz/promotion !!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2628601,
      "author_name": "thomasmeiner",
      "author_url": "",
      "post_date": "01/31/2024 11:59:37",
      "content": "<p>I tried tuning hyperparams in my notebook and used target encoding from <code>category_encoders</code></p>\n<p>Surprisingly the performance was not good. I assume two issues:</p>\n<ul>\n<li>To get all datasets through in one run I have limited tuning time for each dataset. However this is pretty naive and too short for bigger datasets. So here Xgboost would be even better with default params.</li>\n<li>Maybe I am wrong, but I read once upon a time that Xgboost is not that good on regression tasks. Potentially it looses many points here. </li>\n</ul>\n<p>When I tried Xgboost's integrated category handling it always performed worse than target encoding. Maybe the feature is too new?</p>\n<p>Out of curiosity: Can you as competition host see all the submission files? After the competition it would be so interesting to not only see overall score, but also score per task (classification vs regression). I would assume that Catboost makes lots of ground on the regression side.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2630275,
          "author_name": "lucasmorin",
          "author_url": "",
          "post_date": "02/01/2024 08:33:24",
          "content": "<p>Do you have a source for this?</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2776827,
      "author_name": "something4kag",
      "author_url": "",
      "post_date": "04/26/2024 11:12:39",
      "content": "<p>for XGBClassifier, XGBRegressor have public LB 0.0255<br>\nthis is using the comparison code from Tabnet without changes<br>\nfor categorical columns (if any) used OHE via pd. dummies -&gt;</p>\n<p><code>X = pd.get_dummies(X, prefix=catcols, columns=catcols, dtype = int)</code></p>\n<p>seemed to work better than sklearn.preprocessing LabelEncoder for xgb or tabnet (still a work in progress)<br>\nXGB is fast.   Time wise and resource wise (now Kaggle only allows 5 cpu sessions) <br>\nSorry only saw about this competition via your Last week post.  Community competitions need better viz/promotion !!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2622204": "From the @carlmcbrideellis trees benchmark I found weird that xgboost was lagging that much behind in terms of performance. I tought it might be due to categorical encoding. Currently xgboost is trialing a new feature for handling categoricals internally. But it has weird requirements (need to be encoded as continuous ints). I am not sure how to do this properly. I have tried OHE (https://www.kaggle.com/code/lucasmorin/benchtab-v1-ohe-xgboost/) and got similar results. \n\nI tried investigating the performance loss a little bit and funnily the best answer I got was to try CatBoost Encoder (https://datascience.stackexchange.com/questions/126184/handling-categorical-variables-for-xgboost). And I will investigate default parameters / individual datasets.",
    "2628601": "I tried tuning hyperparams in my notebook and used target encoding from `category_encoders`\n\nSurprisingly the performance was not good. I assume two issues:\n* To get all datasets through in one run I have limited tuning time for each dataset. However this is pretty naive and too short for bigger datasets. So here Xgboost would be even better with default params.\n* Maybe I am wrong, but I read once upon a time that Xgboost is not that good on regression tasks. Potentially it looses many points here. \n\nWhen I tried Xgboost's integrated category handling it always performed worse than target encoding. Maybe the feature is too new?\n\nOut of curiosity: Can you as competition host see all the submission files? After the competition it would be so interesting to not only see overall score, but also score per task (classification vs regression). I would assume that Catboost makes lots of ground on the regression side.",
    "2630275": "Do you have a source for this?",
    "2776827": "for XGBClassifier, XGBRegressor have public LB 0.0255\nthis is using the comparison code from Tabnet without changes\nfor categorical columns (if any) used OHE via pd. dummies ->\n\n`X = pd.get_dummies(X, prefix=catcols, columns=catcols, dtype = int)`\n\nseemed to work better than sklearn.preprocessing LabelEncoder for xgb or tabnet (still a work in progress)\nXGB is fast.   Time wise and resource wise (now Kaggle only allows 5 cpu sessions) \nSorry only saw about this competition via your Last week post.  Community competitions need better viz/promotion !!"
  },
  "source": "meta"
}