{ "id": 531362, "title": "#1 Solution", "url": "/competitions/bench-tab-v2/writeups/ambrosm-1-solution", "author_name": "", "post_date": "2024-09-01T04:06:35.756668200Z", "votes": 9, "comment_count": 4, "views": 0, "content": "

Thank you, @lucasmorin, for organizing this competition! Optimizing 59 machine learning models at once certainly is a unique challenge, and I learnt a lot while working on it.

\n

My approach consisted of five steps:

\n
    \n
  1. Determine feature importance for every dataset and drop features which are obviously useless.
    2. \n
  2. Use Optuna to tune XGBoost, LightGBM, CatBoost, HistGradientBoosting and DART models for every task.
    3. \n
  3. Manually analyze datasets with non-anonymous feature names and try to apply some domain knowledge.
    4. \n
  4. Where Optuna diverges to overly complex models (many iterations or very large trees), try to model the task with a neural network.
    5. \n
  5. Ensemble.
    6. \n
\n

The following table shows what was done in addition to the gradient-boosting models:

\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
IdentifierSolution
0334_361110Feature engineering exploiting the fact that the time series has been split randomly into train and test so that we can predict the past from the future
6334_361286Linear support vector machine, ExplainableBoostingClassifier
8335_361094Semi-supervised classification by LabelPropagation
11335_361098Feature engineering and ridge regression
13335_361101Feature engineering (trip duration, and integer and fractional parts of the amount)
16335_361104Feature engineering and ridge regression
18335_361288Anisotropic Gaussian process and kernel ridge regression
20335_361291Duplicate grouping, ExtraTreesRegressor, RandomForestRegressor
22335_361293Feature engineering interpreting the distance as a categorical feature, high max_bin
23335_361294SplineTransformer with ridge regression and ExplainableBoostingRegressor
25336_361073Keras
32336_361081Feature engineering and ridge regression
34336_361083Feature engineering (trip duration, and integer and fractional parts of the amount)
37336_361086SplineTransformer with ridge regression and ExplainableBoostingRegressor
41336_361280Anisotropic Gaussian process and kernel ridge regression
42336_361281Duplicate grouping, ExtraTreesRegressor, RandomForestRegressor
44337_361060Feature engineering and predicting from the future
51337_361069Keras
56337_361276Kernel support vector machine
\n

 

\n

Conclusion: AutoML doesn't yet replace common sense.

", "messages": [ { "id": "2975604", "postDate": "09/01/2024 04:06:35", "content": "

Thank you, @lucasmorin, for organizing this competition! Optimizing 59 machine learning models at once certainly is a unique challenge, and I learnt a lot while working on it.

\n

My approach consisted of five steps:

\n
    \n
  1. Determine feature importance for every dataset and drop features which are obviously useless.
    2. \n
  2. Use Optuna to tune XGBoost, LightGBM, CatBoost, HistGradientBoosting and DART models for every task.
    3. \n
  3. Manually analyze datasets with non-anonymous feature names and try to apply some domain knowledge.
    4. \n
  4. Where Optuna diverges to overly complex models (many iterations or very large trees), try to model the task with a neural network.
    5. \n
  5. Ensemble.
    6. \n
\n

The following table shows what was done in addition to the gradient-boosting models:

\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
IdentifierSolution
0334_361110Feature engineering exploiting the fact that the time series has been split randomly into train and test so that we can predict the past from the future
6334_361286Linear support vector machine, ExplainableBoostingClassifier
8335_361094Semi-supervised classification by LabelPropagation
11335_361098Feature engineering and ridge regression
13335_361101Feature engineering (trip duration, and integer and fractional parts of the amount)
16335_361104Feature engineering and ridge regression
18335_361288Anisotropic Gaussian process and kernel ridge regression
20335_361291Duplicate grouping, ExtraTreesRegressor, RandomForestRegressor
22335_361293Feature engineering interpreting the distance as a categorical feature, high max_bin
23335_361294SplineTransformer with ridge regression and ExplainableBoostingRegressor
25336_361073Keras
32336_361081Feature engineering and ridge regression
34336_361083Feature engineering (trip duration, and integer and fractional parts of the amount)
37336_361086SplineTransformer with ridge regression and ExplainableBoostingRegressor
41336_361280Anisotropic Gaussian process and kernel ridge regression
42336_361281Duplicate grouping, ExtraTreesRegressor, RandomForestRegressor
44337_361060Feature engineering and predicting from the future
51337_361069Keras
56337_361276Kernel support vector machine
\n

 

\n

Conclusion: AutoML doesn't yet replace common sense.

", "rawMarkdown": "Thank you, @lucasmorin, for organizing this competition! Optimizing 59 machine learning models at once certainly is a unique challenge, and I learnt a lot while working on it.\n\nMy approach consisted of five steps:\n1. Determine [feature importance](https://www.kaggle.com/code/ambrosm/tdb2-feature-importance) for every dataset and drop features which are obviously useless.\n1. Use Optuna to tune XGBoost, LightGBM, CatBoost, HistGradientBoosting and DART models for every task.\n1. Manually analyze datasets with non-anonymous feature names and try to apply some domain knowledge.\n1. Where Optuna diverges to overly complex models (many iterations or very large trees), try to model the task with a neural network.\n1. Ensemble.\n\nThe following table shows what was done in addition to the gradient-boosting models:\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
IdentifierSolution
0334_361110Feature engineering exploiting the fact that the time series has been split randomly into train and test so that we can predict the past from the future
6334_361286Linear support vector machine, ExplainableBoostingClassifier
8335_361094Semi-supervised classification by LabelPropagation
11335_361098Feature engineering and ridge regression
13335_361101Feature engineering (trip duration, and integer and fractional parts of the amount)
16335_361104Feature engineering and ridge regression
18335_361288Anisotropic Gaussian process and kernel ridge regression
20335_361291Duplicate grouping, ExtraTreesRegressor, RandomForestRegressor
22335_361293Feature engineering interpreting the distance as a categorical feature, high max_bin
23335_361294SplineTransformer with ridge regression and ExplainableBoostingRegressor
25336_361073Keras
32336_361081Feature engineering and ridge regression
34336_361083Feature engineering (trip duration, and integer and fractional parts of the amount)
37336_361086SplineTransformer with ridge regression and ExplainableBoostingRegressor
41336_361280Anisotropic Gaussian process and kernel ridge regression
42336_361281Duplicate grouping, ExtraTreesRegressor, RandomForestRegressor
44337_361060Feature engineering and predicting from the future
51337_361069Keras
56337_361276Kernel support vector machine
\n\n \n\nConclusion: AutoML doesn't yet replace common sense.", "votes": null }, { "id": "2975671", "postDate": "09/01/2024 05:57:31", "content": "

Great writeup and congratulations!

\n

I would see automl more as a helper and accelerator, a complimentary tool for a data scientist.
\nThe domain knowledge and overall ml knowledge will still be required.

", "rawMarkdown": "Great writeup and congratulations!\n\nI would see automl more as a helper and accelerator, a complimentary tool for a data scientist. \nThe domain knowledge and overall ml knowledge will still be required.", "votes": null }, { "id": "2976453", "postDate": "09/02/2024 00:39:10", "content": "

share your notebook @ambrosm

", "rawMarkdown": "share your notebook @ambrosm", "votes": null }, { "id": "2976533", "postDate": "09/02/2024 03:51:08", "content": "

Thanks @ambrosm for your conclusions. This seems to be a lot of work. I am happy the feature engineering save the day and conclude for the need of good data scientists.

\n

Your decision process (4. Try NN if Optuna doesn't work) appears interesting. Do you see other rules of thumbs for choosing NNs over gbdts ? In the end you have two tasks with NNs models ? (Keras). Similarly, what was the hint that made you go to SVM for one task ? just testing it ?

\n

Also I see two tasks are leaky. Did you guess this because of the name of the features ? or from some more advanced studies ?

", "rawMarkdown": "Thanks @ambrosm for your conclusions. This seems to be a lot of work. I am happy the feature engineering save the day and conclude for the need of good data scientists. \n\nYour decision process (4. Try NN if Optuna doesn't work) appears interesting. Do you see other rules of thumbs for choosing NNs over gbdts ? In the end you have two tasks with NNs models ? (Keras). Similarly, what was the hint that made you go to SVM for one task ? just testing it ?\n\nAlso I see two tasks are leaky. Did you guess this because of the name of the features ? or from some more advanced studies ?", "votes": null }, { "id": "2977262", "postDate": "09/02/2024 16:59:53", "content": "

Hi @lucasmorin

\n
    \n
  1. This is my only rule of thumb. Gradient-boosters approximate the decision surface with axis-parallel facets. I speculate that one can construct datasets which need exponentially many axis-parallel facets for a good approximation. For these datasets manual feature engineering or a neural network can bring an advantage. See this discussion for another dataset where gradient-boosters without feature engineering fail.
    2. \n
  2. I tried two or three more NNs, but at the end I only kept two of them.
    3. \n
  3. I routinely try linear models pipelined with the scikit-learn preprocessors as well as kernel SVMs (if the data size permits it). Sometimes they give good results.
    4. \n
  4. If you do a Google search for the feature names, you'll read that the original purpose of the dataset was forecasting with a time series. As soon as I realized that the competition data were split at random, the solution was obvious.
    5. \n
", "rawMarkdown": "Hi @lucasmorin\n\n1. This is my only rule of thumb. Gradient-boosters approximate the decision surface with axis-parallel facets. I speculate that one can construct datasets which need exponentially many axis-parallel facets for a good approximation. For these datasets manual feature engineering or a neural network can bring an advantage. See [this discussion](https://www.kaggle.com/competitions/playground-series-s4e5/discussion/499274) for another dataset where gradient-boosters without feature engineering fail.\n2. I tried two or three more NNs, but at the end I only kept two of them.\n3. I routinely try linear models pipelined with the scikit-learn preprocessors as well as kernel SVMs (if the data size permits it). Sometimes they give good results.\n4. If you do a Google search for the feature names, you'll read that the original purpose of the dataset was forecasting with a time series. As soon as I realized that the competition data were split at random, the solution was obvious.", "votes": null } ], "comments": [ { "id": 2975671, "author_name": "thomasmeiner", "author_url": "", "post_date": "09/01/2024 05:57:31", "content": "

Great writeup and congratulations!

\n

I would see automl more as a helper and accelerator, a complimentary tool for a data scientist.
\nThe domain knowledge and overall ml knowledge will still be required.

", "votes": null, "replies": [] }, { "id": 2976453, "author_name": "marifel", "author_url": "", "post_date": "09/02/2024 00:39:10", "content": "

share your notebook @ambrosm

", "votes": null, "replies": [] }, { "id": 2976533, "author_name": "lucasmorin", "author_url": "", "post_date": "09/02/2024 03:51:08", "content": "

Thanks @ambrosm for your conclusions. This seems to be a lot of work. I am happy the feature engineering save the day and conclude for the need of good data scientists.

\n

Your decision process (4. Try NN if Optuna doesn't work) appears interesting. Do you see other rules of thumbs for choosing NNs over gbdts ? In the end you have two tasks with NNs models ? (Keras). Similarly, what was the hint that made you go to SVM for one task ? just testing it ?

\n

Also I see two tasks are leaky. Did you guess this because of the name of the features ? or from some more advanced studies ?

", "votes": null, "replies": [ { "id": 2977262, "author_name": "ambrosm", "author_url": "", "post_date": "09/02/2024 16:59:53", "content": "

Hi @lucasmorin

\n
    \n
  1. This is my only rule of thumb. Gradient-boosters approximate the decision surface with axis-parallel facets. I speculate that one can construct datasets which need exponentially many axis-parallel facets for a good approximation. For these datasets manual feature engineering or a neural network can bring an advantage. See this discussion for another dataset where gradient-boosters without feature engineering fail.
    2. \n
  2. I tried two or three more NNs, but at the end I only kept two of them.
    3. \n
  3. I routinely try linear models pipelined with the scikit-learn preprocessors as well as kernel SVMs (if the data size permits it). Sometimes they give good results.
    4. \n
  4. If you do a Google search for the feature names, you'll read that the original purpose of the dataset was forecasting with a time series. As soon as I realized that the competition data were split at random, the solution was obvious.
    5. \n
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2975604": "Thank you, @lucasmorin, for organizing this competition! Optimizing 59 machine learning models at once certainly is a unique challenge, and I learnt a lot while working on it.\n\nMy approach consisted of five steps:\n1. Determine [feature importance](https://www.kaggle.com/code/ambrosm/tdb2-feature-importance) for every dataset and drop features which are obviously useless.\n1. Use Optuna to tune XGBoost, LightGBM, CatBoost, HistGradientBoosting and DART models for every task.\n1. Manually analyze datasets with non-anonymous feature names and try to apply some domain knowledge.\n1. Where Optuna diverges to overly complex models (many iterations or very large trees), try to model the task with a neural network.\n1. Ensemble.\n\nThe following table shows what was done in addition to the gradient-boosting models:\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
IdentifierSolution
0334_361110Feature engineering exploiting the fact that the time series has been split randomly into train and test so that we can predict the past from the future
6334_361286Linear support vector machine, ExplainableBoostingClassifier
8335_361094Semi-supervised classification by LabelPropagation
11335_361098Feature engineering and ridge regression
13335_361101Feature engineering (trip duration, and integer and fractional parts of the amount)
16335_361104Feature engineering and ridge regression
18335_361288Anisotropic Gaussian process and kernel ridge regression
20335_361291Duplicate grouping, ExtraTreesRegressor, RandomForestRegressor
22335_361293Feature engineering interpreting the distance as a categorical feature, high max_bin
23335_361294SplineTransformer with ridge regression and ExplainableBoostingRegressor
25336_361073Keras
32336_361081Feature engineering and ridge regression
34336_361083Feature engineering (trip duration, and integer and fractional parts of the amount)
37336_361086SplineTransformer with ridge regression and ExplainableBoostingRegressor
41336_361280Anisotropic Gaussian process and kernel ridge regression
42336_361281Duplicate grouping, ExtraTreesRegressor, RandomForestRegressor
44337_361060Feature engineering and predicting from the future
51337_361069Keras
56337_361276Kernel support vector machine
\n\n \n\nConclusion: AutoML doesn't yet replace common sense.", "2975671": "Great writeup and congratulations!\n\nI would see automl more as a helper and accelerator, a complimentary tool for a data scientist. \nThe domain knowledge and overall ml knowledge will still be required.", "2976453": "share your notebook @ambrosm", "2976533": "Thanks @ambrosm for your conclusions. This seems to be a lot of work. I am happy the feature engineering save the day and conclude for the need of good data scientists. \n\nYour decision process (4. Try NN if Optuna doesn't work) appears interesting. Do you see other rules of thumbs for choosing NNs over gbdts ? In the end you have two tasks with NNs models ? (Keras). Similarly, what was the hint that made you go to SVM for one task ? just testing it ?\n\nAlso I see two tasks are leaky. Did you guess this because of the name of the features ? or from some more advanced studies ?", "2977262": "Hi @lucasmorin\n\n1. This is my only rule of thumb. Gradient-boosters approximate the decision surface with axis-parallel facets. I speculate that one can construct datasets which need exponentially many axis-parallel facets for a good approximation. For these datasets manual feature engineering or a neural network can bring an advantage. See [this discussion](https://www.kaggle.com/competitions/playground-series-s4e5/discussion/499274) for another dataset where gradient-boosters without feature engineering fail.\n2. I tried two or three more NNs, but at the end I only kept two of them.\n3. I routinely try linear models pipelined with the scikit-learn preprocessors as well as kernel SVMs (if the data size permits it). Sometimes they give good results.\n4. If you do a Google search for the feature names, you'll read that the original purpose of the dataset was forecasting with a time series. As soon as I realized that the competition data were split at random, the solution was obvious." }, "source": "meta" }