I'd like to share some intermediate objectives, if you'd like an idea on what to work on or give some feedback. I'll build dedicated discussions for some of thoses points.
\nOn simpler technical aspects:
\nOn benchmarking models:
\nOn the follow-up:
\nI'd like to share some intermediate objectives, if you'd like an idea on what to work on or give some feedback. I'll build dedicated discussions for some of thoses points.
\nOn simpler technical aspects:
\nOn benchmarking models:
\nOn the follow-up:
\nVery interesting about your last point : recurring competition to know year after the evolution of algorithms
", "rawMarkdown": "Very interesting about your last point : **recurring competition** to know year after the evolution of algorithms", "votes": null }, { "id": "2630582", "postDate": "02/01/2024 11:03:30", "content": "first of all, interesting effort @lucasmorin
secondly, since this is essentially research focused. It is necessary to foster rigorous research. In particular, whether gbdt or DL is better should depend heavily on type of data. Are there some systematic results/papers? For instance, seasonality, data size, low or SNR (signal-noise ratio), feedback mechanisms in the data (financial, economic, weather, etc). In this competition, it seems to be \"average result\" over large dataset, but it is a good benchmarking effort!
thirdly, motivation - monetary prizes (and even medals) drive competitive spirits.
Kaggle history is itself a research, there are some papers as you know better than me. Perhaps, some post-analysis by Kaggle and/or anonymized data about models from finished Kaggle competitions (each year on tabular data)?
", "rawMarkdown": "first of all, interesting effort @lucasmorin \n\n- secondly, since this is essentially research focused. It is necessary to foster rigorous research. In particular, whether gbdt or DL is better should depend heavily on type of data. Are there some systematic results/papers? For instance, seasonality, data size, low or SNR (signal-noise ratio), feedback mechanisms in the data (financial, economic, weather, etc). In this competition, it seems to be \"average result\" over large dataset, but it is a good benchmarking effort!\n\n- thirdly, motivation - monetary prizes (and even medals) drive competitive spirits. \n\nKaggle history is itself a research, there are some papers as you know better than me. Perhaps, some post-analysis by Kaggle and/or anonymized data about models from finished Kaggle competitions (each year on tabular data)?", "votes": null }, { "id": "2631085", "postDate": "02/01/2024 16:06:11", "content": "Well the paper I try to replicate with the competition is one systematic research. I think we might end up discussing the improvement in performance depending on the task and its properties (number of feature, correlation among features, StN ratio… ).
\nRegarding competition spirit. Kaggle don't allow monetary prizes for public datasets. This is not easy to solve (either using a generating process or finding private datasets).
", "rawMarkdown": "Well the paper I try to replicate with the competition is one systematic research. I think we might end up discussing the improvement in performance depending on the task and its properties (number of feature, correlation among features, StN ratio... ).\n\nRegarding competition spirit. Kaggle don't allow monetary prizes for public datasets. This is not easy to solve (either using a generating process or finding private datasets).", "votes": null }, { "id": "2634094", "postDate": "02/03/2024 13:50:32", "content": "\"… end up discussing…\" is better to make quantitative - this would be insightful research and raise attention. I suggested initial thoughts. Perhaps, there can be general information-theory (mutual information, etc) scoring of the nature of the datasets before any modeling?
\nFor example, it is clear that DL outperforms on the cat-dog pictures, isn't it? Perhaps, look intentionally where DL is better on tabular datasets as an important example. And on the opposite, a dataset where DL really struggles to get to the gbdt scores.
\nActually, it seems the question may not be well-defined if arbitrarily-complex DL in principle can mimic gbdt, can it not?
", "rawMarkdown": "\"... end up discussing...\" is better to make quantitative - this would be insightful research and raise attention. I suggested initial thoughts. Perhaps, there can be general information-theory (mutual information, etc) scoring of the nature of the datasets before any modeling?\n\nFor example, it is clear that DL outperforms on the cat-dog pictures, isn't it? Perhaps, look intentionally where DL is better on tabular datasets as an important example. And on the opposite, a dataset where DL really struggles to get to the gbdt scores.\n\nActually, it seems the question may not be well-defined if arbitrarily-complex DL in principle can mimic gbdt, can it not?", "votes": null }, { "id": "2635262", "postDate": "02/04/2024 09:00:58", "content": "Well in principle yes, but here it is about practice.
\nRegarding the exploration of the datasets and performance gain: yes I hope to see some exploratory notebooks, not only specific model tests.
my further thoughts are here
\nthank you for your effort
", "rawMarkdown": "my further thoughts are [here](https://www.kaggle.com/competitions/bench-tab-v1/discussion/473318)\n\nthank you for your effort", "votes": null } ], "comments": [ { "id": 2627408, "author_name": "mathurinache", "author_url": "", "post_date": "01/30/2024 17:08:23", "content": "Very interesting about your last point : recurring competition to know year after the evolution of algorithms
", "votes": null, "replies": [] }, { "id": 2630582, "author_name": "mihailturlakov", "author_url": "", "post_date": "02/01/2024 11:03:30", "content": "first of all, interesting effort @lucasmorin
secondly, since this is essentially research focused. It is necessary to foster rigorous research. In particular, whether gbdt or DL is better should depend heavily on type of data. Are there some systematic results/papers? For instance, seasonality, data size, low or SNR (signal-noise ratio), feedback mechanisms in the data (financial, economic, weather, etc). In this competition, it seems to be \"average result\" over large dataset, but it is a good benchmarking effort!
thirdly, motivation - monetary prizes (and even medals) drive competitive spirits.
Kaggle history is itself a research, there are some papers as you know better than me. Perhaps, some post-analysis by Kaggle and/or anonymized data about models from finished Kaggle competitions (each year on tabular data)?
", "votes": null, "replies": [ { "id": 2631085, "author_name": "lucasmorin", "author_url": "", "post_date": "02/01/2024 16:06:11", "content": "Well the paper I try to replicate with the competition is one systematic research. I think we might end up discussing the improvement in performance depending on the task and its properties (number of feature, correlation among features, StN ratio… ).
\nRegarding competition spirit. Kaggle don't allow monetary prizes for public datasets. This is not easy to solve (either using a generating process or finding private datasets).
", "votes": null, "replies": [ { "id": 2634094, "author_name": "mihailturlakov", "author_url": "", "post_date": "02/03/2024 13:50:32", "content": "\"… end up discussing…\" is better to make quantitative - this would be insightful research and raise attention. I suggested initial thoughts. Perhaps, there can be general information-theory (mutual information, etc) scoring of the nature of the datasets before any modeling?
\nFor example, it is clear that DL outperforms on the cat-dog pictures, isn't it? Perhaps, look intentionally where DL is better on tabular datasets as an important example. And on the opposite, a dataset where DL really struggles to get to the gbdt scores.
\nActually, it seems the question may not be well-defined if arbitrarily-complex DL in principle can mimic gbdt, can it not?
", "votes": null, "replies": [ { "id": 2635262, "author_name": "lucasmorin", "author_url": "", "post_date": "02/04/2024 09:00:58", "content": "Well in principle yes, but here it is about practice.
\nRegarding the exploration of the datasets and performance gain: yes I hope to see some exploratory notebooks, not only specific model tests.
my further thoughts are here
\nthank you for your effort
", "votes": null, "replies": [] } ] } ] } ] } ] } ], "raw_markdown_by_id": { "2610916": "I'd like to share some intermediate objectives, if you'd like an idea on what to work on or give some feedback. I'll build dedicated discussions for some of thoses points.\n\n- First of all it was for me the opportunity to **discover the competition UI**. There was some quirks but it went smoother than expected and I am pretty happy with the current result. Feel free to share if you have some feedback. This is probably not the last competition I build.\n- **Communication**: participation and results seems to highly depends on communication. I'll try to improve on this as my top priority. Feel free to participate and share on social networks. I'll also try to reach out to prominent kagglers.\n\nOn simpler technical aspects:\n\n- **Building good baselines**. I shared some baselines, I'd like to see a bit more stuff 'off the shelf', basically how base libraries works with correct categorical handling / minimal hand tuning.\n- **Data / Feature engineering**: I feel like an important area of improvement should be FE. I'd like to see wheter we can find EDA / FE approaches that work on multiple problems.\n- On **performance and it's validation**: there is a wide range of different datasets. I performed a simple split for validation. I wonder if data selection / split can be improved for future competitions.\n- I am also interested in alternate approaches to performance (think **efficience**, like small run time, small memory usage, small power consumption / CO2 emissions). E.g. anything that score between -1 and 0 but is faster than the catboost baseline.\n\nOn benchmarking models:\n- Different **DL** papers have claimed to outperform gbdts \"off the shelf\". It would be a good opportunity to test these claims. \n- Regarding **gbdts**: implementation are constantly evolving. I wonder if there is some simple (or advanced) parameters / technics than can help significantly improve gbdts performance. (I'll probably participate in this for the first month).\n- I am very curious if there are **other models** out there that could be competitive. \n\nOn the follow-up:\n- Ideally we find interesting stuff and we can **formalise findings in a blog post / paper**. I have not yet tought too much about the best approach and can be interested in your feedback.\n- Building a **reoccuring competition**. With all feedbacks I was thinking to improve and make the event reoccuring. (yearly basis). Again, feedback welcome.", "2627408": "Very interesting about your last point : **recurring competition** to know year after the evolution of algorithms", "2630582": "first of all, interesting effort @lucasmorin \n\n- secondly, since this is essentially research focused. It is necessary to foster rigorous research. In particular, whether gbdt or DL is better should depend heavily on type of data. Are there some systematic results/papers? For instance, seasonality, data size, low or SNR (signal-noise ratio), feedback mechanisms in the data (financial, economic, weather, etc). In this competition, it seems to be \"average result\" over large dataset, but it is a good benchmarking effort!\n\n- thirdly, motivation - monetary prizes (and even medals) drive competitive spirits. \n\nKaggle history is itself a research, there are some papers as you know better than me. Perhaps, some post-analysis by Kaggle and/or anonymized data about models from finished Kaggle competitions (each year on tabular data)?", "2631085": "Well the paper I try to replicate with the competition is one systematic research. I think we might end up discussing the improvement in performance depending on the task and its properties (number of feature, correlation among features, StN ratio... ).\n\nRegarding competition spirit. Kaggle don't allow monetary prizes for public datasets. This is not easy to solve (either using a generating process or finding private datasets).", "2634094": "\"... end up discussing...\" is better to make quantitative - this would be insightful research and raise attention. I suggested initial thoughts. Perhaps, there can be general information-theory (mutual information, etc) scoring of the nature of the datasets before any modeling?\n\nFor example, it is clear that DL outperforms on the cat-dog pictures, isn't it? Perhaps, look intentionally where DL is better on tabular datasets as an important example. And on the opposite, a dataset where DL really struggles to get to the gbdt scores.\n\nActually, it seems the question may not be well-defined if arbitrarily-complex DL in principle can mimic gbdt, can it not?", "2635262": "Well in principle yes, but here it is about practice. \nRegarding the exploration of the datasets and performance gain: yes I hope to see some exploratory notebooks, not only specific model tests.", "2635329": "my further thoughts are [here](https://www.kaggle.com/competitions/bench-tab-v1/discussion/473318)\n\nthank you for your effort" }, "source": "meta" }