Here is a topic discussing differents NNs approaches that I'd like to see being tested here.
\nClassic Architectures:
\nMLP, 1-D CNN, DAE, (Transformers ?)
\nI was thinking about seting up a MLP baseline myself for dicsussion (I have code at hands). But not sure how to select base architecture / parameters. Does any one have an idea about what should constitute MLP default parameters? (Maybe matching sklearn defaults parameters?)
\nAfter that I am generally interested to see how the different arch. performs. (and how they are limited by Kaggle ressources). Maybe this can be a source of discussion regarding the tf-Keras v.s. Pytorch debate. Are you interested in trying some of the classic architectures?
\nAdvanced Architectures:
\nI am mainly thinking about the three architectures below:
\nTabNet: Attentive Interpretable Tabular Learning
\nTabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second
\nTabR: Tabular Deep Learning Meets Nearest Neighbors in 2023
\nGNN ?
Do you know about other DL architectures that claimed to be competitive?
\nExcept maybe for Tabnet (part of the ensemble winning the first Optiver Comp. if I recall), I am not sure about their usage (and performance) here on Kaggle. Anyone knowing ressources on how to use those / interested in trying them?
\nEdit: I have also heard of foundationnal approaches to tabular data (Towards Foundation Models for Learning on Tabular Data). I have some concerns regarding efficience / and data leakage. But could be interested if someone know how to get one of those working here.
", "messages": [ { "id": "2610955", "postDate": "01/20/2024 12:56:07", "content": "Here is a topic discussing differents NNs approaches that I'd like to see being tested here.
\nClassic Architectures:
\nMLP, 1-D CNN, DAE, (Transformers ?)
\nI was thinking about seting up a MLP baseline myself for dicsussion (I have code at hands). But not sure how to select base architecture / parameters. Does any one have an idea about what should constitute MLP default parameters? (Maybe matching sklearn defaults parameters?)
\nAfter that I am generally interested to see how the different arch. performs. (and how they are limited by Kaggle ressources). Maybe this can be a source of discussion regarding the tf-Keras v.s. Pytorch debate. Are you interested in trying some of the classic architectures?
\nAdvanced Architectures:
\nI am mainly thinking about the three architectures below:
\nTabNet: Attentive Interpretable Tabular Learning
\nTabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second
\nTabR: Tabular Deep Learning Meets Nearest Neighbors in 2023
\nGNN ?
Do you know about other DL architectures that claimed to be competitive?
\nExcept maybe for Tabnet (part of the ensemble winning the first Optiver Comp. if I recall), I am not sure about their usage (and performance) here on Kaggle. Anyone knowing ressources on how to use those / interested in trying them?
\nEdit: I have also heard of foundationnal approaches to tabular data (Towards Foundation Models for Learning on Tabular Data). I have some concerns regarding efficience / and data leakage. But could be interested if someone know how to get one of those working here.
", "rawMarkdown": "Here is a topic discussing differents NNs approaches that I'd like to see being tested here. \n\n**Classic Architectures:** \n\nMLP, 1-D CNN, DAE, (Transformers ?)\n\nI was thinking about seting up a MLP baseline myself for dicsussion (I have code at hands). But not sure how to select base architecture / parameters. **Does any one have an idea about what should constitute MLP default parameters?** (Maybe matching sklearn defaults parameters?) \n\nAfter that I am generally interested to see how the different arch. performs. (and how they are limited by Kaggle ressources). Maybe this can be a source of discussion regarding the tf-Keras v.s. Pytorch debate. **Are you interested in trying some of the classic architectures?**\n\n**Advanced Architectures:** \n\nI am mainly thinking about the three architectures below: \n[TabNet: Attentive Interpretable Tabular Learning](https://arxiv.org/abs/1908.07442)\n[TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://arxiv.org/abs/2207.01848)\n[TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023](https://arxiv.org/abs/2307.14338)\nGNN ?\n\n\n**Do you know about other DL architectures that claimed to be competitive?**\n\nExcept maybe for Tabnet (part of the ensemble winning the first Optiver Comp. if I recall), I am not sure about their usage (and performance) here on Kaggle. **Anyone knowing ressources on how to use those / interested in trying them?**\n\nEdit: I have also heard of **foundationnal approaches** to tabular data ([Towards Foundation Models for Learning on Tabular Data](https://arxiv.org/abs/2310.07338)). I have some concerns regarding efficience / and data leakage. But could be interested if someone know how to get one of those working here.", "votes": null }, { "id": "2611011", "postDate": "01/20/2024 13:45:21", "content": "Dear @lucasmorin
\n\n\n\"Do you know about other DL architectures that claimed to be competitive?\"
\n
PyTorch Tabular by Manu Joseph (version 1.1.0 was very recently released) has GANDALF: Gated Adaptive Network for Deep Automated Learning of Features:
\n\n\n\"GANDALF outperforms or stays at-par with SOTA approaches like XGBoost, SAINT, FT-Transformers, etc. by experiments on multiple established public benchmarks.\"
\n
Indeed I believe he has contributed a starter notebook \"BenchTab: PyTorch Tabular Starter\".
\nAll the best,
\ncarl
On tabular data I found the soft ordered 1D-CNN really good:
\nhttps://medium.com/spikelab/convolutional-neural-networks-on-tabular-datasets-part-1-4abdd67795b6
That could be an interesting entry
", "rawMarkdown": "On tabular data I found the soft ordered 1D-CNN really good:\nhttps://medium.com/spikelab/convolutional-neural-networks-on-tabular-datasets-part-1-4abdd67795b6\n\nThat could be an interesting entry", "votes": null } ], "comments": [ { "id": 2611011, "author_name": "carlmcbrideellis", "author_url": "", "post_date": "01/20/2024 13:45:21", "content": "Dear @lucasmorin
\n\n\n\"Do you know about other DL architectures that claimed to be competitive?\"
\n
PyTorch Tabular by Manu Joseph (version 1.1.0 was very recently released) has GANDALF: Gated Adaptive Network for Deep Automated Learning of Features:
\n\n\n\"GANDALF outperforms or stays at-par with SOTA approaches like XGBoost, SAINT, FT-Transformers, etc. by experiments on multiple established public benchmarks.\"
\n
Indeed I believe he has contributed a starter notebook \"BenchTab: PyTorch Tabular Starter\".
\nAll the best,
\ncarl
On tabular data I found the soft ordered 1D-CNN really good:
\nhttps://medium.com/spikelab/convolutional-neural-networks-on-tabular-datasets-part-1-4abdd67795b6
That could be an interesting entry
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2610955": "Here is a topic discussing differents NNs approaches that I'd like to see being tested here. \n\n**Classic Architectures:** \n\nMLP, 1-D CNN, DAE, (Transformers ?)\n\nI was thinking about seting up a MLP baseline myself for dicsussion (I have code at hands). But not sure how to select base architecture / parameters. **Does any one have an idea about what should constitute MLP default parameters?** (Maybe matching sklearn defaults parameters?) \n\nAfter that I am generally interested to see how the different arch. performs. (and how they are limited by Kaggle ressources). Maybe this can be a source of discussion regarding the tf-Keras v.s. Pytorch debate. **Are you interested in trying some of the classic architectures?**\n\n**Advanced Architectures:** \n\nI am mainly thinking about the three architectures below: \n[TabNet: Attentive Interpretable Tabular Learning](https://arxiv.org/abs/1908.07442)\n[TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://arxiv.org/abs/2207.01848)\n[TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023](https://arxiv.org/abs/2307.14338)\nGNN ?\n\n\n**Do you know about other DL architectures that claimed to be competitive?**\n\nExcept maybe for Tabnet (part of the ensemble winning the first Optiver Comp. if I recall), I am not sure about their usage (and performance) here on Kaggle. **Anyone knowing ressources on how to use those / interested in trying them?**\n\nEdit: I have also heard of **foundationnal approaches** to tabular data ([Towards Foundation Models for Learning on Tabular Data](https://arxiv.org/abs/2310.07338)). I have some concerns regarding efficience / and data leakage. But could be interested if someone know how to get one of those working here.", "2611011": "Dear @lucasmorin \n\n> \"*Do you know about other DL architectures that claimed to be competitive?*\"\n\n[PyTorch Tabular](https://pytorch-tabular.readthedocs.io/en/latest/) by [Manu Joseph](https://www.kaggle.com/manujosephv) (version 1.1.0 was very recently released) has [GANDALF: Gated Adaptive Network for Deep Automated Learning of Features](https://arxiv.org/pdf/2207.08548.pdf): \n\n> \"*GANDALF outperforms or stays at-par with SOTA approaches like XGBoost, SAINT, FT-Transformers, etc. by experiments on multiple established public benchmarks.*\"\n\nIndeed I believe he has contributed a starter notebook [\"BenchTab: PyTorch Tabular Starter\"](https://www.kaggle.com/code/manujosephv/benchtab-pytorch-tabular-starter).\n\nAll the best,\ncarl", "2611227": "On tabular data I found the soft ordered 1D-CNN really good:\nhttps://medium.com/spikelab/convolutional-neural-networks-on-tabular-datasets-part-1-4abdd67795b6\n\nThat could be an interesting entry" }, "source": "meta" }