Final Submission Code: here
\nFirst, I'd like to sincerely thank @andreipaulavets for organizing this competition and for providing thoughtful feedback throughout. Your comments helped shape and sharpen many of my experiments. Thanks also to @ravi20076 for introducing such a great learning opportunity.
\nData Overview:
\nThe dataset captures complex, non-linear relationships between card removal patterns and expected value (EV) changes in a 6-deck Blackjack shoe. Models capable of learning these non-linearities, like Neural Networks and GBDTs, performed well.
Model Comparisons:
\nGBDT Models:
\nCatBoost initially performed very well, outperforming XGBoost and LightGBM in early trials. Parameter tuning was critical, with CatBoost requiring deeper trees (depth=6) for best results.
Neural Networks:
\nNeural Networks ultimately outperformed all other models. After noticing that NN-based models consistently ranked high in AutoGluon's leaderboard, I explored different architectures through a grid-style search. The most effective architecture was:
nn_model.add(Dense(, activation=, input_dim=X_train_scaled.shape[]))\nnn_model.add(Dense(, activation=))\nnn_model.add(Dense(, activation=))\nnn_model.add(Dense(, activation=))\nnn_model.add(Dense(, activation=))\nnn_model.add(Dense(, activation=))\nnn_model.add(Dense(, activation=))\nnn_model.(optimizer=, loss=)\nThis deep, dropout-free network achieved extremely low error. To reduce prediction variance, I trained three models with different seeds and averaged their predictions.
While earlier trials showed that engineered features like card counts, averages, and ratios helped GBDT models, the final DNN model performed best using only the original features. This suggests the DNN was able to internally capture the structure and relationships in the data without explicit feature engineering—thanks in part to the clean and low-noise nature of the dataset.
\nThis competition was a valuable lesson in questioning common assumptions. While GBDTs are often considered best for tabular data, this challenge showed that deep neural networks can outperform them in settings where the data is clean and highly structured. Careful tuning of architecture and training strategies enabled the DNN to reach top-tier performance.
", "comments": [ { "id": 3189893, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2025-04-29T22:36:06.790000", "content": "@masayakawamata what was the intuition for removing drop-outs? I think we are almost always ingrained to use layer normalization and dropout across architectures. In fact, using such a large network may goad you to use higher dropout rates to reduce the risk of overfitting.
\nWhat made you think to let go off layer-normalization and drop-outs and did you use weight-decay?
", "votes": 3, "replies": [ { "id": 3189901, "author_name": "Masaya Kawamata", "author_url": "", "post_date": "2025-04-29T22:50:16.503000", "content": "At first, I used dropout and normalization layers, but through a lot of experimentation, I noticed that removing them consistently led to better performance.
\nAs for why BatchNorm might have hurt performance:
\nSince this was a small-scale regression task with pre-standardized 10-dimensional inputs, adding BatchNorm introduced additional learnable parameters (gamma and beta) that rescale and shift the data.
\nThis extra rescaling made the gradients unnecessarily unstable, and because the target values were already small (~10⁻² range), the floating-point errors might have been amplified instead of reduced.
\nFor similar reasons, introducing dropout might have added unnecessary noise to the network, disrupting the small, meaningful signals we needed to capture, rather than helping with generalization.
Really impressive work with the DNN… It's interesting how deep learning ended up outperforming GBDTs, especially with clean and structured data. I also found it surprising that the DNN didn’t rely much on feature engineering, given how crucial it often is for other models…….Congrats on the win
", "votes": 2, "replies": [ { "id": 3190261, "author_name": "Masaya Kawamata", "author_url": "", "post_date": "2025-04-30T11:41:26.793000", "content": "Thanks! I really appreciate it. I tend to think of DNNs as models that can essentially generate their own features. With GBDTs, the idea that performance improves when you create features that help the model make better splits feels quite intuitive. But those same features don’t always translate well to neural networks. In some sense, it feels like the features that matter for a DNN are ones the network is better off discovering for itself.
\nOne reason DNNs aren’t often used for tabular data is, I think, because tabular datasets tend to be noisy, and neural nets are quite prone to overfitting that noise. GBDTs, on the other hand, are relatively robust in that respect. But this project was a good example of how, under ideal conditions, DNNs can actually be very effective—even on structured data.
", "votes": 2, "replies": [] } ] }, { "id": 3189933, "author_name": "shakaut hassain⚡", "author_url": "", "post_date": "2025-04-30T01:24:44.603000", "content": "Thanks a lot for your great contribution, A lot of thinks we have to learn from you.
", "votes": 2, "replies": [] }, { "id": 3189872, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2025-04-29T22:02:48.547000", "content": "Wow such a deep network @masayakawamata
\nI think my shortcoming was the usage of a shallow network in my TabM model- this did not perform well at all and I gave up on this without increasing the depth and width of the network!
", "votes": 2, "replies": [ { "id": 3189877, "author_name": "Masaya Kawamata", "author_url": "", "post_date": "2025-04-29T22:14:47.597000", "content": "Thank you for introducing the competition on Playground Discussions @ravi20076 !
\nI also tried using TabM, but like you, I struggled to get good performance with it. And, I was surprised to find that deeper networks significantly improved the results.
\nAlthough I'm not entirely sure if my understanding of TabM is correct, my impression is that its idea of building many small and shallow networks might not have matched this dataset very well. Since the dataset had very little noise, it was less prone to overfitting, and the main challenge seemed to be how clearly we could capture the signal. That's why I think deeper and wider architectures turned out to be more effective here.
", "votes": 2, "replies": [ { "id": 3189882, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2025-04-29T22:27:55.053000", "content": "When the shallow TabM model did not work, I thought that a deep TabM won't work either - this was my biggest mistake! Without doing it, I assumed that it won't work and left it early - on Kaggle, one needs to experiment and see the results and then infer accordingly!
\nThis is my biggest lesson! If a 22500 iteration catboost with depth 6 worked well, a TabM/ NN model with wide and deep layers would work as well! This should have been my line of thought rather than since a shallow network did not work, I assume that a deeper network will also not work
\n", "votes": 2, "replies": [] }, { "id": 3189884, "author_name": "Andrei Paulavets", "author_url": "", "post_date": "2025-04-29T22:31:01.790000", "content": "Since target values were produced by combinatorial computations, it's a pure signal, with no margin of error.
\nA combinatorial approach is hard to set up and expensive to run, thus, this competition was born.
\nSo, what I assume the NN model was able to capture is the underlying non-linear nature of expected value calculations.
\nLooking back, I'm kind of surprised that I haven't been able to see this. Since deep NNs are indeed well fit for this kind of task.
\nOnce again, beautiful finding. It definitely took a lot of effort to get here.
Thanks for your participation. You defenetly dug far deeper then anyone else, provided a lot of insights as well as the best solution.
\nI'm realy enjoing following your work, also in month or two will publish a new competition, will be happy to see you taking the part.
\nCongratulations with the flawless victory!
Thank you for organizing such a great competition. I truly enjoyed it and learned a lot. I'm really looking forward to the next one and would be happy to take part again!
", "votes": 2, "replies": [] } ] } ], "comment_count": 11, "messages": [ { "id": 3189863, "postDate": "2025-04-29T21:53:42.573Z", "content": "Final Submission Code: here
\nFirst, I'd like to sincerely thank @andreipaulavets for organizing this competition and for providing thoughtful feedback throughout. Your comments helped shape and sharpen many of my experiments. Thanks also to @ravi20076 for introducing such a great learning opportunity.
\nData Overview:
\nThe dataset captures complex, non-linear relationships between card removal patterns and expected value (EV) changes in a 6-deck Blackjack shoe. Models capable of learning these non-linearities, like Neural Networks and GBDTs, performed well.
Model Comparisons:
\nGBDT Models:
\nCatBoost initially performed very well, outperforming XGBoost and LightGBM in early trials. Parameter tuning was critical, with CatBoost requiring deeper trees (depth=6) for best results.
Neural Networks:
\nNeural Networks ultimately outperformed all other models. After noticing that NN-based models consistently ranked high in AutoGluon's leaderboard, I explored different architectures through a grid-style search. The most effective architecture was:
nn_model.add(Dense(, activation=, input_dim=X_train_scaled.shape[]))\nnn_model.add(Dense(, activation=))\nnn_model.add(Dense(, activation=))\nnn_model.add(Dense(, activation=))\nnn_model.add(Dense(, activation=))\nnn_model.add(Dense(, activation=))\nnn_model.add(Dense(, activation=))\nnn_model.(optimizer=, loss=)\nThis deep, dropout-free network achieved extremely low error. To reduce prediction variance, I trained three models with different seeds and averaged their predictions.
While earlier trials showed that engineered features like card counts, averages, and ratios helped GBDT models, the final DNN model performed best using only the original features. This suggests the DNN was able to internally capture the structure and relationships in the data without explicit feature engineering—thanks in part to the clean and low-noise nature of the dataset.
\nThis competition was a valuable lesson in questioning common assumptions. While GBDTs are often considered best for tabular data, this challenge showed that deep neural networks can outperform them in settings where the data is clean and highly structured. Careful tuning of architecture and training strategies enabled the DNN to reach top-tier performance.
", "rawMarkdown": "**Final Submission Code:** [here](https://www.kaggle.com/code/masayakawamata/bj-deep-neural-network?)\n\n---\n\n## Acknowledgements\n\nFirst, I'd like to sincerely thank @andreipaulavets for organizing this competition and for providing thoughtful feedback throughout. Your comments helped shape and sharpen many of my experiments. Thanks also to @ravi20076 for introducing such a great learning opportunity.\n\n---\n\n## TL;DR\n\n- **Best Model:** Deep Neural Network (DNN), trained and averaged over 3 different seeds.\n- **Evaluation:** 10-fold KFold cross-validation.\n - **Final DNN Architecture:** A deep feedforward network without dropout.\n- **Performance:**\n - **CV:** ~0.0000000200\n - **Public LB:** 0.0000000092\n - **Private LB:** 0.00000000109\n\n---\n\n## Dataset Characteristics & Model Performance\n\n- **Data Overview:** \n The dataset captures complex, non-linear relationships between card removal patterns and expected value (EV) changes in a 6-deck Blackjack shoe. Models capable of learning these non-linearities, like Neural Networks and GBDTs, performed well.\n\n- **Model Comparisons:**\n - **GBDT Models:** \n CatBoost initially performed very well, outperforming XGBoost and LightGBM in early trials. Parameter tuning was critical, with CatBoost requiring deeper trees (depth=6) for best results.\n\n - **Neural Networks:** \n Neural Networks ultimately outperformed all other models. After noticing that NN-based models consistently ranked high in AutoGluon's leaderboard, I explored different architectures through a grid-style search. The most effective architecture was:\n\n ```python\n nn_model.add(Dense(256, activation='relu', input_dim=X_train_scaled.shape[1]))\n nn_model.add(Dense(256, activation='relu'))\n nn_model.add(Dense(256, activation='relu'))\n nn_model.add(Dense(256, activation='relu'))\n nn_model.add(Dense(128, activation='relu'))\n nn_model.add(Dense(128, activation='relu'))\n nn_model.add(Dense(1, activation='linear'))\n nn_model.compile(optimizer='adam', loss='mean_squared_error')\n ```\n\n This deep, dropout-free network achieved extremely low error. To reduce prediction variance, I trained three models with different seeds and averaged their predictions.\n\n---\n\n## Feature Engineering\n\nWhile earlier trials showed that engineered features like card counts, averages, and ratios helped GBDT models, the final DNN model performed best using only the original features. This suggests the DNN was able to internally capture the structure and relationships in the data without explicit feature engineering—thanks in part to the clean and low-noise nature of the dataset.\n\n---\n\n## Conclusion\n\nThis competition was a valuable lesson in questioning common assumptions. While GBDTs are often considered best for tabular data, this challenge showed that deep neural networks can outperform them in settings where the data is clean and highly structured. Careful tuning of architecture and training strategies enabled the DNN to reach top-tier performance.", "votes": 8 }, { "id": 3189893, "postDate": "2025-04-29T22:36:06.790Z", "content": "@masayakawamata what was the intuition for removing drop-outs? I think we are almost always ingrained to use layer normalization and dropout across architectures. In fact, using such a large network may goad you to use higher dropout rates to reduce the risk of overfitting.
\nWhat made you think to let go off layer-normalization and drop-outs and did you use weight-decay?
", "rawMarkdown": "@masayakawamata what was the intuition for removing drop-outs? I think we are almost always ingrained to use layer normalization and dropout across architectures. In fact, using such a large network may goad you to use higher dropout rates to reduce the risk of overfitting.\n\nWhat made you think to let go off layer-normalization and drop-outs and did you use weight-decay? ", "votes": 3, "replies": [ { "id": 3189901, "postDate": "2025-04-29T22:50:16.503Z", "content": "At first, I used dropout and normalization layers, but through a lot of experimentation, I noticed that removing them consistently led to better performance.
\nAs for why BatchNorm might have hurt performance:
\nSince this was a small-scale regression task with pre-standardized 10-dimensional inputs, adding BatchNorm introduced additional learnable parameters (gamma and beta) that rescale and shift the data.
\nThis extra rescaling made the gradients unnecessarily unstable, and because the target values were already small (~10⁻² range), the floating-point errors might have been amplified instead of reduced.
\nFor similar reasons, introducing dropout might have added unnecessary noise to the network, disrupting the small, meaningful signals we needed to capture, rather than helping with generalization.
Really impressive work with the DNN… It's interesting how deep learning ended up outperforming GBDTs, especially with clean and structured data. I also found it surprising that the DNN didn’t rely much on feature engineering, given how crucial it often is for other models…….Congrats on the win
", "rawMarkdown": "Really impressive work with the DNN... It's interesting how deep learning ended up outperforming GBDTs, especially with clean and structured data. I also found it surprising that the DNN didn’t rely much on feature engineering, given how crucial it often is for other models.......Congrats on the win", "votes": 2, "replies": [ { "id": 3190261, "postDate": "2025-04-30T11:41:26.793Z", "content": "Thanks! I really appreciate it. I tend to think of DNNs as models that can essentially generate their own features. With GBDTs, the idea that performance improves when you create features that help the model make better splits feels quite intuitive. But those same features don’t always translate well to neural networks. In some sense, it feels like the features that matter for a DNN are ones the network is better off discovering for itself.
\nOne reason DNNs aren’t often used for tabular data is, I think, because tabular datasets tend to be noisy, and neural nets are quite prone to overfitting that noise. GBDTs, on the other hand, are relatively robust in that respect. But this project was a good example of how, under ideal conditions, DNNs can actually be very effective—even on structured data.
", "rawMarkdown": "Thanks! I really appreciate it. I tend to think of DNNs as models that can essentially generate their own features. With GBDTs, the idea that performance improves when you create features that help the model make better splits feels quite intuitive. But those same features don’t always translate well to neural networks. In some sense, it feels like the features that matter for a DNN are ones the network is better off discovering for itself.\n\nOne reason DNNs aren’t often used for tabular data is, I think, because tabular datasets tend to be noisy, and neural nets are quite prone to overfitting that noise. GBDTs, on the other hand, are relatively robust in that respect. But this project was a good example of how, under ideal conditions, DNNs can actually be very effective—even on structured data.", "votes": 2 } ] }, { "id": 3189933, "postDate": "2025-04-30T01:24:44.603Z", "content": "Thanks a lot for your great contribution, A lot of thinks we have to learn from you.
", "rawMarkdown": "Thanks a lot for your great contribution, A lot of thinks we have to learn from you.", "votes": 2 }, { "id": 3189872, "postDate": "2025-04-29T22:02:48.547Z", "content": "Wow such a deep network @masayakawamata
\nI think my shortcoming was the usage of a shallow network in my TabM model- this did not perform well at all and I gave up on this without increasing the depth and width of the network!
", "rawMarkdown": "Wow such a deep network @masayakawamata \n\nI think my shortcoming was the usage of a shallow network in my TabM model- this did not perform well at all and I gave up on this without increasing the depth and width of the network!", "votes": 2, "replies": [ { "id": 3189877, "postDate": "2025-04-29T22:14:47.597Z", "content": "Thank you for introducing the competition on Playground Discussions @ravi20076 !
\nI also tried using TabM, but like you, I struggled to get good performance with it. And, I was surprised to find that deeper networks significantly improved the results.
\nAlthough I'm not entirely sure if my understanding of TabM is correct, my impression is that its idea of building many small and shallow networks might not have matched this dataset very well. Since the dataset had very little noise, it was less prone to overfitting, and the main challenge seemed to be how clearly we could capture the signal. That's why I think deeper and wider architectures turned out to be more effective here.
", "rawMarkdown": "Thank you for introducing the competition on Playground Discussions @ravi20076 ! \n\nI also tried using TabM, but like you, I struggled to get good performance with it. And, I was surprised to find that deeper networks significantly improved the results.\n\nAlthough I'm not entirely sure if my understanding of TabM is correct, my impression is that its idea of building many small and shallow networks might not have matched this dataset very well. Since the dataset had very little noise, it was less prone to overfitting, and the main challenge seemed to be how clearly we could capture the signal. That's why I think deeper and wider architectures turned out to be more effective here.", "votes": 2, "replies": [ { "id": 3189882, "postDate": "2025-04-29T22:27:55.053Z", "content": "When the shallow TabM model did not work, I thought that a deep TabM won't work either - this was my biggest mistake! Without doing it, I assumed that it won't work and left it early - on Kaggle, one needs to experiment and see the results and then infer accordingly!
\nThis is my biggest lesson! If a 22500 iteration catboost with depth 6 worked well, a TabM/ NN model with wide and deep layers would work as well! This should have been my line of thought rather than since a shallow network did not work, I assume that a deeper network will also not work
\n", "rawMarkdown": "When the shallow TabM model did not work, I thought that a deep TabM won't work either - this was my biggest mistake! Without doing it, I assumed that it won't work and left it early - on Kaggle, one needs to experiment and see the results and then infer accordingly! \n\nThis is my biggest lesson! If a 22500 iteration catboost with depth 6 worked well, a TabM/ NN model with wide and deep layers would work as well! This should have been my line of thought rather than *since a shallow network did not work, I assume that a deeper network will also not work*\n\n@masayakawamata ", "votes": 2 }, { "id": 3189884, "postDate": "2025-04-29T22:31:01.790Z", "content": "Since target values were produced by combinatorial computations, it's a pure signal, with no margin of error.
\nA combinatorial approach is hard to set up and expensive to run, thus, this competition was born.
\nSo, what I assume the NN model was able to capture is the underlying non-linear nature of expected value calculations.
\nLooking back, I'm kind of surprised that I haven't been able to see this. Since deep NNs are indeed well fit for this kind of task.
\nOnce again, beautiful finding. It definitely took a lot of effort to get here.
Thanks for your participation. You defenetly dug far deeper then anyone else, provided a lot of insights as well as the best solution.
\nI'm realy enjoing following your work, also in month or two will publish a new competition, will be happy to see you taking the part.
\nCongratulations with the flawless victory!
Thank you for organizing such a great competition. I truly enjoyed it and learned a lot. I'm really looking forward to the next one and would be happy to take part again!
", "rawMarkdown": "Thank you for organizing such a great competition. I truly enjoyed it and learned a lot. I'm really looking forward to the next one and would be happy to take part again!", "votes": 2 } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T02:32:21Z", "competition_id": 94020, "competition_slug": "black-jack-smart-effect-of-removal-ml", "team_id": 13409693, "solution_url": "/competitions/black-jack-smart-effect-of-removal-ml/writeups/masaya-kawamata-1st-place-solution-single-nn-3-ran" }