# (66th) My Reflections On This Competition

First of all, congratulations to all winners and medalists!
I have been reflecting on what I should have done better after my team fell 40+ places on the private leaderboard. Confusingly, looking at the private scores of my team’s submissions, I haven’t found any of my strategies/approaches would have deterministically outperformed the others in the private leaderboard. Here’s my list of thoughts.
Single Model
vs
Model Ensemble
: Most of the top scoring submissions are ensembles, but there are some single models on the top scoring list as well. I will still opt for an ensemble next time though.
Heavy Ensemble
vs
Light/Simple Ensemble
: No obvious pattern found. Many light ensemble models (1-layer stacking with logistic regression on just a few models) outperform heavy ensembles and vice versa.
Specific Ensemble Strategy
: Nothing seems to always work. For each strategy, we have many similar ensembles. They can range from top 30 to maybe 200th with similar CV and public LB.
CV
vs
LB
vs
?
: We have quite a few private 808 subs and none of them are best CV or best LB. I can easily find a similar model with a better CV and LB and scores a lot worse on the private LB.
NN/GBDT Ensemble
vs
Pure-GBDT
: Again no conclusive observation here. Including NN in the ensemble doesn’t seem to deterministically make performance better or worse in my case.
Sadly no immediate takeaways from my own solutions ☹️☹️☹️.
Things I learned post-competition that worked well for others:
[Knowledge Distillation](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347641)
from
[@cdeotte](https://www.kaggle.com/cdeotte)
Capturing time-series dynamics within each column
[SequentialEncoder](https://www.kaggle.com/code/pavelvod/27-place-sequentialencoder?scriptVersionId=104154431)
from
[@pavelvod](https://www.kaggle.com/pavelvod)
and
[Mini-LSTMs](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347651)
from
[@fritzcremer](https://www.kaggle.com/fritzcremer)
[Ensemble of multiple TabNets](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347688)
from
[@raddar](https://www.kaggle.com/raddar)
Hope you can share your experience here regarding what consistently worked great for you or your takeaways.