# 5th Place Solution - Team 💳VISA💳(Summary&zakopuro's part)

I would like to thank the organizers for organizing this competition, the participants for sharing their many insights, and my teammates(
[@baosenguo](https://www.kaggle.com/baosenguo)
[@wimwim](https://www.kaggle.com/wimwim)
[@scumufeng](https://www.kaggle.com/scumufeng)
). Now
[@scumufeng](https://www.kaggle.com/scumufeng)
and I are promoted to kaggle master and
[@wimwim](https://www.kaggle.com/wimwim)
is reaching for GM.
This competition will be unforgettable for me:)
Summary
Our solution is the result of ensembling several GBDT models , Transfomr, 2d-CNN, and GRU.
We noticed that ensemble weights are determined based on Public LB and overfit if based on CV.
![](assets/f911c8f796b3b6040cdd9baf.png)
Features
We are using the dataset shared with us by raddar. The features are based on those shared by ragnar.(Thanks to both of you.)
meta feature
I did not know this is called a meta feature.
This feature was useful not only in GBDT, but also in Transformer.
If added to
[chris's Transformer](https://www.kaggle.com/code/cdeotte/tensorflow-transformer-0-790l)
, the LB will increase from 0.790 to 793.
![](assets/c178006b564f25777a8c0194.png)
Pivot
Combine all features horizontally.
P_2_0 , P_2_1 , P_2_3 , P_2_4 , ... , P_2_12 , B_30_0 , ... 
 XXX  ,  XXX  ,  XXX  ,  XXX  , ... ,   XXX  ,   YYY  , ...
Model
GBDT
LightGBM
Almost no change from
[this Notebook](https://www.kaggle.com/code/ragnar123/amex-lgbm-dart-cv-0-7977)
stratfiedKfold: 10
Catboost
Use GPU(I was surprised at how fast it was.)
parameter : default
Transformer
zakopuro
Based on
chris's Notebook
Some additional features.(Mainly meta features)
Patrick Yam
This is his solution.
[https://www.kaggle.com/competitions/amex-default-prediction/discussion/348118](https://www.kaggle.com/competitions/amex-default-prediction/discussion/348118)
mufeng
Add meta feature to Patrick's transformer.
LB increased from 0.795 to 0.798.
Ensemble
Use 21 models.
Ensemble weight
Determined based on Public LB
In our case, the LB score will be lower if based on CV.(CV is 0.8016 or higher.)
We trusted Public LB more than CV because it is close to Private and has a sufficient amount of data.
Weights are not complicated. (For example, 0.1,0.2,… etc.)
Select Submit
Best LB
Public : 0.80199(2nd)
Priavte : 0.80881(6th)
Best LB*0.5 + Best CV *0.5
Public : 0.80154
Private : 0.80862(Gold zone)
Correlation check
Check the Public and Private correlation values for all predictions used in the ensemble to see that there are no significant differences.
All posts above 0.801 in Public LB were in the Gold zone in Private LB.(I prayed on the last day not to Shake down😣)
Let's enjoy kaggle! Thank you!!!!