# 12th Place Gold (1/2) - lgbm+xgboost+FCN+Transformer

Thank you Amex and Kaggle for hosting! Very fun competition with a lot of insights.
Solution
This is the first of two posts of our ensemble
[@gandagorn](https://www.kaggle.com/gandagorn)
.  My leg of the ensemble consisted of taking different preprocessing methods and passing them through different architectures (Thank you
[@jiweiliu](https://www.kaggle.com/jiweiliu)
[@lucasmorin](https://www.kaggle.com/lucasmorin)
[@raddar](https://www.kaggle.com/raddar)
! for your excellent notebooks)
second part  is
[https://www.kaggle.com/competitions/amex-default-prediction/discussion/348058](url)
![](assets/3ef27a188c9442508342e26e.png)
Preprocessing
Used the
[@raddar](https://www.kaggle.com/raddar)
integer features  and raw features and then created  separate pre-processing pipelines including standard aggregations (min, max means, last, diff)
Added top 50 PCA Features.
Globally ranked all the features and built products of features that shared the same trend. This boosted all the models.
Used embedded layers for all categorical features.
PostProcessing
Sequentially removed features accordingly to split importance.
Models
LGBM with Focal Loss + logloss
Standard lgbm trained with dart with one-sided focal loss added to the log-loss. The addition of the one-side focal loss was with the objective of having a better classification of the positive classes.
XGBoost
Nothing special.
FCN
Fully connected architecture using the same features that were used in the boosted trees. 20 folds CV.
FCN + Transformer Encoder
Probably the most interesting model.  A NN with double inputs: tabular features (from previous preprocessing) and full-time series of ranked raw features. The training process was as follows:
Freeze Transformer weights and train the FCN architecture for ~30 epochs to produce 1d output [0,1]
Freeze FCN weights and train the Transformer for ~30 epochs to produce 1d output [0,1]
Freeze Transformer weights and train the FCN architecture for ~30 epochs to produce 1d output [0,1]
![](assets/4871c3d5df2fa79c68d3e749.png)
For the Transformer, I used the standard TransformerEncoderBlock with 8 heads and 2 layers