# 21st Solution and Code Sharing

A huge huge thank you to
[@andrew60909](https://www.kaggle.com/andrew60909)
and
[@ryotak12](https://www.kaggle.com/ryotak12)
for an excellent team experience. We learned a lot and had fun, and I appreciate their patience with my extended, largely unsuccessful efforts to make a transformer work really well 😂. Thanks also to Amex and Kaggle for hosting this interesting competition and to
[@raddar](https://www.kaggle.com/raddar)
and everyone else who shared resources that made this competition much more accessible.
I've made my personal code public here:
[https://github.com/JEddy92/amex_default_kaggle](https://github.com/JEddy92/amex_default_kaggle)
. I tried to create a reasonably general structure for division of FE and modeling, with reusable functionality for training and logging results across a variety of models. There is still some clean up to do and it's not as polished as would be ideal!
Feel free to ask us anything if you have questions about our approach!
Solution Summary
We used logistic regression to ensemble ~60 sets of predictions with diversity obtained from different models, feature sets, and data views. Models included the usual mix of gradient boosters (LGBM, LGBM dart, XGBoost, CatBoost), MLPs, TabNet, and Transformers with a few special tricks. Data views included aggregated data, fully flattened data (13 raw features per customer), sequential data (Transformer), and augmented data (shifted statements forward by 1 and appended to original). Very detailed feature information to follow -- they are split across our 3 different team members' different models, but this aims to be a complete compilation of what we all used.
Datasets & Notation
Everything is derived from Raddar's dataset
\(\mathcal{D}_{fulltrain}\) : Original full size train-set (5531451 rows)
\(\mathcal{D}_{train}\) : Train-set (458913 rows)
\(\mathcal{D}_{fulltest}\) : Original full size test-set
\(\mathcal{D}_{test}\) : Test-set
Primary Features
The train-sets with the best CV score for each of us are generated by \(\mathcal{D}_{fulltrain}\) with these features:
Numeric
Mean, std, max, min, first, last
Last-mean, last/mean, last/first, last/std, max/min, linear-weighted mean
Similar aggregates as above but limited to last 3, 5, etc. statements
Recent diff features and aggregates of all diffs
Monthly-based ranking
Multiplication and Division: create all possible pairs for all features, except for categorical features and features highly correlated with other features. Multiplication and division is done after scaling with MinMaxScaler(0-1) and clipping outliers
Null aggregates (including starting streak vs. at random)
Date features (e.g. stats on customer's statement gaps)
Trend
:  Oldest month as 1 and the newest month as 13, take the average over the following periods
mean(13,12) - mean(11,10)
mean(13,12,11) - mean(10,9,8)
mean(13,12,11) - mean(3,2,1)
mean(13,12,11,10,9,8) - mean(7,6,5,4,3,2)
Category
first, last
entropy: Shannon entropy on frequency table
Nunique: Number of unique category values
Counts of each value in each category (both raw counts and tf-idf counts)
P_2
mean (across train and test) of each category
SVD factorization over time of each category
freq1name : the number of most frequent category
freq1ratio : the number of most frequent category / group size
freq_last1name : the number of least frequent category
freq_last1ratio : the number of least frequent category / group size
PCA
(N, 13*raw_features) → (N, 64)
KNN-based target encoding methods
We selected the nearest 500 points for each sample with euclidean distance(with 2 groups of most important features, each has 14 features. One with last statement of each customer, another one with mean of all statements), calculate the average of target as features
Follow the above method, using nearest 500 points but applying average with weight by the distance matrix
Aggregate the distance matrix, by mean, max, min, std
We also calculated the cosine similarity of each customer's past N months of data in one dimension.(N=1,3,6,13) The average of the 500 nearest neighbor customers' targets is used as the feature. This feature was binned because the distribution is somewhat different between Train and Test
Nested model
We added the labels to \(\mathcal{D}_{fulltrain}\), then trained a LGBM model. The purpose of doing this is to capture what kinds of records, behavior and attributes will cause default. Instead of aggregating features into 458913 rows, this method can let the model learn some extra information from a more "base" level. After that, for each
customer_ID
we will have a "predicted target" that has the same length as the number of records in each
customer_ID
.
Finally, to merge it into \(\mathcal{D}_{train}\) : Train-set (458913 rows), we aggregate similarly to other features: mean, std, max, min, first, last, man-min, last-mean, last/std.
An interesting point here is that the last value of our "predicted value" with \(\mathcal{D}_{fulltrain}\) scores 0.785 on the Amex metric, while the score of the max value is around 0.62 -- indicating the importance of the last value.
Other Feature Methods:
Target/Count encoding
: On \(\mathcal{D}_{train}\)
Target encoding
with \(\mathcal{D}_{fulltrain}\)
Other functions to aggregate
\(\mathcal{D}_{fulltrain}\)
Exponential weight average
Skewness, Kurtosis
The ratio beyond 1std
Median Absolute Deviation
Mean, max, min, std on 1st derivative(t=1,2,…13), timestamp of max and min also used
Shannon entropy: estimate of the spectral density of sequence
Stability, Lumpiness: variance of the means and variance of the variances on tiles of windows
Crossing points: number of times a sequence crosses the median line
Some measurements from
[here](http://isadoranun.github.io/tsfeat/FeaturesDocumentation.html)
Clustering methods
: Kmeans, DBSCAN on important feature(e.g.
P_2_last
or
B_1_last
(including other measurements)). Using raw clusters largely degraded our CV and we're not sure about the reason, so count/target encoding was used here.
pred0 feature
: using full data to predict specific important features via 2 datasets: \(\mathcal{D}_{train}\) and the fullsize one. For example, if we want to predict
P_2
in the former case or predict
P_2_last
in the latter case, we will excluded all "P-related" feature and train a lgbm model on it.
Probit, Logit models with regularization
on different parts of features (e.g. S-related, B-related sets), like how we make pred0 features, we also exclude related features when we construct the predictions
Time-series based methods
to predict important features in the next statement(e.g.
P_2
,
B_1
….)
Curve fitting, smoothing, denoising
:
Linear fit: return coefficient and the value of the next 1,3,6,12 months as features
Poly fit(2,3,4): take the coefficient, and the prediction in next 1 month
LOWESS fit: only with the
customer_ID
that have >10 statements.
Wavelet based method of denoising: Least Asymmetric, Haar, Daubechies(16) then construct some agg features, this method only applies on several important features
Leaf-embedding
: An old trick. We mixed the approaches from
[this paper](https://scontent-tpe1-1.xx.fbcdn.net/v/t39.8562-6/240842589_204052295113548_74168590424110542_n.pdf?_nc_cat=109&ccb=1-7&_nc_sid=ad8a9d&_nc_ohc=nd2mJEAuPSkAX-EVY6m&_nc_ht=scontent-tpe1-1.xx&oh=00_AT9NGACeKxD36CIc-jVHSZ_hyBN5NsVETQihOZqmm7nZ1A&oe=630C598A)
and
[this kaggle code](https://www.kaggle.com/code/mmueller/categorical-embedding-with-xgb)
. We use lgbm with 1round + huge amount of leaves + super high regularization to embed the "whole" data, which has only 1 dimension. And of course using only categorical data or expanding the embedding matrix to the dimension of
nrounds
is also possible.
Most of the above methods are not really able to improve the best CV in a single model, so we turn to utilize it in different models to generate more diversity. Most of our lgbm models with the above selected methods can have a score ranging from CV.796~.801/ PbLB.797~.80 / PriLB.804~.806,  and some of them can generate pretty good diversity. But we are not going to do a detailed ablation study to check the improvement in the PrivateLB of each method :P
Feature Selection Strategies
LightGBM w/ permutation importance
: During the training process, we observed that the metric D is quite noisy and unstable. Probably as a result, permutation importance selection with Amex-Score didn't work here. So we selected features by monitoring G (Gini score) alone and it worked much better! For example, Ryota's FE generated over 20k features, but the final number of features utilized was about 1200.
Model Details
GBDT
LightGBM (gbdt, dart) - classification, regression: CV 0.796~0.801, Public 0.799, Private 0.805~.806
XGBoost - classification: CV 0.797~0.800, Public 0.798, Private 0.805
CatBoost - classification, regression CV 0.796~0.799, Public 0.797, Private 0.804
LGB linear_tree, ExtraTrees, RF, RGF and rmse objective to generate some diversity
Some LGB dart models trained on the fully flattened view of the data (don't score well individually but ensemble well)
Some LGB dart models trained on features derived from augmented data view
TabNet
TabNet: CV 0.793 Public 0.794 Private 0.801
Residual Learning (LightGBM): CV 0.795, Public 0.796 Private 0.802 (specifically, we predict
ground truth - tabnet’s prediction
with LightGBM and then finalize predictions as
tabnet's prediction + resid prediction
MLP
MLP: CV 0.796 Public 0.794 Private 0.802
Transformer
CV: .793
-> .794 with residual learning
All nulls imputed with LGB trained on all other features, categories encoded as mean
P_2
value from entire train+test. Data Augmented ~2x by statement shift strategy
Example Dart hyper-parameters from Angus, no special sauce here:)  :
lgb_param <- list(boosting_type = 'dart',
                  objective = "binary",
                  metric = amex,
                  learning_rate = 0.02,
                  num_leaves = 48,
                  feature_fraction = 0.1,
                  #bagging_freq = 1,
                  #bagging_fraction = 1,
                  min_child_weight = 1,
                  lambda_l1 = 1,
                  lambda_l2 = 64,
                  skip_drop = 0.8
)
Some Other Interesting Things we Tried
Private set specialization
: We tried using the KNN result mentioned above to estimate which samples are closer to Private LB data. In particular, we calculated the nearest 500 points for each sample, then  the ratio of each set in those 500 points. After that, we took the top 30% of samples which are closer to private LB and ran a logistic regression to decide the weight of our final ensemble. But unfortunately, the CV score dropped a lot. The CV of the top 30% data is ~0.88, but overall(450k data) Amex-score is ~0.7812. We also tried the top 30% "public ratio" one. The CV is ~0.7259 and the overall CV is 0.7896711. It's possible that using adversarial validation predictions trained on top importance features would be a more reliable option.
Transfer learning on
P_2
: this idea unfortunately came up very close to the end before it could pay off and probably was worth pursuing further. This was an attempt at exploiting the test data - pretrain a transformer on train+test to predict
P_2
(because it's strongly related to the target), then fine-tune on the actual target with train. This is like a worse version of the knowledge distillation techniques that were used to exploit the test data more for neural net training.
Psuedo labeling
: We only got 0.00005 improvement on CV (LB doesn't seem tio change), and we have tuned the size and what proportion of test-set can be used for it but didn't have real luck. However, log loss and AUC may be high enough to do this more optimally and get a considerable improvement.
Ensembling methods
, we tried lgbm with 50 seeds bagging, LR and optimization methods like
md1*par1 + md2*par2....
with Nelder-Mead or L-BFGS-B solver. LR performed the best (CV.8031), and lgbm seemed to be severely overfitting.
Reflections
We fell hard from 3rd Public -> 21st Private, with 6 submissions that would have scored gold. In hindsight, CV here did not seem to be a great measurement for the private LB, even though CV improvement aligned very well with public LB. We probably had some bad luck, but 2 things we may have benefited from doing differently are:
Exploiting the test data more with knowledge distillation, pretraining, etc.
Hedged our submission strategy more - the two submissions we chose (best Public LB and best CV) were not that different from each other, and choosing a different backup could have been a better way to game the noisiness here