# 2020 place overview

First and foremost we would like to thank Raddar for his cleaned and compressed dataset
[[1]](https://www.kaggle.com/datasets/raddar/amex-data-integer-dtypes-parquet-format)
(which we used) and Martin for his two magnificent DART notebooks
[[2]](https://www.kaggle.com/code/ragnar123/amex-lgbm-dart-cv-0-7963)
[[3]](https://www.kaggle.com/code/ragnar123/amex-lgbm-dart-cv-0-7977)
. We finally used a stacking of XGBoost, LightGBM and TabNet. When blending our solution with that of Martin we obtained a Private score of 0.80761, and playing the hypothetical
what if
game this would have landed around 138th on the LB. However, although acutely aware that our solution was not competitive, we still went ahead with selecting only our own work, and followed the sage advice of Raddar in submitting our best CV, and (our best CV + our best LB)/2 works. Doing this left our CV score on the wrong side of the CV of the work published by Martin, and consequently way down the LB, but with no regrets whatsoever.
Things perhaps of mention:
Feature engineering: We spent an inordinate  amount of time with the features in conjunction with the 13GB GPU memory limit on kaggle, and with each new feature sadly eventually having to sacrifice in the end around 30 other features (team-mate
[@danielhanchen](https://www.kaggle.com/danielhanchen)
helped with that process) to stay within the notebook memory. In retrospect we should really have heavily sub-sampled the training data (if it wasn't for the noisy
D
component in the metric needing lots of data that would definitely be the way to go) in order to keep as many of these features as possible as almost all of them added just a little something (we did try the random under-sampling
[RUSBoostClassifier](https://imbalanced-ensemble.readthedocs.io/en/latest/api/ensemble/_autosummary/imbalanced_ensemble.ensemble.RUSBoostClassifier.html)
from
[imbalanced-ensemble](https://imbalanced-ensemble.readthedocs.io/en/latest/index.html)
, but it did not work well).
As well as a lot of futher cleaning to remove outliers
etc
, things of particular note were:
The features
S_8
and
S_13
seemed reminiscent of some sort of FICO credit rating scores. With that in mind we created an additional two new binary features with a split (
np.where
) chosen by fitting just these features alone with a RandomForest 'stump' with the idea of 'good' or 'bad' creditor in mind.
As well as generating the usual aggregated features for all of the (up to) 13 statements, we found that also using such aggregations just over the last 3 statements, and the last 6 statements, proved to produce some very informative features.
We found that the feature
S_3/S_7
helped.
We found that the average of the three features
D_115
,
D_118
and
D_119
also helped a little
On the float features we used the
round(2)
trick suggested by Jiwei Liu
[[4]](https://www.kaggle.com/code/jiweiliu/amex-catboost-rounding-trick)
Our final number of features after aggregations was 1145
Estimators:
We started with also using CatBoost, but in the end we found the CV scores were not on par with those of XGBoost and LGBM
We used DART with LGBM thanks to the work of Martin (but not with XGBoost; incredibly slow!)
Our only novelty with XGBoost was using the
num_parallel_tree=7
parameter, which slowed down the training by around 7 times, but made for a smoother training curve and a CV with a lower standard deviation.
As per the magnificent notbook by Chris Deotte
[[5]](https://www.kaggle.com/code/cdeotte/xgboost-starter-0-79391)
we used surprisingly  shallow estimators (
max_depth=4
) as going deeper didn't really seem to help
Combined multiple runs of each estimator, each using different CV splitting seeds
Although TabNet alone did not have a spectacular CV score despite the best efforts of my team-mate
[@optimo](https://www.kaggle.com/optimo)
, it helped every ensemble it touched.
The OOF predictions of these estimators were then calibrated and then fed into a stacking ensemble.
We were also aware that the distribution of some of the features in the test data were significantly different from those in the training data (some wonderful notebooks on this were made on this by Pavel Vodolazov
[[6]](https://www.kaggle.com/code/pavelvod/amex-eda-revealing-time-patterns-of-features)
[[7]](https://www.kaggle.com/code/pavelvod/amex-eda-even-more-insane-time-patterns-revealed)
). In view of the fact that usually tree based estimators cannot extrapolate we tried out the
[linear-tree](https://github.com/cerlymarco/linear-tree)
estimator, and also the
gblinear
booster in XGBoost, but neither were used in the end; they may well have eventually performed better on the test data for the features that had covariate shifts, but there was little way of telling beforehand.
All in all a very enjoyable competition and many thanks to AmEx for providing us with tabular data to play with, and kaggle for the computational resources!
All the best,
carl