# Short summary for our approach (public 5th -> private 212th)

Congrats to all winners. Of course we are a little disappointed at the result, but we would like to share our approach. We believe our experience can be lessons for us, and also for some other people.
Summary
Our best public LB score 1.259 is provided by ensemble of 6 LightGBM models.
Use features of “
[Masters Final Project](https://www.kaggle.com/vettejeep/masters-final-project-model-lb-1-392)
” (Its size is over 800)
Reduced the size up to 449 by:
Kolmogorov–Smirnov test
scipy.stats.pearsonr
adversarial validation
Train LightGBM models with:
gamma regression
rounds different from 10000 to 20000 (separated by 2000)
CV Strategy
KFold
8 fold
shuffle=True
When shuffle=False, cv score in each fold fluctuates a lot
[as olivier says](https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/89366)
. We felt it’s not appropriate for good cv, so I decided to use shuffle=True.
Models
We tried {LightGBM, XGBoost, Catboost, MLP, RNN}. And LightGBM is the best model for us, the others couldn’t be better.
Hyperparams
Gamma regression is suitable for this competition. By just changing objective from
regression
to
gamma
, we can get great gain in public LB score.
Ensemble
We utilize “rounds averaging” for ensemble.
Since we use the augmented features, early stopping doesn’t work properly. We can’t see what is the best round for LightGBM, so we take the average of 6 predictions generated from different rounds models (from 10000 to 20000 separately).
Submission Strategy
We prepared the following 2 submission:
Almost the same model described above
Only one difference is the number of rounds averaging. We use 11 model (from 10000 to 20000 separately)
public: 1.260, private 2.49606
Feature dropped version
We removed all features named ‘mean’ so that we can avoid overfit
public: 1.273, private 2.49699
What didn’t work for us
Convert targets to sqrt(targets) in order that targets follow normal distribution
Remove some train data by adversarial validation
Use meta feature (prediction value as a feature)
Use dart as LightGBM hyperparams
Stacking
Seed averaging
Anyway, I would like to thank every participants especially for my teammates. We would like to learn more from winner solutions!