# 87th place private and 1st place public solution

I want to thank Kaggle and the great community of this competition participants for making this journey possible! This is a write-up of our work with my teammate
[@idog90](/idog90)
.
First, please check out
[the LB probing strategy](https://medium.com/@zaharchikishev/how-to-lb-probe-on-kaggle-c0aa21458bfe?source=friends_link&sk=543b4ebcafee24697979b69efdb35adf)
that gave us the public leader-board first place. In this post I will describe the other components of our solution, focusing on the submission that gave us 87th place.
features and models
We generated 270 features, mostly taken from the public kernels with some modifications. By using recursive feature elimination with permutation importance by LGBM we reduced the feature set to 90 features. Regarding the segment mean, we subtracted the mean from the signal before calculating the features, but left it as an additional feature because it improved both CV and LB. Leaving it was possibly a critical mistake. 
Models that participated are LGBM, XGB, CatBoost, SVR and shallow NN by fastai. Additionally we added a median of classification output of LGBM, by dividing the targets space into 11 classes. 
The 6 models oof predictions were blended together by linear stacking with MAE objective function and non-negativity constraints. Using 6 models and stacking gave more or less similar scores to the best single model scores (XGB or CatBoost), so this setup is probably redundant but no harm.
CV strategy
We used 25k stride to generate train segments, and then assigned every 35 consecutive segment into a group of its own. Then we removed the segments with data overlapping into the next group, and used stratified group 5-fold CV strategy with the groups as defined above. I think it worked well for us.
secret weapon
We used two extra elements that gave us the medal. The second final submission didn't include it and scored somewhere around 500 on the private.
a. First, we merged short quakes 5 and 6 together, considering them as a single long quake. This is of course to make the train distribution to be more similar to the test. This is the change that made the difference.
b. Second, we added public test segments to the train with their best estimated scores from the LB probing strategy linked above. I don't think it had much impact, it was less than 2% of the train data, and noisy.
using MIP solution
As also mentioned in the medium article linked above, our mixed integer programming (MIP) solution for the LB probing had a nice ability to calculate public LB scores of potential submissions with great precision (0.02). In a sense, the MIP model learnt very well the subspace of all "normal" submissions and therefore gave excellent estimations for anything in that subspace. We used it to estimate public LB scores for our final submissions candidates on the last day.
what didn't work
Huge amount of things that we tried didn't work well. To name the major ones:
a.
[batch start identification](https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/94435#latest-543574)
didn't help
b. transformer NNs on raw data gave bad CV and LB scores
c. spectograms with CNN was OK but worse than our best models and we didn't include it at the end.
e. reweighting of the train set into the known test distribution, fitting histograms of the classification models to match the test distribution, - didn't work well on test, still an enigma for me.
f. genetic programming, - but we are not experts, tried only a simple configuration
g. more complex stacking, - seems to overfit
Thanks for reading that far, and hopefully see you in the future competitions!