# My no-test-leak 23rd solution

Hello guys,
After a day both agonising my lost solo gold medal, as well as enjoy my luck to survive the shake, I am now be able to do this mini write-up.
Although I admitted I exploited the advantage of test leak to survive, my 23rd position DID NOT use any kind of test leak information. What I mean is that as the leak test estimates are known, it just made me less suspect my final submission with a very bad public LB (1.350) score as well as bad local MAE score, and select it as the "risky" submission. That submission does not use any kind of leak information.
Feature Engineering
+ The top feature is the 1st coefficient of numpy.polyfit  of order 3 on the columns of STFT spectrogram. 
+ The 2nd top feature is just the magnitude of a specific band of FFT frequency, after some smoothing techniques on the FFT spectrum.
+ Other weaker features are some welch coefficents, roll std, STFT contrast level, C3 unlinearity....
Models
I used LGB, XGB, CatBoost, and RF. All 4 of them score almost identical in my CV as well as LB (around 1.31 to 1.33). A simple blend of them gave me 1.290 on public LB. Hyperparameters were tuned by random search and later by Gaussian optimization. LGB with 40 leaves, 20 min_data_in_leave, depth 7, subsample 0.5. XGB with depth 7, subsample 0.5.
CV
This is the part that made me disappointed at the end. I was disappointed as I spent a lot of time building this trustworthy CV framework, but in the end to get a good score, one would not need it at all. However, this CV framework can serve me as a legacy for future competitions.
- First, I bet so many people may wonder why using sqrt(y) gives better result than raw y. That's because the varied variance of predictions. Higher ttf samples are more unpredictable. So we need to focus more on low ttf samples, because they are achievable. If we use raw y, CV will overfit on high ttf samples, and underfit (not converged) in low ttf samples.
After some careful CV iterations (which will be described next), I chose y^(0.55) to be the best transformation.
Leave-one-quake-out (LOQO) or shuffled CV will give the same result, if the distributions of true train data and early stopping data is the same, with a SEPARATE validation data. As a result, I used 10 stratified folds based on 16 quake_id. For each fold train, I further use another CV of another 10 stratified folds of the training fold (which has 9/10 data of the whole original training set).  So, in the end each innermost loop of training session consists of (9/10)*(9/10) samples from the whole original train set. The inner loop CV serves to get the best averaged iteration for each outer fold train.
I train on 10 different seeds of the outer CV.
It turns out that using shuffled CV in general gives better result than LOQO, because each sample is validated by 10 different subsets of seeds. In LOQO, each data sample is validated by just 1 set of seed, that is the combination of all the quakes other than the one containing that sample.
Stack
Optimizing MSE: Using a simple ridge regression sample on all OOF predictions of 4 models (each with 10 seeds, as said), I got my best submission which score 1.350 public LB and 2.409 private (23rd position). Please note that, this is my best CV only in terms of MSE. The stacked MAE is much worse than any of the 1st level model. This is a gamble that I made, given that the private set has significantly higher ttf samples. Without that knowledge, I would have not selected this submission. This is my "risky" submission, and turned out to be the saviour.
Optimizing MAE: Ridge regression does not allow us to directly optimize MAE. So as usual, transform y to y^(0.55), and do MSE. By this, my stacked MAE is much better, which gives me 1.29x on public LB. This is my "safe" submission, and turned out to be the disaster.
What I regret
I did the exact estimate of private LB based on the image of the p4677 paper. Then, I use the training weights obtained by getting the binned normalised histogram of private set divide the binned normalised histogram of the training set. Then I use just 1 seed split of 1 simple XGB model. That model gives 1.380 on public LB, and 2.375 on private (which could earn me a gold). If I blend more models with these weights, and finally stacking, I will end of in the top 3. I decided not to use this leak at the end, and I regret it. However, it's still lucky to me that my non-leak submission secured my 23rd position.
Thanks for reading.