# 3rd Place Part II

Thank you my briliant team, all teams who competed with us, and all people who participated in this competition. Congrats Kyle Boone, who won this competition with an incredible performance!
I'm happy because I and yuval became kaggle expert with 2 gold medals in this competition, and We finally became Prize Winner :)
It's my second kaggle competition and it was fun to compete with AhmetErdem and CPMP again, who I competed with in my previous competition, TalkingData Adtracking Fraud Detection. Here, I won't explain all of my team's solution, please take a look at
[@yuval6967](https://www.kaggle.com/yuval6967)
and
[@nyanpn](https://www.kaggle.com/nyanpn)
solutions, if you want to see what my teammate has done.
yuval's solution
:
[https://www.kaggle.com/c/PLAsTiCC-2018/discussion/75116](https://www.kaggle.com/c/PLAsTiCC-2018/discussion/75116)
nyanp's solution
:
[https://www.kaggle.com/c/PLAsTiCC-2018/discussion/75222](https://www.kaggle.com/c/PLAsTiCC-2018/discussion/75222)
My github for the host
:
[https://github.com/takashioya/plasticc](https://github.com/takashioya/plasticc)
My Model
my model is based on CatBoost, with lotta feature engineering.
I made 3 types of models.
(1) galactic model, the number of features are about 1000.
(2) extra-galactic model, the number of features are about 2000.
(3) extra-galactic model with hostgal_specz, the number of features are about 2000. It gave 0.004 improvement.
Then, I trained 7 * 3 types models with different feature set and get simple average of them, which scored 0.806 on Public LB, while my best single model scored 0.823.
you may think the number of features of my model is too big, but using small number of features degraded the score. tbh I prefer to use big features, because I think big features lead to big improvement in ensemble, from my experience.
My Features
actually, I used nyanp's features for my model and some of them have much bigger importance than mine (especially salt-2 features). However, some of my features still have high importance, so I'll explain a bit about them.
features related to mjd
:
something like 'variance of mjd whose detected == 1' worked well.
abs_curve_angle_features
:
by calculating np.arctan(flux_diff/mjd_diff), we can calculate 'angle' of light curve.
I calculated many stats from this.
feets features
:
I used
[https://github.com/carpyncho/feets](https://github.com/carpyncho/feets)
, which was effective.
gaussian process features
:
It worked better than the features like 'days_from_peak_n_percent_flux'.
for gaussian process, I used Gpy, but it's too slow and I
had to use 30 instances for feature extraction.
shifted_flux * distance ** 2 features
:
I calculated shifted_flux = flux - flux.min() if flux.min() > 0 per each object_id and passband, which worked better than just calculating
flux * distance ** 2
Ensemble
I used weighted average for ensemble. however, I gave different weights to each class, which gave us 0.006 improvement. I calculated the weights using oof-predictions by hyperopt.
By getting weighted average of mamas's CatBoost(0.806) and nyanp's LightGBM(0.834), we got 0.780 on public LB.Then, in the same way, we calculated weighted average of 0.780 model and yuval's NN (0.791) and got 0.737.
probing class99
Actually, we did nothing special about class99 7 days before the end of the competition.
We noticed class99 probing is very effective 6 days before the end and spent 5 days probing class99 distribution.
Finally, we got 0.057 improvement by probing class99, which took us to the Prize zone (0.680 on public LB).
For extra-galactic, we used this special formula.
c99 = 1.65 * (c42 + c52 + c62 + c95) ** 2.5 *  (1 - c62 - c95) ** 0.635
For galactic, we did nothing special, just a normal method like this.
c99 = 0.2 * (1 - c42) * (1 - c52) * .... * (1 - c90)
Here, I'll explain how we found this special formula.
First, we started with the normal method like this.
c99 = 0.9 * (1 - c42) * (1 - c52) * .... * (1 - c90)
Then, we noticed this way works better, which means c90 is not similar to c99 at all.
c99 = 0.9 * (1 - c42) * (1 - c52) * .... * (1 - c90) ** 2
Then, we also tried changing power for other classes and got this formula. (We spent about 19 subs for this step)
powers = {c42 : 0, c52 : 0, c62 : 0.5, c95 : 1, c90 : 2, c88 : 2, c67 : 2, c64 : 2,  c15 : 2}
c99 = np.prod([(1 - pred[c]) ** powers[c] for c in classes])
This formula means c99 is very similar to c42 and c52, and somewhat similar to c62 and c95. actually, this formula already improved our score by 0.049 and it was the last day of the competition. but, we didn't give up and finally found the problem of this formula.
For object_id whose
c90 = 0.5
and
c88 = 0.5
, we thought we should assign
c99 = 0
.
However, this formula assigns
c99 = 0.5 ** 2 * 0.5 ** 2 = 0.0625
, which looks too high for us. so, we thought we should do something like
c99 *= (1 - c90 - c88)
and finally found
c99 = 1.65 * (1 - c90 - c88 - c67 - c64 - c15) ** 2 * (1 - c62 - c95) ** 0.635
= 1.65 * (c42 + c52 + c62 + c95) ** 2 * (1 - c62 - c95) ** 0.635
is a very good approximation of the formula.
Actually, it was our last submission in this competition and we couldn't tune coefficient at all.
what didn't work for me
some of the experiments that didn't work (Not all)
Band-correction using hostgal_photoz
target encoding using ra, decl
using adversarial validation for feature selection
stacking using GBDT as meta model
UNet-like Autoencoder for feature extraction
make binary classification model and use oof-predictions as features
what worked
class90 pseudo-labelling
:
It gave 0.016 improvements for me.
adversarial pseudo-labelling
:
I selected the objects whose the prediction is very high and similar to train data and used them for training. It didn't give a significant improvement for me, but I suppose it somewhat contributed to diversity in ensemble.
Comments
Actually, I haven't figured out why we dropped from 2nd to 3rd on Private LB.  I'm wondering whether it's because of the luck or not.
As for the competition design, I think making 'class99' is not so good, because it leads to LB probing and is not so practical. However, on the whole, I suppose this competition is really interesting and one of the most successful competition in kaggle. I'm especially really impressed with the yuval's beautiful and practical NN modelling :)
Anyway, I really enjoyed my second kaggle competition. I'll continue kaggle and want to become the winner in the next competition :)
Thanks all, see you again!