# Congrats and 8th place Rapids solution updated!

Update: my solution will be posted
[on github](https://github.com/daxiongshu/Rapids_PLAsTiCC_2018)
. Now you can find a quick end2end demo with rapids. Forgive me that I'm always distracted to other stuff but I promise I'll get the whole solution done eventually!
Big Congrats to the winners and new kaggle masters. I want to thank Olivier
[@ogrellier](/ogrellier)
and Kyle
[@kyleboone](/kyleboone)
specifically. Without your kernels, I can't even start this competition.
Overall this competition is a fantastic learning process for me. If you have no astronomical background, this solution is for you. Considering I started climbing the LB from a model that is just 0.95 LB, what happened is a miracle :P
Here is a preview of my solution and more to come!
![Solution Overview](assets/a0f9a29701836b6983d0846a.png)
Back to my laptop :P Ok, let me walk through this chart.
Baseline Models and feature engineering.
In case you haven't noticed, I'm using
[cudf](https://github.com/rapidsai/cudf)
to replace pandas for this competition whenever I can. In general, it is at least
10x
faster for csv reader and general groupby - aggregation
with a single GPU
than pandas on CPU. Sometimes I got
100x
speedup. I used cudf to generate most features in
[Olivier's kernel](https://www.kaggle.com/ogrellier/plasticc-in-a-kernel-meta-and-data/code)
. However, as a merely two-month old library, you can expect some missing functionalities of cudf. I will start publish cudf tutorial in kaggle competitions and work with our developers to make your favorite pandas trick come true on GPU!  A demo notebook of using
cudf
will be published sometime tomorrow. Stay tuned!
In addition, I also built some other features that gave me a small boost including decline rate from the peak, number of ups and downs of the flux curve, passband count percentage around the peak,
[period features](https://www.kaggle.com/rejpalcz/feature-extraction-using-period-analysis)
(optimized), and features from
[Chia-Ta's kernels](https://www.kaggle.com/cttsai/forked-lgbm-w-ideas-from-kernels-and-discuss)
. A small trick to add diversity of features is to only use detected==1 and redo the above feature engineering. So in the end I have two sets of features, one with whole times series and the other with detected==1 series.
Unfortunately, my many other feature engineering attempts failed miserably. In the end, I have a mediocre lgb model with about 80 features of 0.58 cv and 1.05 LB. By the way, lgb always outperformed xgb by a small margin but the xgb gpu is just so much faster so in the end I used xgb gpu for experimental validation & feature selection, and lgb for submission.
Stacking: non linear ensemble
As the great Kaggler Giba once said, stacking is all about diversity of base models. so I actually have three base models (level 1 modes):
lgb as in
[Chia-Ta's kernels](https://www.kaggle.com/cttsai/forked-lgbm-w-ideas-from-kernels-and-discuss)
with whole series
lgb with features from detected==1 and train for galaxy and ex-galaxy separately
MLP as in
[Siddhartha's kernel](https://www.kaggle.com/meaninglesslives/simple-neural-net-for-time-series-classification)
with same features as 2. Mysteriously, this keras mlp is just better than my own TF implementation with the exact same network structure and hyperparameters.
A key to successful non-linear stacking is to avoid overfitting. I have found two tricks for this:
In level 1 base model, don't use the oof labels for early stopping. Instead, split the train data in that fold again to get its own validation set. This will degrade the level 1 model's score but it will help level 2 stacking.
Make the 2nd level model simpler. For example, my 1st level lgb use depth of 7 and 3. In 2nd level lgb, the depth is always 1. The 1st level MLP has 4 layers but the 2nd level MLP only has 1 layer. And again, mysteriously, now it is my own TF MLP at 2nd level that is always better than the keras counterpart. In the end I just average the lgb/xgb and nn level 2 predictions with equal weights.
At this stage, my stacking is 0.51 cv and 0.95 private LB which is 0.1 better than my best single model. So I have a good stacking framework setup and everything is great except for my non-impressive rank. Now
what I need is just some magic
.
RNN feature extraction with attention
The sudden jump of @hklee made me believe that deep learning could be it. I looked at the data again and the great posts by
[CPMP](https://www.kaggle.com/c/PLAsTiCC-2018/discussion/71949)
and
[hklee](https://www.kaggle.com/c/PLAsTiCC-2018/discussion/72646)
. I did the following preprocessing of flux:
log transform the flux and calculate the diff(periods=1) and just use the diff
calculate mjd the diff(periods=1) and just use the diff
segment the time series and reset the mjd_diff and flux_diff at segment boundaries to 0.
Insert random size of all zero data to the gap between the segments.
And I use an RNN structure as the following:
```
with tf.variable_scope(name):
net1,net2 = net[:,:,:-1],net[:,:,-1] # B, S, F

        net2 = self._get_embedding("%s/passband"%(name),net2,V,E)

        net = tf.concat([net1,net2],axis=2)

        state = None

        net = self._bd_rnn_layer(net,"%s/rnn3"%name,cell_name,args,
            state_fw=state,state_bw=state,output_size=-1,useproject=False)

        args = {"num_units":H//4,"activation":'relu'}

        self.next_flux_diff = self._bd_rnn_layer(net,"%s/rnn5"%name,cell_name,args,
            state_fw=state,state_bw=state,output_size=1,useproject=True)[:,:-1,0]

        args = {"num_units":H//4,"activation":'relu'}

        net = self._bd_rnn_layer(net,"%s/rnn4"%name,cell_name,args,
            state_fw=state,state_bw=state,output_size=-1,useproject=False)

        w = self._get_variable(name, name='attn', shape=[1,args["num_units"]])

        w = tf.expand_dims(w,axis=0)

        atten = tf.nn.softmax(w*net)

        self.bottleneck = tf.reduce_sum(net*atten,axis=1)

        net = self._fc(self.bottleneck, num_classes, layer_name='%s/out'%(name))
```
I start to train this RNN using the 7K training data and as you can image it went nowhere.
Deep learning demands big data unless it can be transferred.
And we do have big test data with 3M samples. And then it struck me that I could use the pseudo label of test data for training purpose as my great kaggle buddy
[@xiaozhouwang](/xiaozhouwang)
did in
[tensorflow speech 3rd place](https://www.kaggle.com/c/tensorflow-speech-recognition-challenge/discussion/47722)
. Intuitively, I used my best model at that time (LB 0.95) to generate pseudo labels which is just np.argmax(test_predictions,axis=1) and
Bazinga!
it just worked! So as the chart shows I did this repeatedly for the last week of competion and my score is improved as [0.95 LB, 0.88, 0.85, 0.83, 0.82, 0.81, 0.80] and it just converged to where I am.
Some tricks of this step:
only provide time series data to the RNN without meta data. In this way, we force the RNN to focus on the timer series patterns only to fit a score that's obtained by all features and complex ensemble.
use all the test data for training and use train data for validation and early stopping. The rational is that we expect the RNN to learn a middle ground to best fit the pseudo labels of test and true labels of train.
multi objective function. in addition to the final classification, the model also predicts the next flux diff. It helps the model from cold start.
Use different RNN cells in a interleaved way. For example, if GRU get the best score in this experiment, use its pseudo labels to train a LSTM  for the next experiment.
Use bottleneck layer as features instead of the softmax output. Remember, we want the middle ground features that could be useful to train data, not the ones that fits the pseudo labels best.
Yep, that's it. I'll also open source my solution on github. Thank you all. Happy kaggling! :D