# 5th Place Solution

Hi all,
This is absolutely a tough one, so congratulations to all those who persevered in this competition until the end.
Thanks to the organizers and congrats to all the winners and my wonderful teammates
[@boliu0](https://www.kaggle.com/boliu0)
[@garybios](https://www.kaggle.com/garybios)
and we must thanks to
[@hengck23](https://www.kaggle.com/hengck23)
and
[@yasufuminakama](https://www.kaggle.com/yasufuminakama)
we couldn't have achieved this without your sharing. Thank you so much!
Summary
We didn't use Transformer Encoder, our architecture is just CNN (384x384 & 512x512) + Transformer Decoder (12~16 layers)
We trained 5 backbones. EffNet B3/B5/B7, ResNet200D, eca-nfnet-l0
Noise Injection for regularization.
Training with some ext data which generated by rdkit. (But not very helpful)
Ensemble during decode phase.
Handling invalid predictions (most crucial trick for going under 0.6).
Noise Injection
This is a very old regularization technique, if I remember correctly someone mentioned it in a paper before 2000. In the field of Image Caption, the accuracy of prediction for the next character is quite high while ensuring that the previous sequence is exactly correct. But once a character is predicted incorrectly, then the next prediction failure rate gradually increases until it runs completely off.
So I came up with noise injection, which is done by randomly replacing GT characters with other characters during training. The replaced characters are ignored in the calculation of the loss, but this does not mean that this technique has no effect on the prediction. Although the character itself is not computed as part of the loss, the model is then forced to correctly predict the next character on the basis that the previous one was wrong.
In addition to this, the technique itself has a regularization effect. This allows us to train 12-16 layers of transformer decoder.
Official Ext Data
The host released 10 million InChI without images as ext data during the competition.
We found that rdkit can use InChI to generate images, so we generated some of the images (about 1~2 million) as ext data and added them to the training. But later we found that this part of data did not bring much improvement. However, since it did not degrade the performance either, we kept it.
Handling Invalid Predictions
After training all models we ensemble them in the decode phase.
The LB of the original prediction is 0.62, then we use rdkit to norm the prediction and get LB 0.60
At this point we find that there are about 14000 rows in the predicted InChI that are not valid. To replace these invalid predictions, we use 3 methods to find their valid predictions:
Prediction generated by single model (fixed 5k~ rows)
Search for top 12 predictions by ensemble (fixed 2k5~ rows)
For the numerical part of the chemical formula, we use top2 predicted values to decode (fixed 500~ rows)
For example, if the original prediction is C12H3, and the top2 predictions for the second and fourth characters are 11 and 4, then we replace them with C11H3, and C12H4 and then decode the next part.
We discovered that replacing invalid predictions with valid predictions could significantly improve LB the night before the competition deadline, due to lack of time, we only had time to dealing with 8000 rows out of the 14000 invalid predictions in the original prediction, which improved our LB score from 0.60 to 0.57
Acknowledge
Special Thanks to Z by HP & NVIDIA for sponsoring me a Z8G4 Workstation with dual RTX6000 GPU and a ZBook with RTX5000 GPU.
This is without a doubt a competition with the biggest dataset in the last half years in kaggle.
So I tried pytorch's DDP parallel training on my dual RTX6000 GPUs and the experience was great, basically it can be twice as fast as single GPU training.