# 50th Place Solution

The source code is available in
[the github repository](https://github.com/affjljoo3581/BMS-Molecular-Translation)
.
Hi all. Thanks to the kaggle and competition host. It was a greate experience. We tried hard and we barely hold the silver medal 🤣.
Our approach is to handle this competition as a translation problem, not image-captioning or OCR. Because the
ViT
model showed that the images can be treated like sentences, we focused on the translation task (from
image
language to
InChI
language). Actually, the model structure and training procedures are not pretty different to other approaches, but the detailed configurations (e.g. hyperparameter tuning) are based on the translation papers.
Our basic strategy is to pretrain the transformer model with large-scale images from
extra_approved_InChIs.csv
, and then finetune the model with the original images.
Model Structure
We use
ViT
as an encoder and
the original transformer
as a decoder. According to
[this paper](https://arxiv.org/abs/1906.01787)
, it is more effective to increase the encoder depth rather than the decoder one. So we make a base version of our model to have 12-layer encoder, 6-layer decoder and a large version to have 24-layer encoder.
Dataset
As I mentioned above, we create the external dataset from
extra_approved_InChIs.csv
by synthesize the molecular images through
rdkit
. The number of original images is about
2.4M
and the number of entire pretraining images is about
12M
. It is comparable to the ImageNet-21k, mention in ViT paper. The authors of ViT paper showed that the performance of the image transformer is highly related to the dataset scale. And the
12M
images are indeed helpful.
Training
We train the base model for about 15 epochs with 224x224
12M
images. And we finetune with 384x384
2.4M
images. The CVs are 0.99 and 0.83 and LBs are 1.82 and 1.55 respectively.
And we increase the model size by copying the 12 layers and stacking to the end of the model. So the large version of our model has 24 layers. We pretrain the model for 5 epochs and finetune for 15 epochs. It achieves 0.74 CV and 1.44 LB.
Basically, we use single A100 GPU on GCP. To accelerate the training, we use
[apex](https://github.com/NVIDIA/apex)
's fused layers and O2 amp mode.
Thanks to
[jarvislabs.ai](https://jarvislabs.ai/)
, we can use 8xRTX5000s. And interestingly, we found that the multi-GPUs environment with DDP (DataDistributedParallel) and
apex amp
do not work correctly. I don't know why it happens, but it is well-known problem in
PyTorch Lightning
. So in 8xRTX5000 environment, we use native AMP and
[ZeRO](https://github.com/microsoft/DeepSpeed)
to increase the computing performance.
Prediction
We use
greedy search
, instead of
beam search
. That's because the inference time is really long (large model with 384x384 spent about 7 hours).
Thanks to
[@nofreewill](https://www.kaggle.com/nofreewill)
, we applied
[rdkit normalization](https://www.kaggle.com/nofreewill/normalize-your-predictions)
and the performance is indeed enhanced to 1.37.
Lastly, specially thanks to
[jarvislabs.ai](https://jarvislabs.ai/)
to support some credits 👍👍