# 4th Place Solution

Thanks to the kaggle and competition host for this competition. And also thanks to my teammate
[@haochensui](https://www.kaggle.com/haochensui)
. We spent one month in this competition and in the first 2 weeks, we were doing experiments on 10% of the data. In the last two week, we rent several V100 and 3090 and switch to larger image size.
Our solution is very simple and mainly composed of 4 models, plus some early models for post processing with rdkit. We use tokenizer from
[https://www.kaggle.com/yasufuminakama/inchi-preprocess-2](https://www.kaggle.com/yasufuminakama/inchi-preprocess-2)
. Thanks for the amazing work.
Model Structures
CNN -> Encoder -> Decoder.
We use resnest101 and efficientnetv2_m as our CNN backbones, and use 6 layers in transformer encoder and 9 layers in transformer decoder. We project embeddings from CNN into 512 channels to match the dimension of transformer.  In the encoder, we inject sinusoid positional embedding into each layer before key and query, pretty similar with DETR. In the decoder, we use relative positional embedding as in T5.
Model Training
Because of GPU limits, we did not train large scale models from scratch. We first train the model on 416x416 images and then increase the image resolutions and finetune the model. We reserve 2% data, ~48k for validation.
We also do pseudo labeling here and find it quite useful. We used a 0.64CV/0.78LB submission(rdkit ensemble of efv2 on 416 and 640) to generate pseudo labels, and selected around 1.3M samples from it. We concatenate these samples with train samples.
Training order
efficientnetv2_m:  416(10 epochs) -> 640(5 epochs)       ->704(5 epochs + pseudo labels)
​                                                                                               -> 384x768 (5 epochs + pseudo labels)
resnest101:            416(10 epochs)  -> 640(5 epochs + pseudo labels)
​                                                           ->  384x768 (5 epochs + pseudo labels)
Results
Model
Size
CV(no norm)
LB(with norm)
efficientnetv2-m
416
0.99
1.18
efficientnetv2-m
640
0.74
0.9
efficientnetv2-m
706
0.65
0.67
efficientnetv2-m
384x768
0.68
0.67
resnest101
640
0.71
0.66
resnest101
384x768
0.74
0.67
Ensemble
We use 4 models with 0.66-0.67 LB to make a step-wise logit ensemble and use a beam search with beam size 3. Instead of choosing the prediction with maximum likelihood, we choose prediction with rdkit. We choose the first valid prediction from rdkit. In this way, we get around 0.02 improvement in CV. The ensemble of 4 models achieves 0.56 on LB. It should have  CV around 0.45.
In addition to that, we merge our early submissions(scores >0.65) with the 0.56 one according to rdkit and got 0.55.
Finally, we select the 3500 invalid prediction in the submission and did a beam search with size 32. This reduces the LB score to 0.54.