# 7th Place Solution : My First Gold And I am in Tears

Hi all,
We would like to thank kaggle and the organizers for such a lovely competition. My learning in this has been immense , while a lot of people might criticize this competition to be a  hardware struggle , for me it more of a learning experience than a GPU pain mainly due to my lovely teammates
[@pheadrus](https://www.kaggle.com/pheadrus)
[@shivamcyborg](https://www.kaggle.com/shivamcyborg)
[@atsunorifujita](https://www.kaggle.com/atsunorifujita)
[@nishcaydnk](https://www.kaggle.com/nishcaydnk)
who allowed and encouraged me to explore more and more.
I always thought that I understood transformers completely but all that was shaken in this competition. We would like to thank
[@hengck23](https://www.kaggle.com/hengck23)
for being so humble in sharing the transformer to transformer baseline with us , without that this medal would not have been possible . The code was easy to understand and a lot of our concepts got cleared during this competition . There are a lot of things we were still not able to implement like Beam search for our transformer model , but at the end of the day we are very happy with the result.
We (
[@nischaydnk](https://www.kaggle.com/nischaydnk)
, me and
[@shivamcyborg](https://www.kaggle.com/shivamcyborg)
) started with the CNN encoder with LSTM decoder like everyone else , quickly made our way to GRU , then we were joined by Rajneesh who brought in the idea of transformer to transformer . We fought hard to get to 1.30 then we were joined
[@atsunorifujita](https://www.kaggle.com/atsunorifujita)
and since then it has been an a rolercoaster ride .
The last day was completely intense for us , we were at 0.65 and we knew we need something other than rdkit post-processing to get into gold , the only thing left was to do logits ensemble at every time step , we were unsuccesful at that before but nevertheless we tried it and it worked , our ensemble gave (0.66 lb 0.71cv) submitted at
11:30 UTC
and then after rdkit post-process we went to (0.60lb cv 0.66) submitted at
11:45 UTC
and jumped to 7th place where we ended as well.
Our final solution consists four major parts :
Training transformer to transformer effectively
Logit Level Ensemble at each time step
Rdkit Post Processing
Most Important : Analysis of all submissions and validation INCHI's
Models Used :
Our final ensemble consisted of
Encoder - 2 ViT 384 Base, Decoder - Vanilla transformer  ( Cv-0.85 , Lb -0.76) and (Cv-0.86 , lb-0.77) respectively
Encoder - 1 Swin Transformer 384 , Decoder -  Vanilla transformer  (Cv - 0.99 , LB -0.96)
Model Training :
We started ViT Training similar to heng's using lookahead optimizer and manually decreasing LR's . We made use of increasing resolution training and trained first with 224 Img_size , then fine-tuned the same model to 384 then finally to 448. This gave us a good amount of boost and we were able to reach 1.27 lb but it still wasn't enough . After analyzing the valid predictions we realized that the models kept failing on very long chained molecules and molecules with noises , our hypothesis was it was overfitting to the normal ones . Owing to the noisy images and our findings we decided to add Label Smoothing and Cutout to training and it worked , we were able to achieve (lb 0.77 ) with the same ViT model . After that we tried with 448 Image size with the same strategy however it didn't work so well and hence was excluded from final ensemble . We trained another ViT model with different LR and increased Random Scale as augmentation and were able to scoer (lb 0.76)
We then started the search for diversifying our model zoo and moved on train SWIN and transformer-in-transformer with the same training Strategy , however Swin Beat tnt and scored (0.96 lb) as compared to tnt (1.07 lb)
RDkit Post-Processing
Everybody was normalizing the predicitons and we were no different , but only after reading
[@nofreewill](https://www.kaggle.com/nofreewill)
post we dug deeper and realized what we were missing.
def normalize_inchi(inchi):
try:
mol = Chem.MolFromInchi(inchi)
except:
pass
if mol is None:
return inchi,'invalid'
else:
try:
norm = Chem.MolToInchi(mol)
if norm == inchi:
return norm,'valid'
else:
return norm,'modified'
except: return inchi,'segmentation_failure'
After adding this code to the normalization script we were able to get the status as well . We tested out the hypothesis that the places where INCHI were suggested invalid by rdkit were really wrong hence to replace them with valid predictions from different other model's prediction were a sure shot to improving score . We didn't stop here and dug even deeper to realize that the invalid INCHI's were coming from where the INCHI length was very long . Hence we fine-tuned our best tnt model on images having INCHi length greater than 150 and it gave great cv (0.99 previously 1.09) however due to lack of time we didn';t test lb and decided to add in post-processing to replace invalid INCHI's only
Models used in Post-Process to replace Invalid INCHI's
Tnt 336 Model
Vit 448 Model
Tnt 448 Model
prunedeffnetb3 + GRU Model
Effnetb4 + LSTM Model
Effnetb7 + GRU Model
Logits Ensemble
I started coding Logit Ensemble on the last day and by the time it was ready only 9 hours were left in the competition , it gave a cv of 0.71 and hence we gave it our best to make it complete before 2 hours of deadline . Things fell into place and we were able to complete it in time.
We realized that in heng's code the final logit gave an output of
(bs,vocab_size) , so untill the max_length and the vocab size of models are same
we can at every time step ensemble the predictions , and since we didn't have a lot of time , we hard-coded it , something like this :
class EnsembleNet(nn.Module):
def __init__(self, model_1, model_2, model_3):
    super(EnsembleNet, self).__init__()
    self.model_1 = model_1
    self.model_2 = model_2
    self.model_3 = model_3

    self.model_1.load_state_dict(torch.load(initial_checkpoint1)['state_dict'], strict=True)
    self.model_2.load_state_dict(torch.load(initial_checkpoint2)['state_dict'], strict=True)
    self.model_3.load_state_dict(torch.load(initial_checkpoint3)['state_dict'], strict=True)

    #self.model_1 = torch.jit.script(self.model_1)
    #self.model_2 = torch.jit.script(self.model_2)

def forward(self, image):

    image_dim = 768
    text_dim = 768
    decoder_dim = 768
    num_layer = 4 #3
    num_head = 8
    ff_dim = 2048#1024

    STOI = {
        '<sos>': 190,
        '<eos>': 191,
        '<pad>': 192,
        #'<mask>': 193,
    }

    image_size = 384
    vocab_size = 193#194
    max_length = 280 #300  # 275

    device = image.device
    batch_size = len(image)

    image_embed_1 = self.model_1.cnn(image) #(bs,img_len,image_dim)
    image_embed_1 = self.model_1.image_encode(image_embed_1).permute(1, 0, 2).contiguous() # (img_len,bs,image_dim)
    image_embed_2 = self.model_2.cnn(image) #(bs,img_len,image_dim)
    image_embed_2 = self.model_2.image_encode(image_embed_2).permute(1, 0, 2).contiguous() # (img_len,bs,image_dim)
    image_embed_3 = self.model_3.cnn(image) #(bs,img_len,image_dim)
    image_embed_3 = self.model_3.image_encode(image_embed_3).permute(1, 0, 2).contiguous() # (img_len,bs,image_dim)

    token = torch.full((batch_size, max_length), STOI['<pad>'], dtype=torch.long, device=device) # (batch_size,max_len) 
    text_pos_1 = self.model_1.text_pos.pos #(1,sequence_len,text_dim) torch.zeros(1, max_length, dim)
    text_pos_2 = self.model_2.text_pos.pos #(1,sequence_len,text_dim) torch.zeros(1, max_length, dim)
    text_pos_3 = self.model_3.text_pos.pos #(1,sequence_len,text_dim) torch.zeros(1, max_length, dim)
    token[:, 0] = STOI['<sos>']
    # -------------------------------------
    eos = STOI['<eos>']
    pad = STOI['<pad>']
    # fast version
    if 1:
        # incremental_state = {}
        incremental_state1 = torch.jit.annotate(
            Dict[str, Dict[str, Optional[torch.Tensor]]],
            torch.jit.annotate(Dict[str, Dict[str, Optional[torch.Tensor]]], {}),
        )
        incremental_state2 = torch.jit.annotate(
            Dict[str, Dict[str, Optional[torch.Tensor]]],
            torch.jit.annotate(Dict[str, Dict[str, Optional[torch.Tensor]]], {}),
        )
        incremental_state3 = torch.jit.annotate(
            Dict[str, Dict[str, Optional[torch.Tensor]]],
            torch.jit.annotate(Dict[str, Dict[str, Optional[torch.Tensor]]], {}),
        )
        for t in range(max_length - 1):
            last_token_1 = token[:, t] # take the whole batch's t'th token
            text_embed_1 = self.model_1.token_embed(last_token_1) #[bs,text_dim] Generate embedding for the t'th token
            text_embed_1 = text_embed_1 + text_pos_1[:, t]  #[bs,text_dim] Combine with pos embed for t'th token
            text_embed_1 = text_embed_1.reshape(1, batch_size, 768)

            last_token_2 = token[:, t] # take the whole batch's t'th token
            text_embed_2 = self.model_2.token_embed(last_token_2) #[bs,text_dim] Generate embedding for the t'th token
            text_embed_2 = text_embed_2 + text_pos_2[:, t]  #[bs,text_dim] Combine with pos embed for t'th token
            text_embed_2 = text_embed_2.reshape(1, batch_size, 768)

            last_token_3 = token[:, t] # take the whole batch's t'th token
            text_embed_3 = self.model_3.token_embed(last_token_3) #[bs,text_dim] Generate embedding for the t'th token
            text_embed_3 = text_embed_3 + text_pos_3[:, t]  #[bs,text_dim] Combine with pos embed for t'th token
            text_embed_3 = text_embed_3.reshape(1, batch_size, 1024)

            #text_embed ---> 1,bs,text_dim(768)
            #image_embed ---> img_pos_embed,bs,image_im
            x_1 = self.model_1.text_decode.forward_one(text_embed_1, image_embed_1, incremental_state1)
            x_2 = self.model_2.text_decode.forward_one(text_embed_2, image_embed_2, incremental_state2)
            x_3 = self.model_3.text_decode.forward_one(text_embed_3, image_embed_3, incremental_state3)
            ## x -----> (1,bs,text_dim)
            x_1 = x_1.reshape(batch_size, 768)
            x_2 = x_2.reshape(batch_size, 768)
            x_3 = x_3.reshape(batch_size, 1024)
            ## x -----> (bs,decoder_dim)
            l_1 = self.model_1.logit(x_1)
            l_2 = self.model_2.logit(x_2)
            l_3 = self.model_3.logit(x_3)
            l = (l_1 + l_2 + l_3)/3
            ## l -----> (bs,num_classes)
            k = torch.argmax(l, -1)
            token[:, t + 1] = k
            if ((k == eos) | (k == pad)).all():
                break

    predict = token[:, 1:]
    return predict
Final Hard-Voting
One Last Layer that we also added was hard-voting using all our submission (11 subs) . After doing RDkit Post Processing , we found that approx 11k invalid still remained and since we were not able to implement beam search for those invalid images we applied hard voting using all our model's submission (
[@nischaydnk](https://www.kaggle.com/nischaydnk)
's Idea)
Gratitude
This competition will remain closest to my heart , we have stayed up all night , its 6 AM in India right now , I will never forget how we passed the time from 1 to 4 while the ensemble inference was running . Congrats to
[@pheadrus](https://www.kaggle.com/pheadrus)
[@shivamcyborg](https://www.kaggle.com/shivamcyborg)
for becoming a master and
[@atsunorifujita](https://www.kaggle.com/atsunorifujita)
[@nischaydnk](https://www.kaggle.com/nischaydnk)
for their second gold . It was a collective team effort and I couldn;t thank you all enough for taking me in your team .
Its late , I will clean up the code soon and post here  very soon . To be continued ….
I hope you liked our solution . Thanks for reading