# 27th place key highlights in 60 seconds

Architecture - Combination of:
VIT encoder with 8 transformer decoder layers. I felt on edge about having no inductive bias so I used 2x 3x3 convolutional kernels as input adaptors to the VIT (inconlusive whether it helped - doubt it).
Edit
Forgot to mention the important bit - the second one was stride = 2 thereby allowing me to do "intelligent" downsampling from larger input images than normal hardware constraints would permit.
TNT encoder with 8 transformer decoder layers.
Used selective patches (thank you
[@hengck23](https://www.kaggle.com/hengck23)
for the idea) making sure to only feed relevant patches to the encoder. This significantly sped up training.
Salt and pepper (especially pepper) augmentation for the win! Check
[this team's discussion](https://www.kaggle.com/c/bms-molecular-translation/discussion/243932)
.
RDKit generated images - I think this also helped because it let me increase training vocab.
[@nofreewill](https://www.kaggle.com/nofreewill)
normalization script but with cascading and agreement:
Cascading = if the inchi was invalid, try the next best model's inchi. Huge improvement!
Agreement = if top model inchi is different from 2nd, and 2nd == 3rd == 4th, then use the latter. Tiny improvement.
As
[@fergusoci](https://www.kaggle.com/fergusoci)
says - be patient and let the model train out for a while.
Psuedolabelling training for 2 epochs - tiny improvement.
What didn't work
Kind of wasted my time a bit by splitting the transformer decoder up into two transformers, one for the chemical formula and another for the rest. The inspiration was similar to the one for
[noise injection as this team did](https://www.kaggle.com/c/bms-molecular-translation/discussion/243943)
. Too bad the idea didn't work.
GG