# 12th place...

Uuuuff. Kaggle is tough! The last two months have felt like a really hard slog. I was trying to go for a solo gold but fell just short - congratulations to all the winners, and everyone in the medal zone, it was a pleasure to compete with you!
Some recognition before I go into solution details:
[@hengck23](https://www.kaggle.com/hengck23)
- delighted you made it to the gold zone. My strongest models were based on the solutions which you so generously opensourced early on. I learnt a huge amount from team mates in the last competition that I entered (@christofhenkel,
[@ilu000](https://www.kaggle.com/ilu000)
,
[@philippsinger](https://www.kaggle.com/philippsinger)
,
[@robikscube](https://www.kaggle.com/robikscube)
) - I don't think I would have done as well had it not been for that. Cheers.
Solution strategy:
Given the size of both the dataset and the highest performing models, it became clear early on that I would need to make a strategy call: train multiple smaller models and ensemble them, or train a single large model and try to push the limit of what that could do.
I felt that this competition was similar to
[Lyft](https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles)
in terms of data size. My main learning from that competition was that patience can be a virtue: there is value in simply training a model for far longer than you think you should. Improvements are slow, but they are steady.
To this end, I focussed on a single VIT transformer (based on
[@hengck23](https://www.kaggle.com/hengck23)
, 384 x 384 inputs). My best single checkpoint ultimately made it to 0.67 LB. Each epoch took c. 24 hours (!!!). Ensembled they reached 0.63.
When experimenting early on I trained a LSTM+attention model based on
[@yasufuminakama](https://www.kaggle.com/yasufuminakama)
's code with a resnest200 backbone. This reached 1.11 LB so I included it in my final ensemble for a small amount of diversification.
Data:
Both models were fit using all of the data provided, plus an additional 10% samples per epoch of randomly generated images from
extra_approved_InChIs.csv
. I based the image generation on
[@stainsby](https://www.kaggle.com/stainsby)
's
[notebook](https://www.kaggle.com/stainsby/improved-synthetic-data-for-bms-competition-v3)
. Augmentation comprised random rotation (plus coarse dropout in the case of the LSTM model). The final ensemble set also included models that had been fit on pseudo-labelled test data.
Ensembling strategy:
Throughout the competition I had been using
[@nofreewill](https://www.kaggle.com/nofreewill)
's InChI normalisation as the basis for ensembling: each prediction was categorized as 1) normalized, where normalized result = prediction; 2) normalized, with modifications; 3) normalization fails.
For each image in the test set I isolated the predictions with the lowest normalization category. Within this group I used simple voting, with ties decided by model preference (i.e. model were ordered highest preference to lowest preference based on their LB scores).
I ensembled 7 checkpoints for the VIT model + 2 for the Resnest200/LSTM model. Each sample was computed twice per checkpoint: once in its original form, once flipped L->R.
Congrats, all.