# 13th Place Team (a story from a Kaggle newbie's view)

Hello all,
It was an amazing competition. I would like to thank my team members (
[@eakdag](https://www.kaggle.com/eakdag)
,
[@ubique](https://www.kaggle.com/ubique)
,
[@aerdem4](https://www.kaggle.com/aerdem4)
,
[@proletheus](https://www.kaggle.com/proletheus)
) for their great effort. We missed the gold medal at the last moment, but I learned a lot and had nice friends.
As a Kaggle newbie, I will explain the story of our team from my view. It could be a bit long so I divided it into sections and I will fill them in time.
[Introduction]
I and Erkut were friends from high school (MFL) at Konya/Turkey. After 12 years we met in Eindhoven/Netherlands by chance. We are both pursuing a PhD at TU/e. I am working on cheminformatics applications and Erkut is working on computer vision. We had discussed trying a Kaggle competition together 2 weeks before the BMS competition started. When this competition popped up, it was an amazing opportunity :D. So we set up "MFL Eindhoven" team.
[Preparetion]
I had no prior experience with computer vision except playing with some toy datasets at the beginning of the competition. I took a quick
[computer vision course](https://www.kaggle.com/learn/computer-vision)
from Kaggle. Then I developed
[my first kernel which detects the stereochemical layer](https://www.kaggle.com/c/bms-molecular-translation/discussion/229168#1255349)
in the image.
At the same time, I analyzed InChI and listed possible solution ideas. (see the picture of my early research in the comments)
[Training First Models]
First, we started image captioning with
[@yasufuminakama](https://www.kaggle.com/yasufuminakama)
's famous
[ResNet+LSTM notebook](https://www.kaggle.com/yasufuminakama/inchi-resnet-lstm-with-attention-starter)
. However, we had limited resources and training was super slow. We wanted to make many different experiments, so we needed a faster way. (
LB:7.06
)
[Mark's TPU notebook](https://www.kaggle.com/markwijkhuizen/tensorflow-tpu-training-baseline-lb-16-92)
:
[@markwijkhuizen](https://www.kaggle.com/markwijkhuizen)
shared an amazing end-to-end TPU notebook. We used eff_B1+LSTM for training our models. Each epoch was taking around (30 min to 1 hour) which allowed us to make the following experiments:
Larger Image size
Different tokenizer
Train with only longer sequences
(see comments for the list of other possible improvements)
Now, we had several models (
LB: btw 4.0 to 7.0
) and it was time to ensemble them which will boost our score.
![](https://i.imgur.com/Hovv3C0.png)
PS: Special thanks to
[@markwijkhuizen](https://www.kaggle.com/markwijkhuizen)
! These experiments would not have been possible without your sharing.
[Boost of Merge Algorithms]
Gold Rush
Our first merge algorithm was "Gold Rush". It simply merges two submissions by selecting valid molecules detected by Rdkit. When we apply Gold Rush to our submissions from different models, our score improved to
LB: ~3.50
.
![](https://i.imgur.com/UX4r21n.png)
The Collector of Lost Souls
After we merge all submissions, there are still around 100k invalid molecules. In order to find them, we developed an algorithm that takes inference from each epoch for only invalid molecules. Using this method, we collected 20k more valid molecules and our score improved to
LB: ~3.30
.
During that time
[@hengck23](https://www.kaggle.com/hengck23)
shared his
[transformer and the inference script](https://www.kaggle.com/c/bms-molecular-translation/discussion/231190)
. We inferred two submissions (
LB:3.10 and 3.11
). Then we also included these submissions into Gold Rush and we got a high improvement to
LB:2.30
.
Multi Merge
After this point, I started to analyze my validation data. I found that avg error of valid molecules was 0.8. So even if we found all valid molecules, 0.8 was the top score we could reach. Then I started to compare different submissions and found different valid InChIs for the same molecules. It meant that valid molecules also need to be validated :D.  So I developed a new algorithm based on voting and minimum distance. That really worked and our score improved to
LB: ~2.00
.
![](https://i.imgur.com/5nHXrdf.png)
Now, we had a great merge algorithm and we were in a good LB position, it was a good time to develop a team-up strategy…
[Team Up Strategy]
Our algorithm boosted 3.xx models to 2.00 level, so if we find new team members with high-scored models we could have played for top positions. Our algorithm gets more benefit from diverse models, so we always took this into account. We decided to go until 5 members. After each member joined our score would increase and we would have a better chance to merge with higher LBs. So we decided to go round by round and started to invite other top LBs.
PS:
There were some suspicious accounts at the top positions, we decided not to invite them.
[Boost of New Members]
In the first round,
[@ubique](https://www.kaggle.com/ubique)
accepted our invitation. He had SMILES model trained with ResNest (
LB:2.03
).  When we added his models in our algorithm we reached (
LB:1.16
).
Before the second round,
[@aerdem4](https://www.kaggle.com/aerdem4)
contacted us for a possible merge. He had just finished another competition and recently joined this one. He had an InChI Transformer model (
LB:3.71
) which predicts each layer separately. We thought that his experience would be very useful for the team and probably will improve his model quickly (it exactly came true). Since his InChI layers ensembled from separate models, valid ones were very reliable. His model boosted us to (
LB:0.99
).
We reached that level, using relatively weak models. We needed a locomotive single model (preferably InChI). In the last round,
[@proletheus](https://www.kaggle.com/proletheus)
joined our team, and we found our locomotive (
LB:1.42
Transformer InChI). By including his model into multi-merge, we reached (
LB:0.78
).
[Developed Models]
After that point, every day we worked hard and improved our models. In the end, our merged score reached (
LB:0.62
). The main models we developed are listed with scores (before pseudo-labeling) below.
InChI / Transformer /
[@proletheus](https://www.kaggle.com/proletheus)
/
LB: 1.37
InChI (layered) / Transformer /
[@aerdem4](https://www.kaggle.com/aerdem4)
/
LB: 2.01
InChI  / eff_B1+LSTM /
[@sorkun](https://www.kaggle.com/sorkun)
&
[@eakdag](https://www.kaggle.com/eakdag)
/
LB: 3.89
SMILES / eff_B1+LSTM /
[@sorkun](https://www.kaggle.com/sorkun)
&
[@eakdag](https://www.kaggle.com/eakdag)
/
LB: 1.89
SMILES / ResNest+GRU /
[@ubique](https://www.kaggle.com/ubique)
/
LB: 2.03
SELFIES / Transformer  /
[@proletheus](https://www.kaggle.com/proletheus)
/
LB: 2.52
SELFIES / eff_B1+LSTM /
[@sorkun](https://www.kaggle.com/sorkun)
&
[@eakdag](https://www.kaggle.com/eakdag)
/
LB: 3.74
[Applied Methods]
Molecular Reps (InChI, SMILES, SELFIES)
OHEM
Pseudo-labeling (improved single models but not improved the merged score)