# 18th place solution

Output:
I used 4096 byte pair encodings created from plain InChI text with
[sentencepiece](https://github.com/google/sentencepiece)
.
Model:
CNN -> Encoder -> Decoder
CNN is a custom model, built up from 5 layers of these:
def forward(self, x_in):
    x = self.bnl1(x_in)
    x = self.bnl2(x) + self.bnl4(self.bnl3(x))
    x = torch.cat((x, self.pool(x_in)), dim=1) if self.r else x
    return x
where bnl[1234] is convolution, followed by batch norm and relu.
Encoder and Decoder are Transformer-based.
Sampling molecules:
I removed samples that had no carbon or had no /b layer after /i (due to too few examples).
I calculated a weight for each molecule based on it's:
complexity:
x.count('-') + x.count('-') * x.count('(') + x.count('(')
atom count: number of all the atoms it's made up of
atom rarity: I calculate how many times (M) an atom is in all samples (N). For each molecule I sum up one of {1/M, 1/(N-M)} values for each atom, depending on if it is in that molecule or not.
layer rarity: the same as for atoms
I take these values to a power and add them together with some reasonable coefficient -> I have a final sampling weight for each molecule.
Training:
Cross entropy loss; Adam; One cycle LR schedule; Mixed Precision + FP32 fine-tuning.
Scheduled attention dropout and drophead. I also used
[sentencepiece](https://github.com/google/sentencepiece)
's BPE-dropout.
I've trained 5 models on different aspect ratios. Then I also trained each of them on yet a different aspect ratio for a few more epochs.
Their individual LD on my validation are as follows:
img_sizes = [(160,224), (192,192),
(256,544), (224,640), (320,480),
(288,512), (320,480), (480,320),
(288,512), (256,576), (320,448),
(384,384), (352,416), (192,768),]
raw_lds = [1.5233, 1.5275,
1.0955, 1.1850, 1.0699,
1.1706, 1.0629, 1.3790,
1.0664, 1.1234, 1.0832,
1.4073, 1.2095, 2.1064,]
norm_lds = [1.3661, 1.3879,
0.9438, 1.0244, 0.9248,
1.0205, 0.9280, 1.2254,
0.9415, 0.9923, 0.9400,
1.2318, 1.0646, 1.9285,]
Prediction:
I've predicted all molecules with all models separately and normalized all of the predictions.
Then I searched those that have the same normalized form.
I've predicted the rest of them (~390k) with ensembled beam search of size 16 and searched those that are not valid predictions. These (~16K) I predicted with beam size of 64.
Taking all these predictions I only had ~5k molecules that I had no valid prediction for.
I've predicted the 390k molecules with a bunch of other settings (with different beam size and ensemble weights), and tried to sample the valid ones with different strategies, but I just couldn't break the 0.70 barrier.
Very interestingly, I had a lot of submissions that had quite an LD between them but all of them were 0.70 on LB. I even had two submissions with 0.11 LD between them, and they're still 0.70 and 0.71 on LB.
I couldn't figure out if there could be any potential in this "phenomenon" with which I could make better predictions.
Didn't work:
Label smoothing
Additional targets on encoder output
Encoder level mixup (decoder works on the two targets individually from the input to the output)
Sorting molecules by InChI BPE lenght for decoder speedup. I forgot to randomize, lost a whole week trying to figure out what went wrong. It felt incredible, as this happened right after I bought a 3090 … :DDD
Thank you all:
This was a very interesting problem to work on, and was a huge pleasure to compete with all of you!
Even though I didn't end up in the gold zone, I don't regret spending that much money on a 3090, as so many worked so hard on this competition, and I'm more than happy that I could reach 18th place among so many so talented people!
Special thanks to:
[@stassl](https://www.kaggle.com/stassl)
and
[@yasufuminakama](https://www.kaggle.com/yasufuminakama)
for the idea of normalization, and
[@hengck23](https://www.kaggle.com/hengck23)
for the importance of validation
Thank you!