# 8th place solution - a densely connected GNN 

Thanks to the organizers for such an interesting competition. It was a lot of fun exploring the current state of the art of graph neural networks and trying to improve it. I also want to thank
[@artgor](/artgor)
,
[@tunguz](/tunguz)
,
[@borisdee](/borisdee)
and
[@psilogram](/psilogram)
for being amazing team mates. In short, our solution is a custom Graph Neural Network with SchNet interaction at its core, which alone scores ~ -2.95 on leaderboard. The model is a single model for all bonding types.
Since my team already had good engineered features for their lgb (with lb score of -2.0) when I joined, I could fully concentrate on what I love: finding a strong deep learning based solution.
The journey was as following
I started with refactoring Hengs starter kit for MPNN which was at around (-1.4 LB) for speed optimization to have a better speed of iterating ideas. After 2 days I got it down from 15 min to 90 sec  per epoch on my GTX1080Ti , without losing any accuracy. That enabled to quickly explore different ideas to improve MPNN which I implemented then. However the leaderboard score was still worse than our lgb.
So I continued to read a lot of papers related to GNN:
papers referenced in forum, and
newer papers referencing that papers
papers and git repos found by stalking top LB positions
papers shared by
[@artgor](/artgor)
who posted new ones in our slack before I have read the previous
I implemented most of them in hacky way to see if they fit for this competition. Besides others I implemented the papers listed below. Some had usable git repositories for others I need to implement more or less from scratch because they used a different input format or were coded in tensorflow/keras whereas I had setup everything in pytorch. All of them are supposed to predict properties on molecular level, so I adjusted where necessary to work on atom pair level:
[MPNN](https://arxiv.org/abs/1704.01212)
[GAT](https://github.com/SeongokRyu/Graph-neural-networks/blob/master/tutorials/gat.ipynb)
[Crystal GNN](https://github.com/txie-93/cgcnn)
[SchNet](https://arxiv.org/abs/1706.08566)
[MegNet](https://github.com/materialsvirtuallab/megnet)
[3DGNN](https://github.com/blackmints/3DGCN)
For me SchNet not only had the best code basis but also got decent results and was relatively fast, so I started building on that one. In general the model consists of 3 parts:
encoder for nodes and edges
consecutive blocks of interaction between encoded nodes and edges
regression head
As mentioned by other solutions SchNet lacks several things, but the core interaction between atoms and edges is good and fast.
So I spent some weeks improving that baseline step by step by adding new stuff I read in other papers  or I found useful during computer vision competitions. From every 10 ideas I implemented a maximum of one worked, and it felt like digging for treasures. At the end not much was left from the original SchNet apart from the core interaction function between nodes and edges.
The most architectural enhancements were the following
using additionally edge features like bondtype, angles
additionally updating edges (as in MegNet)
[densenet similar connections](assets/d39e5921fcfd99a72292bd91.pdf)
between interaction blocks
[squeeze and excitation block](https://arxiv.org/abs/1709.01507)
at the end of an Interaction
encoding and using  also molecule level features as in Hengs starter
using edge level hidden units for final regression
using molecule level hidden units for final regression
using a deep architecture (12 interaction blocks)
some details on training
preprocessing: save graph to disk (40 min)
bs: 64
optimizer: Adam
loss: weighted lmae (weights by inverse frequency of coupling type counts)
lr_schedule: Cyclic Cosine annealing with initial lr 0.0005 and cycles of 200 epochs which enables to  continue training easily by just adding a new cycle.
validation: groupkfold (10fold) per molecule
epochs: 1200 (and still underfitting)
our final submission:
We basically just bagged versions of the above model trained with different hyperparameters which took some resources. So we split training by
[@borisdee](/borisdee)
[@tunguz](/tunguz)
and myself. Some version of MegNet scoring around -2.5 LB made it also in the final blend but with very minor contribution.
[@psilogram](/psilogram)
then used a simple mae optimizer to blend the different models on bonding type level. -> private leaderboard score of -3.001
things that did not work
- additional task of autoencoding edge or node features
- metric learning
- and thousand of others.
Happy to answer any questions and thanks for reading.