# 10th place solution

Thanks the organizers for such a great competition and great dataset. I just remembered the very recent nightmarish LANL competition, where its organizers were totally inactive and leaked dataset, then this Molecule competition is indeed one of the best ever. I love it very much.
And special thanks to all of my teammates, who each one actually did a great deal of work from the beginning. Thanks so much.
In summary, main methods are:
SchNet1 (1st train) with input features: node-features (x, y, z, atom_type, mulliken_charges), edge-features (bond binary (for pairs of atoms with distance 1),  angle (for pairs of atoms with distance 2), and dihedral (for pairs of atoms with distance 3). We predict 4 scc (scalar_coupling_constant) contributions and scc itself. This model gave us around -2.7 LB.
SchNet2 (2nd train): the model is the same as SchNet1, except we feed 4 out-of-fold scc contributions and scc itself as edge features where applicable, EXCEPT the edge of the learning instance itself (so in other words, we prevent leak from this, and this is not a stacking method). This model gave us -2.85 LB.
Scheduling: Cyclic LR.
Train 3 to 4 different seeds of 4-fold for each type. Some small types (xJHN) we also had an extra training of 8-fold.
Quantile regression and median averaging: since MAE cares more about predicting median of the variables, we tried to use quantile regression (with q=0.5), instead of linear regression, for 2 sub-tasks: 1) blending periodical valley model checkpoints of CyclicLR, where we removed the first 50% earlier checkpoints and only kept the last 50% checkpoints. 2) blending oof predictions of seeds of SchNet1 and SchNet2. This gave us around 0.01-0.02 boost on LB. For averaging test folds predictions, we also use median instead of mean, and found that it worked perfectly on LB.
Now, I would like to describe the process of methods from start to end, like a journey.
First we began with tree-based models, and without much chemistry knowledge. 
- Zidmie
[@zidmie](/zidmie)
in our team wrote a custom code to extract bond information from only 3D coords, which was really accurate and impressive (and later proved unnecessary since we have some chemistry toolboxes to do that). 
- We even know aromatic indicator of each atom by his method! His early work laid a good foundation to get our team to top 15, at the time we were strong enough to add in
[@joshxsarah](/joshxsarah)
and Kyle
[@petersk20](/petersk20)
, who had great mindsets and diverse ideas.
It was proved that feature importance permutation is a spectacularly good technique to eliminate useless features. Josh and Kyle with their strong features, together with importance permutation once got us to 6th place. But not late after that we gradually slipped on the LB as we saw tree-based models can't compete GCN in this competition.
We then developed a custom GCN code based on Heng's shared code in the forum, which got us -2.5 to -2.6 LB, but can never reach -2.7.
Then we merged with Akira
[@akirasosa](/akirasosa)
, who has extremely great skills in NN!
In the remaining of the competition onwards when having 5 people, we exploited all possible local GPUs (6) as well as 6 accounts of Google Cloud Platform (each with $300 free credits), to boost the score until the end.
My teammates, especially Akira, will add in necessary information regarding SchNet if I missed something. You can view his separate post about SchNet architecture here:
[https://www.kaggle.com/c/champs-scalar-coupling/discussion/106293#latest-610869](https://www.kaggle.com/c/champs-scalar-coupling/discussion/106293#latest-610869)
Our kernels are here: 
[GCN from Heng's code]
[https://www.kaggle.com/joshxsarah/custom-gcn-10th-place-solution](https://www.kaggle.com/joshxsarah/custom-gcn-10th-place-solution)
[SchNet light Kaggle version]
[https://www.kaggle.com/petersk20/schnet-10th-place-solution](https://www.kaggle.com/petersk20/schnet-10th-place-solution)
Thanks for reading.