# Solution 7th (CPMP view) First of all, I'd like to thank my team mates, without whom I would not have landed where we landed, and even would not have entered the competition. Let me also way that I'm very happy to see Ahmet become competition GM. And Outrunner is only one gold away from it. I'm sure he'll become GM in his next competition. I was very lucky to team with them. It proves 1 + 1 + 1 > 3. I also want to thank [@inversion](/inversion) , Kaggle staff in general, and CHAMPS for setting up this very challenging, unflawed contest. It is refreshing to not have any leak. Using test data wasn't helpful really either. And the way [@inversion](/inversion) managed the two crisis (QM9 use, and unplanned kernel share) was very effective. I was very reluctant to enter this competition, for two reasons. First, I was convinced that people with domain knowledge would have an unfair advantage. I was right as the top of LB shows a high proportion of teams with chemists on board. Second, it was clear from early discussions that graph NN would be key, and I had absolutely no experience with it. This second reason is a bit weak given most of us here are in same situation! Then Ahmet asked me if I would join, because we almost teamed in a previous competition and said we would team ASAP. It motivated me to complete an initial effort using lightgbm. I developed a core set of features, then added features specific to each type. The features are based on the graph defined by bonds computed using open babel (thanks [@borisd](/borisd) for the starter code). Angles for 2 bonds path, and dihedral angle for 3 bonds paths were key. I then added features based on the number of 1, 2, 3 bond paths starting on every atom on the path, using aggregates like number by bond type, angle averages, distance averages, etc. Depending on the type, number of features was between 500 and 2000. I trained one lgb per type. With this I got to 2.0 on LB with one fold the day I entered. Once trained on all folds, this model got 2.1 on LB. There are few things I did to improve the model. First, I use a very large number of leaves, 2**10 -1. I am not sure why this is good, but it was. Other parameter tuning wasn't very important. Second, I duplicated train and test rows with types 2JHH and 3JHH by swapping the two atoms, before feature engineering, because scalar coupling is symmetric. Final prediction for one row in the original train or test was the average of the two rows predictions after expansion. This was mentioned early in the forum I think. Third, for 3Jxx bond types, there is often more than one 3 bonds path between the two atoms. Here again I duplicated, using one row for each 3 bonds path between the two atoms, then averaging their predictions back for final prediction. I did not see this discussed anywhere. Row duplication improves lgb score by about 0.002, not much, but still useful. I then teamed with Ahmet, and given he had a NN with much better performance than my lgb model I gave up on my base model. I'll let Ahmet describe his NN (edit: description is available [here](https://www.kaggle.com/c/champs-scalar-coupling/discussion/106421#611963) ). Let me just say that with some improvements, like using logcosh as loss, and expanding its size, we got to -2.64 on LB using single 4 fold cv run. I was very happy given this moved us way above where lgb could move us to. Lgb was useful still, this time as a post processing step. As I did in a previous competition (Web Traffic Forecasting), I used lgb to predict the residuals of the NN, i.e. predict the difference between the NN prediction and the target. Tuning lgb with huber loss and a decaying learning rate (yes, this common deep learning practice can also be used for lgb) yields a -0.15 boost on LB. This plus bagging several runs eventually led to a score better than -2.8. To make sure we get a gold medal we knew we would need another strong model. I started developing a graph NN from scratch in keras, and was getting CV o -1.9 and LB -2.08 nwith just 4 features (atom and bond type, mulliken charge, and dihedral angle for 3 bond types). I stopped working on it when we teamed with Outrunner. Indeed, Outrunner skills in NN clearly outperfom mine, by far. He had a set of 13 NN models that yield almost -2.9 LB once bagged. He further improved them to yield -2.92 or so soon after merging. I will let him describe these models if he wants to. The first thing we did after teaming was to run my lgb postprocessing on his validation prediction. He was using a single train/val split with only 3% or so of validation data. Yet, even with this limited data lgb could extract about 0.025 more. Blending the result with what we had added another 0.03 or so, which moved us better than -2.95. The last progress we made was based on the NN approach used. A lot of people have used Graph NN where one molecule is a sample, and all coupling for that molecule can be predicted together. Outrunner and Ahmet use another way, with one sample per train and test row. They have features that capture the relative position of every atom compared to the two for which the coupling is computed. For instance distances to these two atoms, and an angle, and more. But distance to the two atoms and an angle doesn't uniquely identify atom positions. There is a circular symmetry. A way to remove this symmetry is to consider other atoms. Ahmet was doing it but not Outrunner. I suggested to use middle atoms. For 2JHx, a0 and a1 are both bond to a middle atom am. For 3JHx, a0 is bond to am0 which is bond to am1 which is bond to a1. Outrunner added distance to am, or am0/am1 depending on the type. This boosted performance to where we finally ended. Unfortunately, given we teamed late, he could only retrain 3 of his models with the new feature. Bagging these leads to -2.99 LB. Running lgb on top of it and blending with Ahmet-CPMP blend yields -3.032. I guess we could have landed higher with more models retrained, but we lacked time and/or GPUs. Bottom line is I am happy this very challenging competition ended. I probably should have entered earlier, my first sub is from July 22, only 5 weeks ago. But teaming with two talented people more than made for the lack of time. Edt, you'll find Ahmet NN description [here](https://www.kaggle.com/c/champs-scalar-coupling/discussion/106421#611963) . Outrunner told me his NN is similar to Ahmet's hence he won't provide a separate description.