# 2nd Place Solution (Draft)

First of all, I would like to thank the organizers for providing such a large, interesting, uniform, but challenging dataset. It taught me a lot of (hopefully correct) lessons, which I want to describe in the following. I'm planning to use the dataset eventually to predict solubility using neighbor information with my Bayesian IBOL (instance-Based Offset Learning) method, which I naïvely used in the beginning out-of-the box failing miserably. The idea will be to make the model adjust for batch artefacts automatically, which I tried with some success, but wasn't able to get the math right so far.
I'm looking forward to the method used by the winning team, which I think must have been able to do real structure-based predictions (probably also explicitly or implicitly correcting for plating structure).
Overview of the Ranked Solution
As detailed below, the solution doesn't really use any new molecular insights into solubility. It might even be considered "cheating" as it guesses non-explicitly available information and uses it to get most of the milage.
The model uses the sequence of ID numbers and the data plate size of 352 to infer the plate/row/column structure of the raw data. It turns out that a large portion of the variability can be modelled with additively combining plate/row/column increments plus some standard cheminformatics regression variables like logP, CMR, etc.  to an assumed normal prior. The predicted normal distributions per molecule are then subjected to (kappa) optimized probability cut-offs to yield the final assignment to solubility categories. The ranked solution used just logP, but I plan to post code which will allow multiple cheminformatics regressors to be used simultaneously.
Detailed Description of the Process that Arrived at the Ranked Method
The following describes the analysis process that lead to the ranked solution. It is from memory, but should be mostly correct.
Naïve Use of IBOL Models
Initially, the challenge dataset looked as if it would be interesting to use my Bayesian extension of kNN regression called IBOL (Instance-Based Offset Learning). As a Bayesian method, it returns predictions as probability distributions, which could utilized for classification by (kappa) optimizing probability thresholds for the classes to be predicted. In short, the IBOL method combines the data probability distributions (value+stdev or KL-optimal Gaussian replacements for censored and interval distributions) of the neighbors to yield predictive distributions for a molecule. Cheminformatics properties like clogP and CMR can be used to correct for systematic (physically motivated) differences of the neighbors to the query structure, and the similarity measure and the importance kernel can also be optimized.
Normalizing Structures
The contents of the competition data files
train.csv
and
test.csv
were read into Pandas data frames. Further processing was done on the standardized so-called MainFragment, which was extracted from the
smiles
column with RDKit using the method described in
[https://github.com/greglandrum/RSC_OpenScience_Standardization_202104/blob/main/MolStandardize%20pieces.ipynb](https://github.com/greglandrum/RSC_OpenScience_Standardization_202104/blob/main/MolStandardize%20pieces.ipynb)
.
The IBOL method can also separately model structure classes which can be assigned using RDKit SMARTS patterns. The preprocessing labelled the presence of quaternary nitrogen atoms (which should be very soluble), amines (mostly soluble in the given buffer), acids (should be soluble), and compounds containing aromatic nitrogens. Decoupling those classes for model building should have helped in modeling, but didn't. However, it later turned out that only the top-up structure subset with IDs above EOS100000 did actually contain any acids, which was rather puzzling.
Computing Cheminformatics Descriptors with RDKit
In addition to standardization, preprocessing also computed CLOGP, CMR, TPSA, and more cheminformatics descriptors for the molecules. IBOL can pick regressors from those sets for neighbor extrapolation.
Using IBOL
After naïvely converting the solubility categories to allowed ranges from the prior distribution and converting the resulting distributions to Gaussians, the IBOL engine did find an optimized model, but it yielded a kappa of 0.004, i.e. a complete failure.
Looking for Supplier Information to Catch Outliers and Artefacts
Reviewing information about the source of the actual samples in
[https://www.eu-openscreen.eu/services/compound-collection.html](https://www.eu-openscreen.eu/services/compound-collection.html)
allow tagging the compounds by the source from which they were taken as the supplier for the non-Pilot structures could be found in the SDF files.
Building separate models for the main sources Enamine, ChemDiv, ChemBridge, Pilot, and Others eventually improved the IBOL predictions by a factor of 22 (kappa=0.089), i.e. still unusable. Structural clustering by supplier seemed to collide with model building as different sources provided different kinds of structures with possibly also different artefacts in the measurement process. Particularly striking was the fact that only the Pilot dataset contained any acids.
Starting to Visualize the Training Data
The latter observation triggered further inspection of the training data statistics, and the different sources had very different percentages of partly and insoluble compounds. It was also apparent that compounds like quaternary ammonium salts, which by chemical intuition should be all soluble, showed unexpected data distributions: They were soluble when not from the Pilot set, but some were insoluble when found in the Pilot set. This unevenness by source was very likely confounding any structural model learning.
Guessing Plate Numbers and Row/Column Layout
In order to look deeper into the possible confounding factors, further information found in the challenge description was reviewed and pointed towards a way to split the data into plates with row and column IDs: The numeric part of the ID numbers was practically consecutive (except for a small number of drop-outs either for purity or for confidentiality reasons) with a gap before EOS100001. This sounded like sequential well assignment with a top-up set plated later. Using this hypothesis, plate numbers (0-286) and row/column numbers were inferred.
Visualizing the Data Distribution by Plate
Doing this showed clear patterns of some plates containing mostly/completely soluble compounds and some much more insoluble ones. As this is a common artefact in screening.
![](assets/22b816e88b363409c7a93769.png)
Quaternary ammonium salts are marked with orange circles and acids show as magenta circles. Plates 244 and 279 (red labels) did not contain any non-soluble compounds. The top-up set after EOS100001 had all the acids and also more diversity in logP. And there were plates with many insoluble compounds sprinkled through the set. If those "anomalous" plates would also have some structural coherence, then any structure-base modeling w/o correcting for plate artefacts would be futile.
Optimizing Plate/Row/Column Model with CI Descriptors
Given this batch structure in the training data, it seemed obvious to try to just use this structure to predict the solubility category. The model was a Gaussian distribution per compound with common variance adding one of 287 trainable offsets per plate and also row and column offsets to account for possible common measurement artefacts across the plates. An additional contribution to the means of the Gaussians was added as an optimized factor times the clogp of the compound. The Maximum Likelihood estimator of this model was then used to (kappa) optimize probability threshold for the actual assignments.
This rather crude model yielded the score of the ranked solution: kappa=0.19395 on the public test set and kappa=0.21748 on the private set.
Lessons Learned
Do not throw your favorite ML model at an arbitrary dataset and expect it to work.
Non-random draws from the structure universe can confuse modelling.
Compare what you know about the problem with the presentation of the data to spot (detrimental/exploitable) peculiarities. Visualization is key.
Finding the major sources of variance can give you a lot of mileage even if it does not help understanding the real problem.
As with many puzzles, there is more information than you think in the problem description.
Probabilistic models can be used to predict uneven classification. (Need to get the math right for IBOL models to prove this beyond the prior model, though.) The additional uncertainty model can even help in making decisions.
Having a competitor much ahead of you makes you think.