First of all, I would like to thank the organizers for providing such a large, interesting, uniform, but challenging dataset. It taught me a lot of (hopefully correct) lessons, which I want to describe in the following. I'm planning to use the dataset eventually to predict solubility using neighbor information with my Bayesian IBOL (instance-Based Offset Learning) method, which I naïvely used in the beginning out-of-the box failing miserably. The idea will be to make the model adjust for batch artefacts automatically, which I tried with some success, but wasn't able to get the math right so far.
\nI'm looking forward to the method used by the winning team, which I think must have been able to do real structure-based predictions (probably also explicitly or implicitly correcting for plating structure).
\nAs detailed below, the solution doesn't really use any new molecular insights into solubility. It might even be considered \"cheating\" as it guesses non-explicitly available information and uses it to get most of the milage.
\nThe model uses the sequence of ID numbers and the data plate size of 352 to infer the plate/row/column structure of the raw data. It turns out that a large portion of the variability can be modelled with additively combining plate/row/column increments plus some standard cheminformatics regression variables like logP, CMR, etc. to an assumed normal prior. The predicted normal distributions per molecule are then subjected to (kappa) optimized probability cut-offs to yield the final assignment to solubility categories. The ranked solution used just logP, but I plan to post code which will allow multiple cheminformatics regressors to be used simultaneously.
\nThe following describes the analysis process that lead to the ranked solution. It is from memory, but should be mostly correct.
\nInitially, the challenge dataset looked as if it would be interesting to use my Bayesian extension of kNN regression called IBOL (Instance-Based Offset Learning). As a Bayesian method, it returns predictions as probability distributions, which could utilized for classification by (kappa) optimizing probability thresholds for the classes to be predicted. In short, the IBOL method combines the data probability distributions (value+stdev or KL-optimal Gaussian replacements for censored and interval distributions) of the neighbors to yield predictive distributions for a molecule. Cheminformatics properties like clogP and CMR can be used to correct for systematic (physically motivated) differences of the neighbors to the query structure, and the similarity measure and the importance kernel can also be optimized.
\nThe contents of the competition data files train.csv and test.csv were read into Pandas data frames. Further processing was done on the standardized so-called MainFragment, which was extracted from the smiles column with RDKit using the method described in https://github.com/greglandrum/RSC_OpenScience_Standardization_202104/blob/main/MolStandardize%20pieces.ipynb.
The IBOL method can also separately model structure classes which can be assigned using RDKit SMARTS patterns. The preprocessing labelled the presence of quaternary nitrogen atoms (which should be very soluble), amines (mostly soluble in the given buffer), acids (should be soluble), and compounds containing aromatic nitrogens. Decoupling those classes for model building should have helped in modeling, but didn't. However, it later turned out that only the top-up structure subset with IDs above EOS100000 did actually contain any acids, which was rather puzzling.
\nIn addition to standardization, preprocessing also computed CLOGP, CMR, TPSA, and more cheminformatics descriptors for the molecules. IBOL can pick regressors from those sets for neighbor extrapolation.
\nAfter naïvely converting the solubility categories to allowed ranges from the prior distribution and converting the resulting distributions to Gaussians, the IBOL engine did find an optimized model, but it yielded a kappa of 0.004, i.e. a complete failure.
\nReviewing information about the source of the actual samples in https://www.eu-openscreen.eu/services/compound-collection.html allow tagging the compounds by the source from which they were taken as the supplier for the non-Pilot structures could be found in the SDF files.
\nBuilding separate models for the main sources Enamine, ChemDiv, ChemBridge, Pilot, and Others eventually improved the IBOL predictions by a factor of 22 (kappa=0.089), i.e. still unusable. Structural clustering by supplier seemed to collide with model building as different sources provided different kinds of structures with possibly also different artefacts in the measurement process. Particularly striking was the fact that only the Pilot dataset contained any acids.
\nThe latter observation triggered further inspection of the training data statistics, and the different sources had very different percentages of partly and insoluble compounds. It was also apparent that compounds like quaternary ammonium salts, which by chemical intuition should be all soluble, showed unexpected data distributions: They were soluble when not from the Pilot set, but some were insoluble when found in the Pilot set. This unevenness by source was very likely confounding any structural model learning.
\nIn order to look deeper into the possible confounding factors, further information found in the challenge description was reviewed and pointed towards a way to split the data into plates with row and column IDs: The numeric part of the ID numbers was practically consecutive (except for a small number of drop-outs either for purity or for confidentiality reasons) with a gap before EOS100001. This sounded like sequential well assignment with a top-up set plated later. Using this hypothesis, plate numbers (0-286) and row/column numbers were inferred.
\nDoing this showed clear patterns of some plates containing mostly/completely soluble compounds and some much more insoluble ones. As this is a common artefact in screening.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6486624%2Fb0490ab8d8f9d04092580c9638fd6795%2FStatisticsByIDAndPlate.png?generation=1673187500209281&alt=media\")
\nQuaternary ammonium salts are marked with orange circles and acids show as magenta circles. Plates 244 and 279 (red labels) did not contain any non-soluble compounds. The top-up set after EOS100001 had all the acids and also more diversity in logP. And there were plates with many insoluble compounds sprinkled through the set. If those \"anomalous\" plates would also have some structural coherence, then any structure-base modeling w/o correcting for plate artefacts would be futile.
Given this batch structure in the training data, it seemed obvious to try to just use this structure to predict the solubility category. The model was a Gaussian distribution per compound with common variance adding one of 287 trainable offsets per plate and also row and column offsets to account for possible common measurement artefacts across the plates. An additional contribution to the means of the Gaussians was added as an optimized factor times the clogp of the compound. The Maximum Likelihood estimator of this model was then used to (kappa) optimize probability threshold for the actual assignments.
\nThis rather crude model yielded the score of the ranked solution: kappa=0.19395 on the public test set and kappa=0.21748 on the private set.
\nI've been using the code of the notebook for the 2nd rank entry to score models of varying components. The table below shows a summary.
\n| Model Components | \nMAP Score | \nKappa Train | \nLower Limit | \nUpper Limit | \nPublic Score | \nPrivate Score | \n
|---|---|---|---|---|---|---|
| CLOGP | \n21277.22 | \n0.04334 | \n0.93043 | \n0.93246 | \n0.04192 | \n0.02923 | \n
| CMR | \n21258.17 | \n0.04985 | \n0.92308 | \n0.92713 | \n0.04192 | \n0.02923 | \n
| Plate | \n19608.21 | \n0.21072 | \n0.86857 | \n0.87046 | \n0.19448 | \n0.19768 | \n
| Plate, Row, Column | \n19475.75 | \n0.22713 | \n0.85684 | \n0.86396 | \n0.19252 | \n0.21781 | \n
| Plate, Row, Column, CLOGP | \n19441.63 | \n0.23196 | \n0.84700 | \n0.86459 | \n0.19565 | \n0.21890 | \n
| Plate, Row, Column, CMR | \n19401.01 | \n0.23478 | \n0.85036 | \n0.86205 | \n0.20293 | \n0.21502 | \n
| Plate, Row, Column, CLOGP, CMR | \n19391.69 | \n0.23760 | \n0.84755 | \n0.85799 | \n0.20307 | \n0.22241 | \n
| Plate, Row, Column, CLOGP, CMR, FCSP3 | \n19385.27 | \n0.23704 | \n0.84047 | \n0.86171 | \n0.20715 | \n0.22434 | \n
It is obvious that plate offsets provide the biggest improvement. The row/column modeling also ads some predictivity. The Cheminformatics descriptor CMR scores better than CLOGP on the public test set but worse on the private set. Using multiple descriptors further improves the model, but nowhere near the contribution from the artefact model components.
", "votes": 1, "replies": [ { "id": 2104396, "author_name": "Beardy Polonium", "author_url": "", "post_date": "2023-01-17T19:41:59.910000", "content": "So I can understand why there are plate-wise artifacts, but do you know what's the source of the row-wise and column-wise artifacts?
", "votes": 1, "replies": [ { "id": 2104506, "author_name": "Bernhard Rohde", "author_url": "", "post_date": "2023-01-17T21:08:53.823000", "content": "If you browse through the notebook 'Plate/Row/Column Model with CLogP for Solubility' with the code for my solution, there are plots that show the size of plate and row/column offsets and you can see that those row/column offsets are much smaller. However, they are to be expected in high throughput screening especially if basically single point data are collected.
\nTypical causes are differences in the buffer dispensing tips (the same ones are used across one of the dimensions) as well as solvent evaporation during incubation and other processing steps, which are larger at the edges than in the middle of the plates. Plate readers also might have such biases. Putting controls on the first and last column and not randomly on the plate is technically a lot easier for HTS, but catching row/column biases per plate is difficult this way.
\nIn the comments of the notebook, there is also a table that illustrates the change in explanatory power if the model variables are added sequentially.
\nUpdate: I forgot that the table was also posted here.
", "votes": 1, "replies": [] }, { "id": 2104543, "author_name": "Bernhard Rohde", "author_url": "", "post_date": "2023-01-17T21:53:07.203000", "content": "As to the plate-wise artefacts, there might be clustering of similar structures on plates e.g. from synthesis series, which should not affect the predictivity of structure-based models, but plates do have their handling history (compound source, storage time, dust, temperature, dispensing equipment, freezing/thawing for cherry picking, …) which can affect the full plate, and this is what I expect to be the major issue here. Having the raw readouts or even HPLC/MS QC might allow spotting this with further analysis.
\nUpdate: In combination with intra-plate clustering, experimental plate artefacts could also simulate structure dependence.
", "votes": 2, "replies": [] } ] } ] }, { "id": 2101924, "author_name": "Bernhard Rohde", "author_url": "", "post_date": "2023-01-16T09:53:26.570000", "content": "Added a functional (cleaned-up) notebook that implements the algorithm used. The notebook includes some tracing output, but be aware that rerunning it might take a few hours.
", "votes": 1, "replies": [ { "id": 2101927, "author_name": "", "author_url": "", "post_date": "2023-01-16T09:56:14.103000", "content": "", "votes": 0, "replies": [] }, { "id": 2101929, "author_name": "John Mitchell", "author_url": "", "post_date": "2023-01-16T09:57:30.010000", "content": "@bernhardrohde - Is the notebook set to public viewable - I can't see it? Also, you should be able to rename it using the box at the top left of the editor screen.
", "votes": 0, "replies": [ { "id": 2101969, "author_name": "Bernhard Rohde", "author_url": "", "post_date": "2023-01-16T10:21:48.853000", "content": "I found the textbox myself and changed it. This did also changed the URL, which I've already updated in the post. Looks like a race condition ;-)
", "votes": 1, "replies": [] } ] } ] }, { "id": 2094186, "author_name": "John Mitchell", "author_url": "", "post_date": "2023-01-10T16:45:09.830000", "content": "Well done @bernhardrohde - hacking the plate structure like that is very clever and well within the spirit of Kaggle competitions. From the perspective of solubility prediction, I guess it makes the point that solubilities of structurally very similar compounds are particularly valuable information, beyond just their contribution to a global ML model. Possibly something like a simple kNN model of the solubility, while typically only a moderately good predictor in itself, might prove a useful feature in a more sophisticated model?
", "votes": 1, "replies": [ { "id": 2094577, "author_name": "Bernhard Rohde", "author_url": "", "post_date": "2023-01-10T21:00:03.257000", "content": "This is the idea of my IBOL models I'm planning to apply to the dataset. Reaching the second place with just modeling the plat/row/column artefacts shows that it would be risky to simply do everything with a structure-based, even a kNN, model as structurally near neighbors might be on different plates with, thus, differing artefacts.
\nMy current plan is to use the IBOL (Instance-Based Offset Learning) models, while simultaneously adjusting for the artefacts. I tried this already, but the math wasn't right and I got close but failed to beat my simplistic model. I think I have an idea for the math now that should work, but didn't get to implement and check it yet. Once that works, I think the method can be used to adjust for many different kinds of batch effects like combining results from different assays/publications.
\nThe IBOL approach actually provides probability distributions for the predicted value based on the neighbor data distributions, and those neighbor distributions can even be truncated/censored or ranges. This is close to Gaussian Processes, but computationally feasible even for such large datasets as this one. It also yields parameters that are better to understand. I've been using this method to support Bayesian Optimization even for multi-objective problems optimizing the Pareto Front simulating the medchem workflow. Unfortunately, I'm in the habit of procrastinating when it comes to publishing as I always have ideas for improving the method.
", "votes": 2, "replies": [] } ] }, { "id": 2143791, "author_name": "John Mitchell", "author_url": "", "post_date": "2023-02-14T14:06:08.863000", "content": "I'm finding this ongoing discussion really interesting, albeit catalysed by some non-ideal circumstances. From my perspective, it is clear that:
\nThe test data are organised such that soluble and insoluble compounds are distributed in a highly non-random fashion, see this discussion by @yumizsui, as well @bernhardrohde's solution.
Bernhard has identified a very plausible link between solubility and plate, which in fact follows logically if compounds are typically put in plates grouped by the given label numbers. Noted also that @davidoloren found an optimal moving window close to 400.
Bernhard has also suggested patterns around (row, column) positions on the plate, which are probably weaker signals and less well established than the previous two points.
Both the top two placed models in the private LB utilised label information as a feature. In Bernhard's case at least this was the major input into his model; for the first placed team of @yumizsui, @tonets and @appaeak this was unintentional.
The organisers @robertharmel and @agiani99 have not as yet accepted the first place solution as the winner and have indicated that Bernhard's solution is also unacceptable.
In a parallel universe somewhere, the labels and ordering of the test set were randomised and we are not having this discussion.
The competion involved \"predicting\" (temporally post hoc, but unseen) the results of a high-throughput kinetic solubility assay (mostly one datum per compound). These were very noisy, as I think is acknowledged by all, and the correlations with traditional cheminformatics descriptors were much weaker than I am used to in computing \"predictions\" for low-throughput (typically) multi-replicate equilibrium intrinsic solubilities. The difference is around an order of magnitude. Given how weak the other features were, any model using it would indeed benefit massively from the label number. This is an unusual (for me) situation where one descriptor dominates so strongly.
\nMy major point is that I don't believe that utilising the label as a feature is cheating. I've done plenty of solubility prediction tasks where the label is a chemical name, and am engaged in one now where the label is (for technical reasons) a SMILES. These things truly carry chemical information and I see nothing unethical about leveraging that.
\nIf the label turns out to indicate the source of the compound, this reflects genuine knowledge that would be available in many (though not all) real-world contexts. If the team doing the solubility assay know that \"Acme Chemical Co. only make brickdust\", \"Our Organic synthesis team are considerate in designing solubility into all their projects\", \"Nothing with that blue-and-yellow label is ever soluble\", or \"Compounds from Chemical Store 2 often dissolve well\", then each of these things is useful knowledge that may genuinely aid both prediction and choosing directions to go in order to enhance solubility. Soluble molecules are unlikely to be most efficiently found by randomly sampling libraries. Indeed, we ourselves found that human experts are just as good as Machine Learning at predicting solubility. Such community wisdom is extremely valuable.
\nI look at the leaderboard and it's very obvious that @yumizsui, @tonets and @appaeak won, and that @bernhardrohde was a strong second. I would urge the organisers to award the prize to the top-placed team, while also giving acknowledgement to whatever entry turns out to be the best based solely on molecular features (\"molecular\" is intended also to include any solid, liquid or solution phase ones, if relevant).
\nFinally, I can't remember any solubility prediction competition that hasn't had some potential issues related to dataset design, data accuracy, or ease-of-cheating. Organising and running such a competition is a hard thing to do, and hindsight is unavailable in advance. I make no criticism of the organisers, but we are where we are.
", "votes": 2, "replies": [] }, { "id": 2143339, "author_name": "Bernhard Rohde", "author_url": "", "post_date": "2023-02-14T07:52:09.930000", "content": "I've created another visualization of the guessed plate layout of the dataset. This is a pretty big image, but you should be able to zoom into it. The frame borders are colored by compound source with the final seven plates from the topup set with IDs >= EOS100001.
\nThe blueish color indicates test set compounds for which there is no sol_category.
The red spots are 'low' and the orange ones are 'medium'. The characters in the cells indicate structural features, e.g. 'Q' for quaternary amine' and 'A' for acids. You can easily see that the topup set is structurally different from the other plates.
", "votes": 0, "replies": [ { "id": 2143350, "author_name": "Bernhard Rohde", "author_url": "", "post_date": "2023-02-14T07:55:54.410000", "content": "An additional picture shows the average frequency of 'low', 'medium', and hit ('low' or 'medium') compounds by well coordinates. The first row is for the base set and the second shows the topup set. There isn't much intelligible structure except for some edge effects.
", "votes": 0, "replies": [] } ] }, { "id": 2137810, "author_name": "Andrea Zaliani", "author_url": "", "post_date": "2023-02-10T10:48:04.273000", "content": "Hi Bernhard,
\nI read with high interest your notebook about possible plate effects, which we anyway did not see experimentally. I thank you for the depth of your approach, but I suspect there might be something to be put back in right order.
\nPlate effects can be always present when we are managing thousands of compounds within few tens of 384-well plates. Even if we remove two or more columns for control the remaining 16row x 22columns plate can be filled by systematic errors by robots generating a pattern. You claimed you found one or more using a guess on locations of compounds (row and column). I really doubt this is real as we did not provide this information.
\nThe guess is still legit until proofed wrong, I think. Anyway, how can a statistical model be used to predict external molecules which don't have this information? This is why your model is for us unacceptable, as it does not root into the chemo-physical nature of the solubility experiment, which we think it is still highly structure-dependent.
\nOn the other hand, I personally have doubts any pattern eventually found can be real, without information on number and position of control columns used, without knowing whether plate filling has been done by row or by column, without knowing whether plate have been filled completely or only, say, 90%. I don't want to underestimate your work, which definitively sets a bar for all future chemoinformatic Kaggle competitions on pharma relevant datasets.
\nI hope that this comments will help critically novel Kaggle users/teams when dealing with this data domain.
\nBest
\nAndrea
Hi @agiani99,
\nLet me go though your comments in turn:
\n\nPlate effects can be always present when we are managing thousands of compounds within few tens of 384-well plates. Even if we remove two or more columns for control the remaining 16row x 22columns plate can be filled by systematic errors by robots generating a pattern. You claimed you found one or more using a guess on locations of compounds (row and column). I really doubt this is real as we did not provide this information.
\n
Agreed. The plate effects my Bayesian model detected were much stronger than the row/column model I added on top to get a better score (See the table above that was generated post deadline). The plating I assumed looked convincing to me as you can see in the picture above e.g. for plates 194, 197, 199 (more than average low compounds), and 244, 279 (less than average low compounds), since the neighboring 'plates' (read: ID stretches) are quite different. Knowing this for sure would of course be better. I also redid the row/column model recently, and it appears that only in the first compound set (before EOS100001) actually has a significant edge effect on one of the row and one of the column edges. Assuming an additive row/column offset seemed to be a little to flexible. I do, however, still think the sequence of compounds on the plates is what the statistics told me. The raw data could prove me wrong, however ;-).
\n\n\nThe guess is still legit until proofed wrong, I think. Anyway, how can a statistical model be used to predict external molecules which don't have this information? This is why your model is for us unacceptable, as it does not root into the chemo-physical nature of the solubility experiment, which we think it is still highly structure-dependent.
\n
What the model actually does is correcting the means of the predicted solubility distributions for the plate effects in a linear model. This can be interpreted as adding the plate model to the real structure-based model (just using clogp in my case) that would govern the data generation if the plate effect wasn't there. It would be better to estimate this from the raw data, but what I've done shows that there is some structure based variance, but not enough to compete with the plate effects. So structure-based prediction (just ignoring the artefact coefficients of the linear model) is possible, but not very useful given the current dataset.
\nAnother (Bayesian) point I raised above has to do with the different distributions for the first compound set and what I called the topup set. They are structurally so different that the only conclusions that can be drawn are specific to the two subsets. This actually also shows by the fact that the Kappa score (training is all I have here) of the two subsets are both higher (~0.21 for the first set and ~0.41 for the topup set) than the training score for the combined dataset.
\nSo any prediction depends on whether the compounds to be predicted were selected like those in the first set or the topup set. Having only classes instead of actual solubilities as in AID1996 also provides a lot less information (I think about 0.3 bits) per compound. The confounding plate effect will thus be even more important.
\n\n\nOn the other hand, I personally have doubts any pattern eventually found can be real, without information on number and position of control columns used, without knowing whether plate filling has been done by row or by column, without knowing whether plate have been filled completely or only, say, 90%. I don't want to underestimate your work, which definitively sets a bar for all future chemoinformatic Kaggle competitions on pharma relevant datasets.
\n
This was actually the real point of my competition contribution and of requesting the complete information for the published dataset once it becomes available.
\nOne final remark, I do read your comment as an indication that there will be no winner and thus no possible discussion at the SLAS Meeting. If this is the case, then there wouldn't be a point in me going there on my own expenses. If, however, the contributors, data producers, and judges would be present, I would still go there to have a scientific exchange.
\nCheers,
\nBernd
", "votes": 1, "replies": [] } ] }, { "id": 2093133, "author_name": "", "author_url": "", "post_date": "2023-01-09T20:27:09.070000", "content": "", "votes": 0, "replies": [] } ], "comment_count": 16, "messages": [ { "id": 2091380, "postDate": "2023-01-08T11:00:21.817Z", "content": "First of all, I would like to thank the organizers for providing such a large, interesting, uniform, but challenging dataset. It taught me a lot of (hopefully correct) lessons, which I want to describe in the following. I'm planning to use the dataset eventually to predict solubility using neighbor information with my Bayesian IBOL (instance-Based Offset Learning) method, which I naïvely used in the beginning out-of-the box failing miserably. The idea will be to make the model adjust for batch artefacts automatically, which I tried with some success, but wasn't able to get the math right so far.
\nI'm looking forward to the method used by the winning team, which I think must have been able to do real structure-based predictions (probably also explicitly or implicitly correcting for plating structure).
\nAs detailed below, the solution doesn't really use any new molecular insights into solubility. It might even be considered \"cheating\" as it guesses non-explicitly available information and uses it to get most of the milage.
\nThe model uses the sequence of ID numbers and the data plate size of 352 to infer the plate/row/column structure of the raw data. It turns out that a large portion of the variability can be modelled with additively combining plate/row/column increments plus some standard cheminformatics regression variables like logP, CMR, etc. to an assumed normal prior. The predicted normal distributions per molecule are then subjected to (kappa) optimized probability cut-offs to yield the final assignment to solubility categories. The ranked solution used just logP, but I plan to post code which will allow multiple cheminformatics regressors to be used simultaneously.
\nThe following describes the analysis process that lead to the ranked solution. It is from memory, but should be mostly correct.
\nInitially, the challenge dataset looked as if it would be interesting to use my Bayesian extension of kNN regression called IBOL (Instance-Based Offset Learning). As a Bayesian method, it returns predictions as probability distributions, which could utilized for classification by (kappa) optimizing probability thresholds for the classes to be predicted. In short, the IBOL method combines the data probability distributions (value+stdev or KL-optimal Gaussian replacements for censored and interval distributions) of the neighbors to yield predictive distributions for a molecule. Cheminformatics properties like clogP and CMR can be used to correct for systematic (physically motivated) differences of the neighbors to the query structure, and the similarity measure and the importance kernel can also be optimized.
\nThe contents of the competition data files train.csv and test.csv were read into Pandas data frames. Further processing was done on the standardized so-called MainFragment, which was extracted from the smiles column with RDKit using the method described in https://github.com/greglandrum/RSC_OpenScience_Standardization_202104/blob/main/MolStandardize%20pieces.ipynb.
The IBOL method can also separately model structure classes which can be assigned using RDKit SMARTS patterns. The preprocessing labelled the presence of quaternary nitrogen atoms (which should be very soluble), amines (mostly soluble in the given buffer), acids (should be soluble), and compounds containing aromatic nitrogens. Decoupling those classes for model building should have helped in modeling, but didn't. However, it later turned out that only the top-up structure subset with IDs above EOS100000 did actually contain any acids, which was rather puzzling.
\nIn addition to standardization, preprocessing also computed CLOGP, CMR, TPSA, and more cheminformatics descriptors for the molecules. IBOL can pick regressors from those sets for neighbor extrapolation.
\nAfter naïvely converting the solubility categories to allowed ranges from the prior distribution and converting the resulting distributions to Gaussians, the IBOL engine did find an optimized model, but it yielded a kappa of 0.004, i.e. a complete failure.
\nReviewing information about the source of the actual samples in https://www.eu-openscreen.eu/services/compound-collection.html allow tagging the compounds by the source from which they were taken as the supplier for the non-Pilot structures could be found in the SDF files.
\nBuilding separate models for the main sources Enamine, ChemDiv, ChemBridge, Pilot, and Others eventually improved the IBOL predictions by a factor of 22 (kappa=0.089), i.e. still unusable. Structural clustering by supplier seemed to collide with model building as different sources provided different kinds of structures with possibly also different artefacts in the measurement process. Particularly striking was the fact that only the Pilot dataset contained any acids.
\nThe latter observation triggered further inspection of the training data statistics, and the different sources had very different percentages of partly and insoluble compounds. It was also apparent that compounds like quaternary ammonium salts, which by chemical intuition should be all soluble, showed unexpected data distributions: They were soluble when not from the Pilot set, but some were insoluble when found in the Pilot set. This unevenness by source was very likely confounding any structural model learning.
\nIn order to look deeper into the possible confounding factors, further information found in the challenge description was reviewed and pointed towards a way to split the data into plates with row and column IDs: The numeric part of the ID numbers was practically consecutive (except for a small number of drop-outs either for purity or for confidentiality reasons) with a gap before EOS100001. This sounded like sequential well assignment with a top-up set plated later. Using this hypothesis, plate numbers (0-286) and row/column numbers were inferred.
\nDoing this showed clear patterns of some plates containing mostly/completely soluble compounds and some much more insoluble ones. As this is a common artefact in screening.
\n
\nQuaternary ammonium salts are marked with orange circles and acids show as magenta circles. Plates 244 and 279 (red labels) did not contain any non-soluble compounds. The top-up set after EOS100001 had all the acids and also more diversity in logP. And there were plates with many insoluble compounds sprinkled through the set. If those \"anomalous\" plates would also have some structural coherence, then any structure-base modeling w/o correcting for plate artefacts would be futile.
Given this batch structure in the training data, it seemed obvious to try to just use this structure to predict the solubility category. The model was a Gaussian distribution per compound with common variance adding one of 287 trainable offsets per plate and also row and column offsets to account for possible common measurement artefacts across the plates. An additional contribution to the means of the Gaussians was added as an optimized factor times the clogp of the compound. The Maximum Likelihood estimator of this model was then used to (kappa) optimize probability threshold for the actual assignments.
\nThis rather crude model yielded the score of the ranked solution: kappa=0.19395 on the public test set and kappa=0.21748 on the private set.
\ntrain.csv and test.csv were read into Pandas data frames. Further processing was done on the standardized so-called MainFragment, which was extracted from the smiles column with RDKit using the method described in [https://github.com/greglandrum/RSC_OpenScience_Standardization_202104/blob/main/MolStandardize%20pieces.ipynb](https://github.com/greglandrum/RSC_OpenScience_Standardization_202104/blob/main/MolStandardize%20pieces.ipynb).\n\nThe IBOL method can also separately model structure classes which can be assigned using RDKit SMARTS patterns. The preprocessing labelled the presence of quaternary nitrogen atoms (which should be very soluble), amines (mostly soluble in the given buffer), acids (should be soluble), and compounds containing aromatic nitrogens. Decoupling those classes for model building should have helped in modeling, but didn't. However, it later turned out that only the top-up structure subset with IDs above EOS100000 did actually contain any acids, which was rather puzzling. \n\n###Computing Cheminformatics Descriptors with RDKit\nIn addition to standardization, preprocessing also computed CLOGP, CMR, TPSA, and more cheminformatics descriptors for the molecules. IBOL can pick regressors from those sets for neighbor extrapolation.\n\n###Using IBOL\nAfter naïvely converting the solubility categories to allowed ranges from the prior distribution and converting the resulting distributions to Gaussians, the IBOL engine did find an optimized model, but it yielded a kappa of 0.004, i.e. a complete failure. \n##Looking for Supplier Information to Catch Outliers and Artefacts\nReviewing information about the source of the actual samples in [https://www.eu-openscreen.eu/services/compound-collection.html](https://www.eu-openscreen.eu/services/compound-collection.html) allow tagging the compounds by the source from which they were taken as the supplier for the non-Pilot structures could be found in the SDF files.\n\nBuilding separate models for the main sources Enamine, ChemDiv, ChemBridge, Pilot, and Others eventually improved the IBOL predictions by a factor of 22 (kappa=0.089), i.e. still unusable. Structural clustering by supplier seemed to collide with model building as different sources provided different kinds of structures with possibly also different artefacts in the measurement process. Particularly striking was the fact that only the Pilot dataset contained any acids.\n\n##Starting to Visualize the Training Data\nThe latter observation triggered further inspection of the training data statistics, and the different sources had very different percentages of partly and insoluble compounds. It was also apparent that compounds like quaternary ammonium salts, which by chemical intuition should be all soluble, showed unexpected data distributions: They were soluble when not from the Pilot set, but some were insoluble when found in the Pilot set. This unevenness by source was very likely confounding any structural model learning.\n\n##Guessing Plate Numbers and Row/Column Layout\nIn order to look deeper into the possible confounding factors, further information found in the challenge description was reviewed and pointed towards a way to split the data into plates with row and column IDs: The numeric part of the ID numbers was practically consecutive (except for a small number of drop-outs either for purity or for confidentiality reasons) with a gap before EOS100001. This sounded like sequential well assignment with a top-up set plated later. Using this hypothesis, plate numbers (0-286) and row/column numbers were inferred.\n\n##Visualizing the Data Distribution by Plate\nDoing this showed clear patterns of some plates containing mostly/completely soluble compounds and some much more insoluble ones. As this is a common artefact in screening.\n\nQuaternary ammonium salts are marked with orange circles and acids show as magenta circles. Plates 244 and 279 (red labels) did not contain any non-soluble compounds. The top-up set after EOS100001 had all the acids and also more diversity in logP. And there were plates with many insoluble compounds sprinkled through the set. If those \"anomalous\" plates would also have some structural coherence, then any structure-base modeling w/o correcting for plate artefacts would be futile.\n \n##Optimizing Plate/Row/Column Model with CI Descriptors\nGiven this batch structure in the training data, it seemed obvious to try to just use this structure to predict the solubility category. The model was a Gaussian distribution per compound with common variance adding one of 287 trainable offsets per plate and also row and column offsets to account for possible common measurement artefacts across the plates. An additional contribution to the means of the Gaussians was added as an optimized factor times the clogp of the compound. The Maximum Likelihood estimator of this model was then used to (kappa) optimize probability threshold for the actual assignments.\n\nThis rather crude model yielded the score of the ranked solution: kappa=0.19395 on the public test set and kappa=0.21748 on the private set.\n\n#Lessons Learned\n- Do not throw your favorite ML model at an arbitrary dataset and expect it to work.\n- Non-random draws from the structure universe can confuse modelling.\n- Compare what you know about the problem with the presentation of the data to spot (detrimental/exploitable) peculiarities. Visualization is key.\n- Finding the major sources of variance can give you a lot of mileage even if it does not help understanding the real problem.\n- As with many puzzles, there is more information than you think in the problem description.\n- Probabilistic models can be used to predict uneven classification. (Need to get the math right for IBOL models to prove this beyond the prior model, though.) The additional uncertainty model can even help in making decisions.\n- Having a competitor much ahead of you makes you think.",
"votes": 13
},
{
"id": 2102852,
"postDate": "2023-01-16T21:35:23.703Z",
"content": "I've been using the code of the notebook for the 2nd rank entry to score models of varying components. The table below shows a summary.
\n| Model Components | \nMAP Score | \nKappa Train | \nLower Limit | \nUpper Limit | \nPublic Score | \nPrivate Score | \n
|---|---|---|---|---|---|---|
| CLOGP | \n21277.22 | \n0.04334 | \n0.93043 | \n0.93246 | \n0.04192 | \n0.02923 | \n
| CMR | \n21258.17 | \n0.04985 | \n0.92308 | \n0.92713 | \n0.04192 | \n0.02923 | \n
| Plate | \n19608.21 | \n0.21072 | \n0.86857 | \n0.87046 | \n0.19448 | \n0.19768 | \n
| Plate, Row, Column | \n19475.75 | \n0.22713 | \n0.85684 | \n0.86396 | \n0.19252 | \n0.21781 | \n
| Plate, Row, Column, CLOGP | \n19441.63 | \n0.23196 | \n0.84700 | \n0.86459 | \n0.19565 | \n0.21890 | \n
| Plate, Row, Column, CMR | \n19401.01 | \n0.23478 | \n0.85036 | \n0.86205 | \n0.20293 | \n0.21502 | \n
| Plate, Row, Column, CLOGP, CMR | \n19391.69 | \n0.23760 | \n0.84755 | \n0.85799 | \n0.20307 | \n0.22241 | \n
| Plate, Row, Column, CLOGP, CMR, FCSP3 | \n19385.27 | \n0.23704 | \n0.84047 | \n0.86171 | \n0.20715 | \n0.22434 | \n
It is obvious that plate offsets provide the biggest improvement. The row/column modeling also ads some predictivity. The Cheminformatics descriptor CMR scores better than CLOGP on the public test set but worse on the private set. Using multiple descriptors further improves the model, but nowhere near the contribution from the artefact model components.
", "rawMarkdown": "### Scores for Kappa-Optimized Bayesian Models\n\nI've been using the code of the notebook for the 2nd rank entry to score models of varying components. The table below shows a summary.\n\nModel Components | MAP Score | Kappa Train | Lower Limit | Upper Limit | Public Score | Private Score\n---:|:---:| --- | --- | --- | --- | ---\n**CLOGP** | 21277.22 | 0.04334 | 0.93043 | 0.93246 | 0.04192 | 0.02923\n**CMR** | 21258.17 | 0.04985 | 0.92308 | 0.92713 | 0.04192 | 0.02923\n**Plate** | 19608.21 | 0.21072 | 0.86857 | 0.87046 | 0.19448 | 0.19768\n**Plate, Row, Column** | 19475.75 | 0.22713 | 0.85684 | 0.86396 | 0.19252 | 0.21781\n**Plate, Row, Column, CLOGP** | 19441.63 | 0.23196 | 0.84700 | 0.86459 | 0.19565 | 0.21890\n**Plate, Row, Column, CMR** | 19401.01 | 0.23478 | 0.85036 | 0.86205 | 0.20293 | 0.21502\n**Plate, Row, Column, CLOGP, CMR** | 19391.69 | 0.23760 | 0.84755 | 0.85799 | 0.20307 | 0.22241\n**Plate, Row, Column, CLOGP, CMR, FCSP3** | 19385.27 | 0.23704 | 0.84047 | 0.86171 | 0.20715 | 0.22434\n\nIt is obvious that plate offsets provide the biggest improvement. The row/column modeling also ads some predictivity. The Cheminformatics descriptor CMR scores better than CLOGP on the public test set but worse on the private set. Using multiple descriptors further improves the model, but nowhere near the contribution from the artefact model components.", "votes": 1, "replies": [ { "id": 2104396, "postDate": "2023-01-17T19:41:59.910Z", "content": "So I can understand why there are plate-wise artifacts, but do you know what's the source of the row-wise and column-wise artifacts?
", "rawMarkdown": "So I can understand why there are plate-wise artifacts, but do you know what's the source of the row-wise and column-wise artifacts?", "votes": 1, "replies": [ { "id": 2104506, "postDate": "2023-01-17T21:08:53.823Z", "content": "If you browse through the notebook 'Plate/Row/Column Model with CLogP for Solubility' with the code for my solution, there are plots that show the size of plate and row/column offsets and you can see that those row/column offsets are much smaller. However, they are to be expected in high throughput screening especially if basically single point data are collected.
\nTypical causes are differences in the buffer dispensing tips (the same ones are used across one of the dimensions) as well as solvent evaporation during incubation and other processing steps, which are larger at the edges than in the middle of the plates. Plate readers also might have such biases. Putting controls on the first and last column and not randomly on the plate is technically a lot easier for HTS, but catching row/column biases per plate is difficult this way.
\nIn the comments of the notebook, there is also a table that illustrates the change in explanatory power if the model variables are added sequentially.
\nUpdate: I forgot that the table was also posted here.
", "rawMarkdown": "If you browse through the notebook 'Plate/Row/Column Model with CLogP for Solubility' with the code for my solution, there are plots that show the size of plate and row/column offsets and you can see that those row/column offsets are much smaller. However, they are to be expected in high throughput screening especially if basically single point data are collected.\n\nTypical causes are differences in the buffer dispensing tips (the same ones are used across one of the dimensions) as well as solvent evaporation during incubation and other processing steps, which are larger at the edges than in the middle of the plates. Plate readers also might have such biases. Putting controls on the first and last column and not randomly on the plate is technically a lot easier for HTS, but catching row/column biases per plate is difficult this way.\n\nIn the comments of the notebook, there is also a table that illustrates the change in explanatory power if the model variables are added sequentially.\n\nUpdate: I forgot that the table was also posted here.", "votes": 1 }, { "id": 2104543, "postDate": "2023-01-17T21:53:07.203Z", "content": "As to the plate-wise artefacts, there might be clustering of similar structures on plates e.g. from synthesis series, which should not affect the predictivity of structure-based models, but plates do have their handling history (compound source, storage time, dust, temperature, dispensing equipment, freezing/thawing for cherry picking, …) which can affect the full plate, and this is what I expect to be the major issue here. Having the raw readouts or even HPLC/MS QC might allow spotting this with further analysis.
\nUpdate: In combination with intra-plate clustering, experimental plate artefacts could also simulate structure dependence.
", "rawMarkdown": "As to the plate-wise artefacts, there might be clustering of similar structures on plates e.g. from synthesis series, which should not affect the predictivity of structure-based models, but plates do have their handling history (compound source, storage time, dust, temperature, dispensing equipment, freezing/thawing for cherry picking, ...) which can affect the full plate, and this is what I expect to be the major issue here. Having the raw readouts or even HPLC/MS QC might allow spotting this with further analysis.\n\nUpdate: In combination with intra-plate clustering, experimental plate artefacts could also *simulate* structure dependence.", "votes": 2 } ] } ] }, { "id": 2101924, "postDate": "2023-01-16T09:53:26.570Z", "content": "Added a functional (cleaned-up) notebook that implements the algorithm used. The notebook includes some tracing output, but be aware that rerunning it might take a few hours.
", "rawMarkdown": "Added a functional (cleaned-up) [notebook](https://www.kaggle.com/code/bernhardrohde/plate-row-column-model-with-clogp-for-solubility) that implements the algorithm used. The notebook includes some tracing output, but be aware that rerunning it might take a few hours.", "votes": 1, "replies": [ { "id": 2101927, "postDate": "2023-01-16T09:56:14.103Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2101929, "postDate": "2023-01-16T09:57:30.010Z", "content": "@bernhardrohde - Is the notebook set to public viewable - I can't see it? Also, you should be able to rename it using the box at the top left of the editor screen.
", "rawMarkdown": "@bernhardrohde - Is the notebook set to public viewable - I can't see it? Also, you should be able to rename it using the box at the top left of the editor screen.", "replies": [ { "id": 2101969, "postDate": "2023-01-16T10:21:48.853Z", "content": "I found the textbox myself and changed it. This did also changed the URL, which I've already updated in the post. Looks like a race condition ;-)
", "rawMarkdown": "I found the textbox myself and changed it. This did also changed the URL, which I've already updated in the post. Looks like a race condition ;-)", "votes": 1 } ] } ] }, { "id": 2094186, "postDate": "2023-01-10T16:45:09.830Z", "content": "Well done @bernhardrohde - hacking the plate structure like that is very clever and well within the spirit of Kaggle competitions. From the perspective of solubility prediction, I guess it makes the point that solubilities of structurally very similar compounds are particularly valuable information, beyond just their contribution to a global ML model. Possibly something like a simple kNN model of the solubility, while typically only a moderately good predictor in itself, might prove a useful feature in a more sophisticated model?
", "rawMarkdown": "Well done @bernhardrohde - hacking the plate structure like that is very clever and well within the spirit of Kaggle competitions. From the perspective of solubility prediction, I guess it makes the point that solubilities of structurally very similar compounds are particularly valuable information, beyond just their contribution to a global ML model. Possibly something like a simple kNN model of the solubility, while typically only a moderately good predictor in itself, might prove a useful feature in a more sophisticated model?", "votes": 1, "replies": [ { "id": 2094577, "postDate": "2023-01-10T21:00:03.257Z", "content": "This is the idea of my IBOL models I'm planning to apply to the dataset. Reaching the second place with just modeling the plat/row/column artefacts shows that it would be risky to simply do everything with a structure-based, even a kNN, model as structurally near neighbors might be on different plates with, thus, differing artefacts.
\nMy current plan is to use the IBOL (Instance-Based Offset Learning) models, while simultaneously adjusting for the artefacts. I tried this already, but the math wasn't right and I got close but failed to beat my simplistic model. I think I have an idea for the math now that should work, but didn't get to implement and check it yet. Once that works, I think the method can be used to adjust for many different kinds of batch effects like combining results from different assays/publications.
\nThe IBOL approach actually provides probability distributions for the predicted value based on the neighbor data distributions, and those neighbor distributions can even be truncated/censored or ranges. This is close to Gaussian Processes, but computationally feasible even for such large datasets as this one. It also yields parameters that are better to understand. I've been using this method to support Bayesian Optimization even for multi-objective problems optimizing the Pareto Front simulating the medchem workflow. Unfortunately, I'm in the habit of procrastinating when it comes to publishing as I always have ideas for improving the method.
", "rawMarkdown": "This is the idea of my IBOL models I'm planning to apply to the dataset. Reaching the second place with just modeling the plat/row/column artefacts shows that it would be risky to simply do everything with a structure-based, even a kNN, model as structurally near neighbors might be on different plates with, thus, differing artefacts.\n\nMy current plan is to use the IBOL (Instance-Based Offset Learning) models, while simultaneously adjusting for the artefacts. I tried this already, but the math wasn't right and I got close but failed to beat my simplistic model. I think I have an idea for the math now that should work, but didn't get to implement and check it yet. Once that works, I think the method can be used to adjust for many different kinds of batch effects like combining results from different assays/publications.\n\nThe IBOL approach actually provides **probability distributions** for the predicted value based on the neighbor data distributions, and those neighbor distributions can even be truncated/censored or ranges. This is close to Gaussian Processes, but computationally feasible even for such large datasets as this one. It also yields parameters that are better to understand. I've been using this method to support *Bayesian Optimization* even for multi-objective problems optimizing the Pareto Front simulating the medchem workflow. Unfortunately, I'm in the habit of procrastinating when it comes to publishing as I always have ideas for improving the method.", "votes": 2 } ] }, { "id": 2143791, "postDate": "2023-02-14T14:06:08.863Z", "content": "I'm finding this ongoing discussion really interesting, albeit catalysed by some non-ideal circumstances. From my perspective, it is clear that:
\nThe test data are organised such that soluble and insoluble compounds are distributed in a highly non-random fashion, see this discussion by @yumizsui, as well @bernhardrohde's solution.
Bernhard has identified a very plausible link between solubility and plate, which in fact follows logically if compounds are typically put in plates grouped by the given label numbers. Noted also that @davidoloren found an optimal moving window close to 400.
Bernhard has also suggested patterns around (row, column) positions on the plate, which are probably weaker signals and less well established than the previous two points.
Both the top two placed models in the private LB utilised label information as a feature. In Bernhard's case at least this was the major input into his model; for the first placed team of @yumizsui, @tonets and @appaeak this was unintentional.
The organisers @robertharmel and @agiani99 have not as yet accepted the first place solution as the winner and have indicated that Bernhard's solution is also unacceptable.
In a parallel universe somewhere, the labels and ordering of the test set were randomised and we are not having this discussion.
The competion involved \"predicting\" (temporally post hoc, but unseen) the results of a high-throughput kinetic solubility assay (mostly one datum per compound). These were very noisy, as I think is acknowledged by all, and the correlations with traditional cheminformatics descriptors were much weaker than I am used to in computing \"predictions\" for low-throughput (typically) multi-replicate equilibrium intrinsic solubilities. The difference is around an order of magnitude. Given how weak the other features were, any model using it would indeed benefit massively from the label number. This is an unusual (for me) situation where one descriptor dominates so strongly.
\nMy major point is that I don't believe that utilising the label as a feature is cheating. I've done plenty of solubility prediction tasks where the label is a chemical name, and am engaged in one now where the label is (for technical reasons) a SMILES. These things truly carry chemical information and I see nothing unethical about leveraging that.
\nIf the label turns out to indicate the source of the compound, this reflects genuine knowledge that would be available in many (though not all) real-world contexts. If the team doing the solubility assay know that \"Acme Chemical Co. only make brickdust\", \"Our Organic synthesis team are considerate in designing solubility into all their projects\", \"Nothing with that blue-and-yellow label is ever soluble\", or \"Compounds from Chemical Store 2 often dissolve well\", then each of these things is useful knowledge that may genuinely aid both prediction and choosing directions to go in order to enhance solubility. Soluble molecules are unlikely to be most efficiently found by randomly sampling libraries. Indeed, we ourselves found that human experts are just as good as Machine Learning at predicting solubility. Such community wisdom is extremely valuable.
\nI look at the leaderboard and it's very obvious that @yumizsui, @tonets and @appaeak won, and that @bernhardrohde was a strong second. I would urge the organisers to award the prize to the top-placed team, while also giving acknowledgement to whatever entry turns out to be the best based solely on molecular features (\"molecular\" is intended also to include any solid, liquid or solution phase ones, if relevant).
\nFinally, I can't remember any solubility prediction competition that hasn't had some potential issues related to dataset design, data accuracy, or ease-of-cheating. Organising and running such a competition is a hard thing to do, and hindsight is unavailable in advance. I make no criticism of the organisers, but we are where we are.
", "rawMarkdown": "I'm finding this ongoing discussion really interesting, albeit catalysed by some non-ideal circumstances. From my perspective, it is clear that:\n\n- The test data are organised such that soluble and insoluble compounds are distributed in a highly non-random fashion, see [this discussion](https://www.kaggle.com/competitions/euos-slas/discussion/377428) by @yumizsui, as well @bernhardrohde's solution.\n\n- Bernhard has identified a very plausible link between solubility and plate, which in fact follows logically if compounds are typically put in plates grouped by the given label numbers. Noted also that @davidoloren found an optimal moving window close to 400.\n\n- Bernhard has also suggested patterns around (row, column) positions on the plate, which are probably weaker signals and less well established than the previous two points.\n\n- Both the top two placed models in the private LB utilised label information as a feature. In Bernhard's case at least this was the major input into his model; for the first placed team of @yumizsui, @tonets and @appaeak this was unintentional.\n\n- The organisers @robertharmel and @agiani99 have not as yet accepted the first place solution as the winner and have indicated that Bernhard's solution is also unacceptable.\n\n- In a parallel universe somewhere, the labels and ordering of the test set were randomised and we are not having this discussion.\n\nThe competion involved \"predicting\" (temporally post hoc, but unseen) the results of a high-throughput kinetic solubility assay (mostly one datum per compound). These were very noisy, as I think is acknowledged by all, and the correlations with traditional cheminformatics descriptors were much weaker than I am used to in computing \"predictions\" for low-throughput (typically) multi-replicate equilibrium intrinsic solubilities. The difference is around an order of magnitude. Given how weak the other features were, any model using it would indeed benefit massively from the label number. This is an unusual (for me) situation where one descriptor dominates so strongly.\n\nMy major point is that I don't believe that utilising the label as a feature is cheating. I've done plenty of solubility prediction tasks where the label is a chemical name, and am engaged in one now where the label is (for technical reasons) a SMILES. These things truly carry chemical information and I see nothing unethical about leveraging that. \n\nIf the label turns out to indicate the source of the compound, this reflects genuine knowledge that would be available in many (though not all) real-world contexts. If the team doing the solubility assay know that \"Acme Chemical Co. only make brickdust\", \"Our Organic synthesis team are considerate in designing solubility into all their projects\", \"Nothing with that blue-and-yellow label is ever soluble\", or \"Compounds from Chemical Store 2 often dissolve well\", then each of these things is useful knowledge that may genuinely aid both prediction and choosing directions to go in order to enhance solubility. Soluble molecules are unlikely to be most efficiently found by randomly sampling libraries. Indeed, we ourselves found that [human experts are just as good as Machine Learning at predicting solubility](https://jcheminf.biomedcentral.com/articles/10.1186/s13321-017-0250-y). Such community wisdom is extremely valuable.\n\nI look at the leaderboard and it's very obvious that @yumizsui, @tonets and @appaeak won, and that @bernhardrohde was a strong second. I would urge the organisers to award the prize to the top-placed team, while also giving acknowledgement to whatever entry turns out to be the best based solely on molecular features (\"molecular\" is intended also to include any solid, liquid or solution phase ones, if relevant).\n\nFinally, I can't remember any solubility prediction competition that hasn't had some potential issues related to dataset design, data accuracy, or ease-of-cheating. Organising and running such a competition is a hard thing to do, and hindsight is unavailable in advance. I make no criticism of the organisers, but we are where we are.", "votes": 2 }, { "id": 2143339, "postDate": "2023-02-14T07:52:09.930Z", "content": "I've created another visualization of the guessed plate layout of the dataset. This is a pretty big image, but you should be able to zoom into it. The frame borders are colored by compound source with the final seven plates from the topup set with IDs >= EOS100001.
\nThe blueish color indicates test set compounds for which there is no sol_category.
The red spots are 'low' and the orange ones are 'medium'. The characters in the cells indicate structural features, e.g. 'Q' for quaternary amine' and 'A' for acids. You can easily see that the topup set is structurally different from the other plates.
", "rawMarkdown": "I've created another visualization of the guessed plate layout of the dataset. This is a pretty big image, but you should be able to zoom into it. The frame borders are colored by compound source with the final seven plates from the topup set with IDs >= EOS100001.\nThe blueish color indicates test set compounds for which there is no sol_category.\n\nThe red spots are 'low' and the orange ones are 'medium'. The characters in the cells indicate structural features, e.g. 'Q' for quaternary amine' and 'A' for acids. You can easily see that the topup set is structurally different from the other plates.", "replies": [ { "id": 2143350, "postDate": "2023-02-14T07:55:54.410Z", "content": "An additional picture shows the average frequency of 'low', 'medium', and hit ('low' or 'medium') compounds by well coordinates. The first row is for the base set and the second shows the topup set. There isn't much intelligible structure except for some edge effects.
", "rawMarkdown": "An additional picture shows the average frequency of 'low', 'medium', and hit ('low' or 'medium') compounds by well coordinates. The first row is for the base set and the second shows the topup set. There isn't much intelligible structure except for some edge effects. " } ] }, { "id": 2137810, "postDate": "2023-02-10T10:48:04.273Z", "content": "Hi Bernhard,
\nI read with high interest your notebook about possible plate effects, which we anyway did not see experimentally. I thank you for the depth of your approach, but I suspect there might be something to be put back in right order.
\nPlate effects can be always present when we are managing thousands of compounds within few tens of 384-well plates. Even if we remove two or more columns for control the remaining 16row x 22columns plate can be filled by systematic errors by robots generating a pattern. You claimed you found one or more using a guess on locations of compounds (row and column). I really doubt this is real as we did not provide this information.
\nThe guess is still legit until proofed wrong, I think. Anyway, how can a statistical model be used to predict external molecules which don't have this information? This is why your model is for us unacceptable, as it does not root into the chemo-physical nature of the solubility experiment, which we think it is still highly structure-dependent.
\nOn the other hand, I personally have doubts any pattern eventually found can be real, without information on number and position of control columns used, without knowing whether plate filling has been done by row or by column, without knowing whether plate have been filled completely or only, say, 90%. I don't want to underestimate your work, which definitively sets a bar for all future chemoinformatic Kaggle competitions on pharma relevant datasets.
\nI hope that this comments will help critically novel Kaggle users/teams when dealing with this data domain.
\nBest
\nAndrea
Hi @agiani99,
\nLet me go though your comments in turn:
\n\nPlate effects can be always present when we are managing thousands of compounds within few tens of 384-well plates. Even if we remove two or more columns for control the remaining 16row x 22columns plate can be filled by systematic errors by robots generating a pattern. You claimed you found one or more using a guess on locations of compounds (row and column). I really doubt this is real as we did not provide this information.
\n
Agreed. The plate effects my Bayesian model detected were much stronger than the row/column model I added on top to get a better score (See the table above that was generated post deadline). The plating I assumed looked convincing to me as you can see in the picture above e.g. for plates 194, 197, 199 (more than average low compounds), and 244, 279 (less than average low compounds), since the neighboring 'plates' (read: ID stretches) are quite different. Knowing this for sure would of course be better. I also redid the row/column model recently, and it appears that only in the first compound set (before EOS100001) actually has a significant edge effect on one of the row and one of the column edges. Assuming an additive row/column offset seemed to be a little to flexible. I do, however, still think the sequence of compounds on the plates is what the statistics told me. The raw data could prove me wrong, however ;-).
\n\n\nThe guess is still legit until proofed wrong, I think. Anyway, how can a statistical model be used to predict external molecules which don't have this information? This is why your model is for us unacceptable, as it does not root into the chemo-physical nature of the solubility experiment, which we think it is still highly structure-dependent.
\n
What the model actually does is correcting the means of the predicted solubility distributions for the plate effects in a linear model. This can be interpreted as adding the plate model to the real structure-based model (just using clogp in my case) that would govern the data generation if the plate effect wasn't there. It would be better to estimate this from the raw data, but what I've done shows that there is some structure based variance, but not enough to compete with the plate effects. So structure-based prediction (just ignoring the artefact coefficients of the linear model) is possible, but not very useful given the current dataset.
\nAnother (Bayesian) point I raised above has to do with the different distributions for the first compound set and what I called the topup set. They are structurally so different that the only conclusions that can be drawn are specific to the two subsets. This actually also shows by the fact that the Kappa score (training is all I have here) of the two subsets are both higher (~0.21 for the first set and ~0.41 for the topup set) than the training score for the combined dataset.
\nSo any prediction depends on whether the compounds to be predicted were selected like those in the first set or the topup set. Having only classes instead of actual solubilities as in AID1996 also provides a lot less information (I think about 0.3 bits) per compound. The confounding plate effect will thus be even more important.
\n\n\nOn the other hand, I personally have doubts any pattern eventually found can be real, without information on number and position of control columns used, without knowing whether plate filling has been done by row or by column, without knowing whether plate have been filled completely or only, say, 90%. I don't want to underestimate your work, which definitively sets a bar for all future chemoinformatic Kaggle competitions on pharma relevant datasets.
\n
This was actually the real point of my competition contribution and of requesting the complete information for the published dataset once it becomes available.
\nOne final remark, I do read your comment as an indication that there will be no winner and thus no possible discussion at the SLAS Meeting. If this is the case, then there wouldn't be a point in me going there on my own expenses. If, however, the contributors, data producers, and judges would be present, I would still go there to have a scientific exchange.
\nCheers,
\nBernd
", "rawMarkdown": "Hi @agiani99,\nLet me go though your comments in turn:\n\n> Plate effects can be always present when we are managing thousands of compounds within few tens of 384-well plates. Even if we remove two or more columns for control the remaining 16row x 22columns plate can be filled by systematic errors by robots generating a pattern. You claimed you found one or more using a guess on locations of compounds (row and column). I really doubt this is real as we did not provide this information.\n\nAgreed. The plate effects my Bayesian model detected were *much* stronger than the row/column model I added on top to get a better score (See the table above that was generated post deadline). The plating I assumed looked convincing to me as you can see in the picture above e.g. for plates 194, 197, 199 (more than average low compounds), and 244, 279 (less than average low compounds), since the neighboring 'plates' (read: ID stretches) are quite different. Knowing this for sure would of course be better. I also redid the row/column model recently, and it appears that only in the first compound set (before EOS100001) actually has a significant edge effect on one of the row and one of the column edges. Assuming an additive row/column offset seemed to be a little to flexible. I do, however, still think the sequence of compounds on the plates is what the statistics told me. The raw data could prove me wrong, however ;-).\n\n> The guess is still legit until proofed wrong, I think. Anyway, how can a statistical model be used to predict external molecules which don't have this information? This is why your model is for us unacceptable, as it does not root into the chemo-physical nature of the solubility experiment, which we think it is still highly structure-dependent.\n\nWhat the model actually does is *correcting* the means of the predicted solubility distributions for the plate effects in a **linear** model. This can be interpreted as adding the plate model to the real structure-based model (just using clogp in my case) that would govern the data generation if the plate effect wasn't there. It would be better to estimate this from the raw data, but what I've done shows that there is some structure based variance, but not enough to compete with the plate effects. So structure-based prediction (just ignoring the artefact coefficients of the **linear** model) is possible, but not very useful given the current dataset.\n\nAnother (Bayesian) point I raised above has to do with the different distributions for the first compound set and what I called the topup set. They are structurally so different that the only conclusions that can be drawn are specific to the two subsets. This actually also shows by the fact that the Kappa score (training is all I have here) of the two subsets are both higher (~0.21 for the first set and ~0.41 for the topup set) than the training score for the combined dataset.\n\nSo any prediction depends on whether the compounds to be predicted were selected like those in the first set or the topup set. Having only classes instead of actual solubilities as in AID1996 also provides a lot less information (I think about 0.3 bits) per compound. The confounding plate effect will thus be even more important.\n\n> On the other hand, I personally have doubts any pattern eventually found can be real, without information on number and position of control columns used, without knowing whether plate filling has been done by row or by column, without knowing whether plate have been filled completely or only, say, 90%. I don't want to underestimate your work, which definitively sets a bar for all future chemoinformatic Kaggle competitions on pharma relevant datasets.\n\nThis was actually the real point of my competition contribution and of requesting the complete information for the published dataset once it becomes available.\n\nOne final remark, I do read your comment as an indication that there will be no winner and thus no possible discussion at the SLAS Meeting. If this is the case, then there wouldn't be a point in me going there on my own expenses. If, however, the contributors, data producers, and judges would be present, I would still go there to have a scientific exchange.\n\nCheers,\n\nBernd", "votes": 1 } ] }, { "id": 2093133, "postDate": "2023-01-09T20:27:09.070Z", "rawMarkdown": "", "isDeleted": true } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T16:45:01Z", "competition_id": 38158, "competition_slug": "euos-slas", "team_id": 9185735, "solution_url": "/competitions/euos-slas/writeups/bernhard-rohde-2nd-place-solution-draft" }