First of alll, thanks to the Competition hosts for this exciting challenge. Also thanks for all the interesting dicussions and findings from various researchers.
\nModel workflow
\nThe development of our best-scoring model, the calculation and selection of descriptors and final consensus model were all performed using the OCHEM platform[1] https://ochem.eu. In order to be in line with challenge requirements (model publicly available), we exclusively used open-source methods and descriptors from https://github.com/openochem and a fully-public version of the model will be made as docker (we currently work on it – either it will be exported from OCHEM or reproduced using openOCHEM). An overview of the process is described below.
\nThe process workflow is as follows:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F12202906%2Fb293eefc84cd0ef6177c33e5995766c7%2Fsolution_table1.png?generation=1677561719492738&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F12202906%2F42ee5aded432bf8ab4e5805c9d333450%2Fsolution_table2.png?generation=1677561739919995&alt=media\")
The default configuration of parameters/hyper-parameters as specified on the OCHEM web site were used for all models and descriptors. A simple average of the 28 models was used to produce a consensus model https://ochem.eu/model/1041. If required, the consensus model and its individual models can be exported and be made publicly available as part of the openOCHEM.
\nThe average value calculated from the consensus model was considered as a probability to predict the target classes. The predictions of the consensus model of the test set compounds were downloaded (with all default values).
\nWe assumed that the proportions of molecules with low, medium and high solubilities in the test sets were identical to the proportions in the training set. This assumption was based on a priori information of the challenge organisers, who mentioned that data for the test were randomly sampled. Therefore, we sorted all molecules according to their predicted solubility scores and assigned class zero to 2041/70170 = 2.9% compounds with lowest scores in the test set and class one to 4% compounds with second set of lowest scores. Because of rounding (OCHEM rounds prediction to 4 digits), several compounds had exactly the same prediction score, and the actual numbers of selected compounds with class zero and class one were 892 and 1268, respectively.
\nData shuffling
\nFollowing request of organisers, we re-uploaded datafile with test set compounds, which order was shuffled. Results calculated for original and shuffled datasets were practical identical with an exception of the rounding errors (the last significant digits). Because of these difference the order of molecules for the ranking procedure was slightly different one and predicted solubility values where changed for 86 molecules as compared to the initial submission. This difference minorly affected the kappa statistics, which changed from 0.14731 and 0.11614 (challenge submission) to 0.1495 and 0.11769 for public and private subset, respectively. The differences could be due to change of CDK (2.7.1 to 2.8) instability of algorithms for descriptor calculations or instability of prediction of deep learning algorithms (we did not investigate it deeply). The factors that contribute to it could be calculation on different computers (we used cluster with different Intel CPUs) and/or bugs in the respective software programs.
\nModel availability
\nThe models are available as https://ochem.eu/article/156753 as well as at https://slas.ochem.eu/article/156753 (dockerised version). The molecules are not available to the users due to SLAS policy. The dockerised version can be installed locally and outside of OCHEM as described in the enclosed instruction. The model files will be provided to the organises on the request.
\nReferences
\n[1] Sushko, I.; Novotarskyi, S.; Körner, R.; Pandey, A. K.; Rupp, M.; Teetz, W.; Brandmaier, S.; Abdelaziz, A.; Prokopenko, V. V.; Tanchuk, V. Y.; Todeschini, R.; Varnek, A.; Marcou, G.; Ertl, P.; Potemkin, V.; Grishina, M.; Gasteiger, J.; Schwab, C.; Baskin, I. I.; Palyulin, V. A.; Radchenko, E. V.; Welsh, W. J.; Kholodovych, V.; Chekmarev, D.; Cherkasov, A.; Aires-de-Sousa, J.; Zhang, Q.-Y.; Bender, A.; Nigsch, F.; Patiny, L.; Williams, A.; Tkachenko, V.; Tetko, I. V. Online Chemical Modeling Environment (OCHEM): Web Platform for Data Storage, Model Development and Publishing of Chemical Information. J. Comput. Aided Mol. Des. 2011, 25 (6), 533–554. https://doi.org/10.1007/s10822-011-9440-2.
\n[2] Willighagen, E. L.; Mayfield, J. W.; Alvarsson, J.; Berg, A.; Carlsson, L.; Jeliazkova, N.; Kuhn, S.; Pluskal, T.; Rojas-Chertó, M.; Spjuth, O.; Torrance, G.; Evelo, C. T.; Guha, R.; Steinbeck, C. The Chemistry Development Kit (CDK) v2.0: Atom Typing, Depiction, Molecular Formulas, and Substructure Searching. J. Cheminformatics 2017, 9 (1), 33. https://doi.org/10.1186/s13321-017-0220-4.
\n[3] Sosnin, S.; Karlov, D.; Tetko, I. V.; Fedorov, M. V. Comparative Study of Multitask Toxicity Modeling on a Broad Chemical Space. J. Chem. Inf. Model. 2019, 59 (3), 1062–1072. https://doi.org/10.1021/acs.jcim.8b00685.
\n[4] Prokhorenkova, L.; Gusev, G.; Vorobev, A.; Veronika Dorogush, A.; Gulin, A. CatBoost: Unbiased Boosting with Categorical Features. ArXiv E-Prints 2017, arXiv:1706.09516. https://doi.org/10.48550/arXiv.1706.09516.
\n[5] Reiser, P.; Eberhard, A.; Friederich, P. Graph Neural Networks in TensorFlow-Keras with RaggedTensor Representation (Kgcnn). Softw. Impacts 2021, 9, 100095. https://doi.org/10.1016/j.simpa.2021.100095.
\n[6] Karpov, P.; Godin, G.; Tetko, I. V. Transformer-CNN: Swiss Knife for QSAR Modeling and Interpretation. J. Cheminformatics 2020, 12 (1), 17. https://doi.org/10.1186/s13321-020-00423-w.
First of alll, thanks to the Competition hosts for this exciting challenge. Also thanks for all the interesting dicussions and findings from various researchers.
\nModel workflow
\nThe development of our best-scoring model, the calculation and selection of descriptors and final consensus model were all performed using the OCHEM platform[1] https://ochem.eu. In order to be in line with challenge requirements (model publicly available), we exclusively used open-source methods and descriptors from https://github.com/openochem and a fully-public version of the model will be made as docker (we currently work on it – either it will be exported from OCHEM or reproduced using openOCHEM). An overview of the process is described below.
\nThe process workflow is as follows:
\nThe default configuration of parameters/hyper-parameters as specified on the OCHEM web site were used for all models and descriptors. A simple average of the 28 models was used to produce a consensus model https://ochem.eu/model/1041. If required, the consensus model and its individual models can be exported and be made publicly available as part of the openOCHEM.
\nThe average value calculated from the consensus model was considered as a probability to predict the target classes. The predictions of the consensus model of the test set compounds were downloaded (with all default values).
\nWe assumed that the proportions of molecules with low, medium and high solubilities in the test sets were identical to the proportions in the training set. This assumption was based on a priori information of the challenge organisers, who mentioned that data for the test were randomly sampled. Therefore, we sorted all molecules according to their predicted solubility scores and assigned class zero to 2041/70170 = 2.9% compounds with lowest scores in the test set and class one to 4% compounds with second set of lowest scores. Because of rounding (OCHEM rounds prediction to 4 digits), several compounds had exactly the same prediction score, and the actual numbers of selected compounds with class zero and class one were 892 and 1268, respectively.
\nData shuffling
\nFollowing request of organisers, we re-uploaded datafile with test set compounds, which order was shuffled. Results calculated for original and shuffled datasets were practical identical with an exception of the rounding errors (the last significant digits). Because of these difference the order of molecules for the ranking procedure was slightly different one and predicted solubility values where changed for 86 molecules as compared to the initial submission. This difference minorly affected the kappa statistics, which changed from 0.14731 and 0.11614 (challenge submission) to 0.1495 and 0.11769 for public and private subset, respectively. The differences could be due to change of CDK (2.7.1 to 2.8) instability of algorithms for descriptor calculations or instability of prediction of deep learning algorithms (we did not investigate it deeply). The factors that contribute to it could be calculation on different computers (we used cluster with different Intel CPUs) and/or bugs in the respective software programs.
\nModel availability
\nThe models are available as https://ochem.eu/article/156753 as well as at https://slas.ochem.eu/article/156753 (dockerised version). The molecules are not available to the users due to SLAS policy. The dockerised version can be installed locally and outside of OCHEM as described in the enclosed instruction. The model files will be provided to the organises on the request.
\nReferences
\n[1] Sushko, I.; Novotarskyi, S.; Körner, R.; Pandey, A. K.; Rupp, M.; Teetz, W.; Brandmaier, S.; Abdelaziz, A.; Prokopenko, V. V.; Tanchuk, V. Y.; Todeschini, R.; Varnek, A.; Marcou, G.; Ertl, P.; Potemkin, V.; Grishina, M.; Gasteiger, J.; Schwab, C.; Baskin, I. I.; Palyulin, V. A.; Radchenko, E. V.; Welsh, W. J.; Kholodovych, V.; Chekmarev, D.; Cherkasov, A.; Aires-de-Sousa, J.; Zhang, Q.-Y.; Bender, A.; Nigsch, F.; Patiny, L.; Williams, A.; Tkachenko, V.; Tetko, I. V. Online Chemical Modeling Environment (OCHEM): Web Platform for Data Storage, Model Development and Publishing of Chemical Information. J. Comput. Aided Mol. Des. 2011, 25 (6), 533–554. https://doi.org/10.1007/s10822-011-9440-2.
\n[2] Willighagen, E. L.; Mayfield, J. W.; Alvarsson, J.; Berg, A.; Carlsson, L.; Jeliazkova, N.; Kuhn, S.; Pluskal, T.; Rojas-Chertó, M.; Spjuth, O.; Torrance, G.; Evelo, C. T.; Guha, R.; Steinbeck, C. The Chemistry Development Kit (CDK) v2.0: Atom Typing, Depiction, Molecular Formulas, and Substructure Searching. J. Cheminformatics 2017, 9 (1), 33. https://doi.org/10.1186/s13321-017-0220-4.
\n[3] Sosnin, S.; Karlov, D.; Tetko, I. V.; Fedorov, M. V. Comparative Study of Multitask Toxicity Modeling on a Broad Chemical Space. J. Chem. Inf. Model. 2019, 59 (3), 1062–1072. https://doi.org/10.1021/acs.jcim.8b00685.
\n[4] Prokhorenkova, L.; Gusev, G.; Vorobev, A.; Veronika Dorogush, A.; Gulin, A. CatBoost: Unbiased Boosting with Categorical Features. ArXiv E-Prints 2017, arXiv:1706.09516. https://doi.org/10.48550/arXiv.1706.09516.
\n[5] Reiser, P.; Eberhard, A.; Friederich, P. Graph Neural Networks in TensorFlow-Keras with RaggedTensor Representation (Kgcnn). Softw. Impacts 2021, 9, 100095. https://doi.org/10.1016/j.simpa.2021.100095.
\n[6] Karpov, P.; Godin, G.; Tetko, I. V. Transformer-CNN: Swiss Knife for QSAR Modeling and Interpretation. J. Cheminformatics 2020, 12 (1), 17. https://doi.org/10.1186/s13321-020-00423-w.