{
  "id": 94941,
  "title": "Is it worth training separate models for each type?",
  "url": "/competitions/champs-scalar-coupling/discussion/94941",
  "author_name": "",
  "post_date": "2019-06-08T11:29:31.927152200Z",
  "votes": 45,
  "comment_count": 26,
  "views": 0,
  "content": "<p><code>Type</code> obviously is important feature judging both from common sense and from models. I have already seen at least one <a href=\"https://www.kaggle.com/kirichenko17roman/simple-lightgbm-in-r\">kernel</a> where author tried to train separate models for each type.</p>\n\n<p>So today I also trained separate models in my <a href=\"https://www.kaggle.com/artgor/brute-force-feature-engineering\">kernel</a></p>\n\n<p>Here is a plot of OOF predictions vs target for two types.\n<img src=\"https://i.imgur.com/W4TWyQf.jpg\" alt=\"\"></p>\n\n<p>We can see that predictions are more robust, the score is also better.</p>\n\n<p>So what do you think: is this a good idea or could it lead to overfitting and other problems?</p>",
  "messages": [
    {
      "id": "547847",
      "postDate": "06/08/2019 11:29:31",
      "content": "<p><code>Type</code> obviously is important feature judging both from common sense and from models. I have already seen at least one <a href=\"https://www.kaggle.com/kirichenko17roman/simple-lightgbm-in-r\">kernel</a> where author tried to train separate models for each type.</p>\n\n<p>So today I also trained separate models in my <a href=\"https://www.kaggle.com/artgor/brute-force-feature-engineering\">kernel</a></p>\n\n<p>Here is a plot of OOF predictions vs target for two types.\n<img src=\"https://i.imgur.com/W4TWyQf.jpg\" alt=\"\"></p>\n\n<p>We can see that predictions are more robust, the score is also better.</p>\n\n<p>So what do you think: is this a good idea or could it lead to overfitting and other problems?</p>",
      "rawMarkdown": "`Type` obviously is important feature judging both from common sense and from models. I have already seen at least one [kernel](https://www.kaggle.com/kirichenko17roman/simple-lightgbm-in-r) where author tried to train separate models for each type.\n\nSo today I also trained separate models in my [kernel](https://www.kaggle.com/artgor/brute-force-feature-engineering)\n\nHere is a plot of OOF predictions vs target for two types.\n![](https://i.imgur.com/W4TWyQf.jpg)\n\nWe can see that predictions are more robust, the score is also better.\n\nSo what do you think: is this a good idea or could it lead to overfitting and other problems?",
      "votes": null
    },
    {
      "id": "547912",
      "postDate": "06/08/2019 14:01:03",
      "content": "<p>I think is a good idea.</p>",
      "rawMarkdown": "I think is a good idea.",
      "votes": null
    },
    {
      "id": "547968",
      "postDate": "06/08/2019 15:15:56",
      "content": "<p>I have been exploring this also and have a similar discussion thread.</p>\n\n<p>I agree with your conclusion, that it probably is a good idea. The reason that it might be worth further investigation , is that 15% of the types account for 50% of the score, according to how the group MAE metric works. Therefore, we might sub-optimize the model for the smaller 4 types and have a 'good enough' simpler model for the other 4. There is probably a small risk of overfit, but the training and test sets seem to exhibit very similar profiles. You get great leverage by concentrating effort on the smaller segments.</p>\n\n<p>['3JHC', '2JHC', '1JHC', '3JHH', '2JHH', '3JHN', '2JHN', '1JHN']\n[1510379, 1140674, 709416, 590611, 378036, 166415, 119253, 43363]</p>\n\n<p>the 1JHN type is less than 1% of the data, but contributes 12.5% of the score.</p>",
      "rawMarkdown": "I have been exploring this also and have a similar discussion thread.\n\nI agree with your conclusion, that it probably is a good idea. The reason that it might be worth further investigation , is that 15% of the types account for 50% of the score, according to how the group MAE metric works. Therefore, we might sub-optimize the model for the smaller 4 types and have a 'good enough' simpler model for the other 4. There is probably a small risk of overfit, but the training and test sets seem to exhibit very similar profiles. You get great leverage by concentrating effort on the smaller segments.\n\n['3JHC', '2JHC', '1JHC', '3JHH', '2JHH', '3JHN', '2JHN', '1JHN']\n[1510379, 1140674, 709416, 590611, 378036, 166415, 119253, 43363]\n\nthe 1JHN type is less than 1% of the data, but contributes 12.5% of the score.",
      "votes": null
    },
    {
      "id": "548093",
      "postDate": "06/08/2019 19:04:46",
      "content": "<p>How would this work? You make if then statements to pick which model to run, based on the <code>TYPE</code>  .</p>",
      "rawMarkdown": "How would this work? You make if then statements to pick which model to run, based on the `TYPE`  .",
      "votes": null
    },
    {
      "id": "548098",
      "postDate": "06/08/2019 19:15:07",
      "content": "<p>I use a different approach - you can see it in my kernel.\n- iterate over unique types;\n- for each type take a subset of train and test data with this value;\n- train model on train and predict on test for each type;\n- correctly combine predictions and indexes;</p>",
      "rawMarkdown": "I use a different approach - you can see it in my kernel.\n- iterate over unique types;\n- for each type take a subset of train and test data with this value;\n- train model on train and predict on test for each type;\n- correctly combine predictions and indexes;",
      "votes": null
    },
    {
      "id": "548099",
      "postDate": "06/08/2019 19:15:11",
      "content": "<p>It's absolutely a good idea! The score is averaged over types and single types are very different in the way outliers are present which have a huge influence on single type scores. </p>",
      "rawMarkdown": "It's absolutely a good idea! The score is averaged over types and single types are very different in the way outliers are present which have a huge influence on single type scores.",
      "votes": null
    },
    {
      "id": "548154",
      "postDate": "06/08/2019 21:57:46",
      "content": "<p>I think it makes sense because feature engineering will differ according to the different types of coupling that are being examined. It's an intensive process, so better to keep it specific. Andrew, what are your MAEs for each coupling type so far? I've only done 2JHH and 3JHH so far and I get errors of 0.366 and 0.336 respectively.</p>",
      "rawMarkdown": "I think it makes sense because feature engineering will differ according to the different types of coupling that are being examined. It's an intensive process, so better to keep it specific. Andrew, what are your MAEs for each coupling type so far? I've only done 2JHH and 3JHH so far and I get errors of 0.366 and 0.336 respectively.",
      "votes": null
    },
    {
      "id": "548243",
      "postDate": "06/09/2019 03:14:47",
      "content": "<p>I agree that it would make sense to do feature engineering separately for different types.</p>\n\n<p>You can see MAE in my kernel. It is really different from type to type: some types have low MAE (0.27, 0.47), some have high (1.7)</p>",
      "rawMarkdown": "I agree that it would make sense to do feature engineering separately for different types.\n\nYou can see MAE in my kernel. It is really different from type to type: some types have low MAE (0.27, 0.47), some have high (1.7)",
      "votes": null
    },
    {
      "id": "548339",
      "postDate": "06/09/2019 08:05:54",
      "content": "<blockquote>\n  <p>the 1JHN type is less than 1% of the data, but contributes 12.5% of the score.</p>\n</blockquote>\n\n<p>Is there a quick way to prove that and what about RMSE? Thanks.</p>",
      "rawMarkdown": "&gt; the 1JHN type is less than 1% of the data, but contributes 12.5% of the score.\n\nIs there a quick way to prove that and what about RMSE? Thanks.",
      "votes": null
    },
    {
      "id": "548376",
      "postDate": "06/09/2019 09:42:18",
      "content": "<p>Yes, because physics and chemistry dictates it. See <a href=\"https://www.chem.wisc.edu/areas/reich/nmr/\">Reich NMR</a></p>\n\n<p>The goal of the current competition is to create a parametrization that has similar accuracies to quantum chemical methods (DFT). That means mean absolute deviations (MAD) of 5 Hz or better. See [DFT spin-spin accuracies] (<a href=\"https://pubs.acs.org/doi/abs/10.1021/ct500676m\">https://pubs.acs.org/doi/abs/10.1021/ct500676m</a>).  Basically we are tasked to build ML or AI models that will be comparable to quantum chemistry, but most importantly orders of magnitudes faster. </p>\n\n<p>While it is possible to build a global model, it is better to focus on single models because at the current stage for most of the public kernel the correct feature space is not correctly covered. Many of the important features are not contained in the CSV files, but in the XYZ coordinate files. </p>\n\n<p>Spin-spin couplings depend on bond angles, bending angles, dihedral angles, dipole moments, coupling partners, atomic (micro) environments, atomic charges, HOMO/LUMO states and many many more factors.  Therefore to gain the most detailed information it is best to perform different regressions models for each type (1JHC, 2JHC, 3JHC, 2JHH, 3JHH, 1JHN, 2JHN, 3JHN). For example:</p>\n\n<p><strong>Geminal proton-proton couplings (2JHH)</strong> depend on sigma-donor/acceptors (fluorine vs methyl groups) and pi-donor/acceptor (carbon carbon double bond C=C vs carbonyl C=O) hence these substituents or bond neighbors need to be coded into the model to obtain sufficient RMSEs.  See: <a href=\"https://www.chem.wisc.edu/areas/reich/nmr/05-hmr-04-2j.htm\">2JHH explained</a></p>\n\n<p><strong>Vicinal proton-proton Coupling (3JHH)</strong> depend on conformational and stereochemical parameters. The Bothner-By/Karplus equations cover these rudimentary by including dihedral angles of the protons, but substituent effects also play a role. See: [3JHH explained] (<a href=\"https://www.chem.wisc.edu/areas/reich/nmr/05-hmr-05-3j.htm\">https://www.chem.wisc.edu/areas/reich/nmr/05-hmr-05-3j.htm</a>) </p>\n\n<p>In conclusion, it is best advised to focus on single class models and to improve them with specific properties. Once all the known variables and effects and all additional features are encoded in the model, one can built a global model. </p>\n\n<p>Bonus: [Karplus: Spinach on the ceiling] (<a href=\"https://www.annualreviews.org/doi/pdf/10.1146/annurev.biophys.33.110502.133350\">https://www.annualreviews.org/doi/pdf/10.1146/annurev.biophys.33.110502.133350</a>)</p>",
      "rawMarkdown": "Yes, because physics and chemistry dictates it. See [Reich NMR](https://www.chem.wisc.edu/areas/reich/nmr/)\n\nThe goal of the current competition is to create a parametrization that has similar accuracies to quantum chemical methods (DFT). That means mean absolute deviations (MAD) of 5 Hz or better. See [DFT spin-spin accuracies] (https://pubs.acs.org/doi/abs/10.1021/ct500676m).  Basically we are tasked to build ML or AI models that will be comparable to quantum chemistry, but most importantly orders of magnitudes faster. \n\nWhile it is possible to build a global model, it is better to focus on single models because at the current stage for most of the public kernel the correct feature space is not correctly covered. Many of the important features are not contained in the CSV files, but in the XYZ coordinate files. \n\nSpin-spin couplings depend on bond angles, bending angles, dihedral angles, dipole moments, coupling partners, atomic (micro) environments, atomic charges, HOMO/LUMO states and many many more factors.  Therefore to gain the most detailed information it is best to perform different regressions models for each type (1JHC, 2JHC, 3JHC, 2JHH, 3JHH, 1JHN, 2JHN, 3JHN). For example:\n\n**Geminal proton-proton couplings (2JHH)** depend on sigma-donor/acceptors (fluorine vs methyl groups) and pi-donor/acceptor (carbon carbon double bond C=C vs carbonyl C=O) hence these substituents or bond neighbors need to be coded into the model to obtain sufficient RMSEs.  See: [2JHH explained](https://www.chem.wisc.edu/areas/reich/nmr/05-hmr-04-2j.htm)\n\n**Vicinal proton-proton Coupling (3JHH)** depend on conformational and stereochemical parameters. The Bothner-By/Karplus equations cover these rudimentary by including dihedral angles of the protons, but substituent effects also play a role. See: [3JHH explained] (https://www.chem.wisc.edu/areas/reich/nmr/05-hmr-05-3j.htm) \n\nIn conclusion, it is best advised to focus on single class models and to improve them with specific properties. Once all the known variables and effects and all additional features are encoded in the model, one can built a global model. \n\nBonus: [Karplus: Spinach on the ceiling] (https://www.annualreviews.org/doi/pdf/10.1146/annurev.biophys.33.110502.133350)",
      "votes": null
    },
    {
      "id": "548385",
      "postDate": "06/09/2019 09:48:38",
      "content": "<p>Thank you for your insights! This is really important. I have already started using information about scalar coupling contributions and understand that there are really a lot of things to do.</p>",
      "rawMarkdown": "Thank you for your insights! This is really important. I have already started using information about scalar coupling contributions and understand that there are really a lot of things to do.",
      "votes": null
    },
    {
      "id": "548660",
      "postDate": "06/09/2019 17:14:31",
      "content": "<p>No need to prove it. It is in the group MAE formula\nSubmissions are evaluated on the Log of the Mean Absolute Error, calculated for each scalar coupling type, and then averaged across types, so that a 1% decrease in MAE for one type provides the same improvement in score as a 1% decrease for another type.</p>\n\n<p>The formula is score= 1/T * log_MAE_for_each_type </p>\n\n<p>Where:</p>\n\n<p>T is the number of scalar coupling types\nThere are 8 scalar coupling types, so each is weighted at 12.5%</p>",
      "rawMarkdown": "No need to prove it. It is in the group MAE formula\nSubmissions are evaluated on the Log of the Mean Absolute Error, calculated for each scalar coupling type, and then averaged across types, so that a 1% decrease in MAE for one type provides the same improvement in score as a 1% decrease for another type.\n\nThe formula is score= 1/T * log_MAE_for_each_type \n\nWhere:\n\nT is the number of scalar coupling types\nThere are 8 scalar coupling types, so each is weighted at 12.5%",
      "votes": null
    },
    {
      "id": "548783",
      "postDate": "06/09/2019 22:26:14",
      "content": "<p>Thanks for this information ;-)</p>",
      "rawMarkdown": "Thanks for this information ;-)",
      "votes": null
    },
    {
      "id": "549131",
      "postDate": "06/10/2019 10:28:04",
      "content": "<p>Thank you very useful information!</p>",
      "rawMarkdown": "Thank you very useful information!",
      "votes": null
    },
    {
      "id": "549981",
      "postDate": "06/11/2019 07:38:32",
      "content": "<p>Tnx.</p>",
      "rawMarkdown": "Tnx.",
      "votes": null
    },
    {
      "id": "550020",
      "postDate": "06/11/2019 08:15:35",
      "content": "<p>I have tried training the following:</p>\n\n<ul>\n<li>1 model per type</li>\n<li>1 model per number of bonds between atoms (1, 2, 3)</li>\n<li>1 model per atom pair type (HC, HH, HN)</li>\n<li>1 model per J-coupling value range (~ -30-30, 0-80, 70-200)</li>\n</ul>\n\n<p>So far 1 model per atom pair type seems to give the best results.</p>",
      "rawMarkdown": "I have tried training the following:\n\n- 1 model per type\n- 1 model per number of bonds between atoms (1, 2, 3)\n- 1 model per atom pair type (HC, HH, HN)\n- 1 model per J-coupling value range (~ -30-30, 0-80, 70-200)\n\nSo far 1 model per atom pair type seems to give the best results.",
      "votes": null
    },
    {
      "id": "550117",
      "postDate": "06/11/2019 10:07:48",
      "content": "<p>This is really important domain knowledge that we can apply to the competition. I wasn't aware of how much can we derivate from the tridimensional arrangement of atomic structures. <a href=\"/tobikaggle\">@tobikaggle</a> is there any advice or resource you recommend to calculate features like bond angles, bending angles, dihedral angles, dipole moments, coupling partners and so on?</p>\n\n<p>Thanks for your insights!</p>",
      "rawMarkdown": "This is really important domain knowledge that we can apply to the competition. I wasn't aware of how much can we derivate from the tridimensional arrangement of atomic structures. @tobikaggle is there any advice or resource you recommend to calculate features like bond angles, bending angles, dihedral angles, dipole moments, coupling partners and so on?\n\nThanks for your insights!",
      "votes": null
    },
    {
      "id": "550124",
      "postDate": "06/11/2019 10:16:11",
      "content": "<p><a href=\"/artgor\">@artgor</a> again, thanks for the amazing discussion you created here. I've been reading the comments on this thread and already learned a lot. It seems unanimous that creating a model for each type will improve the level of accuracy. \nWill you share a kernel with individual models for each type?</p>\n\n<p>Thanks for your contributions.</p>",
      "rawMarkdown": "artgor again, thanks for the amazing discussion you created here. I've been reading the comments on this thread and already learned a lot. It seems unanimous that creating a model for each type will improve the level of accuracy. \nWill you share a kernel with individual models for each type?\n\nThanks for your contributions.",
      "votes": null
    },
    {
      "id": "550137",
      "postDate": "06/11/2019 10:38:44",
      "content": "<p>Thank you!</p>\n\n<p>In fact I have already published this kernel: <a href=\"https://www.kaggle.com/artgor/brute-force-feature-engineering/notebook#Training-model-on-selected-features\">https://www.kaggle.com/artgor/brute-force-feature-engineering/notebook#Training-model-on-selected-features</a></p>",
      "rawMarkdown": "Thank you!\n\nIn fact I have already published this kernel: https://www.kaggle.com/artgor/brute-force-feature-engineering/notebook#Training-model-on-selected-features",
      "votes": null
    },
    {
      "id": "550739",
      "postDate": "06/12/2019 01:32:03",
      "content": "<p>emm, I have different results.</p>\n\n<p>1 model per atom pair type get cv -0.36 and LB -0.422, and per type get cv -0.43 and LB -0.494.</p>\n\n<p>with 24 features and LightGBM.</p>",
      "rawMarkdown": "emm, I have different results.\n\n1 model per atom pair type get cv -0.36 and LB -0.422, and per type get cv -0.43 and LB -0.494.\n\nwith 24 features and LightGBM.",
      "votes": null
    },
    {
      "id": "550813",
      "postDate": "06/12/2019 03:54:30",
      "content": "<p><a href=\"/gilbertomrsilva\">@gilbertomrsilva</a> tools to calculate features like bond angles, bending angles, dihedral angles, dipole moments, coupling partners and so on?</p>\n\n<p>1) <strong>MolMod</strong> can read XYZ, calculate connection matrices, bond and dihedral angles and more <a href=\"https://github.com/molmod/molmod\">https://github.com/molmod/molmod</a> and documentation <a href=\"https://molmod.github.io/molmod/tutorial/internal_coordinates.html\">https://molmod.github.io/molmod/tutorial/internal_coordinates.html</a></p>\n\n<p>2) <strong>OpenBabel</strong> to to read and convert XYZ <a href=\"https://github.com/openbabel\">https://github.com/openbabel</a></p>\n\n<p>3) <strong>RDkit</strong> can do many many things <a href=\"https://github.com/rdkit/rdkit\">https://github.com/rdkit/rdkit</a>\nUse search function for dihedral angle such as <a href=\"http://www.rdkit.org/docs/source/rdkit.Chem.rdMolTransforms.html?highlight=dihedral#rdkit.Chem.rdMolTransforms.GetDihedralDeg\">http://www.rdkit.org/docs/source/rdkit.Chem.rdMolTransforms.html?highlight=dihedral#rdkit.Chem.rdMolTransforms.GetDihedralDeg</a></p>",
      "rawMarkdown": "gilbertomrsilva tools to calculate features like bond angles, bending angles, dihedral angles, dipole moments, coupling partners and so on?\n\n1) **MolMod** can read XYZ, calculate connection matrices, bond and dihedral angles and more https://github.com/molmod/molmod and documentation https://molmod.github.io/molmod/tutorial/internal_coordinates.html\n\n2) **OpenBabel** to to read and convert XYZ https://github.com/openbabel\n\n3) **RDkit** can do many many things https://github.com/rdkit/rdkit\nUse search function for dihedral angle such as http://www.rdkit.org/docs/source/rdkit.Chem.rdMolTransforms.html?highlight=dihedral#rdkit.Chem.rdMolTransforms.GetDihedralDeg",
      "votes": null
    },
    {
      "id": "550936",
      "postDate": "06/12/2019 06:39:49",
      "content": "<p>It was very close for me between per type and per atom pair type.  </p>\n\n<ul>\n<li>per type: cv -0.727327</li>\n<li>per atom pair type: cv -0.756933 </li>\n</ul>\n\n<p>with around 60 features, also LGBM. I don't have a LB yet (haven't submitted because I haven't decided if I want to take this comp seriously or not).\nAlso I didn't do any hyper-parameter tuning for either model types so proper tuning could benefit one or the other more. </p>",
      "rawMarkdown": "It was very close for me between per type and per atom pair type.  \n\n- per type: cv -0.727327\n- per atom pair type: cv -0.756933 \n\nwith around 60 features, also LGBM. I don't have a LB yet (haven't submitted because I haven't decided if I want to take this comp seriously or not).\nAlso I didn't do any hyper-parameter tuning for either model types so proper tuning could benefit one or the other more.",
      "votes": null
    },
    {
      "id": "551023",
      "postDate": "06/12/2019 08:51:09",
      "content": "<p>so am I.</p>",
      "rawMarkdown": "so am I.",
      "votes": null
    },
    {
      "id": "551101",
      "postDate": "06/12/2019 10:44:52",
      "content": "<p><a href=\"/tobikaggle\">@tobikaggle</a> thanks once again. I'll really try to dig deep on this competition.</p>",
      "rawMarkdown": "tobikaggle thanks once again. I'll really try to dig deep on this competition.",
      "votes": null
    },
    {
      "id": "551255",
      "postDate": "06/12/2019 14:04:41",
      "content": "<p>288 forks, 161 votes - some things never change! Kaggle really needs to make a fork an automatic vote. I think it's really rude to copy someone's work without even giving them a little online credit.</p>",
      "rawMarkdown": "288 forks, 161 votes - some things never change! Kaggle really needs to make a fork an automatic vote. I think it's really rude to copy someone's work without even giving them a little online credit.",
      "votes": null
    },
    {
      "id": "551468",
      "postDate": "06/12/2019 18:49:10",
      "content": "<p>For me a single model has worked best. </p>\n\n<p>Best result so far with single model is LB -0.9 CV -0.81\nBest result so far with splitting model LB -0.7 CV -0.48</p>",
      "rawMarkdown": "For me a single model has worked best. \n\nBest result so far with single model is LB -0.9 CV -0.81\nBest result so far with splitting model LB -0.7 CV -0.48",
      "votes": null
    },
    {
      "id": "572496",
      "postDate": "07/11/2019 02:17:31",
      "content": "<p>After detailed study and various modeling efforts, the answer is yes. It is definitely worth training separate models for each of the 8 types. </p>\n\n<p>The reasons for this answer:\n- each of the distributions for the scalar coupling constant is different\n- The various atomic and molecular properties are quite different for each\n- each can be predicted with an independent model without much risk of overfit\n- each of the coupling types might improve with blended models using other techniques, e.g. GB with NN (see several recent posts that support this)</p>\n\n<p>Once one decides on this approach, this competition becomes a model configuration and management issue. We have already created a multitude of test situations to find which features seem to work best with each coupling type. As competitors refine their results, it becomes a mostly a software configuration management problem to put Humpty Dumpty together again! </p>",
      "rawMarkdown": "After detailed study and various modeling efforts, the answer is yes. It is definitely worth training separate models for each of the 8 types. \n\nThe reasons for this answer:\n- each of the distributions for the scalar coupling constant is different\n- The various atomic and molecular properties are quite different for each\n- each can be predicted with an independent model without much risk of overfit\n- each of the coupling types might improve with blended models using other techniques, e.g. GB with NN (see several recent posts that support this)\n\nOnce one decides on this approach, this competition becomes a model configuration and management issue. We have already created a multitude of test situations to find which features seem to work best with each coupling type. As competitors refine their results, it becomes a mostly a software configuration management problem to put Humpty Dumpty together again!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 547912,
      "author_name": "milkshakespeare",
      "author_url": "",
      "post_date": "06/08/2019 14:01:03",
      "content": "<p>I think is a good idea.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 547968,
      "author_name": "wjholst",
      "author_url": "",
      "post_date": "06/08/2019 15:15:56",
      "content": "<p>I have been exploring this also and have a similar discussion thread.</p>\n\n<p>I agree with your conclusion, that it probably is a good idea. The reason that it might be worth further investigation , is that 15% of the types account for 50% of the score, according to how the group MAE metric works. Therefore, we might sub-optimize the model for the smaller 4 types and have a 'good enough' simpler model for the other 4. There is probably a small risk of overfit, but the training and test sets seem to exhibit very similar profiles. You get great leverage by concentrating effort on the smaller segments.</p>\n\n<p>['3JHC', '2JHC', '1JHC', '3JHH', '2JHH', '3JHN', '2JHN', '1JHN']\n[1510379, 1140674, 709416, 590611, 378036, 166415, 119253, 43363]</p>\n\n<p>the 1JHN type is less than 1% of the data, but contributes 12.5% of the score.</p>",
      "votes": null,
      "replies": [
        {
          "id": 548339,
          "author_name": "tobikaggle",
          "author_url": "",
          "post_date": "06/09/2019 08:05:54",
          "content": "<blockquote>\n  <p>the 1JHN type is less than 1% of the data, but contributes 12.5% of the score.</p>\n</blockquote>\n\n<p>Is there a quick way to prove that and what about RMSE? Thanks.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 548660,
          "author_name": "wjholst",
          "author_url": "",
          "post_date": "06/09/2019 17:14:31",
          "content": "<p>No need to prove it. It is in the group MAE formula\nSubmissions are evaluated on the Log of the Mean Absolute Error, calculated for each scalar coupling type, and then averaged across types, so that a 1% decrease in MAE for one type provides the same improvement in score as a 1% decrease for another type.</p>\n\n<p>The formula is score= 1/T * log_MAE_for_each_type </p>\n\n<p>Where:</p>\n\n<p>T is the number of scalar coupling types\nThere are 8 scalar coupling types, so each is weighted at 12.5%</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 549981,
          "author_name": "tobikaggle",
          "author_url": "",
          "post_date": "06/11/2019 07:38:32",
          "content": "<p>Tnx.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 548093,
      "author_name": "lowecoryr",
      "author_url": "",
      "post_date": "06/08/2019 19:04:46",
      "content": "<p>How would this work? You make if then statements to pick which model to run, based on the <code>TYPE</code>  .</p>",
      "votes": null,
      "replies": [
        {
          "id": 548098,
          "author_name": "artgor",
          "author_url": "",
          "post_date": "06/08/2019 19:15:07",
          "content": "<p>I use a different approach - you can see it in my kernel.\n- iterate over unique types;\n- for each type take a subset of train and test data with this value;\n- train model on train and predict on test for each type;\n- correctly combine predictions and indexes;</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 548099,
      "author_name": "allunia",
      "author_url": "",
      "post_date": "06/08/2019 19:15:11",
      "content": "<p>It's absolutely a good idea! The score is averaged over types and single types are very different in the way outliers are present which have a huge influence on single type scores. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 548154,
      "author_name": "bigironsphere",
      "author_url": "",
      "post_date": "06/08/2019 21:57:46",
      "content": "<p>I think it makes sense because feature engineering will differ according to the different types of coupling that are being examined. It's an intensive process, so better to keep it specific. Andrew, what are your MAEs for each coupling type so far? I've only done 2JHH and 3JHH so far and I get errors of 0.366 and 0.336 respectively.</p>",
      "votes": null,
      "replies": [
        {
          "id": 548243,
          "author_name": "artgor",
          "author_url": "",
          "post_date": "06/09/2019 03:14:47",
          "content": "<p>I agree that it would make sense to do feature engineering separately for different types.</p>\n\n<p>You can see MAE in my kernel. It is really different from type to type: some types have low MAE (0.27, 0.47), some have high (1.7)</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 548376,
      "author_name": "tobikaggle",
      "author_url": "",
      "post_date": "06/09/2019 09:42:18",
      "content": "<p>Yes, because physics and chemistry dictates it. See <a href=\"https://www.chem.wisc.edu/areas/reich/nmr/\">Reich NMR</a></p>\n\n<p>The goal of the current competition is to create a parametrization that has similar accuracies to quantum chemical methods (DFT). That means mean absolute deviations (MAD) of 5 Hz or better. See [DFT spin-spin accuracies] (<a href=\"https://pubs.acs.org/doi/abs/10.1021/ct500676m\">https://pubs.acs.org/doi/abs/10.1021/ct500676m</a>).  Basically we are tasked to build ML or AI models that will be comparable to quantum chemistry, but most importantly orders of magnitudes faster. </p>\n\n<p>While it is possible to build a global model, it is better to focus on single models because at the current stage for most of the public kernel the correct feature space is not correctly covered. Many of the important features are not contained in the CSV files, but in the XYZ coordinate files. </p>\n\n<p>Spin-spin couplings depend on bond angles, bending angles, dihedral angles, dipole moments, coupling partners, atomic (micro) environments, atomic charges, HOMO/LUMO states and many many more factors.  Therefore to gain the most detailed information it is best to perform different regressions models for each type (1JHC, 2JHC, 3JHC, 2JHH, 3JHH, 1JHN, 2JHN, 3JHN). For example:</p>\n\n<p><strong>Geminal proton-proton couplings (2JHH)</strong> depend on sigma-donor/acceptors (fluorine vs methyl groups) and pi-donor/acceptor (carbon carbon double bond C=C vs carbonyl C=O) hence these substituents or bond neighbors need to be coded into the model to obtain sufficient RMSEs.  See: <a href=\"https://www.chem.wisc.edu/areas/reich/nmr/05-hmr-04-2j.htm\">2JHH explained</a></p>\n\n<p><strong>Vicinal proton-proton Coupling (3JHH)</strong> depend on conformational and stereochemical parameters. The Bothner-By/Karplus equations cover these rudimentary by including dihedral angles of the protons, but substituent effects also play a role. See: [3JHH explained] (<a href=\"https://www.chem.wisc.edu/areas/reich/nmr/05-hmr-05-3j.htm\">https://www.chem.wisc.edu/areas/reich/nmr/05-hmr-05-3j.htm</a>) </p>\n\n<p>In conclusion, it is best advised to focus on single class models and to improve them with specific properties. Once all the known variables and effects and all additional features are encoded in the model, one can built a global model. </p>\n\n<p>Bonus: [Karplus: Spinach on the ceiling] (<a href=\"https://www.annualreviews.org/doi/pdf/10.1146/annurev.biophys.33.110502.133350\">https://www.annualreviews.org/doi/pdf/10.1146/annurev.biophys.33.110502.133350</a>)</p>",
      "votes": null,
      "replies": [
        {
          "id": 548385,
          "author_name": "artgor",
          "author_url": "",
          "post_date": "06/09/2019 09:48:38",
          "content": "<p>Thank you for your insights! This is really important. I have already started using information about scalar coupling contributions and understand that there are really a lot of things to do.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 548783,
          "author_name": "alvarocalle",
          "author_url": "",
          "post_date": "06/09/2019 22:26:14",
          "content": "<p>Thanks for this information ;-)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 549131,
          "author_name": "hervind",
          "author_url": "",
          "post_date": "06/10/2019 10:28:04",
          "content": "<p>Thank you very useful information!</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 550117,
          "author_name": "gilbertomrsilva",
          "author_url": "",
          "post_date": "06/11/2019 10:07:48",
          "content": "<p>This is really important domain knowledge that we can apply to the competition. I wasn't aware of how much can we derivate from the tridimensional arrangement of atomic structures. <a href=\"/tobikaggle\">@tobikaggle</a> is there any advice or resource you recommend to calculate features like bond angles, bending angles, dihedral angles, dipole moments, coupling partners and so on?</p>\n\n<p>Thanks for your insights!</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 550813,
          "author_name": "tobikaggle",
          "author_url": "",
          "post_date": "06/12/2019 03:54:30",
          "content": "<p><a href=\"/gilbertomrsilva\">@gilbertomrsilva</a> tools to calculate features like bond angles, bending angles, dihedral angles, dipole moments, coupling partners and so on?</p>\n\n<p>1) <strong>MolMod</strong> can read XYZ, calculate connection matrices, bond and dihedral angles and more <a href=\"https://github.com/molmod/molmod\">https://github.com/molmod/molmod</a> and documentation <a href=\"https://molmod.github.io/molmod/tutorial/internal_coordinates.html\">https://molmod.github.io/molmod/tutorial/internal_coordinates.html</a></p>\n\n<p>2) <strong>OpenBabel</strong> to to read and convert XYZ <a href=\"https://github.com/openbabel\">https://github.com/openbabel</a></p>\n\n<p>3) <strong>RDkit</strong> can do many many things <a href=\"https://github.com/rdkit/rdkit\">https://github.com/rdkit/rdkit</a>\nUse search function for dihedral angle such as <a href=\"http://www.rdkit.org/docs/source/rdkit.Chem.rdMolTransforms.html?highlight=dihedral#rdkit.Chem.rdMolTransforms.GetDihedralDeg\">http://www.rdkit.org/docs/source/rdkit.Chem.rdMolTransforms.html?highlight=dihedral#rdkit.Chem.rdMolTransforms.GetDihedralDeg</a></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 551101,
          "author_name": "gilbertomrsilva",
          "author_url": "",
          "post_date": "06/12/2019 10:44:52",
          "content": "<p><a href=\"/tobikaggle\">@tobikaggle</a> thanks once again. I'll really try to dig deep on this competition.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 550020,
      "author_name": "taniaj",
      "author_url": "",
      "post_date": "06/11/2019 08:15:35",
      "content": "<p>I have tried training the following:</p>\n\n<ul>\n<li>1 model per type</li>\n<li>1 model per number of bonds between atoms (1, 2, 3)</li>\n<li>1 model per atom pair type (HC, HH, HN)</li>\n<li>1 model per J-coupling value range (~ -30-30, 0-80, 70-200)</li>\n</ul>\n\n<p>So far 1 model per atom pair type seems to give the best results.</p>",
      "votes": null,
      "replies": [
        {
          "id": 550739,
          "author_name": "hadxu123",
          "author_url": "",
          "post_date": "06/12/2019 01:32:03",
          "content": "<p>emm, I have different results.</p>\n\n<p>1 model per atom pair type get cv -0.36 and LB -0.422, and per type get cv -0.43 and LB -0.494.</p>\n\n<p>with 24 features and LightGBM.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 550936,
          "author_name": "taniaj",
          "author_url": "",
          "post_date": "06/12/2019 06:39:49",
          "content": "<p>It was very close for me between per type and per atom pair type.  </p>\n\n<ul>\n<li>per type: cv -0.727327</li>\n<li>per atom pair type: cv -0.756933 </li>\n</ul>\n\n<p>with around 60 features, also LGBM. I don't have a LB yet (haven't submitted because I haven't decided if I want to take this comp seriously or not).\nAlso I didn't do any hyper-parameter tuning for either model types so proper tuning could benefit one or the other more. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 551023,
          "author_name": "hadxu123",
          "author_url": "",
          "post_date": "06/12/2019 08:51:09",
          "content": "<p>so am I.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 550124,
      "author_name": "gilbertomrsilva",
      "author_url": "",
      "post_date": "06/11/2019 10:16:11",
      "content": "<p><a href=\"/artgor\">@artgor</a> again, thanks for the amazing discussion you created here. I've been reading the comments on this thread and already learned a lot. It seems unanimous that creating a model for each type will improve the level of accuracy. \nWill you share a kernel with individual models for each type?</p>\n\n<p>Thanks for your contributions.</p>",
      "votes": null,
      "replies": [
        {
          "id": 550137,
          "author_name": "artgor",
          "author_url": "",
          "post_date": "06/11/2019 10:38:44",
          "content": "<p>Thank you!</p>\n\n<p>In fact I have already published this kernel: <a href=\"https://www.kaggle.com/artgor/brute-force-feature-engineering/notebook#Training-model-on-selected-features\">https://www.kaggle.com/artgor/brute-force-feature-engineering/notebook#Training-model-on-selected-features</a></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 551255,
          "author_name": "bigironsphere",
          "author_url": "",
          "post_date": "06/12/2019 14:04:41",
          "content": "<p>288 forks, 161 votes - some things never change! Kaggle really needs to make a fork an automatic vote. I think it's really rude to copy someone's work without even giving them a little online credit.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 551468,
      "author_name": "jonassv",
      "author_url": "",
      "post_date": "06/12/2019 18:49:10",
      "content": "<p>For me a single model has worked best. </p>\n\n<p>Best result so far with single model is LB -0.9 CV -0.81\nBest result so far with splitting model LB -0.7 CV -0.48</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 572496,
      "author_name": "wjholst",
      "author_url": "",
      "post_date": "07/11/2019 02:17:31",
      "content": "<p>After detailed study and various modeling efforts, the answer is yes. It is definitely worth training separate models for each of the 8 types. </p>\n\n<p>The reasons for this answer:\n- each of the distributions for the scalar coupling constant is different\n- The various atomic and molecular properties are quite different for each\n- each can be predicted with an independent model without much risk of overfit\n- each of the coupling types might improve with blended models using other techniques, e.g. GB with NN (see several recent posts that support this)</p>\n\n<p>Once one decides on this approach, this competition becomes a model configuration and management issue. We have already created a multitude of test situations to find which features seem to work best with each coupling type. As competitors refine their results, it becomes a mostly a software configuration management problem to put Humpty Dumpty together again! </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "547847": "`Type` obviously is important feature judging both from common sense and from models. I have already seen at least one [kernel](https://www.kaggle.com/kirichenko17roman/simple-lightgbm-in-r) where author tried to train separate models for each type.\n\nSo today I also trained separate models in my [kernel](https://www.kaggle.com/artgor/brute-force-feature-engineering)\n\nHere is a plot of OOF predictions vs target for two types.\n![](https://i.imgur.com/W4TWyQf.jpg)\n\nWe can see that predictions are more robust, the score is also better.\n\nSo what do you think: is this a good idea or could it lead to overfitting and other problems?",
    "547912": "I think is a good idea.",
    "547968": "I have been exploring this also and have a similar discussion thread.\n\nI agree with your conclusion, that it probably is a good idea. The reason that it might be worth further investigation , is that 15% of the types account for 50% of the score, according to how the group MAE metric works. Therefore, we might sub-optimize the model for the smaller 4 types and have a 'good enough' simpler model for the other 4. There is probably a small risk of overfit, but the training and test sets seem to exhibit very similar profiles. You get great leverage by concentrating effort on the smaller segments.\n\n['3JHC', '2JHC', '1JHC', '3JHH', '2JHH', '3JHN', '2JHN', '1JHN']\n[1510379, 1140674, 709416, 590611, 378036, 166415, 119253, 43363]\n\nthe 1JHN type is less than 1% of the data, but contributes 12.5% of the score.",
    "548093": "How would this work? You make if then statements to pick which model to run, based on the `TYPE`  .",
    "548098": "I use a different approach - you can see it in my kernel.\n- iterate over unique types;\n- for each type take a subset of train and test data with this value;\n- train model on train and predict on test for each type;\n- correctly combine predictions and indexes;",
    "548099": "It's absolutely a good idea! The score is averaged over types and single types are very different in the way outliers are present which have a huge influence on single type scores.",
    "548154": "I think it makes sense because feature engineering will differ according to the different types of coupling that are being examined. It's an intensive process, so better to keep it specific. Andrew, what are your MAEs for each coupling type so far? I've only done 2JHH and 3JHH so far and I get errors of 0.366 and 0.336 respectively.",
    "548243": "I agree that it would make sense to do feature engineering separately for different types.\n\nYou can see MAE in my kernel. It is really different from type to type: some types have low MAE (0.27, 0.47), some have high (1.7)",
    "548339": "&gt; the 1JHN type is less than 1% of the data, but contributes 12.5% of the score.\n\nIs there a quick way to prove that and what about RMSE? Thanks.",
    "548376": "Yes, because physics and chemistry dictates it. See [Reich NMR](https://www.chem.wisc.edu/areas/reich/nmr/)\n\nThe goal of the current competition is to create a parametrization that has similar accuracies to quantum chemical methods (DFT). That means mean absolute deviations (MAD) of 5 Hz or better. See [DFT spin-spin accuracies] (https://pubs.acs.org/doi/abs/10.1021/ct500676m).  Basically we are tasked to build ML or AI models that will be comparable to quantum chemistry, but most importantly orders of magnitudes faster. \n\nWhile it is possible to build a global model, it is better to focus on single models because at the current stage for most of the public kernel the correct feature space is not correctly covered. Many of the important features are not contained in the CSV files, but in the XYZ coordinate files. \n\nSpin-spin couplings depend on bond angles, bending angles, dihedral angles, dipole moments, coupling partners, atomic (micro) environments, atomic charges, HOMO/LUMO states and many many more factors.  Therefore to gain the most detailed information it is best to perform different regressions models for each type (1JHC, 2JHC, 3JHC, 2JHH, 3JHH, 1JHN, 2JHN, 3JHN). For example:\n\n**Geminal proton-proton couplings (2JHH)** depend on sigma-donor/acceptors (fluorine vs methyl groups) and pi-donor/acceptor (carbon carbon double bond C=C vs carbonyl C=O) hence these substituents or bond neighbors need to be coded into the model to obtain sufficient RMSEs.  See: [2JHH explained](https://www.chem.wisc.edu/areas/reich/nmr/05-hmr-04-2j.htm)\n\n**Vicinal proton-proton Coupling (3JHH)** depend on conformational and stereochemical parameters. The Bothner-By/Karplus equations cover these rudimentary by including dihedral angles of the protons, but substituent effects also play a role. See: [3JHH explained] (https://www.chem.wisc.edu/areas/reich/nmr/05-hmr-05-3j.htm) \n\nIn conclusion, it is best advised to focus on single class models and to improve them with specific properties. Once all the known variables and effects and all additional features are encoded in the model, one can built a global model. \n\nBonus: [Karplus: Spinach on the ceiling] (https://www.annualreviews.org/doi/pdf/10.1146/annurev.biophys.33.110502.133350)",
    "548385": "Thank you for your insights! This is really important. I have already started using information about scalar coupling contributions and understand that there are really a lot of things to do.",
    "548660": "No need to prove it. It is in the group MAE formula\nSubmissions are evaluated on the Log of the Mean Absolute Error, calculated for each scalar coupling type, and then averaged across types, so that a 1% decrease in MAE for one type provides the same improvement in score as a 1% decrease for another type.\n\nThe formula is score= 1/T * log_MAE_for_each_type \n\nWhere:\n\nT is the number of scalar coupling types\nThere are 8 scalar coupling types, so each is weighted at 12.5%",
    "548783": "Thanks for this information ;-)",
    "549131": "Thank you very useful information!",
    "549981": "Tnx.",
    "550020": "I have tried training the following:\n\n- 1 model per type\n- 1 model per number of bonds between atoms (1, 2, 3)\n- 1 model per atom pair type (HC, HH, HN)\n- 1 model per J-coupling value range (~ -30-30, 0-80, 70-200)\n\nSo far 1 model per atom pair type seems to give the best results.",
    "550117": "This is really important domain knowledge that we can apply to the competition. I wasn't aware of how much can we derivate from the tridimensional arrangement of atomic structures. @tobikaggle is there any advice or resource you recommend to calculate features like bond angles, bending angles, dihedral angles, dipole moments, coupling partners and so on?\n\nThanks for your insights!",
    "550124": "artgor again, thanks for the amazing discussion you created here. I've been reading the comments on this thread and already learned a lot. It seems unanimous that creating a model for each type will improve the level of accuracy. \nWill you share a kernel with individual models for each type?\n\nThanks for your contributions.",
    "550137": "Thank you!\n\nIn fact I have already published this kernel: https://www.kaggle.com/artgor/brute-force-feature-engineering/notebook#Training-model-on-selected-features",
    "550739": "emm, I have different results.\n\n1 model per atom pair type get cv -0.36 and LB -0.422, and per type get cv -0.43 and LB -0.494.\n\nwith 24 features and LightGBM.",
    "550813": "gilbertomrsilva tools to calculate features like bond angles, bending angles, dihedral angles, dipole moments, coupling partners and so on?\n\n1) **MolMod** can read XYZ, calculate connection matrices, bond and dihedral angles and more https://github.com/molmod/molmod and documentation https://molmod.github.io/molmod/tutorial/internal_coordinates.html\n\n2) **OpenBabel** to to read and convert XYZ https://github.com/openbabel\n\n3) **RDkit** can do many many things https://github.com/rdkit/rdkit\nUse search function for dihedral angle such as http://www.rdkit.org/docs/source/rdkit.Chem.rdMolTransforms.html?highlight=dihedral#rdkit.Chem.rdMolTransforms.GetDihedralDeg",
    "550936": "It was very close for me between per type and per atom pair type.  \n\n- per type: cv -0.727327\n- per atom pair type: cv -0.756933 \n\nwith around 60 features, also LGBM. I don't have a LB yet (haven't submitted because I haven't decided if I want to take this comp seriously or not).\nAlso I didn't do any hyper-parameter tuning for either model types so proper tuning could benefit one or the other more.",
    "551023": "so am I.",
    "551101": "tobikaggle thanks once again. I'll really try to dig deep on this competition.",
    "551255": "288 forks, 161 votes - some things never change! Kaggle really needs to make a fork an automatic vote. I think it's really rude to copy someone's work without even giving them a little online credit.",
    "551468": "For me a single model has worked best. \n\nBest result so far with single model is LB -0.9 CV -0.81\nBest result so far with splitting model LB -0.7 CV -0.48",
    "572496": "After detailed study and various modeling efforts, the answer is yes. It is definitely worth training separate models for each of the 8 types. \n\nThe reasons for this answer:\n- each of the distributions for the scalar coupling constant is different\n- The various atomic and molecular properties are quite different for each\n- each can be predicted with an independent model without much risk of overfit\n- each of the coupling types might improve with blended models using other techniques, e.g. GB with NN (see several recent posts that support this)\n\nOnce one decides on this approach, this competition becomes a model configuration and management issue. We have already created a multitude of test situations to find which features seem to work best with each coupling type. As competitors refine their results, it becomes a mostly a software configuration management problem to put Humpty Dumpty together again!"
  },
  "source": "meta"
}