{
  "id": 96574,
  "title": "Is impact on score different for each type?",
  "url": "/competitions/champs-scalar-coupling/discussion/96574",
  "author_name": "",
  "post_date": "2019-06-21T08:36:49.643502800Z",
  "votes": 5,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Rules state that score improvement for different types will have the same effect on the score:\n<em>\"1% decrease in MAE for one type provides the same improvement in score as a 1% decrease for another type\"</em></p>\n\n<p>However, according to the provided formula, it seems that the same <strong>decrease</strong> in MAE between 2 types would have the same impact on the score, but it doesn't work for the <strong>% decrease</strong></p>\n\n<p>Furthermore, standard deviation of scalar_coupling_constant is not the same for different types. Actually, it is very different, and goes from 1.3 (3JHN) to 18.3 (3JHC).\nIf we expect the MAE difference between groups to be in the same order of magnitude than the standard deviation difference between groups, we will have a factor 10 between MAE of 3JHN and MAE of 3JHC.</p>\n\n<p>And according to the scoring formula, this factor 10 will be directly reflected in the score calculation.</p>\n\n<p>As a conclusion, it seems relevant to focus efforts on improving accuracy of models for types where scalar_coupling_constant standard deviation is high instead of the ones where it is low.</p>",
  "messages": [
    {
      "id": "557530",
      "postDate": "06/21/2019 08:36:49",
      "content": "<p>Rules state that score improvement for different types will have the same effect on the score:\n<em>\"1% decrease in MAE for one type provides the same improvement in score as a 1% decrease for another type\"</em></p>\n\n<p>However, according to the provided formula, it seems that the same <strong>decrease</strong> in MAE between 2 types would have the same impact on the score, but it doesn't work for the <strong>% decrease</strong></p>\n\n<p>Furthermore, standard deviation of scalar_coupling_constant is not the same for different types. Actually, it is very different, and goes from 1.3 (3JHN) to 18.3 (3JHC).\nIf we expect the MAE difference between groups to be in the same order of magnitude than the standard deviation difference between groups, we will have a factor 10 between MAE of 3JHN and MAE of 3JHC.</p>\n\n<p>And according to the scoring formula, this factor 10 will be directly reflected in the score calculation.</p>\n\n<p>As a conclusion, it seems relevant to focus efforts on improving accuracy of models for types where scalar_coupling_constant standard deviation is high instead of the ones where it is low.</p>",
      "rawMarkdown": "Rules state that score improvement for different types will have the same effect on the score:\n*\"1% decrease in MAE for one type provides the same improvement in score as a 1% decrease for another type\"*\n\nHowever, according to the provided formula, it seems that the same **decrease** in MAE between 2 types would have the same impact on the score, but it doesn't work for the **% decrease**\n\nFurthermore, standard deviation of scalar_coupling_constant is not the same for different types. Actually, it is very different, and goes from 1.3 (3JHN) to 18.3 (3JHC).\nIf we expect the MAE difference between groups to be in the same order of magnitude than the standard deviation difference between groups, we will have a factor 10 between MAE of 3JHN and MAE of 3JHC.\n\nAnd according to the scoring formula, this factor 10 will be directly reflected in the score calculation.\n\nAs a conclusion, it seems relevant to focus efforts on improving accuracy of models for types where scalar_coupling_constant standard deviation is high instead of the ones where it is low.",
      "votes": null
    },
    {
      "id": "557738",
      "postDate": "06/21/2019 12:36:53",
      "content": "<p><a href=\"/florent31\">@florent31</a> You have made a very good point here. This does make sense!</p>",
      "rawMarkdown": "florent31 You have made a very good point here. This does make sense!",
      "votes": null
    },
    {
      "id": "558592",
      "postDate": "06/22/2019 16:12:46",
      "content": "<p>It is not the standard deviation that counts but the MAE of your prediction mistakes per type. And give that there is a log after, proportionally large mistakes have smaller impact on the final result.\nMy point is that which type to optimize really depends on the prediction errors and some other factors. There is no golden rule for that</p>",
      "rawMarkdown": "It is not the standard deviation that counts but the MAE of your prediction mistakes per type. And give that there is a log after, proportionally large mistakes have smaller impact on the final result.\nMy point is that which type to optimize really depends on the prediction errors and some other factors. There is no golden rule for that",
      "votes": null
    },
    {
      "id": "561481",
      "postDate": "06/26/2019 13:26:51",
      "content": "<p>Halving the MAE for 1JHC should have the same effect on the score as halving the MAE for 2JHN, no? </p>\n\n<p>log(0.5 * MAE_1JHC) + log(MAE_2JHN) ==  log(MAE_1JHC) + log(0.5 * MAE_2JHN)</p>",
      "rawMarkdown": "Halving the MAE for 1JHC should have the same effect on the score as halving the MAE for 2JHN, no? \n\nlog(0.5 * MAE\\_1JHC) + log(MAE\\_2JHN) ==  log(MAE\\_1JHC) + log(0.5 * MAE\\_2JHN)",
      "votes": null
    },
    {
      "id": "561697",
      "postDate": "06/26/2019 16:48:01",
      "content": "<p>it is probably for this reason that you should train separate prediction for different J coupling type. The \"scale of the error is difference\". When trained to together, reduction of loss of one type may inhibits another.</p>\n\n<p><code>\nAs a conclusion, it seems relevant to focus efforts on improving accuracy of models for types where scalarcouplingconstant standard deviation is high instead of the ones where it is low.\n</code></p>\n\n<p>you may note that the winning team would be the one that can reduce the high variance loss (e.g. 18.3 1JHC), when most team cannot. High precision regression (e.g. 0.01 out of 18.3, normalised error of 1e-4) is \"statistically difficult\" but may be \"made easy by some tricks\" or \"better loss function\"</p>",
      "rawMarkdown": "it is probably for this reason that you should train separate prediction for different J coupling type. The \"scale of the error is difference\". When trained to together, reduction of loss of one type may inhibits another.\n\n\n```\nAs a conclusion, it seems relevant to focus efforts on improving accuracy of models for types where scalarcouplingconstant standard deviation is high instead of the ones where it is low.\n```\n\nyou may note that the winning team would be the one that can reduce the high variance loss (e.g. 18.3 1JHC), when most team cannot. High precision regression (e.g. 0.01 out of 18.3, normalised error of 1e-4) is \"statistically difficult\" but may be \"made easy by some tricks\" or \"better loss function\"",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 557738,
      "author_name": "prazhant",
      "author_url": "",
      "post_date": "06/21/2019 12:36:53",
      "content": "<p><a href=\"/florent31\">@florent31</a> You have made a very good point here. This does make sense!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 558592,
      "author_name": "snovik1975",
      "author_url": "",
      "post_date": "06/22/2019 16:12:46",
      "content": "<p>It is not the standard deviation that counts but the MAE of your prediction mistakes per type. And give that there is a log after, proportionally large mistakes have smaller impact on the final result.\nMy point is that which type to optimize really depends on the prediction errors and some other factors. There is no golden rule for that</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 561481,
      "author_name": "larsbratholm",
      "author_url": "",
      "post_date": "06/26/2019 13:26:51",
      "content": "<p>Halving the MAE for 1JHC should have the same effect on the score as halving the MAE for 2JHN, no? </p>\n\n<p>log(0.5 * MAE_1JHC) + log(MAE_2JHN) ==  log(MAE_1JHC) + log(0.5 * MAE_2JHN)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 561697,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "06/26/2019 16:48:01",
      "content": "<p>it is probably for this reason that you should train separate prediction for different J coupling type. The \"scale of the error is difference\". When trained to together, reduction of loss of one type may inhibits another.</p>\n\n<p><code>\nAs a conclusion, it seems relevant to focus efforts on improving accuracy of models for types where scalarcouplingconstant standard deviation is high instead of the ones where it is low.\n</code></p>\n\n<p>you may note that the winning team would be the one that can reduce the high variance loss (e.g. 18.3 1JHC), when most team cannot. High precision regression (e.g. 0.01 out of 18.3, normalised error of 1e-4) is \"statistically difficult\" but may be \"made easy by some tricks\" or \"better loss function\"</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "557530": "Rules state that score improvement for different types will have the same effect on the score:\n*\"1% decrease in MAE for one type provides the same improvement in score as a 1% decrease for another type\"*\n\nHowever, according to the provided formula, it seems that the same **decrease** in MAE between 2 types would have the same impact on the score, but it doesn't work for the **% decrease**\n\nFurthermore, standard deviation of scalar_coupling_constant is not the same for different types. Actually, it is very different, and goes from 1.3 (3JHN) to 18.3 (3JHC).\nIf we expect the MAE difference between groups to be in the same order of magnitude than the standard deviation difference between groups, we will have a factor 10 between MAE of 3JHN and MAE of 3JHC.\n\nAnd according to the scoring formula, this factor 10 will be directly reflected in the score calculation.\n\nAs a conclusion, it seems relevant to focus efforts on improving accuracy of models for types where scalar_coupling_constant standard deviation is high instead of the ones where it is low.",
    "557738": "florent31 You have made a very good point here. This does make sense!",
    "558592": "It is not the standard deviation that counts but the MAE of your prediction mistakes per type. And give that there is a log after, proportionally large mistakes have smaller impact on the final result.\nMy point is that which type to optimize really depends on the prediction errors and some other factors. There is no golden rule for that",
    "561481": "Halving the MAE for 1JHC should have the same effect on the score as halving the MAE for 2JHN, no? \n\nlog(0.5 * MAE\\_1JHC) + log(MAE\\_2JHN) ==  log(MAE\\_1JHC) + log(0.5 * MAE\\_2JHN)",
    "561697": "it is probably for this reason that you should train separate prediction for different J coupling type. The \"scale of the error is difference\". When trained to together, reduction of loss of one type may inhibits another.\n\n\n```\nAs a conclusion, it seems relevant to focus efforts on improving accuracy of models for types where scalarcouplingconstant standard deviation is high instead of the ones where it is low.\n```\n\nyou may note that the winning team would be the one that can reduce the high variance loss (e.g. 18.3 1JHC), when most team cannot. High precision regression (e.g. 0.01 out of 18.3, normalised error of 1e-4) is \"statistically difficult\" but may be \"made easy by some tricks\" or \"better loss function\""
  },
  "source": "meta"
}