{
  "id": 94237,
  "title": "Split models lift",
  "url": "/competitions/champs-scalar-coupling/discussion/94237",
  "author_name": "",
  "post_date": "2019-06-03T11:27:06.350294Z",
  "votes": 22,
  "comment_count": 11,
  "views": 0,
  "content": "<p>Hello,</p>\n\n<p>I just wanted to share that training one model for types <code>1JHC</code> and <code>1JHN</code>, and another one for the rest of the types gave me a 0.1 lift. Specifically I went from -0.291 to -0.396. I did this because I noticed that <code>type</code> was the feature with the highest gain according to LightGBM. Moreover the first split of each tree was nearly always done based on values from <code>type</code>.</p>",
  "messages": [
    {
      "id": "542044",
      "postDate": "06/03/2019 11:27:06",
      "content": "<p>Hello,</p>\n\n<p>I just wanted to share that training one model for types <code>1JHC</code> and <code>1JHN</code>, and another one for the rest of the types gave me a 0.1 lift. Specifically I went from -0.291 to -0.396. I did this because I noticed that <code>type</code> was the feature with the highest gain according to LightGBM. Moreover the first split of each tree was nearly always done based on values from <code>type</code>.</p>",
      "rawMarkdown": "Hello,\n\nI just wanted to share that training one model for types `1JHC` and `1JHN`, and another one for the rest of the types gave me a 0.1 lift. Specifically I went from -0.291 to -0.396. I did this because I noticed that `type` was the feature with the highest gain according to LightGBM. Moreover the first split of each tree was nearly always done based on values from `type`.",
      "votes": null
    },
    {
      "id": "542057",
      "postDate": "06/03/2019 11:54:12",
      "content": "<p>Is it feasible to train 8 models for each type and then blend them? Or at least the 1J, 2J and 3J couplings? Sorry if this is a naive question... I'm not a ML expert.</p>",
      "rawMarkdown": "Is it feasible to train 8 models for each type and then blend them? Or at least the 1J, 2J and 3J couplings? Sorry if this is a naive question... I'm not a ML expert.",
      "votes": null
    },
    {
      "id": "542065",
      "postDate": "06/03/2019 12:04:37",
      "content": "<p>You could, but my experience tells me this won't help. The reason why I did the split the way I mentioned was because types <code>1JHC</code> and <code>1JHN</code> have a very different target distribution. The rest of the types have somewhat similar target distributions.</p>\n\n<p>I could be wrong though. What's more, everything is worth trying considering there's still 3 months left!</p>",
      "rawMarkdown": "You could, but my experience tells me this won't help. The reason why I did the split the way I mentioned was because types `1JHC` and `1JHN` have a very different target distribution. The rest of the types have somewhat similar target distributions.\n\nI could be wrong though. What's more, everything is worth trying considering there's still 3 months left!",
      "votes": null
    },
    {
      "id": "542067",
      "postDate": "06/03/2019 12:07:31",
      "content": "<p>Interesting.  Thanks, Max.</p>",
      "rawMarkdown": "Interesting.  Thanks, Max.",
      "votes": null
    },
    {
      "id": "542391",
      "postDate": "06/03/2019 20:14:55",
      "content": "<p>i got a similar amount of lift, but i split my model into 8 groups - one for each type. </p>",
      "rawMarkdown": "i got a similar amount of lift, but i split my model into 8 groups - one for each type.",
      "votes": null
    },
    {
      "id": "542394",
      "postDate": "06/03/2019 20:31:14",
      "content": "<p>Good to know! Thanks.</p>",
      "rawMarkdown": "Good to know! Thanks.",
      "votes": null
    },
    {
      "id": "543955",
      "postDate": "06/05/2019 00:57:28",
      "content": "<p>Thank you this is a great tip! I will try this out on my model and report back.</p>",
      "rawMarkdown": "Thank you this is a great tip! I will try this out on my model and report back.",
      "votes": null
    },
    {
      "id": "545925",
      "postDate": "06/06/2019 03:48:50",
      "content": "<p>Thanks for this advice!!!\nI will try it at my end.</p>",
      "rawMarkdown": "Thanks for this advice!!!\nI will try it at my end.",
      "votes": null
    },
    {
      "id": "546797",
      "postDate": "06/06/2019 22:23:05",
      "content": "<p>I'm just creating 8 models and datasets for each of the 8 types since the features we're looking for will vary according to the relevant chemistry. So far I've just been working on 3JHH since that seems to give people the largest error. My MAE is currently around 0.6 for it. Out of curiousity, what is everyone else's error for 3JHH?</p>",
      "rawMarkdown": "I'm just creating 8 models and datasets for each of the 8 types since the features we're looking for will vary according to the relevant chemistry. So far I've just been working on 3JHH since that seems to give people the largest error. My MAE is currently around 0.6 for it. Out of curiousity, what is everyone else's error for 3JHH?",
      "votes": null
    },
    {
      "id": "564538",
      "postDate": "06/29/2019 15:14:43",
      "content": "<p>So If I get it right, you create 8 different models for each type of molecule. Are you blending those models and using the blended model for inference, or are you using for each type of molecule the adequate model? </p>",
      "rawMarkdown": "So If I get it right, you create 8 different models for each type of molecule. Are you blending those models and using the blended model for inference, or are you using for each type of molecule the adequate model?",
      "votes": null
    },
    {
      "id": "564544",
      "postDate": "06/29/2019 15:25:25",
      "content": "<p>8 models are trained on separate subsets of train data and predict on separate subsets of test data. They can't be blended.</p>",
      "rawMarkdown": "8 models are trained on separate subsets of train data and predict on separate subsets of test data. They can't be blended.",
      "votes": null
    },
    {
      "id": "564591",
      "postDate": "06/29/2019 17:05:41",
      "content": "<p>a few comments:</p>\n\n<ol>\n<li>by splitting dataset, you will have less training samples for each coupling type</li>\n</ol>\n\n<p>2.it may be possible that some type are related, e.g. 1jhc,2jhc,3jhc</p>\n\n<ol>\n<li>in my experiments, some  performances best when trained alone, e.g. 3jhn local cv is about -2.5. however, some actually performs much worse when trained alone (maybe due to less train samples???)</li>\n</ol>",
      "rawMarkdown": "a few comments:\n\n1. by splitting dataset, you will have less training samples for each coupling type\n\n2.it may be possible that some type are related, e.g. 1jhc,2jhc,3jhc\n\n3. in my experiments, some  performances best when trained alone, e.g. 3jhn local cv is about -2.5. however, some actually performs much worse when trained alone (maybe due to less train samples???)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 542057,
      "author_name": "phucco",
      "author_url": "",
      "post_date": "06/03/2019 11:54:12",
      "content": "<p>Is it feasible to train 8 models for each type and then blend them? Or at least the 1J, 2J and 3J couplings? Sorry if this is a naive question... I'm not a ML expert.</p>",
      "votes": null,
      "replies": [
        {
          "id": 542065,
          "author_name": "maxhalford",
          "author_url": "",
          "post_date": "06/03/2019 12:04:37",
          "content": "<p>You could, but my experience tells me this won't help. The reason why I did the split the way I mentioned was because types <code>1JHC</code> and <code>1JHN</code> have a very different target distribution. The rest of the types have somewhat similar target distributions.</p>\n\n<p>I could be wrong though. What's more, everything is worth trying considering there's still 3 months left!</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 542067,
      "author_name": "abendorf",
      "author_url": "",
      "post_date": "06/03/2019 12:07:31",
      "content": "<p>Interesting.  Thanks, Max.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 542391,
      "author_name": "kublakahn",
      "author_url": "",
      "post_date": "06/03/2019 20:14:55",
      "content": "<p>i got a similar amount of lift, but i split my model into 8 groups - one for each type. </p>",
      "votes": null,
      "replies": [
        {
          "id": 542394,
          "author_name": "maxhalford",
          "author_url": "",
          "post_date": "06/03/2019 20:31:14",
          "content": "<p>Good to know! Thanks.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 543955,
      "author_name": "venkykrishna",
      "author_url": "",
      "post_date": "06/05/2019 00:57:28",
      "content": "<p>Thank you this is a great tip! I will try this out on my model and report back.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 545925,
      "author_name": "iamctr",
      "author_url": "",
      "post_date": "06/06/2019 03:48:50",
      "content": "<p>Thanks for this advice!!!\nI will try it at my end.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 546797,
      "author_name": "bigironsphere",
      "author_url": "",
      "post_date": "06/06/2019 22:23:05",
      "content": "<p>I'm just creating 8 models and datasets for each of the 8 types since the features we're looking for will vary according to the relevant chemistry. So far I've just been working on 3JHH since that seems to give people the largest error. My MAE is currently around 0.6 for it. Out of curiousity, what is everyone else's error for 3JHH?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 564538,
      "author_name": "rftexas",
      "author_url": "",
      "post_date": "06/29/2019 15:14:43",
      "content": "<p>So If I get it right, you create 8 different models for each type of molecule. Are you blending those models and using the blended model for inference, or are you using for each type of molecule the adequate model? </p>",
      "votes": null,
      "replies": [
        {
          "id": 564544,
          "author_name": "artgor",
          "author_url": "",
          "post_date": "06/29/2019 15:25:25",
          "content": "<p>8 models are trained on separate subsets of train data and predict on separate subsets of test data. They can't be blended.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 564591,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "06/29/2019 17:05:41",
      "content": "<p>a few comments:</p>\n\n<ol>\n<li>by splitting dataset, you will have less training samples for each coupling type</li>\n</ol>\n\n<p>2.it may be possible that some type are related, e.g. 1jhc,2jhc,3jhc</p>\n\n<ol>\n<li>in my experiments, some  performances best when trained alone, e.g. 3jhn local cv is about -2.5. however, some actually performs much worse when trained alone (maybe due to less train samples???)</li>\n</ol>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "542044": "Hello,\n\nI just wanted to share that training one model for types `1JHC` and `1JHN`, and another one for the rest of the types gave me a 0.1 lift. Specifically I went from -0.291 to -0.396. I did this because I noticed that `type` was the feature with the highest gain according to LightGBM. Moreover the first split of each tree was nearly always done based on values from `type`.",
    "542057": "Is it feasible to train 8 models for each type and then blend them? Or at least the 1J, 2J and 3J couplings? Sorry if this is a naive question... I'm not a ML expert.",
    "542065": "You could, but my experience tells me this won't help. The reason why I did the split the way I mentioned was because types `1JHC` and `1JHN` have a very different target distribution. The rest of the types have somewhat similar target distributions.\n\nI could be wrong though. What's more, everything is worth trying considering there's still 3 months left!",
    "542067": "Interesting.  Thanks, Max.",
    "542391": "i got a similar amount of lift, but i split my model into 8 groups - one for each type.",
    "542394": "Good to know! Thanks.",
    "543955": "Thank you this is a great tip! I will try this out on my model and report back.",
    "545925": "Thanks for this advice!!!\nI will try it at my end.",
    "546797": "I'm just creating 8 models and datasets for each of the 8 types since the features we're looking for will vary according to the relevant chemistry. So far I've just been working on 3JHH since that seems to give people the largest error. My MAE is currently around 0.6 for it. Out of curiousity, what is everyone else's error for 3JHH?",
    "564538": "So If I get it right, you create 8 different models for each type of molecule. Are you blending those models and using the blended model for inference, or are you using for each type of molecule the adequate model?",
    "564544": "8 models are trained on separate subsets of train data and predict on separate subsets of test data. They can't be blended.",
    "564591": "a few comments:\n\n1. by splitting dataset, you will have less training samples for each coupling type\n\n2.it may be possible that some type are related, e.g. 1jhc,2jhc,3jhc\n\n3. in my experiments, some  performances best when trained alone, e.g. 3jhn local cv is about -2.5. however, some actually performs much worse when trained alone (maybe due to less train samples???)"
  },
  "source": "meta"
}