{
  "id": 299035,
  "title": "Is this a good MSE with a LGBM model?",
  "url": "/competitions/g-research-crypto-forecasting/discussion/299035",
  "author_name": "",
  "post_date": "2022-01-05T21:33:53.212313400Z",
  "votes": -1,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Hello,</p>\n<p>I used a LGBM model and got an MSE of <strong>3.98e-06</strong> on my validation set.</p>\n<p>Is that good?   Did anyone get a lower number? </p>",
  "messages": [
    {
      "id": "1639734",
      "postDate": "01/05/2022 21:33:53",
      "content": "<p>Hello,</p>\n<p>I used a LGBM model and got an MSE of <strong>3.98e-06</strong> on my validation set.</p>\n<p>Is that good?   Did anyone get a lower number? </p>",
      "rawMarkdown": "Hello,\n\nI used a LGBM model and got an MSE of **3.98e-06** on my validation set.\n\nIs that good?   Did anyone get a lower number?",
      "votes": null
    },
    {
      "id": "1640200",
      "postDate": "01/06/2022 09:21:19",
      "content": "<p>Difficult to say, did you run your val set through the weighted correlation?</p>",
      "rawMarkdown": "Difficult to say, did you run your val set through the weighted correlation?",
      "votes": null
    },
    {
      "id": "1640442",
      "postDate": "01/06/2022 13:42:24",
      "content": "<p><a href=\"https://www.kaggle.com/jonnydossantos\" target=\"_blank\">@jonnydossantos</a>  I'm not sure I know how to run my validation set through the weighted correlation. All I did was a simple train test split, and train a model.  Then I predicted the Target with an LGBM, and noted the MSE. </p>",
      "rawMarkdown": "jonnydossantos  I'm not sure I know how to run my validation set through the weighted correlation. All I did was a simple train test split, and train a model.  Then I predicted the Target with an LGBM, and noted the MSE.",
      "votes": null
    },
    {
      "id": "1640464",
      "postDate": "01/06/2022 13:59:47",
      "content": "<p>No worries, makes sure you have the below in your code:</p>\n<p>def weighted_correlation(a, b, weights):<br>\n    w = np.ravel(weights)<br>\n    a = np.ravel(a)<br>\n    b = np.ravel(b)</p>\n<pre><code>sum_w = np.sum(w)\nmean_a = np.sum(a * w) / sum_w\nmean_b = np.sum(b * w) / sum_w\nvar_a = np.sum(w * np.square(a - mean_a)) / sum_w\nvar_b = np.sum(w * np.square(b - mean_b)) / sum_w\n\ncov = np.sum((a * b * w)) / np.sum(w) - mean_a * mean_b\ncorr = cov / np.sqrt(var_a * var_b)\n\nreturn corr\n</code></pre>\n<p>then predict your val using LGBM so</p>\n<p>val_predict = lgbmregressormodel.fit(val_set) pass this through to </p>\n<p>weighted_correlation(val['Target'], val_predict, val['Weight'])</p>\n<p>This will give you the competition metric for your validation set.  Anything north of 0.03 ish would be fairly good.</p>\n<p>Good luck</p>",
      "rawMarkdown": "No worries, makes sure you have the below in your code:\n\ndef weighted_correlation(a, b, weights):\n    w = np.ravel(weights)\n    a = np.ravel(a)\n    b = np.ravel(b)\n\n    sum_w = np.sum(w)\n    mean_a = np.sum(a * w) / sum_w\n    mean_b = np.sum(b * w) / sum_w\n    var_a = np.sum(w * np.square(a - mean_a)) / sum_w\n    var_b = np.sum(w * np.square(b - mean_b)) / sum_w\n\n    cov = np.sum((a * b * w)) / np.sum(w) - mean_a * mean_b\n    corr = cov / np.sqrt(var_a * var_b)\n\n    return corr\n\nthen predict your val using LGBM so\n\nval_predict = lgbmregressormodel.fit(val_set) pass this through to \n\nweighted_correlation(val['Target'], val_predict, val['Weight'])\n\nThis will give you the competition metric for your validation set.  Anything north of 0.03 ish would be fairly good.\n\nGood luck",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1640200,
      "author_name": "jonnydossantos",
      "author_url": "",
      "post_date": "01/06/2022 09:21:19",
      "content": "<p>Difficult to say, did you run your val set through the weighted correlation?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1640442,
      "author_name": "jonathansilverman",
      "author_url": "",
      "post_date": "01/06/2022 13:42:24",
      "content": "<p><a href=\"https://www.kaggle.com/jonnydossantos\" target=\"_blank\">@jonnydossantos</a>  I'm not sure I know how to run my validation set through the weighted correlation. All I did was a simple train test split, and train a model.  Then I predicted the Target with an LGBM, and noted the MSE. </p>",
      "votes": null,
      "replies": [
        {
          "id": 1640464,
          "author_name": "jonnydossantos",
          "author_url": "",
          "post_date": "01/06/2022 13:59:47",
          "content": "<p>No worries, makes sure you have the below in your code:</p>\n<p>def weighted_correlation(a, b, weights):<br>\n    w = np.ravel(weights)<br>\n    a = np.ravel(a)<br>\n    b = np.ravel(b)</p>\n<pre><code>sum_w = np.sum(w)\nmean_a = np.sum(a * w) / sum_w\nmean_b = np.sum(b * w) / sum_w\nvar_a = np.sum(w * np.square(a - mean_a)) / sum_w\nvar_b = np.sum(w * np.square(b - mean_b)) / sum_w\n\ncov = np.sum((a * b * w)) / np.sum(w) - mean_a * mean_b\ncorr = cov / np.sqrt(var_a * var_b)\n\nreturn corr\n</code></pre>\n<p>then predict your val using LGBM so</p>\n<p>val_predict = lgbmregressormodel.fit(val_set) pass this through to </p>\n<p>weighted_correlation(val['Target'], val_predict, val['Weight'])</p>\n<p>This will give you the competition metric for your validation set.  Anything north of 0.03 ish would be fairly good.</p>\n<p>Good luck</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1639734": "Hello,\n\nI used a LGBM model and got an MSE of **3.98e-06** on my validation set.\n\nIs that good?   Did anyone get a lower number?",
    "1640200": "Difficult to say, did you run your val set through the weighted correlation?",
    "1640442": "jonnydossantos  I'm not sure I know how to run my validation set through the weighted correlation. All I did was a simple train test split, and train a model.  Then I predicted the Target with an LGBM, and noted the MSE.",
    "1640464": "No worries, makes sure you have the below in your code:\n\ndef weighted_correlation(a, b, weights):\n    w = np.ravel(weights)\n    a = np.ravel(a)\n    b = np.ravel(b)\n\n    sum_w = np.sum(w)\n    mean_a = np.sum(a * w) / sum_w\n    mean_b = np.sum(b * w) / sum_w\n    var_a = np.sum(w * np.square(a - mean_a)) / sum_w\n    var_b = np.sum(w * np.square(b - mean_b)) / sum_w\n\n    cov = np.sum((a * b * w)) / np.sum(w) - mean_a * mean_b\n    corr = cov / np.sqrt(var_a * var_b)\n\n    return corr\n\nthen predict your val using LGBM so\n\nval_predict = lgbmregressormodel.fit(val_set) pass this through to \n\nweighted_correlation(val['Target'], val_predict, val['Weight'])\n\nThis will give you the competition metric for your validation set.  Anything north of 0.03 ish would be fairly good.\n\nGood luck"
  },
  "source": "meta"
}