{
  "id": 299774,
  "title": "Custom lgbm loss?",
  "url": "/competitions/g-research-crypto-forecasting/discussion/299774",
  "author_name": "Lucas Morin",
  "post_date": "2022-01-09T21:53:33.447000",
  "votes": 5,
  "comment_count": 1,
  "views": 0,
  "content": "<p>I was trying to build a <strong>custom lgbm loss</strong>, but the lack of automatic differentiation imply that gradient and hessian are needed to build a custom lgbm loss. I didn't manage to build their formula by hand either. I tried getting trough it with a tf custom loss with automatic differentiation (see below). Not my proudest moment but it seems to work with basic tf loss. It doesn't work with the custom correlation loss. </p>\n<p>Tf keras custom loss  (see notebook here: <a href=\"https://www.kaggle.com/lucasmorin/online-fe-nn-loss-shap):\" target=\"_blank\">https://www.kaggle.com/lucasmorin/online-fe-nn-loss-shap):</a> </p>\n<pre><code>def wcorr_loss(y_true, y_pred, w):\n    x = tf.cast(y_true, tf.float32)\n    y = tf.cast(y_pred, tf.float32)\n    w = tf.cast(w, tf.float32)\n    wmx = K.sum(tf.multiply(x, w)) / K.sum(w)\n    wmy = K.sum(tf.multiply(y, w)) / K.sum(w)\n    xm, ym = x-wmx, y-wmy\n    tfwcovxy = K.sum( tf.multiply(tf.multiply(xm, w),  ym)) / K.sum(w)\n    tfwcovxx = K.sum( tf.multiply(tf.multiply(xm, w),  xm)) / K.sum(w)\n    tfwcovyy = K.sum( tf.multiply(tf.multiply(ym, w),  ym)) / K.sum(w)\n    r = tfwcovxy / K.sqrt(tf.multiply(tfwcovxx , tfwcovyy))\n    r = K.maximum(K.minimum(r, 1.0), -1.0)\n    return - r\n\ndef wcorr_fn(w):\n    def wcorr_eval(y_true, y_pred):\n        return wcorr_loss(y_true, y_pred, w) \n</code></pre>\n<p>and automatic differentiation:</p>\n<pre><code>def lgbm_custom_loss(y_hat, y_true):\n\n    y_hat = tf.Variable(y_hat)\n    y_true = tf.Variable(y_true)\n\n    with tf.GradientTape() as t1:\n        with tf.GradientTape() as t2:\n            L = wcorr_loss(y_true, y_pred, w)\n        grad = t2.gradient(L, y_hat)\n    hess = t1.gradient(grad, y_hat)\n\n    return grad.numpy(), hess.numpy()\n</code></pre>\n<p>Any idea how to build a custom lgbm loss?</p>",
  "messages": [
    {
      "id": 1643952,
      "postDate": "2022-01-09T21:53:33.447Z",
      "content": "<p>I was trying to build a <strong>custom lgbm loss</strong>, but the lack of automatic differentiation imply that gradient and hessian are needed to build a custom lgbm loss. I didn't manage to build their formula by hand either. I tried getting trough it with a tf custom loss with automatic differentiation (see below). Not my proudest moment but it seems to work with basic tf loss. It doesn't work with the custom correlation loss. </p>\n<p>Tf keras custom loss  (see notebook here: <a href=\"https://www.kaggle.com/lucasmorin/online-fe-nn-loss-shap):\" target=\"_blank\">https://www.kaggle.com/lucasmorin/online-fe-nn-loss-shap):</a> </p>\n<pre><code>def wcorr_loss(y_true, y_pred, w):\n    x = tf.cast(y_true, tf.float32)\n    y = tf.cast(y_pred, tf.float32)\n    w = tf.cast(w, tf.float32)\n    wmx = K.sum(tf.multiply(x, w)) / K.sum(w)\n    wmy = K.sum(tf.multiply(y, w)) / K.sum(w)\n    xm, ym = x-wmx, y-wmy\n    tfwcovxy = K.sum( tf.multiply(tf.multiply(xm, w),  ym)) / K.sum(w)\n    tfwcovxx = K.sum( tf.multiply(tf.multiply(xm, w),  xm)) / K.sum(w)\n    tfwcovyy = K.sum( tf.multiply(tf.multiply(ym, w),  ym)) / K.sum(w)\n    r = tfwcovxy / K.sqrt(tf.multiply(tfwcovxx , tfwcovyy))\n    r = K.maximum(K.minimum(r, 1.0), -1.0)\n    return - r\n\ndef wcorr_fn(w):\n    def wcorr_eval(y_true, y_pred):\n        return wcorr_loss(y_true, y_pred, w) \n</code></pre>\n<p>and automatic differentiation:</p>\n<pre><code>def lgbm_custom_loss(y_hat, y_true):\n\n    y_hat = tf.Variable(y_hat)\n    y_true = tf.Variable(y_true)\n\n    with tf.GradientTape() as t1:\n        with tf.GradientTape() as t2:\n            L = wcorr_loss(y_true, y_pred, w)\n        grad = t2.gradient(L, y_hat)\n    hess = t1.gradient(grad, y_hat)\n\n    return grad.numpy(), hess.numpy()\n</code></pre>\n<p>Any idea how to build a custom lgbm loss?</p>",
      "rawMarkdown": "I was trying to build a **custom lgbm loss**, but the lack of automatic differentiation imply that gradient and hessian are needed to build a custom lgbm loss. I didn't manage to build their formula by hand either. I tried getting trough it with a tf custom loss with automatic differentiation (see below). Not my proudest moment but it seems to work with basic tf loss. It doesn't work with the custom correlation loss. \n\nTf keras custom loss  (see notebook here: https://www.kaggle.com/lucasmorin/online-fe-nn-loss-shap): \n\n```\ndef wcorr_loss(y_true, y_pred, w):\n    x = tf.cast(y_true, tf.float32)\n    y = tf.cast(y_pred, tf.float32)\n    w = tf.cast(w, tf.float32)\n    wmx = K.sum(tf.multiply(x, w)) / K.sum(w)\n    wmy = K.sum(tf.multiply(y, w)) / K.sum(w)\n    xm, ym = x-wmx, y-wmy\n    tfwcovxy = K.sum( tf.multiply(tf.multiply(xm, w),  ym)) / K.sum(w)\n    tfwcovxx = K.sum( tf.multiply(tf.multiply(xm, w),  xm)) / K.sum(w)\n    tfwcovyy = K.sum( tf.multiply(tf.multiply(ym, w),  ym)) / K.sum(w)\n    r = tfwcovxy / K.sqrt(tf.multiply(tfwcovxx , tfwcovyy))\n    r = K.maximum(K.minimum(r, 1.0), -1.0)\n    return - r\n\ndef wcorr_fn(w):\n    def wcorr_eval(y_true, y_pred):\n        return wcorr_loss(y_true, y_pred, w) \n```\n\nand automatic differentiation:\n\n```\ndef lgbm_custom_loss(y_hat, y_true):\n\n    y_hat = tf.Variable(y_hat)\n    y_true = tf.Variable(y_true)\n    \n    with tf.GradientTape() as t1:\n        with tf.GradientTape() as t2:\n            L = wcorr_loss(y_true, y_pred, w)\n        grad = t2.gradient(L, y_hat)\n    hess = t1.gradient(grad, y_hat)\n    \n    return grad.numpy(), hess.numpy()\n```\n\nAny idea how to build a custom lgbm loss?\n\n\n\n",
      "votes": 5
    },
    {
      "id": 1646860,
      "postDate": "2022-01-12T05:30:38.710Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 1646860,
      "author_name": "",
      "author_url": "",
      "post_date": "2022-01-12T05:30:38.710000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1643952": "I was trying to build a **custom lgbm loss**, but the lack of automatic differentiation imply that gradient and hessian are needed to build a custom lgbm loss. I didn't manage to build their formula by hand either. I tried getting trough it with a tf custom loss with automatic differentiation (see below). Not my proudest moment but it seems to work with basic tf loss. It doesn't work with the custom correlation loss. \n\nTf keras custom loss  (see notebook here: https://www.kaggle.com/lucasmorin/online-fe-nn-loss-shap): \n\n```\ndef wcorr_loss(y_true, y_pred, w):\n    x = tf.cast(y_true, tf.float32)\n    y = tf.cast(y_pred, tf.float32)\n    w = tf.cast(w, tf.float32)\n    wmx = K.sum(tf.multiply(x, w)) / K.sum(w)\n    wmy = K.sum(tf.multiply(y, w)) / K.sum(w)\n    xm, ym = x-wmx, y-wmy\n    tfwcovxy = K.sum( tf.multiply(tf.multiply(xm, w),  ym)) / K.sum(w)\n    tfwcovxx = K.sum( tf.multiply(tf.multiply(xm, w),  xm)) / K.sum(w)\n    tfwcovyy = K.sum( tf.multiply(tf.multiply(ym, w),  ym)) / K.sum(w)\n    r = tfwcovxy / K.sqrt(tf.multiply(tfwcovxx , tfwcovyy))\n    r = K.maximum(K.minimum(r, 1.0), -1.0)\n    return - r\n\ndef wcorr_fn(w):\n    def wcorr_eval(y_true, y_pred):\n        return wcorr_loss(y_true, y_pred, w) \n```\n\nand automatic differentiation:\n\n```\ndef lgbm_custom_loss(y_hat, y_true):\n\n    y_hat = tf.Variable(y_hat)\n    y_true = tf.Variable(y_true)\n    \n    with tf.GradientTape() as t1:\n        with tf.GradientTape() as t2:\n            L = wcorr_loss(y_true, y_pred, w)\n        grad = t2.gradient(L, y_hat)\n    hess = t1.gradient(grad, y_hat)\n    \n    return grad.numpy(), hess.numpy()\n```\n\nAny idea how to build a custom lgbm loss?\n\n\n\n",
    "1646860": ""
  }
}