{
  "id": 329482,
  "title": "Kendall tau TensorFlow loss",
  "url": "/competitions/AI4Code/discussion/329482",
  "author_name": "The Devastator",
  "post_date": "2022-06-07T00:01:08.761000",
  "votes": 5,
  "comment_count": 0,
  "views": 0,
  "content": "<h1>Kendall tau TensorFlow loss</h1>\n<p>Hi, </p>\n<p>I am trying to write a function to approximate the competition's metric in a loss function. <br>\nSo far I came up with the following implementation which I think might not be so far off as far as I understand the metric. </p>\n<pre><code>def kendall_tau_loss(labels, predictions):\n    n = BATCH_SIZE\n    kp = 0\n    kn = 0\n    for i in range(n):\n        for j in range(i + 1, n):\n            dp = tf.sign(predictions[i] - predictions[j])\n            c = tf.reduce_sum(tf.math.abs(labels[i] - labels[j]))\n            dsp = dp * tf.expand_dims(c, 0)\n            kp = kp + tf.sign(dsp)\n            kn = kn + tf.sign(-dsp)\n    tau = (kp - kn) / (n * (n - 1) / 2)\n    return tau\n</code></pre>\n<p>It does work and the model trains with it [locally, not submitted yet. CV results are on the way..] but there are some things to take into consideration.</p>\n<p>There are some differences between the two, the original definition of the competition's kendall tau computes the kendall tau between the ground truth and the prediction but it assumes that prediction is already in the form of the indices, so it can be applied directly whereas kendall tau in the loss function is just a loss function so it is for measuring the kendall tau between two indices vectors that get passed, so it just computes the loss by mini-batches [sort of]. </p>\n<p>To be honest, I don't expect much out of this since there had already been multiple attempts at implementing a correlation metric for loss function, <a href=\"https://www.kaggle.com/competitions/us-patent-phrase-to-phrase-matching/discussion/327159\" target=\"_blank\">here</a> and <a href=\"https://www.kaggle.com/competitions/us-patent-phrase-to-phrase-matching/discussion/319272\" target=\"_blank\">here</a> for example.</p>\n<p>But it is a cool experiment and I just want to know what this approach leads to :)<br>\nWhat do you think?</p>",
  "messages": [
    {
      "id": 1813521,
      "postDate": "2022-06-07T00:01:08.760Z",
      "content": "<h1>Kendall tau TensorFlow loss</h1>\n<p>Hi, </p>\n<p>I am trying to write a function to approximate the competition's metric in a loss function. <br>\nSo far I came up with the following implementation which I think might not be so far off as far as I understand the metric. </p>\n<pre><code>def kendall_tau_loss(labels, predictions):\n    n = BATCH_SIZE\n    kp = 0\n    kn = 0\n    for i in range(n):\n        for j in range(i + 1, n):\n            dp = tf.sign(predictions[i] - predictions[j])\n            c = tf.reduce_sum(tf.math.abs(labels[i] - labels[j]))\n            dsp = dp * tf.expand_dims(c, 0)\n            kp = kp + tf.sign(dsp)\n            kn = kn + tf.sign(-dsp)\n    tau = (kp - kn) / (n * (n - 1) / 2)\n    return tau\n</code></pre>\n<p>It does work and the model trains with it [locally, not submitted yet. CV results are on the way..] but there are some things to take into consideration.</p>\n<p>There are some differences between the two, the original definition of the competition's kendall tau computes the kendall tau between the ground truth and the prediction but it assumes that prediction is already in the form of the indices, so it can be applied directly whereas kendall tau in the loss function is just a loss function so it is for measuring the kendall tau between two indices vectors that get passed, so it just computes the loss by mini-batches [sort of]. </p>\n<p>To be honest, I don't expect much out of this since there had already been multiple attempts at implementing a correlation metric for loss function, <a href=\"https://www.kaggle.com/competitions/us-patent-phrase-to-phrase-matching/discussion/327159\" target=\"_blank\">here</a> and <a href=\"https://www.kaggle.com/competitions/us-patent-phrase-to-phrase-matching/discussion/319272\" target=\"_blank\">here</a> for example.</p>\n<p>But it is a cool experiment and I just want to know what this approach leads to :)<br>\nWhat do you think?</p>",
      "rawMarkdown": "# Kendall tau TensorFlow loss  \n\nHi, \n\nI am trying to write a function to approximate the competition's metric in a loss function. \nSo far I came up with the following implementation which I think might not be so far off as far as I understand the metric. \n\n```\ndef kendall_tau_loss(labels, predictions):\n    n = BATCH_SIZE\n    kp = 0\n    kn = 0\n    for i in range(n):\n        for j in range(i + 1, n):\n            dp = tf.sign(predictions[i] - predictions[j])\n            c = tf.reduce_sum(tf.math.abs(labels[i] - labels[j]))\n            dsp = dp * tf.expand_dims(c, 0)\n            kp = kp + tf.sign(dsp)\n            kn = kn + tf.sign(-dsp)\n    tau = (kp - kn) / (n * (n - 1) / 2)\n    return tau\n```\n\nIt does work and the model trains with it [locally, not submitted yet. CV results are on the way..] but there are some things to take into consideration.\n\nThere are some differences between the two, the original definition of the competition's kendall tau computes the kendall tau between the ground truth and the prediction but it assumes that prediction is already in the form of the indices, so it can be applied directly whereas kendall tau in the loss function is just a loss function so it is for measuring the kendall tau between two indices vectors that get passed, so it just computes the loss by mini-batches [sort of]. \n\nTo be honest, I don't expect much out of this since there had already been multiple attempts at implementing a correlation metric for loss function, [here](https://www.kaggle.com/competitions/us-patent-phrase-to-phrase-matching/discussion/327159) and [here](https://www.kaggle.com/competitions/us-patent-phrase-to-phrase-matching/discussion/319272) for example.\n\nBut it is a cool experiment and I just want to know what this approach leads to :)\nWhat do you think?\n\n",
      "votes": 5
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "1813521": "# Kendall tau TensorFlow loss  \n\nHi, \n\nI am trying to write a function to approximate the competition's metric in a loss function. \nSo far I came up with the following implementation which I think might not be so far off as far as I understand the metric. \n\n```\ndef kendall_tau_loss(labels, predictions):\n    n = BATCH_SIZE\n    kp = 0\n    kn = 0\n    for i in range(n):\n        for j in range(i + 1, n):\n            dp = tf.sign(predictions[i] - predictions[j])\n            c = tf.reduce_sum(tf.math.abs(labels[i] - labels[j]))\n            dsp = dp * tf.expand_dims(c, 0)\n            kp = kp + tf.sign(dsp)\n            kn = kn + tf.sign(-dsp)\n    tau = (kp - kn) / (n * (n - 1) / 2)\n    return tau\n```\n\nIt does work and the model trains with it [locally, not submitted yet. CV results are on the way..] but there are some things to take into consideration.\n\nThere are some differences between the two, the original definition of the competition's kendall tau computes the kendall tau between the ground truth and the prediction but it assumes that prediction is already in the form of the indices, so it can be applied directly whereas kendall tau in the loss function is just a loss function so it is for measuring the kendall tau between two indices vectors that get passed, so it just computes the loss by mini-batches [sort of]. \n\nTo be honest, I don't expect much out of this since there had already been multiple attempts at implementing a correlation metric for loss function, [here](https://www.kaggle.com/competitions/us-patent-phrase-to-phrase-matching/discussion/327159) and [here](https://www.kaggle.com/competitions/us-patent-phrase-to-phrase-matching/discussion/319272) for example.\n\nBut it is a cool experiment and I just want to know what this approach leads to :)\nWhat do you think?\n\n"
  }
}