{
  "id": 396912,
  "title": "Evaluation metric",
  "url": "/competitions/benetech-making-graphs-accessible/discussion/396912",
  "author_name": "",
  "post_date": "2023-03-23T09:56:59.570425500Z",
  "votes": 3,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Suppose we have a categorical series and we exactly predict the series values.  That means all the Levenshtein distances are 0.  So, NLev = sigma(0) = 2 - 2/(1-e^(-0)) = 2 - 2/(1- 1) = 2 - 2/0.  This seems to give either infinity or negative infinity (depending on how you interpret the sign of 0) which seems wrong.  The text on the evaluation seems to suggest the optimal value should be 1.  The wrong length of series is assigned a value of 0. So I was expecting 0 &lt;= NLev &lt;= 1.  Also, as our predictions get worse (Levenshtein distance increases) x gets larger, so sigma(x) gets larger with a max value of sigma(infty) = 2 - 2/(1-0) = 2 - 2 = 0.</p>\n<p>I'm wondering if the sigma function should be sigma(x) = 2 - 2/(1+e^(-x)).  (Change the subtraction in the denominator to addition.)  This change would give sigma(0) = 2 - 2/(1+1) = 2 - 1 = 1.  I'm guessing that's what we want since 0 as the argument for sigma would be a perfect prediction (in either the numerical or categorical case).  Larger values of x would result in smaller values of sigma(x).  We would also get sigma(infty) = 2 - 2/(1+e^(-infty)) = 2 - 2/(1+0) = 2 - 2/1 = 2-2 = 0.  So, then 0 &lt;= NLev &lt;= 1 and 0 &lt;= NRMSE &lt;= 1.</p>\n<p>Or am I misinterpreting the evaluation metric?</p>",
  "messages": [
    {
      "id": "2193447",
      "postDate": "03/23/2023 09:56:59",
      "content": "<p>Suppose we have a categorical series and we exactly predict the series values.  That means all the Levenshtein distances are 0.  So, NLev = sigma(0) = 2 - 2/(1-e^(-0)) = 2 - 2/(1- 1) = 2 - 2/0.  This seems to give either infinity or negative infinity (depending on how you interpret the sign of 0) which seems wrong.  The text on the evaluation seems to suggest the optimal value should be 1.  The wrong length of series is assigned a value of 0. So I was expecting 0 &lt;= NLev &lt;= 1.  Also, as our predictions get worse (Levenshtein distance increases) x gets larger, so sigma(x) gets larger with a max value of sigma(infty) = 2 - 2/(1-0) = 2 - 2 = 0.</p>\n<p>I'm wondering if the sigma function should be sigma(x) = 2 - 2/(1+e^(-x)).  (Change the subtraction in the denominator to addition.)  This change would give sigma(0) = 2 - 2/(1+1) = 2 - 1 = 1.  I'm guessing that's what we want since 0 as the argument for sigma would be a perfect prediction (in either the numerical or categorical case).  Larger values of x would result in smaller values of sigma(x).  We would also get sigma(infty) = 2 - 2/(1+e^(-infty)) = 2 - 2/(1+0) = 2 - 2/1 = 2-2 = 0.  So, then 0 &lt;= NLev &lt;= 1 and 0 &lt;= NRMSE &lt;= 1.</p>\n<p>Or am I misinterpreting the evaluation metric?</p>",
      "rawMarkdown": "Suppose we have a categorical series and we exactly predict the series values.  That means all the Levenshtein distances are 0.  So, NLev = sigma(0) = 2 - 2/(1-e^(-0)) = 2 - 2/(1- 1) = 2 - 2/0.  This seems to give either infinity or negative infinity (depending on how you interpret the sign of 0) which seems wrong.  The text on the evaluation seems to suggest the optimal value should be 1.  The wrong length of series is assigned a value of 0. So I was expecting 0 <= NLev <= 1.  Also, as our predictions get worse (Levenshtein distance increases) x gets larger, so sigma(x) gets larger with a max value of sigma(infty) = 2 - 2/(1-0) = 2 - 2 = 0.\n\nI'm wondering if the sigma function should be sigma(x) = 2 - 2/(1+e^(-x)).  (Change the subtraction in the denominator to addition.)  This change would give sigma(0) = 2 - 2/(1+1) = 2 - 1 = 1.  I'm guessing that's what we want since 0 as the argument for sigma would be a perfect prediction (in either the numerical or categorical case).  Larger values of x would result in smaller values of sigma(x).  We would also get sigma(infty) = 2 - 2/(1+e^(-infty)) = 2 - 2/(1+0) = 2 - 2/1 = 2-2 = 0.  So, then 0 <= NLev <= 1 and 0 <= NRMSE <= 1.\n\nOr am I misinterpreting the evaluation metric?",
      "votes": null
    },
    {
      "id": "2193663",
      "postDate": "03/23/2023 12:05:06",
      "content": "<p>Hey <a href=\"https://www.kaggle.com/pabyrnes\" target=\"_blank\">@pabyrnes</a>, You are correct. That was a typo on the description page. The actual metric has the <code>2 - 2/(1 + e^(-x))</code> version. Thanks for heads up!</p>",
      "rawMarkdown": "Hey @pabyrnes, You are correct. That was a typo on the description page. The actual metric has the `2 - 2/(1 + e^(-x))` version. Thanks for heads up!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2193663,
      "author_name": "ryanholbrook",
      "author_url": "",
      "post_date": "03/23/2023 12:05:06",
      "content": "<p>Hey <a href=\"https://www.kaggle.com/pabyrnes\" target=\"_blank\">@pabyrnes</a>, You are correct. That was a typo on the description page. The actual metric has the <code>2 - 2/(1 + e^(-x))</code> version. Thanks for heads up!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2193447": "Suppose we have a categorical series and we exactly predict the series values.  That means all the Levenshtein distances are 0.  So, NLev = sigma(0) = 2 - 2/(1-e^(-0)) = 2 - 2/(1- 1) = 2 - 2/0.  This seems to give either infinity or negative infinity (depending on how you interpret the sign of 0) which seems wrong.  The text on the evaluation seems to suggest the optimal value should be 1.  The wrong length of series is assigned a value of 0. So I was expecting 0 <= NLev <= 1.  Also, as our predictions get worse (Levenshtein distance increases) x gets larger, so sigma(x) gets larger with a max value of sigma(infty) = 2 - 2/(1-0) = 2 - 2 = 0.\n\nI'm wondering if the sigma function should be sigma(x) = 2 - 2/(1+e^(-x)).  (Change the subtraction in the denominator to addition.)  This change would give sigma(0) = 2 - 2/(1+1) = 2 - 1 = 1.  I'm guessing that's what we want since 0 as the argument for sigma would be a perfect prediction (in either the numerical or categorical case).  Larger values of x would result in smaller values of sigma(x).  We would also get sigma(infty) = 2 - 2/(1+e^(-infty)) = 2 - 2/(1+0) = 2 - 2/1 = 2-2 = 0.  So, then 0 <= NLev <= 1 and 0 <= NRMSE <= 1.\n\nOr am I misinterpreting the evaluation metric?",
    "2193663": "Hey @pabyrnes, You are correct. That was a typo on the description page. The actual metric has the `2 - 2/(1 + e^(-x))` version. Thanks for heads up!"
  },
  "source": "meta"
}