{ "id": 403126, "title": "Question on NRMSE", "url": "/competitions/benetech-making-graphs-accessible/discussion/403126", "author_name": "", "post_date": "2023-04-21T10:39:15.435178800Z", "votes": 1, "comment_count": 1, "views": 0, "content": "

I've been playing around with the competition metric and I am not sure if the NRMSE's takes into consideration the scale of the data series values.

\n

The competition metric is defined as:

\n
 ():\n      -  / ( + np.exp(-x))\n\n\n ():\n    \n    \n     sigmoid(( - r2_score(y_true, y_pred)) ** )\n
\n

So for example, lets say for the sake of simplicity there are 3 points all of which their values are 100 and we predict a value of 90 for each:

\n
y_true = np.array([, , ])\ny_pred = np.array([, , ])\n\nnormalized_rmse(y_true, y_pred)\n\n
\n

On the other hand, lets say now we have also 3 points but all of which their values are 0.1 and we predict a value of 0.09 for each:

\n
y_true = np.array([, , ])\ny_pred = np.array([, , ])\n\nnormalized_rmse(y_true, y_pred)\n\n
\n

It seems that in this case data series with high values will get penalized harder only for the fact that they are on a different scale. In a way this would be unfair, as being accurate on data series with small ranges of values would not contribute to the overall metric.

\n

Is this behavior right @ryanholbrook ?

", "messages": [ { "id": "2229407", "postDate": "04/21/2023 10:39:15", "content": "

I've been playing around with the competition metric and I am not sure if the NRMSE's takes into consideration the scale of the data series values.

\n

The competition metric is defined as:

\n
 ():\n      -  / ( + np.exp(-x))\n\n\n ():\n    \n    \n     sigmoid(( - r2_score(y_true, y_pred)) ** )\n
\n

So for example, lets say for the sake of simplicity there are 3 points all of which their values are 100 and we predict a value of 90 for each:

\n
y_true = np.array([, , ])\ny_pred = np.array([, , ])\n\nnormalized_rmse(y_true, y_pred)\n\n
\n

On the other hand, lets say now we have also 3 points but all of which their values are 0.1 and we predict a value of 0.09 for each:

\n
y_true = np.array([, , ])\ny_pred = np.array([, , ])\n\nnormalized_rmse(y_true, y_pred)\n\n
\n

It seems that in this case data series with high values will get penalized harder only for the fact that they are on a different scale. In a way this would be unfair, as being accurate on data series with small ranges of values would not contribute to the overall metric.

\n

Is this behavior right @ryanholbrook ?

", "rawMarkdown": "I've been playing around with the competition metric and I am not sure if the NRMSE's takes into consideration the scale of the data series values.\n\nThe competition metric is defined as:\n```python\ndef sigmoid(x):\n return 2 - 2 / (1 + np.exp(-x))\n\n\ndef normalized_rmse(y_true, y_pred):\n # The argument to the sigmoid transform is equal to \n # rmse(y_true, y_pred) / rmse(y_true, np.mean(y_true))\n return sigmoid((1 - r2_score(y_true, y_pred)) ** 0.5)\n```\n\nSo for example, lets say for the sake of simplicity there are 3 points all of which their values are 100 and we predict a value of 90 for each:\n```python\ny_true = np.array([100, 100, 100])\ny_pred = np.array([90, 90, 90])\n\nnormalized_rmse(y_true, y_pred)\n>>> 0.53788\n```\n\nOn the other hand, lets say now we have also 3 points but all of which their values are 0.1 and we predict a value of 0.09 for each:\n```python\ny_true = np.array([0.100, 0.100, 0.100])\ny_pred = np.array([0.090, 0.090, 0.090])\n\nnormalized_rmse(y_true, y_pred)\n>>> 0.0\n```\n\nIt seems that in this case data series with high values will get penalized harder only for the fact that they are on a different scale. In a way this would be unfair, as being accurate on data series with small ranges of values would not contribute to the overall metric. \n\nIs this behavior right @ryanholbrook ?", "votes": null }, { "id": "2229426", "postDate": "04/21/2023 10:59:26", "content": "

Sorry @ryanholbrook it seems I was just unlucky with the example I chose, as the series is a constant series:

\n
\n

In the particular case when y_true is constant, the score is not finite: it is either NaN (perfect predictions) or -Inf (imperfect predictions).

\n
\n

The metric performs well in other cases. It's very unlikely to have a constant series in the images (unless maybe a constant line).

", "rawMarkdown": "Sorry @ryanholbrook it seems I was just unlucky with the example I chose, as the series is a constant series:\n\n> In the particular case when y_true is constant, the score is not finite: it is either NaN (perfect predictions) or -Inf (imperfect predictions).\n\nThe metric performs well in other cases. It's very unlikely to have a constant series in the images (unless maybe a constant line).", "votes": null } ], "comments": [ { "id": 2229426, "author_name": "alejopaullier", "author_url": "", "post_date": "04/21/2023 10:59:26", "content": "

Sorry @ryanholbrook it seems I was just unlucky with the example I chose, as the series is a constant series:

\n
\n

In the particular case when y_true is constant, the score is not finite: it is either NaN (perfect predictions) or -Inf (imperfect predictions).

\n
\n

The metric performs well in other cases. It's very unlikely to have a constant series in the images (unless maybe a constant line).

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2229407": "I've been playing around with the competition metric and I am not sure if the NRMSE's takes into consideration the scale of the data series values.\n\nThe competition metric is defined as:\n```python\ndef sigmoid(x):\n return 2 - 2 / (1 + np.exp(-x))\n\n\ndef normalized_rmse(y_true, y_pred):\n # The argument to the sigmoid transform is equal to \n # rmse(y_true, y_pred) / rmse(y_true, np.mean(y_true))\n return sigmoid((1 - r2_score(y_true, y_pred)) ** 0.5)\n```\n\nSo for example, lets say for the sake of simplicity there are 3 points all of which their values are 100 and we predict a value of 90 for each:\n```python\ny_true = np.array([100, 100, 100])\ny_pred = np.array([90, 90, 90])\n\nnormalized_rmse(y_true, y_pred)\n>>> 0.53788\n```\n\nOn the other hand, lets say now we have also 3 points but all of which their values are 0.1 and we predict a value of 0.09 for each:\n```python\ny_true = np.array([0.100, 0.100, 0.100])\ny_pred = np.array([0.090, 0.090, 0.090])\n\nnormalized_rmse(y_true, y_pred)\n>>> 0.0\n```\n\nIt seems that in this case data series with high values will get penalized harder only for the fact that they are on a different scale. In a way this would be unfair, as being accurate on data series with small ranges of values would not contribute to the overall metric. \n\nIs this behavior right @ryanholbrook ?", "2229426": "Sorry @ryanholbrook it seems I was just unlucky with the example I chose, as the series is a constant series:\n\n> In the particular case when y_true is constant, the score is not finite: it is either NaN (perfect predictions) or -Inf (imperfect predictions).\n\nThe metric performs well in other cases. It's very unlikely to have a constant series in the images (unless maybe a constant line)." }, "source": "meta" }