{ "id": 396947, "title": "Competition Metric in Notebook", "url": "/competitions/benetech-making-graphs-accessible/discussion/396947", "author_name": "", "post_date": "2023-03-23T13:11:14.450049200Z", "votes": 15, "comment_count": 6, "views": 0, "content": "

For your convenience, I just added a Python implementation of the competition metric in this notebook: Benetech Competition Metric.

", "messages": [ { "id": "2193750", "postDate": "03/23/2023 13:11:14", "content": "

For your convenience, I just added a Python implementation of the competition metric in this notebook: Benetech Competition Metric.

", "rawMarkdown": "For your convenience, I just added a Python implementation of the competition metric in this notebook: [Benetech Competition Metric](https://www.kaggle.com/code/ryanholbrook/competition-metric-benetech-mixed-match).", "votes": null }, { "id": "2193798", "postDate": "03/23/2023 13:45:57", "content": "

@ryanholbrook thank you, but there is still 404 error, did you public the notebook?

", "rawMarkdown": "ryanholbrook thank you, but there is still 404 error, did you public the notebook?", "votes": null }, { "id": "2193807", "postDate": "03/23/2023 13:58:18", "content": "

Done! Sorry ๐Ÿ˜…

", "rawMarkdown": "Done! Sorry ๐Ÿ˜…", "votes": null }, { "id": "2196076", "postDate": "03/25/2023 06:08:20", "content": "

Hi @ryanholbrook,

\n

Many thanks for sharing the competition metric๐Ÿ˜Š However, I think there are a few cases where the metric computation is unclear:

\n\n
 ():\n     sigmoid(rmse(y_true, y_pred) / rmse(y_true, np.mean(y_true)))\n
\n

Some examples from training set: 116904a5e992, 8e71f5f4f2d2

\n\n

Additionally, if we predict non-float values for a numerical series, does it result in scoring error or it gets zero score?

\n

Thanks!

", "rawMarkdown": "Hi @ryanholbrook, \n\nMany thanks for sharing the competition metric๐Ÿ˜Š However, I think there are a few cases where the metric computation is unclear:\n\n* `y_ture` is a constant function e.g. y_ture = [1.0, 1.0]. The `normalized_rmse` will have nan values.\n```python\ndef normalized_rmse(y_true, y_pred):\n return sigmoid(rmse(y_true, y_pred) / rmse(y_true, np.mean(y_true)))\n```\n Some examples from training set: `116904a5e992`, `8e71f5f4f2d2`\n\n* A few ground truth series contain nan values e.g. `4163f70a77b3` where true y series is `[2, 3, 4, 7, 9, nan]`. They all relate to `vertical_bar` chart type (histogram). As per graph convention: `Bar charts may also appear as histograms. In this case, the x labels occur at the ends of each bar and the x data series will have one more value than the y data series.`. So, I think this is expected, but I'm not sure how this is handled in scoring function.\n\nAdditionally, if we predict non-float values for a `numerical` series, does it result in scoring error or it gets zero score?\n\nThanks!", "votes": null }, { "id": "2196286", "postDate": "03/25/2023 09:05:02", "content": "

Hi @conjuring92 ,

\n

Thanks for your questions. In the case of constant y_true, our actual metric will follow the sklearn.metrics.r2_score conventions of force_finite=True. Specifically, when y_true == y_pred the score is 1.0 and otherwise 0.0. When I get a chance, I'll update the notebook metric to account for this.

\n

The nan values should not be predicted. I think they are present as a side effect of the data series in both axes generally having equal numbers of values. A fully correct prediction for 4163f70a77b3 would be 2;3;4;7;9. I'll update the documentation to clarify this.

\n

If you predict non-numerical values for a numerical series, the scoring will error. However, the numerical / categorical distinction will depend on the chart type (per the Graph Conventions) and the score will be 0 if the chart type is wrong. (Dot plots are an exception. Perhaps we should check the data type in this case. ๐Ÿค”)

", "rawMarkdown": "Hi @conjuring92 ,\n\nThanks for your questions. In the case of constant `y_true`, our actual metric will follow the `sklearn.metrics.r2_score` conventions of `force_finite=True`. Specifically, when `y_true == y_pred` the score is `1.0` and otherwise `0.0`. When I get a chance, I'll update the notebook metric to account for this.\n\nThe `nan` values should not be predicted. I think they are present as a side effect of the data series in both axes generally having equal numbers of values. A fully correct prediction for `4163f70a77b3` would be `2;3;4;7;9`. I'll update the documentation to clarify this.\n\nIf you predict non-numerical values for a numerical series, the scoring will error. However, the numerical / categorical distinction will depend on the chart type (per the Graph Conventions) and the score will be 0 if the chart type is wrong. (Dot plots are an exception. Perhaps we should check the data type in this case. ๐Ÿค”)", "votes": null }, { "id": "2196555", "postDate": "03/25/2023 13:41:18", "content": "

Noted with thanks!

", "rawMarkdown": "Noted with thanks!", "votes": null }, { "id": "2203695", "postDate": "03/31/2023 03:57:55", "content": "

I want to participate in your competition metric group. I must try to write the code for this dataset.

", "rawMarkdown": "I want to participate in your competition metric group. I must try to write the code for this dataset.", "votes": null } ], "comments": [ { "id": 2193798, "author_name": "thanhhau097a", "author_url": "", "post_date": "03/23/2023 13:45:57", "content": "

@ryanholbrook thank you, but there is still 404 error, did you public the notebook?

", "votes": null, "replies": [ { "id": 2193807, "author_name": "ryanholbrook", "author_url": "", "post_date": "03/23/2023 13:58:18", "content": "

Done! Sorry ๐Ÿ˜…

", "votes": null, "replies": [] } ] }, { "id": 2196076, "author_name": "conjuring92", "author_url": "", "post_date": "03/25/2023 06:08:20", "content": "

Hi @ryanholbrook,

\n

Many thanks for sharing the competition metric๐Ÿ˜Š However, I think there are a few cases where the metric computation is unclear:

\n\n
 ():\n     sigmoid(rmse(y_true, y_pred) / rmse(y_true, np.mean(y_true)))\n
\n

Some examples from training set: 116904a5e992, 8e71f5f4f2d2

\n\n

Additionally, if we predict non-float values for a numerical series, does it result in scoring error or it gets zero score?

\n

Thanks!

", "votes": null, "replies": [ { "id": 2196286, "author_name": "ryanholbrook", "author_url": "", "post_date": "03/25/2023 09:05:02", "content": "

Hi @conjuring92 ,

\n

Thanks for your questions. In the case of constant y_true, our actual metric will follow the sklearn.metrics.r2_score conventions of force_finite=True. Specifically, when y_true == y_pred the score is 1.0 and otherwise 0.0. When I get a chance, I'll update the notebook metric to account for this.

\n

The nan values should not be predicted. I think they are present as a side effect of the data series in both axes generally having equal numbers of values. A fully correct prediction for 4163f70a77b3 would be 2;3;4;7;9. I'll update the documentation to clarify this.

\n

If you predict non-numerical values for a numerical series, the scoring will error. However, the numerical / categorical distinction will depend on the chart type (per the Graph Conventions) and the score will be 0 if the chart type is wrong. (Dot plots are an exception. Perhaps we should check the data type in this case. ๐Ÿค”)

", "votes": null, "replies": [ { "id": 2196555, "author_name": "conjuring92", "author_url": "", "post_date": "03/25/2023 13:41:18", "content": "

Noted with thanks!

", "votes": null, "replies": [] } ] } ] }, { "id": 2203695, "author_name": "hsuwai01", "author_url": "", "post_date": "03/31/2023 03:57:55", "content": "

I want to participate in your competition metric group. I must try to write the code for this dataset.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2193750": "For your convenience, I just added a Python implementation of the competition metric in this notebook: [Benetech Competition Metric](https://www.kaggle.com/code/ryanholbrook/competition-metric-benetech-mixed-match).", "2193798": "ryanholbrook thank you, but there is still 404 error, did you public the notebook?", "2193807": "Done! Sorry ๐Ÿ˜…", "2196076": "Hi @ryanholbrook, \n\nMany thanks for sharing the competition metric๐Ÿ˜Š However, I think there are a few cases where the metric computation is unclear:\n\n* `y_ture` is a constant function e.g. y_ture = [1.0, 1.0]. The `normalized_rmse` will have nan values.\n```python\ndef normalized_rmse(y_true, y_pred):\n return sigmoid(rmse(y_true, y_pred) / rmse(y_true, np.mean(y_true)))\n```\n Some examples from training set: `116904a5e992`, `8e71f5f4f2d2`\n\n* A few ground truth series contain nan values e.g. `4163f70a77b3` where true y series is `[2, 3, 4, 7, 9, nan]`. They all relate to `vertical_bar` chart type (histogram). As per graph convention: `Bar charts may also appear as histograms. In this case, the x labels occur at the ends of each bar and the x data series will have one more value than the y data series.`. So, I think this is expected, but I'm not sure how this is handled in scoring function.\n\nAdditionally, if we predict non-float values for a `numerical` series, does it result in scoring error or it gets zero score?\n\nThanks!", "2196286": "Hi @conjuring92 ,\n\nThanks for your questions. In the case of constant `y_true`, our actual metric will follow the `sklearn.metrics.r2_score` conventions of `force_finite=True`. Specifically, when `y_true == y_pred` the score is `1.0` and otherwise `0.0`. When I get a chance, I'll update the notebook metric to account for this.\n\nThe `nan` values should not be predicted. I think they are present as a side effect of the data series in both axes generally having equal numbers of values. A fully correct prediction for `4163f70a77b3` would be `2;3;4;7;9`. I'll update the documentation to clarify this.\n\nIf you predict non-numerical values for a numerical series, the scoring will error. However, the numerical / categorical distinction will depend on the chart type (per the Graph Conventions) and the score will be 0 if the chart type is wrong. (Dot plots are an exception. Perhaps we should check the data type in this case. ๐Ÿค”)", "2196555": "Noted with thanks!", "2203695": "I want to participate in your competition metric group. I must try to write the code for this dataset." }, "source": "meta" }