Hello Kaggle, I have been trying to replicate the evaluation metric for the competition. To understand how my model performs against the leaderboard (better CV calibration). Currently, I'm using RMSE and some implementation of the completion metric Weighted version of the Pearson Correlation Coefficient.
\n\n\nSubmissions are evaluated on a weighted version of the Pearson correlation coefficient. You can find additional details in the 'Prediction Details and Evaluation' section of this tutorial notebook.
\n
`def m(x, w):
\n \"\"\"Weighted Mean\"\"\"
\n return np.sum(x * w) / np.sum(w)
def cov(x, y, w):
\n \"\"\"Weighted Covariance\"\"\"
\n return np.sum(w * (x - m(x, w)) * (y - m(y, w))) / np.sum(w)
def corr(x, y, w):
\n \"\"\"Weighted Correlation\"\"\"
\n return cov(x, y, w) / np.sqrt(cov(x, x, w) * cov(y, y, w))`
The issue is that my model Score locally: 0.001606 (Using the function described), but in the leaderboard, I'm getting: 0.0179
\nAny recommendation to build a model metric that resembles the values of the LB closely
\nThanks
Hello Kaggle, I have been trying to replicate the evaluation metric for the competition. To understand how my model performs against the leaderboard (better CV calibration). Currently, I'm using RMSE and some implementation of the completion metric Weighted version of the Pearson Correlation Coefficient.
\n\n\nSubmissions are evaluated on a weighted version of the Pearson correlation coefficient. You can find additional details in the 'Prediction Details and Evaluation' section of this tutorial notebook.
\n
`def m(x, w):
\n \"\"\"Weighted Mean\"\"\"
\n return np.sum(x * w) / np.sum(w)
def cov(x, y, w):
\n \"\"\"Weighted Covariance\"\"\"
\n return np.sum(w * (x - m(x, w)) * (y - m(y, w))) / np.sum(w)
def corr(x, y, w):
\n \"\"\"Weighted Correlation\"\"\"
\n return cov(x, y, w) / np.sqrt(cov(x, x, w) * cov(y, y, w))`
The issue is that my model Score locally: 0.001606 (Using the function described), but in the leaderboard, I'm getting: 0.0179
\nAny recommendation to build a model metric that resembles the values of the LB closely
\nThanks
I'm doing some testing and I think is just the weighted average of Pearson correlations calculated separately. Also, you have to be sure you are not using test labels to train.
\nDon't bother I found the answer, thanks @yamqwe.
\n", "rawMarkdown": "I'm doing some testing and I think is just the weighted average of Pearson correlations calculated separately. Also, you have to be sure you are not using test labels to train.\n\nDon't bother I found the answer, thanks @yamqwe.\n\n[Look here](https://www.kaggle.com/yamqwe/what-if-we-just-submit-the-target-median)", "votes": null }, { "id": "1591932", "postDate": "11/22/2021 18:00:29", "content": "That is not that bad, my suggestion is consider using it and move on with other experiments :)
", "rawMarkdown": "That is not that bad, my suggestion is consider using it and move on with other experiments :)", "votes": null }, { "id": "1592087", "postDate": "11/22/2021 21:54:05", "content": "Asked the question in the official thread. No response so far.
", "rawMarkdown": "Asked the question in the official thread. No response so far.", "votes": null }, { "id": "1592272", "postDate": "11/23/2021 02:39:15", "content": "Thanks, @lucasmorin do you have the link to the official thread so I can follow for updates.
", "rawMarkdown": "Thanks, @lucasmorin do you have the link to the official thread so I can follow for updates.", "votes": null }, { "id": "1592274", "postDate": "11/23/2021 02:41:26", "content": "Thanks, @agenlu yes I'm only training on the strict dataset before 2021-06-13 00:00:00.
", "rawMarkdown": "Thanks, @agenlu yes I'm only training on the strict dataset before 2021-06-13 00:00:00.", "votes": null }, { "id": "1597088", "postDate": "11/27/2021 07:43:46", "content": "see my notebook What if we just submit the target median? 🤔.
\nIt simulates the leaderboard nearly perfectly
Thanks, @yamqwe reviewing it right now to see how I can apply some of the strategies to it
", "rawMarkdown": "Thanks, @yamqwe reviewing it right now to see how I can apply some of the strategies to it", "votes": null }, { "id": "1605547", "postDate": "12/04/2021 10:54:29", "content": "I already said it in the comment section
", "rawMarkdown": "I already said it in the comment section", "votes": null } ], "comments": [ { "id": 1591422, "author_name": "agenlu", "author_url": "", "post_date": "11/22/2021 10:30:05", "content": "I'm doing some testing and I think is just the weighted average of Pearson correlations calculated separately. Also, you have to be sure you are not using test labels to train.
\nDon't bother I found the answer, thanks @yamqwe.
\n", "votes": null, "replies": [ { "id": 1592274, "author_name": "cv13j0", "author_url": "", "post_date": "11/23/2021 02:41:26", "content": "Thanks, @agenlu yes I'm only training on the strict dataset before 2021-06-13 00:00:00.
", "votes": null, "replies": [] } ] }, { "id": 1591932, "author_name": "avivlevi815", "author_url": "", "post_date": "11/22/2021 18:00:29", "content": "That is not that bad, my suggestion is consider using it and move on with other experiments :)
", "votes": null, "replies": [] }, { "id": 1592087, "author_name": "lucasmorin", "author_url": "", "post_date": "11/22/2021 21:54:05", "content": "Asked the question in the official thread. No response so far.
", "votes": null, "replies": [ { "id": 1592272, "author_name": "cv13j0", "author_url": "", "post_date": "11/23/2021 02:39:15", "content": "Thanks, @lucasmorin do you have the link to the official thread so I can follow for updates.
", "votes": null, "replies": [] } ] }, { "id": 1597088, "author_name": "", "author_url": "", "post_date": "11/27/2021 07:43:46", "content": "see my notebook What if we just submit the target median? 🤔.
\nIt simulates the leaderboard nearly perfectly
Thanks, @yamqwe reviewing it right now to see how I can apply some of the strategies to it
", "votes": null, "replies": [] } ] }, { "id": 1605547, "author_name": "swaralipibose", "author_url": "", "post_date": "12/04/2021 10:54:29", "content": "I already said it in the comment section
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1591197": "Hello Kaggle, I have been trying to replicate the evaluation metric for the competition. To understand how my model performs against the leaderboard (better CV calibration). Currently, I'm using RMSE and some implementation of the completion metric Weighted version of the Pearson Correlation Coefficient.\n\n> Submissions are evaluated on a weighted version of the Pearson correlation coefficient. You can find additional details in the 'Prediction Details and Evaluation' section of this tutorial notebook.\n\n`def m(x, w):\n \"\"\"Weighted Mean\"\"\"\n return np.sum(x * w) / np.sum(w)\n\ndef cov(x, y, w):\n \"\"\"Weighted Covariance\"\"\"\n return np.sum(w * (x - m(x, w)) * (y - m(y, w))) / np.sum(w)\n\ndef corr(x, y, w):\n \"\"\"Weighted Correlation\"\"\"\n return cov(x, y, w) / np.sqrt(cov(x, x, w) * cov(y, y, w))`\n\nThe issue is that my model Score locally: 0.001606 (Using the function described), but in the leaderboard, I'm getting: 0.0179\n\nAny recommendation to build a model metric that resembles the values of the LB closely\nThanks", "1591422": "I'm doing some testing and I think is just the weighted average of Pearson correlations calculated separately. Also, you have to be sure you are not using test labels to train.\n\nDon't bother I found the answer, thanks @yamqwe.\n\n[Look here](https://www.kaggle.com/yamqwe/what-if-we-just-submit-the-target-median)", "1591932": "That is not that bad, my suggestion is consider using it and move on with other experiments :)", "1592087": "Asked the question in the official thread. No response so far.", "1592272": "Thanks, @lucasmorin do you have the link to the official thread so I can follow for updates.", "1592274": "Thanks, @agenlu yes I'm only training on the strict dataset before 2021-06-13 00:00:00.", "1597088": "see my notebook [What if we just submit the target median? 🤔](https://www.kaggle.com/yamqwe/what-if-we-just-submit-the-target-median).\nIt simulates the leaderboard nearly perfectly", "1605136": "Thanks, @yamqwe reviewing it right now to see how I can apply some of the strategies to it", "1605547": "I already said it in the comment section" }, "source": "meta" }