Hi Kagglers,
\nWe would like to clarify how the evaluation metric has been calculated to address your related questions.
\nThe evaluation metric is weighted correlation as opposed to a weighted mean of correlation.
\nThe metric is defined as follows, where 'a', 'b' and 'weights' are vectors of the same length.
\n'a' and 'b' are the expected and predicted targets, and ' weights' include the weight of each row, determined by its asset:
\ndef weighted_correlation(a, b, weights):\n\n w = np.ravel(weights)\n a = np.ravel(a)\n b = np.ravel(b)\n\n sum_w = np.sum(w)\n mean_a = np.sum(a * w) / sum_w\n mean_b = np.sum(b * w) / sum_w\n var_a = np.sum(w * np.square(a - mean_a)) / sum_w\n var_b = np.sum(w * np.square(b - mean_b)) / sum_w\n\n cov = np.sum((a * b * w)) / np.sum(w) - mean_a * mean_b\n corr = cov / np.sqrt(var_a * var_b)\n\n return corr\nThanks and happy Kaggling!
", "messages": [ { "id": 1601355, "postDate": "2021-12-01T07:39:54.923Z", "content": "Hi Kagglers,
\nWe would like to clarify how the evaluation metric has been calculated to address your related questions.
\nThe evaluation metric is weighted correlation as opposed to a weighted mean of correlation.
\nThe metric is defined as follows, where 'a', 'b' and 'weights' are vectors of the same length.
\n'a' and 'b' are the expected and predicted targets, and ' weights' include the weight of each row, determined by its asset:
\ndef weighted_correlation(a, b, weights):\n\n w = np.ravel(weights)\n a = np.ravel(a)\n b = np.ravel(b)\n\n sum_w = np.sum(w)\n mean_a = np.sum(a * w) / sum_w\n mean_b = np.sum(b * w) / sum_w\n var_a = np.sum(w * np.square(a - mean_a)) / sum_w\n var_b = np.sum(w * np.square(b - mean_b)) / sum_w\n\n cov = np.sum((a * b * w)) / np.sum(w) - mean_a * mean_b\n corr = cov / np.sqrt(var_a * var_b)\n\n return corr\nThanks and happy Kaggling!
", "rawMarkdown": "Hi Kagglers,\n\nWe would like to clarify how the evaluation metric has been calculated to address your related questions.\n\nThe evaluation metric is weighted correlation as opposed to a weighted mean of correlation. \n\nThe metric is defined as follows, where 'a', 'b' and 'weights' are vectors of the same length. \n\n'a' and 'b' are the expected and predicted targets, and ' weights' include the weight of each row, determined by its asset:\n\n```\ndef weighted_correlation(a, b, weights):\n\n w = np.ravel(weights)\n a = np.ravel(a)\n b = np.ravel(b)\n\n sum_w = np.sum(w)\n mean_a = np.sum(a * w) / sum_w\n mean_b = np.sum(b * w) / sum_w\n var_a = np.sum(w * np.square(a - mean_a)) / sum_w\n var_b = np.sum(w * np.square(b - mean_b)) / sum_w\n\n cov = np.sum((a * b * w)) / np.sum(w) - mean_a * mean_b\n corr = cov / np.sqrt(var_a * var_b)\n\n return corr\n```\n\nThanks and happy Kaggling!\n", "votes": 59 }, { "id": 1615082, "postDate": "2021-12-11T19:31:40.797Z", "content": "Isn't it the \"Pearson correlation coefficient\" - scipy.stats.pearsonr?
@kierangarvey, Thanks for the info. I suppose that if the targets are NAN, are not included in the calculation.
\nIt seems that it is clear now.
", "rawMarkdown": "@kierangarvey, Thanks for the info. I suppose that if the targets are NAN, are not included in the calculation.\n\nIt seems that it is clear now.", "votes": 1 }, { "id": 1668049, "postDate": "2022-01-29T08:43:30.703Z", "content": "Is not the mean of x, for example, is : sum(x)/n , where n = len(x)
\nand then if values in x are weighted by another vector w, then it becomes:
\nsum(x*w) / sum_w / n
\nwhere sum_w = sum(w), and sum(w/sum_w) = 1
\nIf so, then the calculation of mean_a & mean_b should be like
\nmean_a = np.sum(a * w) / sum_w / n
\nmean_b = np.sum(b * w) / sum_w / n
where n = len(a) = len(b) = len(w)
", "rawMarkdown": "Is not the mean of x, for example, is : sum(x)/n , where n = len(x)\n\nand then if values in x are weighted by another vector w, then it becomes:\n\nsum(x*w) / sum_w / n\n\nwhere sum_w = sum(w), and sum(w/sum_w) = 1\n\nIf so, then the calculation of mean_a & mean_b should be like \n\nmean_a = np.sum(a * w) / sum_w / **n**\nmean_b = np.sum(b * w) / sum_w / **n**\n\nwhere n = len(a) = len(b) = len(w)" }, { "id": 1653795, "postDate": "2022-01-17T21:36:13.590Z", "content": "très merci à vous!
", "rawMarkdown": "très merci à vous!" }, { "id": 1603203, "postDate": "2021-12-02T11:51:36.903Z", "content": "Pretty sure it does not work i measured the metric between the volume feature and the overlapped targets https://www.kaggle.com/swaralipibose/metric here is the code for that and also submitted it got different score the code is here https://www.kaggle.com/swaralipibose/detailed-api-introduction-300351 I might be wrong either
", "rawMarkdown": "Pretty sure it does not work i measured the metric between the volume feature and the overlapped targets https://www.kaggle.com/swaralipibose/metric here is the code for that and also submitted it got different score the code is here https://www.kaggle.com/swaralipibose/detailed-api-introduction-300351 I might be wrong either ", "replies": [ { "id": 1603398, "postDate": "2021-12-02T13:49:05.900Z", "content": "\nWhy do you use the correlation between Volume and Target?
\n\n\nweighted_correlation(test['Volume'],public_lb['Target'],test['Weight'])
\n
maybe you should use the target, no?
", "rawMarkdown": "@swaralipibose ,\n\nWhy do you use the correlation between Volume and Target?\n\n>weighted_correlation(test['Volume'],public_lb['Target'],test['Weight'])\n\nmaybe you should use the target, no?" }, { "id": 1603400, "postDate": "2021-12-02T13:50:37.330Z", "content": "This is just for example @agenlu just like random useless predictions but it does not matter as I am just comparing here
", "rawMarkdown": "This is just for example @agenlu just like random useless predictions but it does not matter as I am just comparing here" }, { "id": 1603571, "postDate": "2021-12-02T15:23:46.233Z", "content": "@swaralipibose, sorry, I didn’t read all your comment. Maybe the problem is that example_test doesn’t include all the public_lb
", "rawMarkdown": "@swaralipibose, sorry, I didn’t read all your comment. Maybe the problem is that example_test doesn’t include all the public_lb" }, { "id": 1604934, "postDate": "2021-12-03T21:21:35.783Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1605401, "postDate": "2021-12-04T07:32:22.093Z", "content": "@dmk2021, read this:
\nhttps://www.kaggle.com/c/g-research-crypto-forecasting/discussion/290265
\npublic LB is included in the train data.
", "rawMarkdown": "@dmk2021, read this:\n\n[https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/290265](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/290265)\n\npublic LB is included in the train data.\n" }, { "id": 1611459, "postDate": "2021-12-08T01:58:18.133Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1611806, "postDate": "2021-12-08T09:33:05.283Z", "content": "@dmk2021, example_test doesn’t include all the data in the public lb, is just an example on how the test data is going to be.
\nAs I have pointed out in the above link post, the public leaderboard has the train rows above 2021-06-13. So, if you want to make your experiments you need to cut the train data, as explained in the above post.
\nRead this:
\nhttps://www.kaggle.com/c/g-research-crypto-forecasting/discussion/290265
", "rawMarkdown": "@dmk2021, example_test doesn’t include all the data in the public lb, is just an example on how the test data is going to be.\n\nAs I have pointed out in the above link post, the public leaderboard has the train rows above 2021-06-13. So, if you want to make your experiments you need to cut the train data, as explained in the above post.\n\nRead this:\n\n\n[https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/290265](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/290265)\n" }, { "id": 1612098, "postDate": "2021-12-08T14:49:10Z", "rawMarkdown": "", "votes": 3, "isDeleted": true }, { "id": 1612519, "postDate": "2021-12-09T03:45:38.457Z", "content": "Interesting
", "rawMarkdown": "Interesting " }, { "id": 1613154, "postDate": "2021-12-09T17:06:55.203Z", "rawMarkdown": "", "votes": 2, "isDeleted": true }, { "id": 1652492, "postDate": "2022-01-16T17:20:48.463Z", "content": "yes, would be nice to have this clarified. Why are the organizers so secretive of this weight? They can just publish the code which calculates the weights.
", "rawMarkdown": "yes, would be nice to have this clarified. Why are the organizers so secretive of this weight? They can just publish the code which calculates the weights." } ] }, { "id": 1641296, "postDate": "2022-01-07T09:20:18.567Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1612357, "postDate": "2021-12-08T19:51:22.567Z", "content": "Thanks for the explanation!
", "rawMarkdown": "Thanks for the explanation!", "votes": 1 }, { "id": 1611955, "postDate": "2021-12-08T12:19:05.020Z", "content": "Thanks for your explanation.
\nThx
Isn't it the \"Pearson correlation coefficient\" - scipy.stats.pearsonr?
@kierangarvey, Thanks for the info. I suppose that if the targets are NAN, are not included in the calculation.
\nIt seems that it is clear now.
", "votes": 1, "replies": [] }, { "id": 1668049, "author_name": "Mohammad Tarek", "author_url": "", "post_date": "2022-01-29T08:43:30.703000", "content": "Is not the mean of x, for example, is : sum(x)/n , where n = len(x)
\nand then if values in x are weighted by another vector w, then it becomes:
\nsum(x*w) / sum_w / n
\nwhere sum_w = sum(w), and sum(w/sum_w) = 1
\nIf so, then the calculation of mean_a & mean_b should be like
\nmean_a = np.sum(a * w) / sum_w / n
\nmean_b = np.sum(b * w) / sum_w / n
where n = len(a) = len(b) = len(w)
", "votes": 0, "replies": [] }, { "id": 1653795, "author_name": "Hansheng LIANG", "author_url": "", "post_date": "2022-01-17T21:36:13.590000", "content": "très merci à vous!
", "votes": 0, "replies": [] }, { "id": 1603203, "author_name": "Sayantan Mazumdar", "author_url": "", "post_date": "2021-12-02T11:51:36.903000", "content": "Pretty sure it does not work i measured the metric between the volume feature and the overlapped targets https://www.kaggle.com/swaralipibose/metric here is the code for that and also submitted it got different score the code is here https://www.kaggle.com/swaralipibose/detailed-api-introduction-300351 I might be wrong either
", "votes": 0, "replies": [ { "id": 1603398, "author_name": "agenlu", "author_url": "", "post_date": "2021-12-02T13:49:05.900000", "content": "\nWhy do you use the correlation between Volume and Target?
\n\n\nweighted_correlation(test['Volume'],public_lb['Target'],test['Weight'])
\n
maybe you should use the target, no?
", "votes": 0, "replies": [] }, { "id": 1603400, "author_name": "Sayantan Mazumdar", "author_url": "", "post_date": "2021-12-02T13:50:37.330000", "content": "This is just for example @agenlu just like random useless predictions but it does not matter as I am just comparing here
", "votes": 0, "replies": [] }, { "id": 1603571, "author_name": "agenlu", "author_url": "", "post_date": "2021-12-02T15:23:46.233000", "content": "@swaralipibose, sorry, I didn’t read all your comment. Maybe the problem is that example_test doesn’t include all the public_lb
", "votes": 0, "replies": [] }, { "id": 1604934, "author_name": "", "author_url": "", "post_date": "2021-12-03T21:21:35.783000", "content": "", "votes": 0, "replies": [] }, { "id": 1605401, "author_name": "agenlu", "author_url": "", "post_date": "2021-12-04T07:32:22.093000", "content": "@dmk2021, read this:
\nhttps://www.kaggle.com/c/g-research-crypto-forecasting/discussion/290265
\npublic LB is included in the train data.
", "votes": 0, "replies": [] }, { "id": 1611459, "author_name": "", "author_url": "", "post_date": "2021-12-08T01:58:18.133000", "content": "", "votes": 0, "replies": [] }, { "id": 1611806, "author_name": "agenlu", "author_url": "", "post_date": "2021-12-08T09:33:05.283000", "content": "@dmk2021, example_test doesn’t include all the data in the public lb, is just an example on how the test data is going to be.
\nAs I have pointed out in the above link post, the public leaderboard has the train rows above 2021-06-13. So, if you want to make your experiments you need to cut the train data, as explained in the above post.
\nRead this:
\nhttps://www.kaggle.com/c/g-research-crypto-forecasting/discussion/290265
", "votes": 0, "replies": [] }, { "id": 1612098, "author_name": "", "author_url": "", "post_date": "2021-12-08T14:49:10", "content": "", "votes": 3, "replies": [] }, { "id": 1612519, "author_name": "Sayantan Mazumdar", "author_url": "", "post_date": "2021-12-09T03:45:38.457000", "content": "Interesting
", "votes": 0, "replies": [] }, { "id": 1613154, "author_name": "", "author_url": "", "post_date": "2021-12-09T17:06:55.203000", "content": "", "votes": 2, "replies": [] }, { "id": 1652492, "author_name": "Endre Moen", "author_url": "", "post_date": "2022-01-16T17:20:48.463000", "content": "yes, would be nice to have this clarified. Why are the organizers so secretive of this weight? They can just publish the code which calculates the weights.
", "votes": 0, "replies": [] } ] }, { "id": 1641296, "author_name": "", "author_url": "", "post_date": "2022-01-07T09:20:18.567000", "content": "", "votes": 0, "replies": [] }, { "id": 1612357, "author_name": "JONALLDAY", "author_url": "", "post_date": "2021-12-08T19:51:22.567000", "content": "Thanks for the explanation!
", "votes": 1, "replies": [] }, { "id": 1611955, "author_name": "Hongkai WANG", "author_url": "", "post_date": "2021-12-08T12:19:05.020000", "content": "Thanks for your explanation.
\nThx