In this discussion https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/286778
\nthe host explains: \"correlation has the useful property that it tends to normalize leading-order volatility out of the covariance between target and prediction\"
\nWhat does that mean?
", "messages": [ { "id": "1653229", "postDate": "01/17/2022 11:17:10", "content": "In this discussion https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/286778
\nthe host explains: \"correlation has the useful property that it tends to normalize leading-order volatility out of the covariance between target and prediction\"
\nWhat does that mean?
", "rawMarkdown": "In this discussion https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/286778\n\nthe host explains: \"correlation has the useful property that it tends to normalize leading-order volatility out of the covariance between target and prediction\"\n\nWhat does that mean?", "votes": null }, { "id": "1653694", "postDate": "01/17/2022 19:47:21", "content": "An earlier spot-on explanation from: @fcfc21
\nComes from the definition of correlation:
\n\n", "rawMarkdown": "An earlier spot-on explanation from: @fcfc21 \n\nComes from the definition of correlation:\n> $$\n> \\rho_{XY} = \\frac{cov(X,Y)}{\\sigma_X \\sigma_Y},\n> $$\n> where sigma is the volatility of X or Y. We want to know how much our predictions correlate with the target because that's all we need to know to decide whether to buy/sell. If you have a predictor that correlates perfectly (rho=1) with the target, you basically have a crystal ball telling you whether you will profit or lose in the next 15 min (your predictor goes up => market goes up and vice versa). Now suppose you have such a predictor P. I can equally create a perfect one by multiplying by 2: predictor Q=2P. We deserve the same score, and indeed this is the case by using correlation. If we were to use, say, covariance, my predictor Q would score twice as much as yours just because it has double volatility (standard dev). This would be unfair. That's why they \"normalise the volatility out\": so that we can compare scores with each other.", "votes": null }, { "id": "1653900", "postDate": "01/18/2022 01:06:07", "content": "$$
\n
\n \\rho_{XY} = \\frac{cov(X,Y)}{\\sigma_X \\sigma_Y},
\n $$
\n where sigma is the volatility of X or Y. We want to know how much our predictions correlate with the target because that's all we need to know to decide whether to buy/sell. If you have a predictor that correlates perfectly (rho=1) with the target, you basically have a crystal ball telling you whether you will profit or lose in the next 15 min (your predictor goes up => market goes up and vice versa). Now suppose you have such a predictor P. I can equally create a perfect one by multiplying by 2: predictor Q=2P. We deserve the same score, and indeed this is the case by using correlation. If we were to use, say, covariance, my predictor Q would score twice as much as yours just because it has double volatility (standard dev). This would be unfair. That's why they \"normalise the volatility out\": so that we can compare scores with each other.
The use of correlation as metrics starts making sense to me. Thanks!
", "rawMarkdown": "The use of correlation as metrics starts making sense to me. Thanks!", "votes": null }, { "id": "1654454", "postDate": "01/18/2022 14:26:44", "content": "When they say \"normalise the volatility out\", is this the same as saying - we are using correlation instead of covariance?
", "rawMarkdown": "When they say \"normalise the volatility out\", is this the same as saying - we are using correlation instead of covariance?", "votes": null }, { "id": "1654462", "postDate": "01/18/2022 14:35:01", "content": "'Normalizing the volatility out' here means: dividing by the sigmas (standard deviations: here called as volatility), as apparent in the denominator in the rho definition above. So correlation is not affected by the variances of each entity.
\nIn other words, 'normalizing the volatility out' is just a wording choice to explain correlation (formula).
", "rawMarkdown": "'Normalizing the volatility out' here means: dividing by the sigmas (standard deviations: here called as volatility), as apparent in the denominator in the rho definition above. So correlation is not affected by the variances of each entity. \n\nIn other words, 'normalizing the volatility out' is just a wording choice to explain correlation (formula).", "votes": null }, { "id": "1654758", "postDate": "01/18/2022 19:17:39", "content": "Yeah - thanks:) I know correlation. Its in the spirit of this competition.
", "rawMarkdown": "Yeah - thanks:) I know correlation. Its in the spirit of this competition.", "votes": null }, { "id": "1656121", "postDate": "01/19/2022 05:19:41", "content": "oh until now, I thought it would only be used for removing extra features which are correlated before entering into model creation
", "rawMarkdown": "oh until now, I thought it would only be used for removing extra features which are correlated before entering into model creation", "votes": null } ], "comments": [ { "id": 1653694, "author_name": "abdulkadirguner", "author_url": "", "post_date": "01/17/2022 19:47:21", "content": "An earlier spot-on explanation from: @fcfc21
\nComes from the definition of correlation:
\n\n", "votes": null, "replies": [ { "id": 1653900, "author_name": "yus002", "author_url": "", "post_date": "01/18/2022 01:06:07", "content": "$$
\n
\n \\rho_{XY} = \\frac{cov(X,Y)}{\\sigma_X \\sigma_Y},
\n $$
\n where sigma is the volatility of X or Y. We want to know how much our predictions correlate with the target because that's all we need to know to decide whether to buy/sell. If you have a predictor that correlates perfectly (rho=1) with the target, you basically have a crystal ball telling you whether you will profit or lose in the next 15 min (your predictor goes up => market goes up and vice versa). Now suppose you have such a predictor P. I can equally create a perfect one by multiplying by 2: predictor Q=2P. We deserve the same score, and indeed this is the case by using correlation. If we were to use, say, covariance, my predictor Q would score twice as much as yours just because it has double volatility (standard dev). This would be unfair. That's why they \"normalise the volatility out\": so that we can compare scores with each other.
The use of correlation as metrics starts making sense to me. Thanks!
", "votes": null, "replies": [] }, { "id": 1654454, "author_name": "endremoen", "author_url": "", "post_date": "01/18/2022 14:26:44", "content": "When they say \"normalise the volatility out\", is this the same as saying - we are using correlation instead of covariance?
", "votes": null, "replies": [] }, { "id": 1654462, "author_name": "abdulkadirguner", "author_url": "", "post_date": "01/18/2022 14:35:01", "content": "'Normalizing the volatility out' here means: dividing by the sigmas (standard deviations: here called as volatility), as apparent in the denominator in the rho definition above. So correlation is not affected by the variances of each entity.
\nIn other words, 'normalizing the volatility out' is just a wording choice to explain correlation (formula).
", "votes": null, "replies": [] }, { "id": 1654758, "author_name": "endremoen", "author_url": "", "post_date": "01/18/2022 19:17:39", "content": "Yeah - thanks:) I know correlation. Its in the spirit of this competition.
", "votes": null, "replies": [] } ] }, { "id": 1656121, "author_name": "bollarapuphanindra", "author_url": "", "post_date": "01/19/2022 05:19:41", "content": "oh until now, I thought it would only be used for removing extra features which are correlated before entering into model creation
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1653229": "In this discussion https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/286778\n\nthe host explains: \"correlation has the useful property that it tends to normalize leading-order volatility out of the covariance between target and prediction\"\n\nWhat does that mean?", "1653694": "An earlier spot-on explanation from: @fcfc21 \n\nComes from the definition of correlation:\n> $$\n> \\rho_{XY} = \\frac{cov(X,Y)}{\\sigma_X \\sigma_Y},\n> $$\n> where sigma is the volatility of X or Y. We want to know how much our predictions correlate with the target because that's all we need to know to decide whether to buy/sell. If you have a predictor that correlates perfectly (rho=1) with the target, you basically have a crystal ball telling you whether you will profit or lose in the next 15 min (your predictor goes up => market goes up and vice versa). Now suppose you have such a predictor P. I can equally create a perfect one by multiplying by 2: predictor Q=2P. We deserve the same score, and indeed this is the case by using correlation. If we were to use, say, covariance, my predictor Q would score twice as much as yours just because it has double volatility (standard dev). This would be unfair. That's why they \"normalise the volatility out\": so that we can compare scores with each other.", "1653900": "The use of correlation as metrics starts making sense to me. Thanks!", "1654454": "When they say \"normalise the volatility out\", is this the same as saying - we are using correlation instead of covariance?", "1654462": "'Normalizing the volatility out' here means: dividing by the sigmas (standard deviations: here called as volatility), as apparent in the denominator in the rho definition above. So correlation is not affected by the variances of each entity. \n\nIn other words, 'normalizing the volatility out' is just a wording choice to explain correlation (formula).", "1654758": "Yeah - thanks:) I know correlation. Its in the spirit of this competition.", "1656121": "oh until now, I thought it would only be used for removing extra features which are correlated before entering into model creation" }, "source": "meta" }