Thanks Kagglers for your questions. We will group the key themes beginning with the eval metric, target formulation and weights:
\nEvaluation Metric
\nWhile mean squared error, R^2, explained variance, and correlation are all very closely related, correlation has the useful property that it tends to normalize leading-order volatility out of the covariance between target and prediction. In financial markets (especially crypto ones!), predicting volatility is a difficult (but interesting!) question in its own right. By using correlation as a metric we hope to remove some noise from the prediction problem and provide a more stable metric to evaluate against.
Target
\nThe target calculation is based on the close price of the asset and can be derived from the provided data using the methodology in https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition. The relevant function is provided below:
def ResidualizeMarket(df, mktColumn, window):\n if mktColumn not in df.columns:\n return df\n\n mkt = df[mktColumn]\n\n num = df.multiply(mkt.values, axis=0).rolling(window).mean().values #numerator of linear regression coefficient\n denom = mkt.multiply(mkt.values, axis=0).rolling(window).mean().values #denominator of linear regression coefficient\n beta = np.nan_to_num( num.T / denom, nan=0., posinf=0., neginf=0.) #if regression fell over, use beta of 0\n\n resultRet = df - (beta * mkt.values).T #perform residualization\n resultBeta = 0.*df + beta.T #shape beta\n\n return resultRet.drop(columns=[mktColumn]), resultBeta.drop(columns=[mktColumn])\nWeights
\nIn this competition, the weights are determined by the logarithm of each product's market cap (in USD), of the cryptocurrencies at a fixed point in time. Weights were assigned to give more relevance to cryptocurrencies with higher market volumes to ensure smaller cryptocurrencies do not disproportionately impact your models.
Thanks everyone. Keep on Kaggling!
", "messages": [ { "id": 1578078, "postDate": "2021-11-10T16:01:45.603Z", "content": "Thanks Kagglers for your questions. We will group the key themes beginning with the eval metric, target formulation and weights:
\nEvaluation Metric
\nWhile mean squared error, R^2, explained variance, and correlation are all very closely related, correlation has the useful property that it tends to normalize leading-order volatility out of the covariance between target and prediction. In financial markets (especially crypto ones!), predicting volatility is a difficult (but interesting!) question in its own right. By using correlation as a metric we hope to remove some noise from the prediction problem and provide a more stable metric to evaluate against.
Target
\nThe target calculation is based on the close price of the asset and can be derived from the provided data using the methodology in https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition. The relevant function is provided below:
def ResidualizeMarket(df, mktColumn, window):\n if mktColumn not in df.columns:\n return df\n\n mkt = df[mktColumn]\n\n num = df.multiply(mkt.values, axis=0).rolling(window).mean().values #numerator of linear regression coefficient\n denom = mkt.multiply(mkt.values, axis=0).rolling(window).mean().values #denominator of linear regression coefficient\n beta = np.nan_to_num( num.T / denom, nan=0., posinf=0., neginf=0.) #if regression fell over, use beta of 0\n\n resultRet = df - (beta * mkt.values).T #perform residualization\n resultBeta = 0.*df + beta.T #shape beta\n\n return resultRet.drop(columns=[mktColumn]), resultBeta.drop(columns=[mktColumn])\nWeights
\nIn this competition, the weights are determined by the logarithm of each product's market cap (in USD), of the cryptocurrencies at a fixed point in time. Weights were assigned to give more relevance to cryptocurrencies with higher market volumes to ensure smaller cryptocurrencies do not disproportionately impact your models.
Thanks everyone. Keep on Kaggling!
", "rawMarkdown": "Thanks Kagglers for your questions. We will group the key themes beginning with the eval metric, target formulation and weights: \n\n**Evaluation Metric**\nWhile mean squared error, R^2, explained variance, and correlation are all very closely related, correlation has the useful property that it tends to normalize leading-order volatility out of the covariance between target and prediction. In financial markets (especially crypto ones!), predicting volatility is a difficult (but interesting!) question in its own right. By using correlation as a metric we hope to remove some noise from the prediction problem and provide a more stable metric to evaluate against.\n\n**Target**\nThe target calculation is based on the close price of the asset and can be derived from the provided data using the methodology in https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition. The relevant function is provided below:\n\n```\ndef ResidualizeMarket(df, mktColumn, window):\n if mktColumn not in df.columns:\n return df\n\n mkt = df[mktColumn]\n\n num = df.multiply(mkt.values, axis=0).rolling(window).mean().values #numerator of linear regression coefficient\n denom = mkt.multiply(mkt.values, axis=0).rolling(window).mean().values #denominator of linear regression coefficient\n beta = np.nan_to_num( num.T / denom, nan=0., posinf=0., neginf=0.) #if regression fell over, use beta of 0\n\n resultRet = df - (beta * mkt.values).T #perform residualization\n resultBeta = 0.*df + beta.T #shape beta\n\n return resultRet.drop(columns=[mktColumn]), resultBeta.drop(columns=[mktColumn])\n```\n\n**Weights**\nIn this competition, the weights are determined by the logarithm of each product's market cap (in USD), of the cryptocurrencies at a fixed point in time. Weights were assigned to give more relevance to cryptocurrencies with higher market volumes to ensure smaller cryptocurrencies do not disproportionately impact your models.\n\n \nThanks everyone. Keep on Kaggling!\n", "votes": 110 }, { "id": 1588962, "postDate": "2021-11-19T19:51:37.493Z", "content": "@andrewscherer: is still unclear to me how the eval metric is calculated. In the base exemple you mention:
\n\" The competition performance metric is weighted correlation. However, for now we will use simple correlation to evaluate the two baseline models built.\" But you don't give a definition of weighted correlation anywhere. Some definition can be found on wikipedia. However you don't mention the formula and there are other interpretation for what you mention.
Is it a simple weighing of the results ? (In the exemple you give that: BTC 0.01, ETH 0.05
\nI see the weights are respectively 6.779922 and 5.894403. Does that mean the final eval would be: 0.01 * 6.77 + 0.05 * 5.89 / (6.77 + 5.89) ?
Or is it the use of a weighted correlation metric as shown below ?
\n# https://stackoverflow.com/questions/38641691/weighted-correlation-coefficient-with-pandas\ndef wmean(x, w):\n return np.sum(x * w) / np.sum(w)\n\ndef wcov(x, y, w):\n return np.sum(w * (x - wmean(x, w)) * (y - wmean(y, w))) / np.sum(w)\n\ndef wcorr(x, y, w):\n return wcov(x, y, w) / np.sqrt(wcov(x, x, w) * wcov(y, y, w))\n\ndef eval_wcorr(preds, train_data):\n w = train_data.add_w.values.flatten()\n y_true = train_data.get_label()\n return 'eval_wcorr', wcorr(preds, y_true, w), True\nHi, where does add_w.values.flatten() come from? I tried implementing the code, but got error from both XGB and LGBM. XGB: AttributeError: 'DMatrix' object has no attribute 'add_w'; LGBM: AttributeError: 'numpy.ndarray' object has no attribute 'add_w'. Thanks!
", "rawMarkdown": "Hi, where does add_w.values.flatten() come from? I tried implementing the code, but got error from both XGB and LGBM. XGB: AttributeError: 'DMatrix' object has no attribute 'add_w'; LGBM: AttributeError: 'numpy.ndarray' object has no attribute 'add_w'. Thanks!", "votes": 1 }, { "id": 1590760, "postDate": "2021-11-21T16:18:45.003Z", "content": "I think you can see how it is implemented in my lgbm notebook (https://www.kaggle.com/lucasmorin/crypto-forecasting-lgbm-feval-feature-importance). You must add weight in your lgbm data set.
", "rawMarkdown": "I think you can see how it is implemented in my lgbm notebook (https://www.kaggle.com/lucasmorin/crypto-forecasting-lgbm-feval-feature-importance). You must add weight in your lgbm data set.", "votes": 1 }, { "id": 1594407, "postDate": "2021-11-24T19:47:01.973Z", "rawMarkdown": "", "votes": 2, "isDeleted": true }, { "id": 1598626, "postDate": "2021-11-28T16:24:14.797Z", "content": "It would be great to have a clear definition of the way the score will be calculated.
", "rawMarkdown": "It would be great to have a clear definition of the way the score will be calculated.", "votes": 1 } ] }, { "id": 1579014, "postDate": "2021-11-11T13:07:15.410Z", "content": "Thank you for sharing this code! It was really hard to replicate the target for some of us here!
\nBut I have a question:
\nbeta = np.nan_to_num( num.T / denom, nan=0., posinf=0., neginf=0.) #if regression fell over, use beta of 0
Why? It doesn't make any sense to assign beta=0 if there are some problems with data. Beta is a sensitivity of an asset to market movements, it doesn't suddenly turn to 0 if you encounter None in your data. As a result, you end up with beta being mostly zero for some assets. Here we already discussed it: https://www.kaggle.com/alexfir/recreating-target/comments
\nI actually calculated \"implied beta\" from provided targets:
\ndata['beta_implied'] = (data['r'] - data['Target']) / data['m']
\n# try different Asset_ID
\nplt_data = np.clip(data.query(\"Asset_ID==0\")[['beta_implied']].values, -1, 2)
\nplt.hist(plt_data,bins=100);
Histograms of \"beta_implied\":
\nAs you can see, Asset_ID==0 mostly has beta equal to zero, which doesn't make any financial sense, and also makes it's impossible to predict the target, since we'll need to predict whatever next datapoints will be NA or not.
\nI think the organizers should fix the bug in their code and provide updated targets.
", "rawMarkdown": "Thank you for sharing this code! It was really hard to replicate the target for some of us here!\n\nBut I have a question:\n\n`beta = np.nan_to_num( num.T / denom, nan=0., posinf=0., neginf=0.) #if regression fell over, use beta of 0`\n\nWhy? It doesn't make any sense to assign beta=0 if there are some problems with data. Beta is a sensitivity of an asset to market movements, it doesn't suddenly turn to 0 if you encounter None in your data. As a result, you end up with beta being mostly zero for some assets. Here we already discussed it: https://www.kaggle.com/alexfir/recreating-target/comments\n\nI actually calculated \"implied beta\" from provided targets:\n\n`data['beta_implied'] = (data['r'] - data['Target']) / data['m']`\n`# try different Asset_ID`\n`plt_data = np.clip(data.query(\"Asset_ID==0\")[['beta_implied']].values, -1, 2)`\n`plt.hist(plt_data,bins=100);`\n\nHistograms of \"beta_implied\":\n\n - Asset_ID==0 - https://ibb.co/ZWhNX89\n - Asset_ID==1 - https://ibb.co/YNvHk07\n - Asset_ID==6 - imagehttps://ibb.co/7k0F3BR\n\nAs you can see, Asset_ID==0 mostly has beta equal to zero, which doesn't make any financial sense, and also **makes it's impossible to predict the target, since we'll need to predict whatever next datapoints will be NA or not**.\n\nI think the organizers should fix the bug in their code and provide updated targets.", "votes": 14, "replies": [ { "id": 1627548, "postDate": "2021-12-24T02:31:02.717Z", "content": "Totally agreed! I just entered this competition and have discovered this same beta=0 issue because of missing data from the future. I am 100% with you that using beta = 0 to construct the target doesn't make any financial sense (If beta is 0 at one minute and not 0 the next minute, it is indicating the correlation between asset and market changes dramatically on a minute by minute basis), and makes it very hard to predict based on whether or not the data will be NaN in the future. I have a separate post listing the details of other side effects caused by future missing data see this notebook.
\nI agree that the organizers should clarify what detailed methodology they used in constructing the training data target (it's clearly not the same as what they described in the tutorial), and also what methodology they intend on using during scoring. But most importantly, I think missing data should be handled by padding with previous available data (like what's described in tutorial). Because missing data normally is caused by error in the network, or maybe simply no trade for that asset at that moment. These missing data shouldn't imply that the stock price has dropped to 0 (please see details in this notebook notebook.)
", "rawMarkdown": "Totally agreed! I just entered this competition and have discovered this same beta=0 issue because of missing data from the future. I am 100% with you that using beta = 0 to construct the target doesn't make any financial sense (If beta is 0 at one minute and not 0 the next minute, it is indicating the correlation between asset and market changes dramatically on a minute by minute basis), and makes it very hard to predict based on whether or not the data will be NaN in the future. I have a separate post listing the details of other side effects caused by future missing data [see this notebook](https://www.kaggle.com/gengdaiziwang/are-we-training-models-on-an-inaccurate-target/edit/run/83054475). \n\nI agree that the organizers should clarify what detailed methodology they used in constructing the training data target (it's clearly not the same as what they described in the tutorial), and also what methodology they intend on using during scoring. But most importantly, I think missing data should be handled by padding with previous available data (like what's described in tutorial). Because missing data normally is caused by error in the network, or maybe simply no trade for that asset at that moment. These missing data shouldn't imply that the stock price has dropped to 0 (please see details in this notebook [notebook](https://www.kaggle.com/gengdaiziwang/are-we-training-models-on-an-inaccurate-target/edit/run/83054475).)\n ", "votes": 2 } ] }, { "id": 1626395, "postDate": "2021-12-22T19:05:32.230Z", "content": "Trying to bring clarification of the Target calculation back into discussion, because the chain of messages here might have lost some readers!
\nThe upshot is that we can reproduce the Target across the whole train set with max absolute error < 1e-13.
\nThis had to be achieved using a method which was materially different from the definition suggested in the tutorial notebook which states that \"the rolling average over time (3750 minute windows)\" was used. It seems that the calculation uses the rolling average over 3750 (non-empty) timestamps instead.
Reprex: https://www.kaggle.com/jagofc/target-reconstruction-quick-summary-of-sota
\nHappy to play by whatever rules the hosts set for the game. Given that we have had to use a different definition of the target in order to reproduce the train set values, there is still some uncertainty around what the rules are though : )
\nPlease could one of the helpful people from Kaggle @sohier or G-Research @andrewscherer confirm whether the Target calculation method will remain the same for the evaluation period of the competition as it was for the train set, or whether it will be altered to align with the definition in the tutorial notebook?
", "rawMarkdown": "Trying to bring clarification of the Target calculation back into discussion, because the chain of messages [here](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/286778#1582306) might have lost some readers!\n\nThe upshot is that we can reproduce the Target across the whole train set with max absolute error < 1e-13. \nThis had to be achieved using a method which was materially different from the definition suggested in the [tutorial notebook](https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition) which states that \"the rolling average over time (3750 minute windows)\" was used. It seems that the calculation uses the rolling average over 3750 (non-empty) timestamps instead. \n\nReprex: https://www.kaggle.com/jagofc/target-reconstruction-quick-summary-of-sota\n\nHappy to play by whatever rules the hosts set for the game. Given that we have had to use a different definition of the target in order to reproduce the train set values, there is still some uncertainty around what the rules are though : )\n\nPlease could one of the helpful people from Kaggle @sohier or G-Research @andrewscherer confirm whether the Target calculation method will remain the same for the evaluation period of the competition as it was for the train set, or whether it will be altered to align with the definition in the tutorial notebook?", "votes": 12, "replies": [ { "id": 1627563, "postDate": "2021-12-24T02:54:22.977Z", "content": "Thanks @jagofc for bringing this to attention! I totally agree that there are inconsistencies between what's suggested in the tutorial book and the method we inferred G-Research has used to calculate the target. Would be great if G-Research can clarify what methodology is or will be used when calculating target.
\nMoreover, rolling average using 3750 minute windows vs 3750 (non-empty) timestamps is one inconsistency. The most troublesome inconsistency is considering asset price as 0 for missing data vs considering no price change in asset price (from tutorial: pad missing data with previous available price). Since market return (M) is calculated based on future prices, if we follow the current target calculation, we basically need to predict whether or not a stock price is missing in the next minute or not which will be very noisy and unpredictable.
\nI am also happy to play by whatever rules the hosts set for the game. But it makes more financial sense to use the methodology and padding approach mentioned in the tutorial. Would be great if G-Research can clarify.
", "rawMarkdown": "Thanks @jagofc for bringing this to attention! I totally agree that there are inconsistencies between what's suggested in the tutorial book and the method we inferred G-Research has used to calculate the target. Would be great if G-Research can clarify what methodology is or will be used when calculating target. \n\nMoreover, rolling average using 3750 minute windows vs 3750 (non-empty) timestamps is one inconsistency. The most troublesome inconsistency is considering asset price as 0 for missing data vs considering no price change in asset price (from tutorial: pad missing data with previous available price). Since market return (M) is calculated based on future prices, if we follow the current target calculation, we basically need to predict whether or not a stock price is missing in the next minute or not which will be very noisy and unpredictable. \n\nI am also happy to play by whatever rules the hosts set for the game. But it makes more financial sense to use the methodology and padding approach mentioned in the tutorial. Would be great if G-Research can clarify. ", "votes": 5 } ] }, { "id": 1582965, "postDate": "2021-11-15T12:59:25.133Z", "content": "\nWouldn't it be easier for you guys to share a notebook to re-create the target than sharing part of the script?
", "rawMarkdown": "@andrewscherer \n\nWouldn't it be easier for you guys to share a notebook to re-create the target than sharing part of the script? ", "votes": 11 }, { "id": 1578201, "postDate": "2021-11-10T18:31:51.840Z", "content": "The ResidualizeMarket function you provided produces a SyntaxError because of resultBeta = 0.*df +**** beta.T #shape beta. What is +**** supposed to mean?
Also, is resultRet supposed to be what returns the Target column? The results do not match what is in the Target column and actually have a higher error than what others have tried to reproduce in the forum/notebooks.
Thanks for flagging @brandenkmurray, we have corrected the post above to remove the **** which should not have been there.
Thanks. Can we also get clarification on the proper use of that function to reproduce the targets? We are still unable to reproduce them.
", "rawMarkdown": "Thanks. Can we also get clarification on the proper use of that function to reproduce the targets? We are still unable to reproduce them.", "votes": 1 } ] }, { "id": 1578387, "postDate": "2021-11-11T00:52:57.020Z", "content": "Hello, @andrewscherer !
\nThank you very much for sharing ResidualizeMarket function.
But it doesn't help in recreating target values 😪
\nLots of people in that competition including me are blowing their minds to recreate targets and no solution for now!
Can you please share the full code on how we can recreate targets in train.csv?
\nHere is a notebook from @ebudia trying to recreate target. I've gone through that notebook and found only one mistake with weights agregation:
\nm = time_group['weighted_asset_r'].sum() / time_group['w'].sum()\n# I think here only weights, where R(t) is not NaN should be summed, but this code sum all weights\nI've also played a lot with different prices, time windows and agregations, but can not recreate target!
\nThe very simple test can not be passed by my algorithms. Just recreate target for first minute (2018-01-01 00:01:00) in train dataset.
\nAccording to your code I assume, that for that first minute:
\n<M*R> = M*R and <M^2> = M^2, because we don't have previous times and you use just .rolling(window)
Thanks in advance for your feedback!
\nP.S. Google Sheet with all intermediate calculations for first minute in train dataset.
", "rawMarkdown": "Hello, @andrewscherer !\nThank you very much for sharing `ResidualizeMarket` function.\n\nBut it doesn't help in recreating target values 😪\n**Lots of people** in that competition including me are **blowing their minds to recreate targets** and no solution for now!\n\n**Can you please share the full code on how we can recreate targets in train.csv?**\n\nHere is a[ notebook](https://www.kaggle.com/ebudia/recreating-target-min-periods-3750) from @ebudia trying to recreate target. I've gone through that notebook and found only one mistake with weights agregation:\n```\nm = time_group['weighted_asset_r'].sum() / time_group['w'].sum()\n# I think here only weights, where R(t) is not NaN should be summed, but this code sum all weights\n\n```\n\nI've also played a lot with different prices, time windows and agregations, but can not recreate target!\nThe very simple test can not be passed by my algorithms. Just recreate target for first minute (2018-01-01 00:01:00) in train dataset. \nAccording to your code I assume, that for that first minute:\n`For those who are still looking for replicating target computation, this how I did it:
\nhttps://www.kaggle.com/vi2018/g-research-crypto-repro-target-computation
\nThe computed values seems to be in agreement with those of \"training.csv\".
", "rawMarkdown": "For those who are still looking for replicating target computation, this how I did it: \n\nhttps://www.kaggle.com/vi2018/g-research-crypto-repro-target-computation\n\nThe computed values seems to be in agreement with those of \"training.csv\".\n", "votes": 7, "replies": [ { "id": 1582764, "postDate": "2021-11-15T08:43:23.347Z", "content": "By changing the calculation of returns from log(P(t+16)/P(t+1)) to P(t+16)/P(t+1) - 1 I was able to reduce the MAE and maximum error in both your and @alexfir notebooks (https://www.kaggle.com/alexfir/recreating-target ). I'm starting to think they didn't actually use log returns when calculating their targets…
Found that (x - 1) is a upper bound for log(x). It looks like around x=1.0, log(x) and (x-1) are very close. Though I do not know why x-1 has better fit to Target than log(x).
", "rawMarkdown": "Found that (x - 1) is a upper bound for log(x). It looks like around x=1.0, log(x) and (x-1) are very close. Though I do not know why x-1 has better fit to Target than log(x).", "votes": 4 }, { "id": 1583953, "postDate": "2021-11-16T06:54:47.597Z", "content": "@brandenkmurray for the rows with beta==0 your formula gives 0 error. For the others, we have to figure out how they calculate m.
", "rawMarkdown": "@brandenkmurray for the rows with beta==0 your formula gives 0 error. For the others, we have to figure out how they calculate m." }, { "id": 1584677, "postDate": "2021-11-16T17:23:40.923Z", "content": "@vi2018 it's not a 100% replication of the target; any insights on such target representation?
", "rawMarkdown": "@vi2018 it's not a 100% replication of the target; any insights on such target representation?" }, { "id": 1585065, "postDate": "2021-11-17T03:29:21.333Z", "content": "I was able to modify @vi2018's notebook a bit to get the MAE down to 0.00003 and max error down to 0.015.
\nMain changes were
\nweighted_avg_market_log_returns calculation Still not perfect, but getting closer…
\n", "rawMarkdown": "I was able to modify @vi2018's notebook a bit to get the MAE down to 0.00003 and max error down to 0.015.\n\nMain changes were \n1. changing the log returns function from log(P(t+16)/P(t+1)) to P(t+16)/P(t+1) - 1 \n2. adjusting the `weighted_avg_market_log_returns` calculation \n3. removing the first minute of each month because as I noted [here](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/286095), they seem to ignore it when calculating returns, . \n\nStill not perfect, but getting closer...\n\nhttps://www.kaggle.com/brandenkmurray/g-research-crypto-repro-target-computation-fc4875?scriptVersionId=79997498", "votes": 6, "replies": [ { "id": 1586086, "postDate": "2021-11-17T19:37:58.793Z", "content": "Updated Target recreation by replacing log(x) => x-1 and using only available timestamps.
\nResults:
\nAverage absolute error 0.000000
\nMax absolute error 0.003798
\nStandard deviation 0.000001
@brandenkmurray I think the link you shared is not working or it isn't public. Thanks for the insights.
", "rawMarkdown": "@brandenkmurray I think the link you shared is not working or it isn't public. Thanks for the insights." }, { "id": 1586046, "postDate": "2021-11-17T18:50:58.083Z", "content": "@ambank1 Oops, you're right. Made it public.
", "rawMarkdown": "@ambank1 Oops, you're right. Made it public.", "votes": 2 }, { "id": 1588091, "postDate": "2021-11-19T07:54:12.210Z", "content": "Just for those unfamiliar with calculus. Replacing $\\log(x)$ by $x-1$ comes from the first order Taylor series expansion:
\n$$
\n\\log(x) = \\log(1+x-1 ) = x-1 + O((x-1)^2)
\n$$
\nSo for
\n$$x\\approx 1, \\quad \\log(x)\\approx x-1.$$
\nReturns over 15 minutes tend to satisfy this condition (unless really wild volatility is present) and the log(x) function takes longer to calculate than the simple algebraic x-1. So I suppose the host used the approximation to speed up computation without losing much precision. Such performance gain might not play a big role with minute-long windows here, but I imagine it is relevant for very high frequency trading where low latency is paramount.
\n\nSo I suppose the host used the approximation to speed up computation without losing much precision
\n
Yeah, makes sense. But why did the hosts tell us they used log(x) when they didn't? And they haven't commented on it since it was raised 3 days ago that they actually used (x-1)…. Really wish they were more active in responding to questions/concerns.
", "rawMarkdown": "> So I suppose the host used the approximation to speed up computation without losing much precision\n\nYeah, makes sense. But why did the hosts tell us they used log(x) when they didn't? And they haven't commented on it since it was raised 3 days ago that they actually used (x-1).... Really wish they were more active in responding to questions/concerns.\n\n", "votes": 6 } ] }, { "id": 1596306, "postDate": "2021-11-26T11:10:47.493Z", "content": "@brandenkmurray maybe the company has forgotten to allocate some employee in this; plus crypto market is in the red sea now… XD
", "rawMarkdown": "@brandenkmurray maybe the company has forgotten to allocate some employee in this; plus crypto market is in the red sea now... XD" }, { "id": 1613127, "postDate": "2021-12-09T16:28:15.387Z", "content": "Hi all, not sure where we're at with target reconstruction, but noticed that there are just 4 rows that are throwing off the reconstruction in the notebook by @alexfir .
\nHave produced a light notebook highlighting the offending rows. After removing the 4 rows, the max absolute error is down from 3.8e-03 to 2.8e-15.
\nInterested in your thoughts as to what's up with those 4 rows!
", "rawMarkdown": "Hi all, not sure where we're at with target reconstruction, but noticed that there are just 4 rows that are throwing off the reconstruction in the [notebook](https://www.kaggle.com/alexfir/recreating-target/) by @alexfir .\n\nHave produced a light [notebook](https://www.kaggle.com/jagofc/target-reconstruction-what-s-up-with-2018-01-03/) highlighting the offending rows. After removing the 4 rows, the max absolute error is down from 3.8e-03 to 2.8e-15. \n\nInterested in your thoughts as to what's up with those 4 rows!", "votes": 1 }, { "id": 1626405, "postDate": "2021-12-22T19:16:43.723Z", "content": "Those 4 rows were 3749 timestamps after 2018-01-01 00:01. Adding in a row for 2018-01-01 00:00 reproduces the Target perfectly (max absolute error across whole train set < 1e-13).
\nFor anyone who's found their way to the end of this message chain - congrats! Trying to get some more clarification on how Target will be calculated going forward in a new post in this topic.
", "rawMarkdown": "Those 4 rows were 3749 timestamps after 2018-01-01 00:01. Adding in a row for 2018-01-01 00:00 [reproduces the Target perfectly](https://www.kaggle.com/jagofc/target-reconstruction-quick-summary-of-sota) (max absolute error across whole train set < 1e-13). \n\nFor anyone who's found their way to the end of this message chain - congrats! Trying to get some more clarification on how Target will be calculated going forward [in a new post in this topic.](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/286778#1626395)", "votes": 3 } ] }, { "id": 1651338, "postDate": "2022-01-15T17:22:12.417Z", "content": "I understand the Target formula as the residuals same as the following linear model:
\n$$R = \\beta * M + u$$
\nWhy not run linear regression with intercept and take the intercept($\\alpha$) as the residuals, which follows the regression model below?
\n$$R = \\alpha + \\beta * M + u$$
", "rawMarkdown": "I understand the Target formula as the residuals same as the following linear model:\n\n$$R = \\beta * M + u$$\n\nWhy not run linear regression with intercept and take the intercept($\\alpha$) as the residuals, which follows the regression model below? \n\n$$R = \\alpha + \\beta * M + u$$", "votes": 1 }, { "id": 1588049, "postDate": "2021-11-19T07:21:42.440Z", "content": "What's the window value in the ResidualizeMarket function if we want to replicate the given target values?
\nThanks!
Do we replace nan value by 0.0 during evaluation? Or we remove the nan values?
", "rawMarkdown": "Do we replace nan value by 0.0 during evaluation? Or we remove the nan values?", "votes": 2, "replies": [ { "id": 1653636, "postDate": "2022-01-17T18:27:17.643Z", "content": "See here
\n\n\nSome rows have null values for targets due to missing values in future prices. Rows with nulls in the test set ground truth are ignored for scoring purposes.
\n
and here
\n\n", "rawMarkdown": "See [here](https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition?scriptVersionId=84976518&cellId=51)\n\n> Some rows have null values for targets due to missing values in future prices. Rows with nulls in the test set ground truth are ignored for scoring purposes.\n\nand [here](https://www.kaggle.com/c/g-research-crypto-forecasting/overview/evaluation)\n\n> You will get an error if you[r] submission includes nulls or infinities.", "votes": 5 } ] }, { "id": 1585322, "postDate": "2021-11-17T08:37:10.577Z", "content": "You will get an error if you[r] submission includes nulls or infinities.
\n
\"correlation has the useful property that it tends to normalize leading-order volatility out of the covariance between target and prediction\"
\nI would love to get some demonstration on this point but couldn't find one. Could you send me some references? Thanks
normalize leading-order volatility out of the covariance between target and prediction.
Please expand
", "rawMarkdown": "`normalize leading-order volatility out of the covariance between target and prediction.`\n\nPlease expand", "votes": 2, "replies": [ { "id": 1589014, "postDate": "2021-11-19T20:44:38.903Z", "content": "Comes from the definition of correlation:
\n$$
\n\\rho_{XY} = \\frac{cov(X,Y)}{\\sigma_X \\sigma_Y},
\n$$
\nwhere sigma is the volatility of X or Y. We want to know how much our predictions correlate with the target because that's all we need to know to decide whether to buy/sell. If you have a predictor that correlates perfectly (rho=1) with the target, you basically have a crystal ball telling you whether you will profit or lose in the next 15 min (your predictor goes up => market goes up and vice versa). Now suppose you have such a predictor P. I can equally create a perfect one by multiplying by 2: predictor Q=2P. We deserve the same score, and indeed this is the case by using correlation. If we were to use, say, covariance, my predictor Q would score twice as much as yours just because it has double volatility (standard dev). This would be unfair. That's why they \"normalise the volatility out\": so that we can compare scores with each other.
The use of correlation as metrics starts making sense to me. Thanks!
", "rawMarkdown": "The use of correlation as metrics starts making sense to me. Thanks!", "votes": 1 } ] }, { "id": 1661835, "postDate": "2022-01-23T18:31:48.577Z", "content": "thanks for sharing!
", "rawMarkdown": "thanks for sharing!\n", "replies": [ { "id": 1662894, "postDate": "2022-01-24T16:10:20.967Z", "content": "Hi. Want to team up? You can find me on discord.
", "rawMarkdown": "Hi. Want to team up? You can find me on discord." } ] }, { "id": 1587982, "postDate": "2021-11-19T05:19:02.283Z", "content": "I am sorry to ask this.I have no knowledge about stock market. There are so many target calculations seeing in this forum.are there any standards for target calculations?
\nWhat I would to know is at a particular time instance what is the actual value of a particular stock.if any one provide the way to study stock market I will be grateful to you.without knowing domain I cant do anything.
\nThanks in advance.
\n\nWhat I would to know is at a particular time instance what is the actual value of a particular stock.
\n
It is important to interpret what you mean by \"value\". If you mean the price of a share of a stock (or of 1 cryptocurrency as in here) you have to look who is selling for cheaper (if you want to buy) and who is buying for most (if you are willing to sell). This means that the \"price\" now depends on whether you want to buy or sell :) . The difference in these prices is the \"bid-ask spread\". This \"spread\" is always positive because if they were equal, a transaction would occur instantly and the spread reappears.
\nThis is not the data we have here though, we are looking at trades that already occurred. When someone actually buys (resp sells) the price they paid (resp received) is recorded. Over 1 minute you calculate how many transactions occurred (\"Count\") and how many cryptocurrencies changed hands (\"Volume\", because one trade might involve more than one unit) along with the other features given by the data.
\nMaybe what you wanted to know is simply what \"P(t)\" is in the returns formula. They defined it to be the close price in the end of minute t.
Thank you for sharing! Just as a side note, I think in the section Data -> \"example_test\" is missing the column row_id.
", "rawMarkdown": "Thank you for sharing! Just as a side note, I think in the section Data -> \"example_test\" is missing the column *row_id*. " }, { "id": 1662915, "postDate": "2022-01-24T16:39:09.673Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1659841, "postDate": "2022-01-22T06:26:04.373Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1579987, "postDate": "2021-11-12T11:53:57.220Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1579594, "postDate": "2021-11-12T03:16:49.083Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1578386, "postDate": "2021-11-11T00:49:51.317Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1659326, "postDate": "2022-01-21T17:49:02.717Z", "content": "Thank you!
", "rawMarkdown": "Thank you!" } ], "comments": [ { "id": 1588962, "author_name": "Lucas Morin", "author_url": "", "post_date": "2021-11-19T19:51:37.493000", "content": "@andrewscherer: is still unclear to me how the eval metric is calculated. In the base exemple you mention:
\n\" The competition performance metric is weighted correlation. However, for now we will use simple correlation to evaluate the two baseline models built.\" But you don't give a definition of weighted correlation anywhere. Some definition can be found on wikipedia. However you don't mention the formula and there are other interpretation for what you mention.
Is it a simple weighing of the results ? (In the exemple you give that: BTC 0.01, ETH 0.05
\nI see the weights are respectively 6.779922 and 5.894403. Does that mean the final eval would be: 0.01 * 6.77 + 0.05 * 5.89 / (6.77 + 5.89) ?
Or is it the use of a weighted correlation metric as shown below ?
\n# https://stackoverflow.com/questions/38641691/weighted-correlation-coefficient-with-pandas\ndef wmean(x, w):\n return np.sum(x * w) / np.sum(w)\n\ndef wcov(x, y, w):\n return np.sum(w * (x - wmean(x, w)) * (y - wmean(y, w))) / np.sum(w)\n\ndef wcorr(x, y, w):\n return wcov(x, y, w) / np.sqrt(wcov(x, x, w) * wcov(y, y, w))\n\ndef eval_wcorr(preds, train_data):\n w = train_data.add_w.values.flatten()\n y_true = train_data.get_label()\n return 'eval_wcorr', wcorr(preds, y_true, w), True\nHi, where does add_w.values.flatten() come from? I tried implementing the code, but got error from both XGB and LGBM. XGB: AttributeError: 'DMatrix' object has no attribute 'add_w'; LGBM: AttributeError: 'numpy.ndarray' object has no attribute 'add_w'. Thanks!
", "votes": 1, "replies": [] }, { "id": 1590760, "author_name": "Lucas Morin", "author_url": "", "post_date": "2021-11-21T16:18:45.003000", "content": "I think you can see how it is implemented in my lgbm notebook (https://www.kaggle.com/lucasmorin/crypto-forecasting-lgbm-feval-feature-importance). You must add weight in your lgbm data set.
", "votes": 1, "replies": [] }, { "id": 1594407, "author_name": "", "author_url": "", "post_date": "2021-11-24T19:47:01.973000", "content": "", "votes": 2, "replies": [] }, { "id": 1598626, "author_name": "Luca Bartoccioni", "author_url": "", "post_date": "2021-11-28T16:24:14.797000", "content": "It would be great to have a clear definition of the way the score will be calculated.
", "votes": 1, "replies": [] } ] }, { "id": 1579014, "author_name": "Nikolay Pismennyi", "author_url": "", "post_date": "2021-11-11T13:07:15.410000", "content": "Thank you for sharing this code! It was really hard to replicate the target for some of us here!
\nBut I have a question:
\nbeta = np.nan_to_num( num.T / denom, nan=0., posinf=0., neginf=0.) #if regression fell over, use beta of 0
Why? It doesn't make any sense to assign beta=0 if there are some problems with data. Beta is a sensitivity of an asset to market movements, it doesn't suddenly turn to 0 if you encounter None in your data. As a result, you end up with beta being mostly zero for some assets. Here we already discussed it: https://www.kaggle.com/alexfir/recreating-target/comments
\nI actually calculated \"implied beta\" from provided targets:
\ndata['beta_implied'] = (data['r'] - data['Target']) / data['m']
\n# try different Asset_ID
\nplt_data = np.clip(data.query(\"Asset_ID==0\")[['beta_implied']].values, -1, 2)
\nplt.hist(plt_data,bins=100);
Histograms of \"beta_implied\":
\nAs you can see, Asset_ID==0 mostly has beta equal to zero, which doesn't make any financial sense, and also makes it's impossible to predict the target, since we'll need to predict whatever next datapoints will be NA or not.
\nI think the organizers should fix the bug in their code and provide updated targets.
", "votes": 14, "replies": [ { "id": 1627548, "author_name": "Gengdaizi Wang", "author_url": "", "post_date": "2021-12-24T02:31:02.717000", "content": "Totally agreed! I just entered this competition and have discovered this same beta=0 issue because of missing data from the future. I am 100% with you that using beta = 0 to construct the target doesn't make any financial sense (If beta is 0 at one minute and not 0 the next minute, it is indicating the correlation between asset and market changes dramatically on a minute by minute basis), and makes it very hard to predict based on whether or not the data will be NaN in the future. I have a separate post listing the details of other side effects caused by future missing data see this notebook.
\nI agree that the organizers should clarify what detailed methodology they used in constructing the training data target (it's clearly not the same as what they described in the tutorial), and also what methodology they intend on using during scoring. But most importantly, I think missing data should be handled by padding with previous available data (like what's described in tutorial). Because missing data normally is caused by error in the network, or maybe simply no trade for that asset at that moment. These missing data shouldn't imply that the stock price has dropped to 0 (please see details in this notebook notebook.)
", "votes": 2, "replies": [] } ] }, { "id": 1626395, "author_name": "jagofc", "author_url": "", "post_date": "2021-12-22T19:05:32.230000", "content": "Trying to bring clarification of the Target calculation back into discussion, because the chain of messages here might have lost some readers!
\nThe upshot is that we can reproduce the Target across the whole train set with max absolute error < 1e-13.
\nThis had to be achieved using a method which was materially different from the definition suggested in the tutorial notebook which states that \"the rolling average over time (3750 minute windows)\" was used. It seems that the calculation uses the rolling average over 3750 (non-empty) timestamps instead.
Reprex: https://www.kaggle.com/jagofc/target-reconstruction-quick-summary-of-sota
\nHappy to play by whatever rules the hosts set for the game. Given that we have had to use a different definition of the target in order to reproduce the train set values, there is still some uncertainty around what the rules are though : )
\nPlease could one of the helpful people from Kaggle @sohier or G-Research @andrewscherer confirm whether the Target calculation method will remain the same for the evaluation period of the competition as it was for the train set, or whether it will be altered to align with the definition in the tutorial notebook?
", "votes": 12, "replies": [ { "id": 1627563, "author_name": "Gengdaizi Wang", "author_url": "", "post_date": "2021-12-24T02:54:22.977000", "content": "Thanks @jagofc for bringing this to attention! I totally agree that there are inconsistencies between what's suggested in the tutorial book and the method we inferred G-Research has used to calculate the target. Would be great if G-Research can clarify what methodology is or will be used when calculating target.
\nMoreover, rolling average using 3750 minute windows vs 3750 (non-empty) timestamps is one inconsistency. The most troublesome inconsistency is considering asset price as 0 for missing data vs considering no price change in asset price (from tutorial: pad missing data with previous available price). Since market return (M) is calculated based on future prices, if we follow the current target calculation, we basically need to predict whether or not a stock price is missing in the next minute or not which will be very noisy and unpredictable.
\nI am also happy to play by whatever rules the hosts set for the game. But it makes more financial sense to use the methodology and padding approach mentioned in the tutorial. Would be great if G-Research can clarify.
", "votes": 5, "replies": [] } ] }, { "id": 1582965, "author_name": "ambank1", "author_url": "", "post_date": "2021-11-15T12:59:25.133000", "content": "\nWouldn't it be easier for you guys to share a notebook to re-create the target than sharing part of the script?
", "votes": 11, "replies": [] }, { "id": 1578201, "author_name": "Branden Murray", "author_url": "", "post_date": "2021-11-10T18:31:51.840000", "content": "The ResidualizeMarket function you provided produces a SyntaxError because of resultBeta = 0.*df +**** beta.T #shape beta. What is +**** supposed to mean?
Also, is resultRet supposed to be what returns the Target column? The results do not match what is in the Target column and actually have a higher error than what others have tried to reproduce in the forum/notebooks.
Thanks for flagging @brandenkmurray, we have corrected the post above to remove the **** which should not have been there.
Thanks. Can we also get clarification on the proper use of that function to reproduce the targets? We are still unable to reproduce them.
", "votes": 1, "replies": [] } ] }, { "id": 1578387, "author_name": "Evgeniy Shubin", "author_url": "", "post_date": "2021-11-11T00:52:57.020000", "content": "Hello, @andrewscherer !
\nThank you very much for sharing ResidualizeMarket function.
But it doesn't help in recreating target values 😪
\nLots of people in that competition including me are blowing their minds to recreate targets and no solution for now!
Can you please share the full code on how we can recreate targets in train.csv?
\nHere is a notebook from @ebudia trying to recreate target. I've gone through that notebook and found only one mistake with weights agregation:
\nm = time_group['weighted_asset_r'].sum() / time_group['w'].sum()\n# I think here only weights, where R(t) is not NaN should be summed, but this code sum all weights\nI've also played a lot with different prices, time windows and agregations, but can not recreate target!
\nThe very simple test can not be passed by my algorithms. Just recreate target for first minute (2018-01-01 00:01:00) in train dataset.
\nAccording to your code I assume, that for that first minute:
\n<M*R> = M*R and <M^2> = M^2, because we don't have previous times and you use just .rolling(window)
Thanks in advance for your feedback!
\nP.S. Google Sheet with all intermediate calculations for first minute in train dataset.
", "votes": 8, "replies": [] }, { "id": 1582306, "author_name": "vion", "author_url": "", "post_date": "2021-11-14T18:33:16.583000", "content": "For those who are still looking for replicating target computation, this how I did it:
\nhttps://www.kaggle.com/vi2018/g-research-crypto-repro-target-computation
\nThe computed values seems to be in agreement with those of \"training.csv\".
", "votes": 7, "replies": [ { "id": 1582764, "author_name": "Branden Murray", "author_url": "", "post_date": "2021-11-15T08:43:23.347000", "content": "By changing the calculation of returns from log(P(t+16)/P(t+1)) to P(t+16)/P(t+1) - 1 I was able to reduce the MAE and maximum error in both your and @alexfir notebooks (https://www.kaggle.com/alexfir/recreating-target ). I'm starting to think they didn't actually use log returns when calculating their targets…
Found that (x - 1) is a upper bound for log(x). It looks like around x=1.0, log(x) and (x-1) are very close. Though I do not know why x-1 has better fit to Target than log(x).
", "votes": 4, "replies": [] }, { "id": 1583953, "author_name": "agenlu", "author_url": "", "post_date": "2021-11-16T06:54:47.597000", "content": "@brandenkmurray for the rows with beta==0 your formula gives 0 error. For the others, we have to figure out how they calculate m.
", "votes": 0, "replies": [] }, { "id": 1584677, "author_name": "ambank1", "author_url": "", "post_date": "2021-11-16T17:23:40.923000", "content": "@vi2018 it's not a 100% replication of the target; any insights on such target representation?
", "votes": 0, "replies": [] }, { "id": 1585065, "author_name": "Branden Murray", "author_url": "", "post_date": "2021-11-17T03:29:21.333000", "content": "I was able to modify @vi2018's notebook a bit to get the MAE down to 0.00003 and max error down to 0.015.
\nMain changes were
\nweighted_avg_market_log_returns calculation Still not perfect, but getting closer…
\n", "votes": 6, "replies": [ { "id": 1586086, "author_name": "Alexander Firsov", "author_url": "", "post_date": "2021-11-17T19:37:58.793000", "content": "Updated Target recreation by replacing log(x) => x-1 and using only available timestamps.
\nResults:
\nAverage absolute error 0.000000
\nMax absolute error 0.003798
\nStandard deviation 0.000001
@brandenkmurray I think the link you shared is not working or it isn't public. Thanks for the insights.
", "votes": 0, "replies": [] }, { "id": 1586046, "author_name": "Branden Murray", "author_url": "", "post_date": "2021-11-17T18:50:58.083000", "content": "@ambank1 Oops, you're right. Made it public.
", "votes": 2, "replies": [] }, { "id": 1588091, "author_name": "FCFCFC", "author_url": "", "post_date": "2021-11-19T07:54:12.210000", "content": "Just for those unfamiliar with calculus. Replacing $\\log(x)$ by $x-1$ comes from the first order Taylor series expansion:
\n$$
\n\\log(x) = \\log(1+x-1 ) = x-1 + O((x-1)^2)
\n$$
\nSo for
\n$$x\\approx 1, \\quad \\log(x)\\approx x-1.$$
\nReturns over 15 minutes tend to satisfy this condition (unless really wild volatility is present) and the log(x) function takes longer to calculate than the simple algebraic x-1. So I suppose the host used the approximation to speed up computation without losing much precision. Such performance gain might not play a big role with minute-long windows here, but I imagine it is relevant for very high frequency trading where low latency is paramount.
\n\nSo I suppose the host used the approximation to speed up computation without losing much precision
\n
Yeah, makes sense. But why did the hosts tell us they used log(x) when they didn't? And they haven't commented on it since it was raised 3 days ago that they actually used (x-1)…. Really wish they were more active in responding to questions/concerns.
", "votes": 6, "replies": [] } ] }, { "id": 1596306, "author_name": "ambank1", "author_url": "", "post_date": "2021-11-26T11:10:47.493000", "content": "@brandenkmurray maybe the company has forgotten to allocate some employee in this; plus crypto market is in the red sea now… XD
", "votes": 0, "replies": [] }, { "id": 1613127, "author_name": "jagofc", "author_url": "", "post_date": "2021-12-09T16:28:15.387000", "content": "Hi all, not sure where we're at with target reconstruction, but noticed that there are just 4 rows that are throwing off the reconstruction in the notebook by @alexfir .
\nHave produced a light notebook highlighting the offending rows. After removing the 4 rows, the max absolute error is down from 3.8e-03 to 2.8e-15.
\nInterested in your thoughts as to what's up with those 4 rows!
", "votes": 1, "replies": [] }, { "id": 1626405, "author_name": "jagofc", "author_url": "", "post_date": "2021-12-22T19:16:43.723000", "content": "Those 4 rows were 3749 timestamps after 2018-01-01 00:01. Adding in a row for 2018-01-01 00:00 reproduces the Target perfectly (max absolute error across whole train set < 1e-13).
\nFor anyone who's found their way to the end of this message chain - congrats! Trying to get some more clarification on how Target will be calculated going forward in a new post in this topic.
", "votes": 3, "replies": [] } ] }, { "id": 1651338, "author_name": "Ding", "author_url": "", "post_date": "2022-01-15T17:22:12.417000", "content": "I understand the Target formula as the residuals same as the following linear model:
\n$$R = \\beta * M + u$$
\nWhy not run linear regression with intercept and take the intercept($\\alpha$) as the residuals, which follows the regression model below?
\n$$R = \\alpha + \\beta * M + u$$
", "votes": 1, "replies": [] }, { "id": 1588049, "author_name": "LIGHT", "author_url": "", "post_date": "2021-11-19T07:21:42.440000", "content": "What's the window value in the ResidualizeMarket function if we want to replicate the given target values?
\nThanks!
Do we replace nan value by 0.0 during evaluation? Or we remove the nan values?
", "votes": 2, "replies": [ { "id": 1653636, "author_name": "jagofc", "author_url": "", "post_date": "2022-01-17T18:27:17.643000", "content": "See here
\n\n\nSome rows have null values for targets due to missing values in future prices. Rows with nulls in the test set ground truth are ignored for scoring purposes.
\n
and here
\n\n", "votes": 5, "replies": [] } ] }, { "id": 1585322, "author_name": "Xuân Tùng Nguyễn", "author_url": "", "post_date": "2021-11-17T08:37:10.577000", "content": "You will get an error if you[r] submission includes nulls or infinities.
\n
\"correlation has the useful property that it tends to normalize leading-order volatility out of the covariance between target and prediction\"
\nI would love to get some demonstration on this point but couldn't find one. Could you send me some references? Thanks
normalize leading-order volatility out of the covariance between target and prediction.
Please expand
", "votes": 2, "replies": [ { "id": 1589014, "author_name": "FCFCFC", "author_url": "", "post_date": "2021-11-19T20:44:38.903000", "content": "Comes from the definition of correlation:
\n$$
\n\\rho_{XY} = \\frac{cov(X,Y)}{\\sigma_X \\sigma_Y},
\n$$
\nwhere sigma is the volatility of X or Y. We want to know how much our predictions correlate with the target because that's all we need to know to decide whether to buy/sell. If you have a predictor that correlates perfectly (rho=1) with the target, you basically have a crystal ball telling you whether you will profit or lose in the next 15 min (your predictor goes up => market goes up and vice versa). Now suppose you have such a predictor P. I can equally create a perfect one by multiplying by 2: predictor Q=2P. We deserve the same score, and indeed this is the case by using correlation. If we were to use, say, covariance, my predictor Q would score twice as much as yours just because it has double volatility (standard dev). This would be unfair. That's why they \"normalise the volatility out\": so that we can compare scores with each other.
The use of correlation as metrics starts making sense to me. Thanks!
", "votes": 1, "replies": [] } ] }, { "id": 1661835, "author_name": "Maryna Antonevych", "author_url": "", "post_date": "2022-01-23T18:31:48.577000", "content": "thanks for sharing!
", "votes": 0, "replies": [ { "id": 1662894, "author_name": "Mark Babayev", "author_url": "", "post_date": "2022-01-24T16:10:20.967000", "content": "Hi. Want to team up? You can find me on discord.
", "votes": 0, "replies": [] } ] }, { "id": 1587982, "author_name": "rajesh", "author_url": "", "post_date": "2021-11-19T05:19:02.283000", "content": "I am sorry to ask this.I have no knowledge about stock market. There are so many target calculations seeing in this forum.are there any standards for target calculations?
\nWhat I would to know is at a particular time instance what is the actual value of a particular stock.if any one provide the way to study stock market I will be grateful to you.without knowing domain I cant do anything.
\nThanks in advance.
\n\nWhat I would to know is at a particular time instance what is the actual value of a particular stock.
\n
It is important to interpret what you mean by \"value\". If you mean the price of a share of a stock (or of 1 cryptocurrency as in here) you have to look who is selling for cheaper (if you want to buy) and who is buying for most (if you are willing to sell). This means that the \"price\" now depends on whether you want to buy or sell :) . The difference in these prices is the \"bid-ask spread\". This \"spread\" is always positive because if they were equal, a transaction would occur instantly and the spread reappears.
\nThis is not the data we have here though, we are looking at trades that already occurred. When someone actually buys (resp sells) the price they paid (resp received) is recorded. Over 1 minute you calculate how many transactions occurred (\"Count\") and how many cryptocurrencies changed hands (\"Volume\", because one trade might involve more than one unit) along with the other features given by the data.
\nMaybe what you wanted to know is simply what \"P(t)\" is in the returns formula. They defined it to be the close price in the end of minute t.
Thank you for sharing! Just as a side note, I think in the section Data -> \"example_test\" is missing the column row_id.
", "votes": 0, "replies": [] }, { "id": 1662915, "author_name": "", "author_url": "", "post_date": "2022-01-24T16:39:09.673000", "content": "", "votes": 0, "replies": [] }, { "id": 1659841, "author_name": "", "author_url": "", "post_date": "2022-01-22T06:26:04.373000", "content": "", "votes": 0, "replies": [] }, { "id": 1579987, "author_name": "", "author_url": "", "post_date": "2021-11-12T11:53:57.220000", "content": "", "votes": 0, "replies": [] }, { "id": 1579594, "author_name": "", "author_url": "", "post_date": "2021-11-12T03:16:49.083000", "content": "", "votes": 0, "replies": [] }, { "id": 1578386, "author_name": "", "author_url": "", "post_date": "2021-11-11T00:49:51.317000", "content": "", "votes": 0, "replies": [] }, { "id": 1659326, "author_name": "Jay Moghariya", "author_url": "", "post_date": "2022-01-21T17:49:02.717000", "content": "Thank you!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1578078": "Thanks Kagglers for your questions. We will group the key themes beginning with the eval metric, target formulation and weights: \n\n**Evaluation Metric**\nWhile mean squared error, R^2, explained variance, and correlation are all very closely related, correlation has the useful property that it tends to normalize leading-order volatility out of the covariance between target and prediction. In financial markets (especially crypto ones!), predicting volatility is a difficult (but interesting!) question in its own right. By using correlation as a metric we hope to remove some noise from the prediction problem and provide a more stable metric to evaluate against.\n\n**Target**\nThe target calculation is based on the close price of the asset and can be derived from the provided data using the methodology in https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition. The relevant function is provided below:\n\n```\ndef ResidualizeMarket(df, mktColumn, window):\n if mktColumn not in df.columns:\n return df\n\n mkt = df[mktColumn]\n\n num = df.multiply(mkt.values, axis=0).rolling(window).mean().values #numerator of linear regression coefficient\n denom = mkt.multiply(mkt.values, axis=0).rolling(window).mean().values #denominator of linear regression coefficient\n beta = np.nan_to_num( num.T / denom, nan=0., posinf=0., neginf=0.) #if regression fell over, use beta of 0\n\n resultRet = df - (beta * mkt.values).T #perform residualization\n resultBeta = 0.*df + beta.T #shape beta\n\n return resultRet.drop(columns=[mktColumn]), resultBeta.drop(columns=[mktColumn])\n```\n\n**Weights**\nIn this competition, the weights are determined by the logarithm of each product's market cap (in USD), of the cryptocurrencies at a fixed point in time. Weights were assigned to give more relevance to cryptocurrencies with higher market volumes to ensure smaller cryptocurrencies do not disproportionately impact your models.\n\n \nThanks everyone. Keep on Kaggling!\n", "1588962": "@andrewscherer: is still unclear to me how the eval metric is calculated. In the base exemple you mention: \n\" The competition performance metric is weighted correlation. However, for now we will use simple correlation to evaluate the two baseline models built.\" But you don't give a definition of weighted correlation anywhere. Some definition can be found on wikipedia. However you don't mention the formula and there are other interpretation for what you mention. \n\nIs it a simple weighing of the results ? (In the exemple you give that: BTC 0.01, ETH 0.05\nI see the weights are respectively 6.779922 and 5.894403. Does that mean the final eval would be: 0.01 * 6.77 + 0.05 * 5.89 / (6.77 + 5.89) ? \n\nOr is it the use of a weighted correlation metric as shown below ?\n\n```\n# https://stackoverflow.com/questions/38641691/weighted-correlation-coefficient-with-pandas\ndef wmean(x, w):\n return np.sum(x * w) / np.sum(w)\n\ndef wcov(x, y, w):\n return np.sum(w * (x - wmean(x, w)) * (y - wmean(y, w))) / np.sum(w)\n\ndef wcorr(x, y, w):\n return wcov(x, y, w) / np.sqrt(wcov(x, x, w) * wcov(y, y, w))\n\ndef eval_wcorr(preds, train_data):\n w = train_data.add_w.values.flatten()\n y_true = train_data.get_label()\n return 'eval_wcorr', wcorr(preds, y_true, w), True\n```\n", "1579014": "Thank you for sharing this code! It was really hard to replicate the target for some of us here!\n\nBut I have a question:\n\n`beta = np.nan_to_num( num.T / denom, nan=0., posinf=0., neginf=0.) #if regression fell over, use beta of 0`\n\nWhy? It doesn't make any sense to assign beta=0 if there are some problems with data. Beta is a sensitivity of an asset to market movements, it doesn't suddenly turn to 0 if you encounter None in your data. As a result, you end up with beta being mostly zero for some assets. Here we already discussed it: https://www.kaggle.com/alexfir/recreating-target/comments\n\nI actually calculated \"implied beta\" from provided targets:\n\n`data['beta_implied'] = (data['r'] - data['Target']) / data['m']`\n`# try different Asset_ID`\n`plt_data = np.clip(data.query(\"Asset_ID==0\")[['beta_implied']].values, -1, 2)`\n`plt.hist(plt_data,bins=100);`\n\nHistograms of \"beta_implied\":\n\n - Asset_ID==0 - https://ibb.co/ZWhNX89\n - Asset_ID==1 - https://ibb.co/YNvHk07\n - Asset_ID==6 - imagehttps://ibb.co/7k0F3BR\n\nAs you can see, Asset_ID==0 mostly has beta equal to zero, which doesn't make any financial sense, and also **makes it's impossible to predict the target, since we'll need to predict whatever next datapoints will be NA or not**.\n\nI think the organizers should fix the bug in their code and provide updated targets.", "1626395": "Trying to bring clarification of the Target calculation back into discussion, because the chain of messages [here](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/286778#1582306) might have lost some readers!\n\nThe upshot is that we can reproduce the Target across the whole train set with max absolute error < 1e-13. \nThis had to be achieved using a method which was materially different from the definition suggested in the [tutorial notebook](https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition) which states that \"the rolling average over time (3750 minute windows)\" was used. It seems that the calculation uses the rolling average over 3750 (non-empty) timestamps instead. \n\nReprex: https://www.kaggle.com/jagofc/target-reconstruction-quick-summary-of-sota\n\nHappy to play by whatever rules the hosts set for the game. Given that we have had to use a different definition of the target in order to reproduce the train set values, there is still some uncertainty around what the rules are though : )\n\nPlease could one of the helpful people from Kaggle @sohier or G-Research @andrewscherer confirm whether the Target calculation method will remain the same for the evaluation period of the competition as it was for the train set, or whether it will be altered to align with the definition in the tutorial notebook?", "1582965": "@andrewscherer \n\nWouldn't it be easier for you guys to share a notebook to re-create the target than sharing part of the script? ", "1578201": "The `ResidualizeMarket` function you provided produces a SyntaxError because of `resultBeta = 0.*df +**** beta.T #shape beta`. What is `+****` supposed to mean?\n\nAlso, is `resultRet` supposed to be what returns the `Target` column? The results do not match what is in the `Target` column and actually have a higher error than what others have tried to reproduce in the forum/notebooks. ", "1578387": "Hello, @andrewscherer !\nThank you very much for sharing `ResidualizeMarket` function.\n\nBut it doesn't help in recreating target values 😪\n**Lots of people** in that competition including me are **blowing their minds to recreate targets** and no solution for now!\n\n**Can you please share the full code on how we can recreate targets in train.csv?**\n\nHere is a[ notebook](https://www.kaggle.com/ebudia/recreating-target-min-periods-3750) from @ebudia trying to recreate target. I've gone through that notebook and found only one mistake with weights agregation:\n```\nm = time_group['weighted_asset_r'].sum() / time_group['w'].sum()\n# I think here only weights, where R(t) is not NaN should be summed, but this code sum all weights\n\n```\n\nI've also played a lot with different prices, time windows and agregations, but can not recreate target!\nThe very simple test can not be passed by my algorithms. Just recreate target for first minute (2018-01-01 00:01:00) in train dataset. \nAccording to your code I assume, that for that first minute:\n`