{ "id": 285624, "title": "**Not clear the way beta_a is calculated**", "url": "/competitions/g-research-crypto-forecasting/discussion/285624", "author_name": "", "post_date": "2021-11-05T14:42:38.687885800Z", "votes": 16, "comment_count": 15, "views": 0, "content": "

I'm trying to reproduce the target creation

\n

First I create log_returns series for each asset

\n
def log_return(series, periods=1):\n    return np.log(series).diff(periods=periods)\n# grouping on asset\ncrypto_df[\"log_return\"] = crypto_df.groupby(\"Asset_ID\")[\"Close\"].transform(lambda x: log_return(x, 15))\n
\n

Then I'm going to calculate M(t)

\n
def weight_average_market_returns(df):\n    return (df.Weight * df.log_return).sum() / df.Weight.sum()\n# Add weight\ncrypto_df= crypto_df.merge(asset_details.drop(\"Asset_Name\", axis=1), on=\"Asset_ID\", how=\"left\")\n# grouping on timestamp\nm_df = crypto_df_sample.groupby(\"timestamp\").apply(weight_average_market_returns).rename(\"M\").reset_index()\ncrypto_df_sample = crypto_df_sample.merge(m_df, on=\"timestamp\", how='left')\n
\n

But, for the beta step is not clear to me what to do, the project documentation specifies:

\n

This forecasting competition aims to predict returns in the near future for prices $P^a$, for each asset $a$. For each row in the dataset, we include the target for prediction, Target. Target is derived from log returns ($R^a$) over 15 minutes.

\n

$$R^a(t) = log (P^a(t+16)\\ /\\ P^a(t+1))$$

\n

Crypto asset returns are highly correlated, following to a large extend the overall crypto market. As we want to test your ability to predict returns for individual assets, we perform a linear residualization, removing the market signal from individual asset returns when creating the target. In more detail, if $M(t)$ is the weighted average market returns, the target is:

\n

$$M(t) = \\frac{\\sum_a w^a R^a(t)}{\\sum_a w^a} \\
\n\\beta^a = \\frac{\\langle M \\cdot R^a \\rangle}{\\langle M^2 \\rangle} \\
\n\\text{Target}^a(t) = R^a(t) - \\beta^a M(t)$$

\n

where the bracket $\\langle .\\rangle$ represent the rolling average over time (3750 minute windows), and same asset weights $w^a$ used for the evaluation metric.

\n

I tried this approach, but it doesn't seem to be right:

\n
def beta(df):\n    df.set_index(\"timestamp\", inplace=True) \n    return (df.M * df.log_return).rolling(3750).mean().sum() / (df.M * df.M).rolling(3750).mean().sum()\nbeta_df = crypto_df_sample.groupby(\"Asset_ID\").apply(beta).rename(\"beta\").to_frame().reset_index()\ncrypto_df_sample = crypto_df_sample.merge(beta_df, on=\"Asset_ID\", how='left')\n
\n

Is there anybody who solve this?

", "messages": [ { "id": "1572236", "postDate": "11/05/2021 14:42:38", "content": "

I'm trying to reproduce the target creation

\n

First I create log_returns series for each asset

\n
def log_return(series, periods=1):\n    return np.log(series).diff(periods=periods)\n# grouping on asset\ncrypto_df[\"log_return\"] = crypto_df.groupby(\"Asset_ID\")[\"Close\"].transform(lambda x: log_return(x, 15))\n
\n

Then I'm going to calculate M(t)

\n
def weight_average_market_returns(df):\n    return (df.Weight * df.log_return).sum() / df.Weight.sum()\n# Add weight\ncrypto_df= crypto_df.merge(asset_details.drop(\"Asset_Name\", axis=1), on=\"Asset_ID\", how=\"left\")\n# grouping on timestamp\nm_df = crypto_df_sample.groupby(\"timestamp\").apply(weight_average_market_returns).rename(\"M\").reset_index()\ncrypto_df_sample = crypto_df_sample.merge(m_df, on=\"timestamp\", how='left')\n
\n

But, for the beta step is not clear to me what to do, the project documentation specifies:

\n

This forecasting competition aims to predict returns in the near future for prices $P^a$, for each asset $a$. For each row in the dataset, we include the target for prediction, Target. Target is derived from log returns ($R^a$) over 15 minutes.

\n

$$R^a(t) = log (P^a(t+16)\\ /\\ P^a(t+1))$$

\n

Crypto asset returns are highly correlated, following to a large extend the overall crypto market. As we want to test your ability to predict returns for individual assets, we perform a linear residualization, removing the market signal from individual asset returns when creating the target. In more detail, if $M(t)$ is the weighted average market returns, the target is:

\n

$$M(t) = \\frac{\\sum_a w^a R^a(t)}{\\sum_a w^a} \\
\n\\beta^a = \\frac{\\langle M \\cdot R^a \\rangle}{\\langle M^2 \\rangle} \\
\n\\text{Target}^a(t) = R^a(t) - \\beta^a M(t)$$

\n

where the bracket $\\langle .\\rangle$ represent the rolling average over time (3750 minute windows), and same asset weights $w^a$ used for the evaluation metric.

\n

I tried this approach, but it doesn't seem to be right:

\n
def beta(df):\n    df.set_index(\"timestamp\", inplace=True) \n    return (df.M * df.log_return).rolling(3750).mean().sum() / (df.M * df.M).rolling(3750).mean().sum()\nbeta_df = crypto_df_sample.groupby(\"Asset_ID\").apply(beta).rename(\"beta\").to_frame().reset_index()\ncrypto_df_sample = crypto_df_sample.merge(beta_df, on=\"Asset_ID\", how='left')\n
\n

Is there anybody who solve this?

", "rawMarkdown": "I'm trying to reproduce the target creation\n\nFirst I create log_returns series for each asset\n\n```python\ndef log_return(series, periods=1):\n return np.log(series).diff(periods=periods)\n# grouping on asset\ncrypto_df[\"log_return\"] = crypto_df.groupby(\"Asset_ID\")[\"Close\"].transform(lambda x: log_return(x, 15))\n```\n\nThen I'm going to calculate M(t)\n\n```python\ndef weight_average_market_returns(df):\n return (df.Weight * df.log_return).sum() / df.Weight.sum()\n# Add weight\ncrypto_df= crypto_df.merge(asset_details.drop(\"Asset_Name\", axis=1), on=\"Asset_ID\", how=\"left\")\n# grouping on timestamp\nm_df = crypto_df_sample.groupby(\"timestamp\").apply(weight_average_market_returns).rename(\"M\").reset_index()\ncrypto_df_sample = crypto_df_sample.merge(m_df, on=\"timestamp\", how='left')\n```\n\n**But**, for the beta step is not clear to me what to do, the project documentation specifies:\n\nThis forecasting competition aims to predict returns in the near future for prices $P^a$, for each asset $a$. For each row in the dataset, we include the target for prediction, `Target`. `Target` is derived from log returns ($R^a$) over 15 minutes.\n\n$$R^a(t) = log (P^a(t+16)\\ /\\ P^a(t+1))$$\n\nCrypto asset returns are highly correlated, following to a large extend the overall crypto market. As we want to test your ability to predict returns for individual assets, we perform a linear residualization, removing the market signal from individual asset returns when creating the target. In more detail, if $M(t)$ is the weighted average market returns, the target is:\n\n$$M(t) = \\frac{\\sum_a w^a R^a(t)}{\\sum_a w^a} \\\\\n\\beta^a = \\frac{\\langle M \\cdot R^a \\rangle}{\\langle M^2 \\rangle} \\\\\n\\text{Target}^a(t) = R^a(t) - \\beta^a M(t)$$\n\nwhere the bracket $\\langle .\\rangle$ represent the rolling average over time (3750 minute windows), and same asset weights $w^a$ used for the evaluation metric.\n\nI tried this approach, but it doesn't seem to be right:\n\n```python\ndef beta(df):\n df.set_index(\"timestamp\", inplace=True) \n return (df.M * df.log_return).rolling(3750).mean().sum() / (df.M * df.M).rolling(3750).mean().sum()\nbeta_df = crypto_df_sample.groupby(\"Asset_ID\").apply(beta).rename(\"beta\").to_frame().reset_index()\ncrypto_df_sample = crypto_df_sample.merge(beta_df, on=\"Asset_ID\", how='left')\n```\n\nIs there anybody who solve this?", "votes": null }, { "id": "1572386", "postDate": "11/05/2021 16:17:44", "content": "

Thank you for sharing this. I've been trying to replicate targets as well, but with no success so far. I think organizers should share their code to avoid any misunderstandings on target computation.

", "rawMarkdown": "Thank you for sharing this. I've been trying to replicate targets as well, but with no success so far. I think organizers should share their code to avoid any misunderstandings on target computation.", "votes": null }, { "id": "1572408", "postDate": "11/05/2021 16:30:55", "content": "

Yes, I agree with you, it would be helpful

", "rawMarkdown": "Yes, I agree with you, it would be helpful", "votes": null }, { "id": "1573254", "postDate": "11/06/2021 12:44:15", "content": "

You can check my notebook of how beta is calculated (Please note the resulting Target is close, but not ideal match)

", "rawMarkdown": "You can check my [notebook ](https://www.kaggle.com/alexfir/recreating-target)of how beta is calculated (Please note the resulting Target is close, but not ideal match)", "votes": null }, { "id": "1575107", "postDate": "11/08/2021 06:59:15", "content": "

I don't think they've properly described how the Target is calculated. Posted this in @alexfir's notebook, but I'll post it again here:

\n

One thing I've noticed is that when Target==0, then R(t)==0.

\n

If they are both 0 then Target(t)=R(t)−βM(t) implies that 0 = βM(t), so either β or M(t) must be 0, but this is pretty much never the case. If R(t) = 0 then the Target should be non-zero after removing the market signal (unless there is no market signal, which is almost never). Something is clearly wrong with the way they've described how the Target, β, and/or M(t) are calculated.

", "rawMarkdown": "I don't think they've properly described how the Target is calculated. Posted this in @alexfir's notebook, but I'll post it again here:\n\nOne thing I've noticed is that when Target==0, then R(t)==0.\n\nIf they are both 0 then Target(t)=R(t)−βM(t) implies that 0 = βM(t), so either β or M(t) must be 0, but this is pretty much never the case. If R(t) = 0 then the Target should be non-zero after removing the market signal (unless there is no market signal, which is almost never). Something is clearly wrong with the way they've described how the Target, β, and/or M(t) are calculated.", "votes": null }, { "id": "1575379", "postDate": "11/08/2021 11:06:31", "content": "

Your comment helped me make the following discovery:

\n", "rawMarkdown": "Your comment helped me make the following discovery:\n- for the assets 0, 8, 10, 11, 13: Target(t)=R(t) - without any removal of the market signal\n- for all other assets Target(t)=R(t)−βM(t) as stated\nwith this adjustment, I can reproduce the targets near perfectly, but still it seems very strange. Can someone confirm this?", "votes": null }, { "id": "1575711", "postDate": "11/08/2021 15:37:21", "content": "

Awesome finding, did you calculate the mean absolute error? It positive, could you share us what error value you got?

\n

Also, did you use 0, 8, 10, 11, 13 to calculate Mt or did you skip them?

", "rawMarkdown": "Awesome finding, did you calculate the mean absolute error? It positive, could you share us what error value you got?\n\nAlso, did you use 0, 8, 10, 11, 13 to calculate Mt or did you skip them?", "votes": null }, { "id": "1576398", "postDate": "11/09/2021 06:26:58", "content": "

If it is indeed like this, then it is really weird. Definitely need clarification from the competition organizers on this matter. I hope that kaggle team is working on it.

", "rawMarkdown": "If it is indeed like this, then it is really weird. Definitely need clarification from the competition organizers on this matter. I hope that kaggle team is working on it.", "votes": null }, { "id": "1576512", "postDate": "11/09/2021 09:08:12", "content": "

As an example, target values for all 14 transaction at timestamp=1632096360, sorted by the Asset_ID:

\n\n

We can see that the second method reproduces the targets almost exactly. As for the MAE, the first method gives 2.11e-3, while the second gives 2.33e-5. In all the cases I always used all 14 assets to calculate M(t).

\n

That being said, I only analyzed the 20000 most recent transactions. When I looked at earlier transactions, sometimes the first method gives the exact answers for all the assets! So we can conclude that either Target(t)=R(t)−βM(t) or Target(t)=R(t) always matches the target, but which one to take depends on the time and the asset. Definitely this need more digging in to understand what's going on.

", "rawMarkdown": "As an example, target values for all 14 transaction at `timestamp=1632096360`, sorted by the Asset_ID:\n- from the train dataset:\n`[-0.00842883 0.00215716 -0.00118596 -0.0037673 -0.00648235 -0.00971452 0.00361398 -0.00157314 -0.01497166 0.00697933 -0.00972777 -0.00874452\n -0.00327159 -0.01140754]`\n- reconstructed using Target(t)=R(t)−βM(t) for all assets:\n`[ 0.00123604 0.0021575 -0.00124577 -0.00384373 -0.00657207 -0.00997308 0.00361239 -0.00162466 0.00047076 0.00700432 0.00219473 0.00232475 -0.00333273 0.00151785]`\n- reconstructed using Target(t)=R(t) for assets 0, 8, 10, 11, 13:\n`[-0.00846456 0.0021575 -0.00124577 -0.00384373 -0.00657207 -0.00997308 0.00361239 -0.00162466 -0.01508486 0.00700432 -0.0097754 -0.00878297 -0.00333273 -0.0114731 ]`\n\nWe can see that the second method reproduces the targets almost exactly. As for the MAE, the first method gives 2.11e-3, while the second gives 2.33e-5. In all the cases I always used all 14 assets to calculate M(t).\n\nThat being said, I only analyzed the 20000 most recent transactions. When I looked at earlier transactions, sometimes the first method gives the exact answers for all the assets! So we can conclude that either Target(t)=R(t)−βM(t) or Target(t)=R(t) always matches the target, but which one to take depends on the time and the asset. Definitely this need more digging in to understand what's going on.", "votes": null }, { "id": "1576527", "postDate": "11/09/2021 09:24:29", "content": "

This is a very interesting find! Earlier in a @alexfir notebook I saw that while MAE was low, max error was still quite high (for very few examples of course), can you please check max errors with your method and see whether they have improved?

", "rawMarkdown": "This is a very interesting find! Earlier in a @alexfir [notebook](https://www.kaggle.com/alexfir/recreating-target/notebook) I saw that while MAE was low, max error was still quite high (for very few examples of course), can you please check max errors with your method and see whether they have improved?", "votes": null }, { "id": "1576673", "postDate": "11/09/2021 11:36:00", "content": "

I suspect it is going to be nearly impossible to create good models with that level of noise on target calculation.

", "rawMarkdown": "I suspect it is going to be nearly impossible to create good models with that level of noise on target calculation.", "votes": null }, { "id": "1577087", "postDate": "11/09/2021 18:44:23", "content": "

βM(t) = 0 if the previous 3750 rows there is a gap in the data, look this:

\n

https://www.kaggle.com/ebudia/recreating-target-min-periods-3750

", "rawMarkdown": "βM(t) = 0 if the previous 3750 rows there is a gap in the data, look this:\n\nhttps://www.kaggle.com/ebudia/recreating-target-min-periods-3750", "votes": null }, { "id": "1577646", "postDate": "11/10/2021 09:14:00", "content": "

Max error also goes down significantly: from 4.6e-2 to 1.1e-3, but again this is for the 20000 most recent transactions. Not sure what happens with earlier transactions, but I see a sharp drop in MAE vs time, which suggests that the data was assembled from different parts, and the targets for these parts were calculated differently.

", "rawMarkdown": "Max error also goes down significantly: from 4.6e-2 to 1.1e-3, but again this is for the 20000 most recent transactions. Not sure what happens with earlier transactions, but I see a sharp drop in MAE vs time, which suggests that the data was assembled from different parts, and the targets for these parts were calculated differently.", "votes": null }, { "id": "1577836", "postDate": "11/10/2021 12:59:24", "content": "

It seems that rolling(window, min_periods=window) better match the target.

", "rawMarkdown": "It seems that rolling(window, min_periods=window) better match the target.", "votes": null }, { "id": "1578280", "postDate": "11/10/2021 20:11:00", "content": "

I similarly cannot calculate Target accurately…

\n

For example, with Bitcoin @ 1514764860 (the very first observation):

\n

Target_bitcoin = -0.014643224
\nlog_return_bitcoin(t+1 to t+16) = ln(13625.614 / 13828.102) = -0.014751495

\n

If beta is supposed to be zero, then these should match. If beta is nonzero, then you need to calculate M, which I get as -0.010012 in this first observation. I can't get these values to match up in a sensible way as beta is very close to zero then (should be around 1). Really just posting here to track additional comments.

", "rawMarkdown": "I similarly cannot calculate Target accurately...\n\nFor example, with Bitcoin @ 1514764860 (the very first observation):\n\nTarget_bitcoin = -0.014643224\nlog_return_bitcoin(t+1 to t+16) = ln(13625.614 / 13828.102) = -0.014751495\n\nIf beta is supposed to be zero, then these should match. If beta is nonzero, then you need to calculate M, which I get as -0.010012 in this first observation. I can't get these values to match up in a sensible way as beta is very close to zero then (should be around 1). Really just posting here to track additional comments.", "votes": null }, { "id": "1599604", "postDate": "11/29/2021 15:45:04", "content": "

Ofcourse beta is just the rolling average of M squared which if 0 means R is 0 too you get 0 as the Target

", "rawMarkdown": "Ofcourse beta is just the rolling average of M squared which if 0 means R is 0 too you get 0 as the Target", "votes": null } ], "comments": [ { "id": 1572386, "author_name": "npismenniy", "author_url": "", "post_date": "11/05/2021 16:17:44", "content": "

Thank you for sharing this. I've been trying to replicate targets as well, but with no success so far. I think organizers should share their code to avoid any misunderstandings on target computation.

", "votes": null, "replies": [ { "id": 1572408, "author_name": "romulomadu", "author_url": "", "post_date": "11/05/2021 16:30:55", "content": "

Yes, I agree with you, it would be helpful

", "votes": null, "replies": [] } ] }, { "id": 1573254, "author_name": "alexfir", "author_url": "", "post_date": "11/06/2021 12:44:15", "content": "

You can check my notebook of how beta is calculated (Please note the resulting Target is close, but not ideal match)

", "votes": null, "replies": [] }, { "id": 1575107, "author_name": "brandenkmurray", "author_url": "", "post_date": "11/08/2021 06:59:15", "content": "

I don't think they've properly described how the Target is calculated. Posted this in @alexfir's notebook, but I'll post it again here:

\n

One thing I've noticed is that when Target==0, then R(t)==0.

\n

If they are both 0 then Target(t)=R(t)−βM(t) implies that 0 = βM(t), so either β or M(t) must be 0, but this is pretty much never the case. If R(t) = 0 then the Target should be non-zero after removing the market signal (unless there is no market signal, which is almost never). Something is clearly wrong with the way they've described how the Target, β, and/or M(t) are calculated.

", "votes": null, "replies": [ { "id": 1575379, "author_name": "sergears", "author_url": "", "post_date": "11/08/2021 11:06:31", "content": "

Your comment helped me make the following discovery:

\n", "votes": null, "replies": [] }, { "id": 1575711, "author_name": "romulomadu", "author_url": "", "post_date": "11/08/2021 15:37:21", "content": "

Awesome finding, did you calculate the mean absolute error? It positive, could you share us what error value you got?

\n

Also, did you use 0, 8, 10, 11, 13 to calculate Mt or did you skip them?

", "votes": null, "replies": [] }, { "id": 1576398, "author_name": "olegsidorshin", "author_url": "", "post_date": "11/09/2021 06:26:58", "content": "

If it is indeed like this, then it is really weird. Definitely need clarification from the competition organizers on this matter. I hope that kaggle team is working on it.

", "votes": null, "replies": [] }, { "id": 1576512, "author_name": "sergears", "author_url": "", "post_date": "11/09/2021 09:08:12", "content": "

As an example, target values for all 14 transaction at timestamp=1632096360, sorted by the Asset_ID:

\n\n

We can see that the second method reproduces the targets almost exactly. As for the MAE, the first method gives 2.11e-3, while the second gives 2.33e-5. In all the cases I always used all 14 assets to calculate M(t).

\n

That being said, I only analyzed the 20000 most recent transactions. When I looked at earlier transactions, sometimes the first method gives the exact answers for all the assets! So we can conclude that either Target(t)=R(t)−βM(t) or Target(t)=R(t) always matches the target, but which one to take depends on the time and the asset. Definitely this need more digging in to understand what's going on.

", "votes": null, "replies": [] }, { "id": 1576527, "author_name": "olegsidorshin", "author_url": "", "post_date": "11/09/2021 09:24:29", "content": "

This is a very interesting find! Earlier in a @alexfir notebook I saw that while MAE was low, max error was still quite high (for very few examples of course), can you please check max errors with your method and see whether they have improved?

", "votes": null, "replies": [] }, { "id": 1576673, "author_name": "romulomadu", "author_url": "", "post_date": "11/09/2021 11:36:00", "content": "

I suspect it is going to be nearly impossible to create good models with that level of noise on target calculation.

", "votes": null, "replies": [] }, { "id": 1577646, "author_name": "sergears", "author_url": "", "post_date": "11/10/2021 09:14:00", "content": "

Max error also goes down significantly: from 4.6e-2 to 1.1e-3, but again this is for the 20000 most recent transactions. Not sure what happens with earlier transactions, but I see a sharp drop in MAE vs time, which suggests that the data was assembled from different parts, and the targets for these parts were calculated differently.

", "votes": null, "replies": [] }, { "id": 1599604, "author_name": "swaralipibose", "author_url": "", "post_date": "11/29/2021 15:45:04", "content": "

Ofcourse beta is just the rolling average of M squared which if 0 means R is 0 too you get 0 as the Target

", "votes": null, "replies": [] } ] }, { "id": 1577087, "author_name": "ebudia", "author_url": "", "post_date": "11/09/2021 18:44:23", "content": "

βM(t) = 0 if the previous 3750 rows there is a gap in the data, look this:

\n

https://www.kaggle.com/ebudia/recreating-target-min-periods-3750

", "votes": null, "replies": [] }, { "id": 1577836, "author_name": "lucasmorin", "author_url": "", "post_date": "11/10/2021 12:59:24", "content": "

It seems that rolling(window, min_periods=window) better match the target.

", "votes": null, "replies": [] }, { "id": 1578280, "author_name": "mmojz642", "author_url": "", "post_date": "11/10/2021 20:11:00", "content": "

I similarly cannot calculate Target accurately…

\n

For example, with Bitcoin @ 1514764860 (the very first observation):

\n

Target_bitcoin = -0.014643224
\nlog_return_bitcoin(t+1 to t+16) = ln(13625.614 / 13828.102) = -0.014751495

\n

If beta is supposed to be zero, then these should match. If beta is nonzero, then you need to calculate M, which I get as -0.010012 in this first observation. I can't get these values to match up in a sensible way as beta is very close to zero then (should be around 1). Really just posting here to track additional comments.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1572236": "I'm trying to reproduce the target creation\n\nFirst I create log_returns series for each asset\n\n```python\ndef log_return(series, periods=1):\n return np.log(series).diff(periods=periods)\n# grouping on asset\ncrypto_df[\"log_return\"] = crypto_df.groupby(\"Asset_ID\")[\"Close\"].transform(lambda x: log_return(x, 15))\n```\n\nThen I'm going to calculate M(t)\n\n```python\ndef weight_average_market_returns(df):\n return (df.Weight * df.log_return).sum() / df.Weight.sum()\n# Add weight\ncrypto_df= crypto_df.merge(asset_details.drop(\"Asset_Name\", axis=1), on=\"Asset_ID\", how=\"left\")\n# grouping on timestamp\nm_df = crypto_df_sample.groupby(\"timestamp\").apply(weight_average_market_returns).rename(\"M\").reset_index()\ncrypto_df_sample = crypto_df_sample.merge(m_df, on=\"timestamp\", how='left')\n```\n\n**But**, for the beta step is not clear to me what to do, the project documentation specifies:\n\nThis forecasting competition aims to predict returns in the near future for prices $P^a$, for each asset $a$. For each row in the dataset, we include the target for prediction, `Target`. `Target` is derived from log returns ($R^a$) over 15 minutes.\n\n$$R^a(t) = log (P^a(t+16)\\ /\\ P^a(t+1))$$\n\nCrypto asset returns are highly correlated, following to a large extend the overall crypto market. As we want to test your ability to predict returns for individual assets, we perform a linear residualization, removing the market signal from individual asset returns when creating the target. In more detail, if $M(t)$ is the weighted average market returns, the target is:\n\n$$M(t) = \\frac{\\sum_a w^a R^a(t)}{\\sum_a w^a} \\\\\n\\beta^a = \\frac{\\langle M \\cdot R^a \\rangle}{\\langle M^2 \\rangle} \\\\\n\\text{Target}^a(t) = R^a(t) - \\beta^a M(t)$$\n\nwhere the bracket $\\langle .\\rangle$ represent the rolling average over time (3750 minute windows), and same asset weights $w^a$ used for the evaluation metric.\n\nI tried this approach, but it doesn't seem to be right:\n\n```python\ndef beta(df):\n df.set_index(\"timestamp\", inplace=True) \n return (df.M * df.log_return).rolling(3750).mean().sum() / (df.M * df.M).rolling(3750).mean().sum()\nbeta_df = crypto_df_sample.groupby(\"Asset_ID\").apply(beta).rename(\"beta\").to_frame().reset_index()\ncrypto_df_sample = crypto_df_sample.merge(beta_df, on=\"Asset_ID\", how='left')\n```\n\nIs there anybody who solve this?", "1572386": "Thank you for sharing this. I've been trying to replicate targets as well, but with no success so far. I think organizers should share their code to avoid any misunderstandings on target computation.", "1572408": "Yes, I agree with you, it would be helpful", "1573254": "You can check my [notebook ](https://www.kaggle.com/alexfir/recreating-target)of how beta is calculated (Please note the resulting Target is close, but not ideal match)", "1575107": "I don't think they've properly described how the Target is calculated. Posted this in @alexfir's notebook, but I'll post it again here:\n\nOne thing I've noticed is that when Target==0, then R(t)==0.\n\nIf they are both 0 then Target(t)=R(t)−βM(t) implies that 0 = βM(t), so either β or M(t) must be 0, but this is pretty much never the case. If R(t) = 0 then the Target should be non-zero after removing the market signal (unless there is no market signal, which is almost never). Something is clearly wrong with the way they've described how the Target, β, and/or M(t) are calculated.", "1575379": "Your comment helped me make the following discovery:\n- for the assets 0, 8, 10, 11, 13: Target(t)=R(t) - without any removal of the market signal\n- for all other assets Target(t)=R(t)−βM(t) as stated\nwith this adjustment, I can reproduce the targets near perfectly, but still it seems very strange. Can someone confirm this?", "1575711": "Awesome finding, did you calculate the mean absolute error? It positive, could you share us what error value you got?\n\nAlso, did you use 0, 8, 10, 11, 13 to calculate Mt or did you skip them?", "1576398": "If it is indeed like this, then it is really weird. Definitely need clarification from the competition organizers on this matter. I hope that kaggle team is working on it.", "1576512": "As an example, target values for all 14 transaction at `timestamp=1632096360`, sorted by the Asset_ID:\n- from the train dataset:\n`[-0.00842883 0.00215716 -0.00118596 -0.0037673 -0.00648235 -0.00971452 0.00361398 -0.00157314 -0.01497166 0.00697933 -0.00972777 -0.00874452\n -0.00327159 -0.01140754]`\n- reconstructed using Target(t)=R(t)−βM(t) for all assets:\n`[ 0.00123604 0.0021575 -0.00124577 -0.00384373 -0.00657207 -0.00997308 0.00361239 -0.00162466 0.00047076 0.00700432 0.00219473 0.00232475 -0.00333273 0.00151785]`\n- reconstructed using Target(t)=R(t) for assets 0, 8, 10, 11, 13:\n`[-0.00846456 0.0021575 -0.00124577 -0.00384373 -0.00657207 -0.00997308 0.00361239 -0.00162466 -0.01508486 0.00700432 -0.0097754 -0.00878297 -0.00333273 -0.0114731 ]`\n\nWe can see that the second method reproduces the targets almost exactly. As for the MAE, the first method gives 2.11e-3, while the second gives 2.33e-5. In all the cases I always used all 14 assets to calculate M(t).\n\nThat being said, I only analyzed the 20000 most recent transactions. When I looked at earlier transactions, sometimes the first method gives the exact answers for all the assets! So we can conclude that either Target(t)=R(t)−βM(t) or Target(t)=R(t) always matches the target, but which one to take depends on the time and the asset. Definitely this need more digging in to understand what's going on.", "1576527": "This is a very interesting find! Earlier in a @alexfir [notebook](https://www.kaggle.com/alexfir/recreating-target/notebook) I saw that while MAE was low, max error was still quite high (for very few examples of course), can you please check max errors with your method and see whether they have improved?", "1576673": "I suspect it is going to be nearly impossible to create good models with that level of noise on target calculation.", "1577087": "βM(t) = 0 if the previous 3750 rows there is a gap in the data, look this:\n\nhttps://www.kaggle.com/ebudia/recreating-target-min-periods-3750", "1577646": "Max error also goes down significantly: from 4.6e-2 to 1.1e-3, but again this is for the 20000 most recent transactions. Not sure what happens with earlier transactions, but I see a sharp drop in MAE vs time, which suggests that the data was assembled from different parts, and the targets for these parts were calculated differently.", "1577836": "It seems that rolling(window, min_periods=window) better match the target.", "1578280": "I similarly cannot calculate Target accurately...\n\nFor example, with Bitcoin @ 1514764860 (the very first observation):\n\nTarget_bitcoin = -0.014643224\nlog_return_bitcoin(t+1 to t+16) = ln(13625.614 / 13828.102) = -0.014751495\n\nIf beta is supposed to be zero, then these should match. If beta is nonzero, then you need to calculate M, which I get as -0.010012 in this first observation. I can't get these values to match up in a sensible way as beta is very close to zero then (should be around 1). Really just posting here to track additional comments.", "1599604": "Ofcourse beta is just the rolling average of M squared which if 0 means R is 0 too you get 0 as the Target" }, "source": "meta" }