{
  "id": 297999,
  "title": "Eval Metric, Target, Weights",
  "url": "/competitions/g-research-crypto-forecasting/discussion/297999",
  "author_name": "dewoweb",
  "post_date": "2021-12-31T02:55:47.429000",
  "votes": -3,
  "comment_count": 0,
  "views": 0,
  "content": "<p>Eval Metric, Target, Weights<br>\nPosted in g-research-crypto-forecasting 2 months ago<br>\n80</p>\n<p>Thanks Kagglers for your questions. We will group the key themes beginning with the eval metric, target formulation and weights:</p>\n<p>Evaluation Metric<br>\nWhile mean squared error, R^2, explained variance, and correlation are all very closely related, correlation has the useful property that it tends to normalize leading-order volatility out of the covariance between target and prediction. In financial markets (especially crypto ones!), predicting volatility is a difficult (but interesting!) question in its own right. By using correlation as a metric we hope to remove some noise from the prediction problem and provide a more stable metric to evaluate against.</p>\n<p>Target<br>\nThe target calculation is based on the close price of the asset and can be derived from the provided data using the methodology in <a href=\"https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition\" target=\"_blank\">https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition</a>. The relevant function is provided below:</p>\n<p>def ResidualizeMarket(df, mktColumn, window):<br>\n  if mktColumn not in df.columns:<br>\n    return df</p>\n<p>mkt = df[mktColumn]</p>\n<p>num = df.multiply(mkt.values, axis=0).rolling(window).mean().values  #numerator of linear regression coefficient<br>\n  denom = mkt.multiply(mkt.values, axis=0).rolling(window).mean().values  #denominator of linear regression coefficient<br>\n  beta = np.nan_to_num( num.T / denom, nan=0., posinf=0., neginf=0.)  #if regression fell over, use beta of 0</p>\n<p>resultRet = df - (beta * mkt.values).T  #perform residualization<br>\n  resultBeta = 0.*df + beta.T  #shape beta</p>\n<p>return resultRet.drop(columns=[mktColumn]), resultBeta.drop(columns=[mktColumn])<br>\nWeights<br>\nIn this competition, the weights are determined by the logarithm of each product's market cap (in USD), of the cryptocurrencies at a fixed point in time. Weights were assigned to give more relevance to cryptocurrencies with higher market volumes to ensure smaller cryptocurrencies do not disproportionately impact your models.</p>\n<p>Thanks everyone. Keep on Kaggling!<br>\n<a href=\"https://www.dewoweb.com\" target=\"_blank\">dewoweb</a></p>",
  "messages": [
    {
      "id": 1633747,
      "postDate": "2021-12-31T02:55:47.430Z",
      "content": "<p>Eval Metric, Target, Weights<br>\nPosted in g-research-crypto-forecasting 2 months ago<br>\n80</p>\n<p>Thanks Kagglers for your questions. We will group the key themes beginning with the eval metric, target formulation and weights:</p>\n<p>Evaluation Metric<br>\nWhile mean squared error, R^2, explained variance, and correlation are all very closely related, correlation has the useful property that it tends to normalize leading-order volatility out of the covariance between target and prediction. In financial markets (especially crypto ones!), predicting volatility is a difficult (but interesting!) question in its own right. By using correlation as a metric we hope to remove some noise from the prediction problem and provide a more stable metric to evaluate against.</p>\n<p>Target<br>\nThe target calculation is based on the close price of the asset and can be derived from the provided data using the methodology in <a href=\"https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition\" target=\"_blank\">https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition</a>. The relevant function is provided below:</p>\n<p>def ResidualizeMarket(df, mktColumn, window):<br>\n  if mktColumn not in df.columns:<br>\n    return df</p>\n<p>mkt = df[mktColumn]</p>\n<p>num = df.multiply(mkt.values, axis=0).rolling(window).mean().values  #numerator of linear regression coefficient<br>\n  denom = mkt.multiply(mkt.values, axis=0).rolling(window).mean().values  #denominator of linear regression coefficient<br>\n  beta = np.nan_to_num( num.T / denom, nan=0., posinf=0., neginf=0.)  #if regression fell over, use beta of 0</p>\n<p>resultRet = df - (beta * mkt.values).T  #perform residualization<br>\n  resultBeta = 0.*df + beta.T  #shape beta</p>\n<p>return resultRet.drop(columns=[mktColumn]), resultBeta.drop(columns=[mktColumn])<br>\nWeights<br>\nIn this competition, the weights are determined by the logarithm of each product's market cap (in USD), of the cryptocurrencies at a fixed point in time. Weights were assigned to give more relevance to cryptocurrencies with higher market volumes to ensure smaller cryptocurrencies do not disproportionately impact your models.</p>\n<p>Thanks everyone. Keep on Kaggling!<br>\n<a href=\"https://www.dewoweb.com\" target=\"_blank\">dewoweb</a></p>",
      "rawMarkdown": "Eval Metric, Target, Weights\nPosted in g-research-crypto-forecasting 2 months ago\n80\n\nThanks Kagglers for your questions. We will group the key themes beginning with the eval metric, target formulation and weights:\n\nEvaluation Metric\nWhile mean squared error, R^2, explained variance, and correlation are all very closely related, correlation has the useful property that it tends to normalize leading-order volatility out of the covariance between target and prediction. In financial markets (especially crypto ones!), predicting volatility is a difficult (but interesting!) question in its own right. By using correlation as a metric we hope to remove some noise from the prediction problem and provide a more stable metric to evaluate against.\n\nTarget\nThe target calculation is based on the close price of the asset and can be derived from the provided data using the methodology in https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition. The relevant function is provided below:\n\ndef ResidualizeMarket(df, mktColumn, window):\n  if mktColumn not in df.columns:\n    return df\n\n  mkt = df[mktColumn]\n\n  num = df.multiply(mkt.values, axis=0).rolling(window).mean().values  #numerator of linear regression coefficient\n  denom = mkt.multiply(mkt.values, axis=0).rolling(window).mean().values  #denominator of linear regression coefficient\n  beta = np.nan_to_num( num.T / denom, nan=0., posinf=0., neginf=0.)  #if regression fell over, use beta of 0\n\n  resultRet = df - (beta * mkt.values).T  #perform residualization\n  resultBeta = 0.*df + beta.T  #shape beta\n\n  return resultRet.drop(columns=[mktColumn]), resultBeta.drop(columns=[mktColumn])\nWeights\nIn this competition, the weights are determined by the logarithm of each product's market cap (in USD), of the cryptocurrencies at a fixed point in time. Weights were assigned to give more relevance to cryptocurrencies with higher market volumes to ensure smaller cryptocurrencies do not disproportionately impact your models.\n\nThanks everyone. Keep on Kaggling!\n[dewoweb](https://www.dewoweb.com)\n\n",
      "votes": -3
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "1633747": "Eval Metric, Target, Weights\nPosted in g-research-crypto-forecasting 2 months ago\n80\n\nThanks Kagglers for your questions. We will group the key themes beginning with the eval metric, target formulation and weights:\n\nEvaluation Metric\nWhile mean squared error, R^2, explained variance, and correlation are all very closely related, correlation has the useful property that it tends to normalize leading-order volatility out of the covariance between target and prediction. In financial markets (especially crypto ones!), predicting volatility is a difficult (but interesting!) question in its own right. By using correlation as a metric we hope to remove some noise from the prediction problem and provide a more stable metric to evaluate against.\n\nTarget\nThe target calculation is based on the close price of the asset and can be derived from the provided data using the methodology in https://www.kaggle.com/cstein06/tutorial-to-the-g-research-crypto-competition. The relevant function is provided below:\n\ndef ResidualizeMarket(df, mktColumn, window):\n  if mktColumn not in df.columns:\n    return df\n\n  mkt = df[mktColumn]\n\n  num = df.multiply(mkt.values, axis=0).rolling(window).mean().values  #numerator of linear regression coefficient\n  denom = mkt.multiply(mkt.values, axis=0).rolling(window).mean().values  #denominator of linear regression coefficient\n  beta = np.nan_to_num( num.T / denom, nan=0., posinf=0., neginf=0.)  #if regression fell over, use beta of 0\n\n  resultRet = df - (beta * mkt.values).T  #perform residualization\n  resultBeta = 0.*df + beta.T  #shape beta\n\n  return resultRet.drop(columns=[mktColumn]), resultBeta.drop(columns=[mktColumn])\nWeights\nIn this competition, the weights are determined by the logarithm of each product's market cap (in USD), of the cryptocurrencies at a fixed point in time. Weights were assigned to give more relevance to cryptocurrencies with higher market volumes to ensure smaller cryptocurrencies do not disproportionately impact your models.\n\nThanks everyone. Keep on Kaggling!\n[dewoweb](https://www.dewoweb.com)\n\n"
  }
}