{
  "id": 301333,
  "title": "Some potentially useful features",
  "url": "/competitions/g-research-crypto-forecasting/discussion/301333",
  "author_name": "",
  "post_date": "2022-01-17T06:48:14.507277400Z",
  "votes": 6,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Please see below some potentially useful features discovered through research done via arxiv, Google Scholar, etc…</p>\n<pre><code>from statsmodels.tsa.filters.hp_filter import hpfilter\nfrom statsmodels.tsa.seasonal import seasonal_decompose\n\n# Log Returns\ndef log_return(series, win):\n    return np.log(series).diff(periods=periods)\n\n# Hodrick-Prescott filter of prices/returns\ndef hpfilterfunc(series):\n    # try lamb 1, 10, 100, 1000, 10000, 100000\n    remaining, trend = hpfilter(series, lamb)\n    return remaining, trend\n\n# Seasonal decomposition of prices/returns\ndef seasdecfunc(series):\n    result = seasonal_decompose(series, model='multiplicative')\n    return result['_seasonal'], result['_trend'], result['_resid']\n\n# Simple moving average of returns\ndef sma(series, win):\n    return series.rolling(win).mean()\n\n# Moving average crossover of returns\ndef mac(series, win1, win2):\n    return (sma(series, win1)/sma(series, win2)).replace([np.inf, -np.inf], np.nan)\n\n# Autocorrelation\ndef acorr(series, win):\n    return series.rolling(win).corr(series.shift())\n\n# Exponentially weighted moving average of returns\ndef ewm(series, com):\n    return series.ewm(com).mean()\n\n# Realized volatility of returns\ndef realized_volatility(series):\n    return np.sqrt(np.sum(series**2))\n\n# Realized vol moving window\ndef realized_vol_m(series, win):\n    return series.rolling(win).apply(realized_volatility)\n\n# Realized absolute variation of returns\ndef realized_absvar(series):\n    return np.sqrt(np.pi/(2*series.count()))*np.sum(np.abs(series))\n\n# Realized abs var moving window\ndef realized_absvar_m(series, win):\n    return series.rolling(win).apply(realized_absvar)\n\n# Realized bipower variation of returns \ndef realized_bipowvar(series):\n    cnt = series.count()\n    if cnt&lt;3:\n        return np.nan\n    else:\n        cons = (np.pi/2)*(cnt/(cnt-2))\n        bipow = cons*np.nansum(np.abs(series)*np.abs(series.shift()))\n        vol = np.sqrt(np.sum(series**2))\n        jump = max(0, vol-bipow)\n        return jump\n\n# Realized bipowvar moving window\ndef realized_bipowvar_m(series, win):\n    return series.rolling(win).apply(realized_bipowvar)\n\n# Price momentum (use returns for series)\ndef momentum(series, win):\n    series2 = np.where(series&gt;=0, 1, -1)\n    return series2.rolling(win).mean()\n\n# Relative strength index\ndef rsi(series, win):\n    u = np.where(series&gt;=0, 1, 0).rolling(win).mean()\n    d = np.where(series&lt;0, 1, 0).rolling(win).mean()\n    return 1 - 1/(1+u/d)\n\n# Aroon oscillator of prices\ndef aroon(series, win)\n    maxloc = series.index.get_loc(series[series==series.max()].index[0])\n    u = maxloc / win\n    minloc = series.index.get_loc(series[series==series.min()].index[0])\n    d = minloc / win\n    return u, d, (u-d)\n\n# Commodity channel index of prices\ndef cci(df, win):\n    typical_price = (df.High+df.Low+df.Close)/3\n    ma = typical_price.rolling(win).mean()\n    md = typical_price.rolling(win).apply(lambda x: np.fabs(x - x.mean()).mean())\n    return (typical_price - ma) / md\n</code></pre>",
  "messages": [
    {
      "id": "1652993",
      "postDate": "01/17/2022 06:48:14",
      "content": "<p>Please see below some potentially useful features discovered through research done via arxiv, Google Scholar, etc…</p>\n<pre><code>from statsmodels.tsa.filters.hp_filter import hpfilter\nfrom statsmodels.tsa.seasonal import seasonal_decompose\n\n# Log Returns\ndef log_return(series, win):\n    return np.log(series).diff(periods=periods)\n\n# Hodrick-Prescott filter of prices/returns\ndef hpfilterfunc(series):\n    # try lamb 1, 10, 100, 1000, 10000, 100000\n    remaining, trend = hpfilter(series, lamb)\n    return remaining, trend\n\n# Seasonal decomposition of prices/returns\ndef seasdecfunc(series):\n    result = seasonal_decompose(series, model='multiplicative')\n    return result['_seasonal'], result['_trend'], result['_resid']\n\n# Simple moving average of returns\ndef sma(series, win):\n    return series.rolling(win).mean()\n\n# Moving average crossover of returns\ndef mac(series, win1, win2):\n    return (sma(series, win1)/sma(series, win2)).replace([np.inf, -np.inf], np.nan)\n\n# Autocorrelation\ndef acorr(series, win):\n    return series.rolling(win).corr(series.shift())\n\n# Exponentially weighted moving average of returns\ndef ewm(series, com):\n    return series.ewm(com).mean()\n\n# Realized volatility of returns\ndef realized_volatility(series):\n    return np.sqrt(np.sum(series**2))\n\n# Realized vol moving window\ndef realized_vol_m(series, win):\n    return series.rolling(win).apply(realized_volatility)\n\n# Realized absolute variation of returns\ndef realized_absvar(series):\n    return np.sqrt(np.pi/(2*series.count()))*np.sum(np.abs(series))\n\n# Realized abs var moving window\ndef realized_absvar_m(series, win):\n    return series.rolling(win).apply(realized_absvar)\n\n# Realized bipower variation of returns \ndef realized_bipowvar(series):\n    cnt = series.count()\n    if cnt&lt;3:\n        return np.nan\n    else:\n        cons = (np.pi/2)*(cnt/(cnt-2))\n        bipow = cons*np.nansum(np.abs(series)*np.abs(series.shift()))\n        vol = np.sqrt(np.sum(series**2))\n        jump = max(0, vol-bipow)\n        return jump\n\n# Realized bipowvar moving window\ndef realized_bipowvar_m(series, win):\n    return series.rolling(win).apply(realized_bipowvar)\n\n# Price momentum (use returns for series)\ndef momentum(series, win):\n    series2 = np.where(series&gt;=0, 1, -1)\n    return series2.rolling(win).mean()\n\n# Relative strength index\ndef rsi(series, win):\n    u = np.where(series&gt;=0, 1, 0).rolling(win).mean()\n    d = np.where(series&lt;0, 1, 0).rolling(win).mean()\n    return 1 - 1/(1+u/d)\n\n# Aroon oscillator of prices\ndef aroon(series, win)\n    maxloc = series.index.get_loc(series[series==series.max()].index[0])\n    u = maxloc / win\n    minloc = series.index.get_loc(series[series==series.min()].index[0])\n    d = minloc / win\n    return u, d, (u-d)\n\n# Commodity channel index of prices\ndef cci(df, win):\n    typical_price = (df.High+df.Low+df.Close)/3\n    ma = typical_price.rolling(win).mean()\n    md = typical_price.rolling(win).apply(lambda x: np.fabs(x - x.mean()).mean())\n    return (typical_price - ma) / md\n</code></pre>",
      "rawMarkdown": "Please see below some potentially useful features discovered through research done via arxiv, Google Scholar, etc...\n\n````\nfrom statsmodels.tsa.filters.hp_filter import hpfilter\nfrom statsmodels.tsa.seasonal import seasonal_decompose\n\n# Log Returns\ndef log_return(series, win):\n    return np.log(series).diff(periods=periods)\n\n# Hodrick-Prescott filter of prices/returns\ndef hpfilterfunc(series):\n    # try lamb 1, 10, 100, 1000, 10000, 100000\n    remaining, trend = hpfilter(series, lamb)\n    return remaining, trend\n\n# Seasonal decomposition of prices/returns\ndef seasdecfunc(series):\n    result = seasonal_decompose(series, model='multiplicative')\n    return result['_seasonal'], result['_trend'], result['_resid']\n\n# Simple moving average of returns\ndef sma(series, win):\n    return series.rolling(win).mean()\n\n# Moving average crossover of returns\ndef mac(series, win1, win2):\n    return (sma(series, win1)/sma(series, win2)).replace([np.inf, -np.inf], np.nan)\n\n# Autocorrelation\ndef acorr(series, win):\n    return series.rolling(win).corr(series.shift())\n\n# Exponentially weighted moving average of returns\ndef ewm(series, com):\n    return series.ewm(com).mean()\n\n# Realized volatility of returns\ndef realized_volatility(series):\n    return np.sqrt(np.sum(series**2))\n\n# Realized vol moving window\ndef realized_vol_m(series, win):\n    return series.rolling(win).apply(realized_volatility)\n\n# Realized absolute variation of returns\ndef realized_absvar(series):\n    return np.sqrt(np.pi/(2*series.count()))*np.sum(np.abs(series))\n\n# Realized abs var moving window\ndef realized_absvar_m(series, win):\n    return series.rolling(win).apply(realized_absvar)\n\n# Realized bipower variation of returns \ndef realized_bipowvar(series):\n    cnt = series.count()\n    if cnt<3:\n        return np.nan\n    else:\n        cons = (np.pi/2)*(cnt/(cnt-2))\n        bipow = cons*np.nansum(np.abs(series)*np.abs(series.shift()))\n        vol = np.sqrt(np.sum(series**2))\n        jump = max(0, vol-bipow)\n        return jump\n\n# Realized bipowvar moving window\ndef realized_bipowvar_m(series, win):\n    return series.rolling(win).apply(realized_bipowvar)\n\n# Price momentum (use returns for series)\ndef momentum(series, win):\n    series2 = np.where(series>=0, 1, -1)\n    return series2.rolling(win).mean()\n\n# Relative strength index\ndef rsi(series, win):\n    u = np.where(series>=0, 1, 0).rolling(win).mean()\n    d = np.where(series<0, 1, 0).rolling(win).mean()\n    return 1 - 1/(1+u/d)\n\n# Aroon oscillator of prices\ndef aroon(series, win)\n    maxloc = series.index.get_loc(series[series==series.max()].index[0])\n    u = maxloc / win\n    minloc = series.index.get_loc(series[series==series.min()].index[0])\n    d = minloc / win\n    return u, d, (u-d)\n\n# Commodity channel index of prices\ndef cci(df, win):\n    typical_price = (df.High+df.Low+df.Close)/3\n    ma = typical_price.rolling(win).mean()\n    md = typical_price.rolling(win).apply(lambda x: np.fabs(x - x.mean()).mean())\n    return (typical_price - ma) / md\n````",
      "votes": null
    },
    {
      "id": "1652994",
      "postDate": "01/17/2022 06:50:02",
      "content": "<p>It seems that most features are useless for the target of this competition.</p>\n<p>ps: I meant the features I made, not feature from this post.</p>",
      "rawMarkdown": "It seems that most features are useless for the target of this competition.\n\nps: I meant the features I made, not feature from this post.",
      "votes": null
    },
    {
      "id": "1653003",
      "postDate": "01/17/2022 06:56:19",
      "content": "<p>Yes you are probably right. Although the bipower variation might be. This particular function measures the magnitude of spikes in returns via volatility calculations. More generally, if one could effectively capture the large spikes in returns in (as measured by a jump process), it could be useful. Haven't tested it though</p>",
      "rawMarkdown": "Yes you are probably right. Although the bipower variation might be. This particular function measures the magnitude of spikes in returns via volatility calculations. More generally, if one could effectively capture the large spikes in returns in (as measured by a jump process), it could be useful. Haven't tested it though",
      "votes": null
    },
    {
      "id": "1653004",
      "postDate": "01/17/2022 06:58:41",
      "content": "<p>Search for \"Jump Process\" in Google and you will know what I mean</p>",
      "rawMarkdown": "Search for \"Jump Process\" in Google and you will know what I mean",
      "votes": null
    },
    {
      "id": "1653025",
      "postDate": "01/17/2022 07:13:13",
      "content": "<p>I meant the basic features I made were almost useless (not the features in your post). </p>",
      "rawMarkdown": "I meant the basic features I made were almost useless (not the features in your post).",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1652994,
      "author_name": "yuanzhezhou",
      "author_url": "",
      "post_date": "01/17/2022 06:50:02",
      "content": "<p>It seems that most features are useless for the target of this competition.</p>\n<p>ps: I meant the features I made, not feature from this post.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1653003,
          "author_name": "nrcjea001",
          "author_url": "",
          "post_date": "01/17/2022 06:56:19",
          "content": "<p>Yes you are probably right. Although the bipower variation might be. This particular function measures the magnitude of spikes in returns via volatility calculations. More generally, if one could effectively capture the large spikes in returns in (as measured by a jump process), it could be useful. Haven't tested it though</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1653004,
          "author_name": "nrcjea001",
          "author_url": "",
          "post_date": "01/17/2022 06:58:41",
          "content": "<p>Search for \"Jump Process\" in Google and you will know what I mean</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1653025,
          "author_name": "yuanzhezhou",
          "author_url": "",
          "post_date": "01/17/2022 07:13:13",
          "content": "<p>I meant the basic features I made were almost useless (not the features in your post). </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1652993": "Please see below some potentially useful features discovered through research done via arxiv, Google Scholar, etc...\n\n````\nfrom statsmodels.tsa.filters.hp_filter import hpfilter\nfrom statsmodels.tsa.seasonal import seasonal_decompose\n\n# Log Returns\ndef log_return(series, win):\n    return np.log(series).diff(periods=periods)\n\n# Hodrick-Prescott filter of prices/returns\ndef hpfilterfunc(series):\n    # try lamb 1, 10, 100, 1000, 10000, 100000\n    remaining, trend = hpfilter(series, lamb)\n    return remaining, trend\n\n# Seasonal decomposition of prices/returns\ndef seasdecfunc(series):\n    result = seasonal_decompose(series, model='multiplicative')\n    return result['_seasonal'], result['_trend'], result['_resid']\n\n# Simple moving average of returns\ndef sma(series, win):\n    return series.rolling(win).mean()\n\n# Moving average crossover of returns\ndef mac(series, win1, win2):\n    return (sma(series, win1)/sma(series, win2)).replace([np.inf, -np.inf], np.nan)\n\n# Autocorrelation\ndef acorr(series, win):\n    return series.rolling(win).corr(series.shift())\n\n# Exponentially weighted moving average of returns\ndef ewm(series, com):\n    return series.ewm(com).mean()\n\n# Realized volatility of returns\ndef realized_volatility(series):\n    return np.sqrt(np.sum(series**2))\n\n# Realized vol moving window\ndef realized_vol_m(series, win):\n    return series.rolling(win).apply(realized_volatility)\n\n# Realized absolute variation of returns\ndef realized_absvar(series):\n    return np.sqrt(np.pi/(2*series.count()))*np.sum(np.abs(series))\n\n# Realized abs var moving window\ndef realized_absvar_m(series, win):\n    return series.rolling(win).apply(realized_absvar)\n\n# Realized bipower variation of returns \ndef realized_bipowvar(series):\n    cnt = series.count()\n    if cnt<3:\n        return np.nan\n    else:\n        cons = (np.pi/2)*(cnt/(cnt-2))\n        bipow = cons*np.nansum(np.abs(series)*np.abs(series.shift()))\n        vol = np.sqrt(np.sum(series**2))\n        jump = max(0, vol-bipow)\n        return jump\n\n# Realized bipowvar moving window\ndef realized_bipowvar_m(series, win):\n    return series.rolling(win).apply(realized_bipowvar)\n\n# Price momentum (use returns for series)\ndef momentum(series, win):\n    series2 = np.where(series>=0, 1, -1)\n    return series2.rolling(win).mean()\n\n# Relative strength index\ndef rsi(series, win):\n    u = np.where(series>=0, 1, 0).rolling(win).mean()\n    d = np.where(series<0, 1, 0).rolling(win).mean()\n    return 1 - 1/(1+u/d)\n\n# Aroon oscillator of prices\ndef aroon(series, win)\n    maxloc = series.index.get_loc(series[series==series.max()].index[0])\n    u = maxloc / win\n    minloc = series.index.get_loc(series[series==series.min()].index[0])\n    d = minloc / win\n    return u, d, (u-d)\n\n# Commodity channel index of prices\ndef cci(df, win):\n    typical_price = (df.High+df.Low+df.Close)/3\n    ma = typical_price.rolling(win).mean()\n    md = typical_price.rolling(win).apply(lambda x: np.fabs(x - x.mean()).mean())\n    return (typical_price - ma) / md\n````",
    "1652994": "It seems that most features are useless for the target of this competition.\n\nps: I meant the features I made, not feature from this post.",
    "1653003": "Yes you are probably right. Although the bipower variation might be. This particular function measures the magnitude of spikes in returns via volatility calculations. More generally, if one could effectively capture the large spikes in returns in (as measured by a jump process), it could be useful. Haven't tested it though",
    "1653004": "Search for \"Jump Process\" in Google and you will know what I mean",
    "1653025": "I meant the basic features I made were almost useless (not the features in your post)."
  },
  "source": "meta"
}