{
  "id": 308100,
  "title": "List of features to increase your score",
  "url": "/competitions/g-research-crypto-forecasting/discussion/308100",
  "author_name": "Ugibugibu",
  "post_date": "2022-02-17T04:36:49.424000",
  "votes": 10,
  "comment_count": 9,
  "views": 0,
  "content": "<p>Hi,</p>\n<p>I created a notebook with a big list of features I was using during this competition.<br>\nThey are quite an improvement from the basic ones. I chose 8 features for the model and it is scoring 0.0356 during training on a basic LGBM model.<br>\nHave a look and give feedback</p>\n<p>I am a professional trader, but new to Kaggle. So coming up with features was something more familiar to me, but the coding part was the hard part. Unfortunately I did not manage to submit my notebook properly, but you might still find the input useful.</p>\n<p><a href=\"https://www.kaggle.com/rafalradwanski/g-research-features-to-increase-your-score\" target=\"_blank\">https://www.kaggle.com/rafalradwanski/g-research-features-to-increase-your-score</a></p>\n<p>8 features selected by me:</p>\n<pre><code>df_feat['Z-score_15_Close'] = zscore(df_feat['Close'], 15)\ndf_feat['lower_shadow_15'] = zscore(lower_shadow_15(df_feat), 1440*30*12)\ndf_feat['upper_shadow_15'] = zscore(upper_shadow_15(df_feat), 1440*30*12)\ndf_feat['Return15_%'] = (df_feat['Close'] / df_feat['Close'].shift(15)) - 1\ndf_feat['Return60_%'] = (df_feat['Close'] / df_feat['Close'].shift(60)) - 1\ndf_feat['Candle_body_%'] = (df_feat['Close'] / df_feat['Open']) - 1\ndf_feat['ATR_15_%'] = ((df_feat['High'].rolling(window=15, min_periods=1).max())/ (df_feat['Low'].rolling(window=15, min_periods=1).min()) - 1)\ndf_feat['Z-score_return15_60_%'] = zscore(df_feat['Return15_%'], 60)\n</code></pre>\n<p>All the features created by me or found in other people's notebooks    </p>\n<pre><code>df_feat['Volume / Count'] = df_feat['Volume'] / df_feat['Count'] \ndf_feat['Close - VWAP'] = df_feat['Close'] - df_feat['VWAP'] \ndf_feat['(Close / VWAP_15) - 1'] = (df_feat['Close'] / df_feat['VWAP'].rolling(window=15, min_periods=1).mean()) - 1\ndf_feat['Close - VWAP_15'] = (df_feat['Close'] - df_feat['VWAP'].rolling(window=15, min_periods=1).mean())\ndf_feat['sma15'] = df_feat['Close'] / df_feat['Close'].rolling(15).mean() -1\ndf_feat['sma60'] = df_feat['Close'] / df_feat['Close'].rolling(60).mean() -1\ndf_feat['sma240'] = df_feat['Close'] / df_feat['Close'].rolling(240).mean() -1\n\ndf_feat['Bar_Range'] = df_feat['High'] - df_feat['Low']  \ndf_feat['Bar_Range_%'] = (df_feat['High'] / df_feat['Low']) -1\n\ndf_feat['lower_Shadow'] = lower_shadow(df_feat)\ndf_feat['upper_Shadow'] = upper_shadow(df_feat)\ndf_feat['lower_shadow_percent'] = lower_shadow_percent(df_feat)\ndf_feat['upper_Shadow_percent'] = upper_shadow_percent(df_feat)\ndf_feat['lower_shadow_15'] = lower_shadow_15(df_feat)\ndf_feat['upper_shadow_15'] = upper_shadow_15(df_feat)\ndf_feat['lower_shadow_15_perc'] = lower_shadow_15_perc(df_feat)\ndf_feat['upper_shadow_15_perc'] = upper_shadow_15_perc(df_feat)\n\ndf_feat['Return1'] = df_feat['Close'] - df_feat['Close'].shift(1) \ndf_feat['Return1_%'] = (df_feat['Close'] / df_feat['Close'].shift(1)) - 1\ndf_feat['Return15'] = df_feat['Close'] - df_feat['Close'].shift(15)\ndf_feat['Return15_%'] = (df_feat['Close'] / df_feat['Close'].shift(15)) - 1\ndf_feat['Return60_%'] = (df_feat['Close'] / df_feat['Close'].shift(60)) - 1\ndf_feat['Return240_%'] = (df_feat['Close'] / df_feat['Close'].shift(240)) - 1\ndf_feat['Candle_body'] = df_feat['Close'] - df_feat['Open'] \ndf_feat['Candle_body_%'] = (df_feat['Close'] / df_feat['Open']) - 1\ndf_feat['Candle_body15'] = df_feat['Close'] - df_feat['Open'].shift(14)\ndf_feat['Candle_body15_%'] = (df_feat['Close'] / df_feat['Open'].shift(14)) -1\n\ndf_feat['Z-score_15_Close'] = zscore(df_feat['Close'], 15)\ndf_feat['Z-score_60_Volume'] = zscore(df_feat['Volume'], 60)\n\ndf_feat['Mean_15_Close'] = df_feat['Close'].rolling(window=15, min_periods=1).mean()\ndf_feat['Mean_60_Volume'] = df_feat['Volume'].rolling(window=60, min_periods=1).mean()\ndf_feat['Mean_60_Count'] = df_feat['Count'].rolling(window=60, min_periods=1).mean()\ndf_feat['Z-score_60_Count'] = zscore(df_feat['Count'], 60)\ndf_feat['ATR_15'] = (df_feat['High'].rolling(window=15, min_periods=1).max() - df_feat['Low'].rolling(window=15, min_periods=1).min())\ndf_feat['ATR_15_%'] = ((df_feat['High'].rolling(window=15, min_periods=1).max())/ (df_feat['Low'].rolling(window=15, min_periods=1).min()) - 1)\n\ndf_feat['Z-score_return15_15'] = zscore(df_feat['Return15'], 15)\ndf_feat['Z-score_return15_15_%'] = zscore(df_feat['Return15_%'], 15)\ndf_feat['Z-score_return15_60'] = zscore(df_feat['Return15'], 60)\ndf_feat['Z-score_return15_60_%'] = zscore(df_feat['Return15_%'], 60)\ndf_feat['Z-score_return15_1440'] = zscore(df_feat['Return15'], 1440)\ndf_feat['Z-score_return15_1440_%'] = zscore(df_feat['Return15_%'], 1440)\ndf_feat['Z-score_return15_month'] = zscore(df_feat['Return15'], 1440*30)\ndf_feat['Z-score_return15_month_%'] = zscore(df_feat['Return15_%'], 1440*30)\n\ndf_feat['VWAP15_median'] = df_feat['VWAP'].rolling(window=15, min_periods=1).median()\ndf_feat['VWAP60_median'] = df_feat['VWAP'].rolling(window=60, min_periods=1).median()\ndf_feat['VWAP240_median'] = df_feat['VWAP'].rolling(window=240, min_periods=1).median()\ndf_feat['VWAPday_median'] = df_feat['VWAP'].rolling(window=1440, min_periods=1).median()\n\ndf_feat['Median_return15_15'] = df_feat['Return15'].rolling(window=15, min_periods=1).median()\ndf_feat['Median_return15_15_%'] = df_feat['Return15_%'].rolling(window=15, min_periods=1).median()\ndf_feat['Median_return15_60'] = df_feat['Return15'].rolling(window=60, min_periods=1).median()\ndf_feat['Median_return15_60_%'] = df_feat['Return15_%'].rolling(window=60, min_periods=1).median()\ndf_feat['Median_return15_1440'] = df_feat['Return15'].rolling(window=1440, min_periods=1).median()\ndf_feat['Median_return15_1440_%'] = df_feat['Return15_%'].rolling(window=1440, min_periods=1).median()\ndf_feat['Median_return15_month'] = df_feat['Return15'].rolling(window=1440*30, min_periods=1).median()\ndf_feat['Median_return15_month_%'] = df_feat['Return15_%'].rolling(window=1440*30, min_periods=1).median()\ndf_feat['Median_return15_year'] = df_feat['Return15'].rolling(window=1440*30*12, min_periods=1).median()\ndf_feat['Median_return15_year_%'] = df_feat['Return15_%'].rolling(window=1440*30*12, min_periods=1).median()\ndf_feat['Median_return15_3years_%'] = df_feat['Return15_%'].rolling(window=1440*30*12*3, min_periods=1).median()\ndf_feat['ROC'] =  ((df_feat['Close'] - df_feat['Close'].shift(15)) / df_feat['Close'].shift(15))*100\n</code></pre>\n<h1>Seasonality features</h1>\n<pre><code>df_feat[\"hour\"] = df_feat[\"datetime\"].dt.hour\ndf_feat[\"day of week\"] = df_feat[\"datetime\"].dt.dayofweek \ndf_feat[\"day\"] = df_feat[\"datetime\"].dt.day\n</code></pre>",
  "messages": [
    {
      "id": 1693949,
      "postDate": "2022-02-17T04:36:49.423Z",
      "content": "<p>Hi,</p>\n<p>I created a notebook with a big list of features I was using during this competition.<br>\nThey are quite an improvement from the basic ones. I chose 8 features for the model and it is scoring 0.0356 during training on a basic LGBM model.<br>\nHave a look and give feedback</p>\n<p>I am a professional trader, but new to Kaggle. So coming up with features was something more familiar to me, but the coding part was the hard part. Unfortunately I did not manage to submit my notebook properly, but you might still find the input useful.</p>\n<p><a href=\"https://www.kaggle.com/rafalradwanski/g-research-features-to-increase-your-score\" target=\"_blank\">https://www.kaggle.com/rafalradwanski/g-research-features-to-increase-your-score</a></p>\n<p>8 features selected by me:</p>\n<pre><code>df_feat['Z-score_15_Close'] = zscore(df_feat['Close'], 15)\ndf_feat['lower_shadow_15'] = zscore(lower_shadow_15(df_feat), 1440*30*12)\ndf_feat['upper_shadow_15'] = zscore(upper_shadow_15(df_feat), 1440*30*12)\ndf_feat['Return15_%'] = (df_feat['Close'] / df_feat['Close'].shift(15)) - 1\ndf_feat['Return60_%'] = (df_feat['Close'] / df_feat['Close'].shift(60)) - 1\ndf_feat['Candle_body_%'] = (df_feat['Close'] / df_feat['Open']) - 1\ndf_feat['ATR_15_%'] = ((df_feat['High'].rolling(window=15, min_periods=1).max())/ (df_feat['Low'].rolling(window=15, min_periods=1).min()) - 1)\ndf_feat['Z-score_return15_60_%'] = zscore(df_feat['Return15_%'], 60)\n</code></pre>\n<p>All the features created by me or found in other people's notebooks    </p>\n<pre><code>df_feat['Volume / Count'] = df_feat['Volume'] / df_feat['Count'] \ndf_feat['Close - VWAP'] = df_feat['Close'] - df_feat['VWAP'] \ndf_feat['(Close / VWAP_15) - 1'] = (df_feat['Close'] / df_feat['VWAP'].rolling(window=15, min_periods=1).mean()) - 1\ndf_feat['Close - VWAP_15'] = (df_feat['Close'] - df_feat['VWAP'].rolling(window=15, min_periods=1).mean())\ndf_feat['sma15'] = df_feat['Close'] / df_feat['Close'].rolling(15).mean() -1\ndf_feat['sma60'] = df_feat['Close'] / df_feat['Close'].rolling(60).mean() -1\ndf_feat['sma240'] = df_feat['Close'] / df_feat['Close'].rolling(240).mean() -1\n\ndf_feat['Bar_Range'] = df_feat['High'] - df_feat['Low']  \ndf_feat['Bar_Range_%'] = (df_feat['High'] / df_feat['Low']) -1\n\ndf_feat['lower_Shadow'] = lower_shadow(df_feat)\ndf_feat['upper_Shadow'] = upper_shadow(df_feat)\ndf_feat['lower_shadow_percent'] = lower_shadow_percent(df_feat)\ndf_feat['upper_Shadow_percent'] = upper_shadow_percent(df_feat)\ndf_feat['lower_shadow_15'] = lower_shadow_15(df_feat)\ndf_feat['upper_shadow_15'] = upper_shadow_15(df_feat)\ndf_feat['lower_shadow_15_perc'] = lower_shadow_15_perc(df_feat)\ndf_feat['upper_shadow_15_perc'] = upper_shadow_15_perc(df_feat)\n\ndf_feat['Return1'] = df_feat['Close'] - df_feat['Close'].shift(1) \ndf_feat['Return1_%'] = (df_feat['Close'] / df_feat['Close'].shift(1)) - 1\ndf_feat['Return15'] = df_feat['Close'] - df_feat['Close'].shift(15)\ndf_feat['Return15_%'] = (df_feat['Close'] / df_feat['Close'].shift(15)) - 1\ndf_feat['Return60_%'] = (df_feat['Close'] / df_feat['Close'].shift(60)) - 1\ndf_feat['Return240_%'] = (df_feat['Close'] / df_feat['Close'].shift(240)) - 1\ndf_feat['Candle_body'] = df_feat['Close'] - df_feat['Open'] \ndf_feat['Candle_body_%'] = (df_feat['Close'] / df_feat['Open']) - 1\ndf_feat['Candle_body15'] = df_feat['Close'] - df_feat['Open'].shift(14)\ndf_feat['Candle_body15_%'] = (df_feat['Close'] / df_feat['Open'].shift(14)) -1\n\ndf_feat['Z-score_15_Close'] = zscore(df_feat['Close'], 15)\ndf_feat['Z-score_60_Volume'] = zscore(df_feat['Volume'], 60)\n\ndf_feat['Mean_15_Close'] = df_feat['Close'].rolling(window=15, min_periods=1).mean()\ndf_feat['Mean_60_Volume'] = df_feat['Volume'].rolling(window=60, min_periods=1).mean()\ndf_feat['Mean_60_Count'] = df_feat['Count'].rolling(window=60, min_periods=1).mean()\ndf_feat['Z-score_60_Count'] = zscore(df_feat['Count'], 60)\ndf_feat['ATR_15'] = (df_feat['High'].rolling(window=15, min_periods=1).max() - df_feat['Low'].rolling(window=15, min_periods=1).min())\ndf_feat['ATR_15_%'] = ((df_feat['High'].rolling(window=15, min_periods=1).max())/ (df_feat['Low'].rolling(window=15, min_periods=1).min()) - 1)\n\ndf_feat['Z-score_return15_15'] = zscore(df_feat['Return15'], 15)\ndf_feat['Z-score_return15_15_%'] = zscore(df_feat['Return15_%'], 15)\ndf_feat['Z-score_return15_60'] = zscore(df_feat['Return15'], 60)\ndf_feat['Z-score_return15_60_%'] = zscore(df_feat['Return15_%'], 60)\ndf_feat['Z-score_return15_1440'] = zscore(df_feat['Return15'], 1440)\ndf_feat['Z-score_return15_1440_%'] = zscore(df_feat['Return15_%'], 1440)\ndf_feat['Z-score_return15_month'] = zscore(df_feat['Return15'], 1440*30)\ndf_feat['Z-score_return15_month_%'] = zscore(df_feat['Return15_%'], 1440*30)\n\ndf_feat['VWAP15_median'] = df_feat['VWAP'].rolling(window=15, min_periods=1).median()\ndf_feat['VWAP60_median'] = df_feat['VWAP'].rolling(window=60, min_periods=1).median()\ndf_feat['VWAP240_median'] = df_feat['VWAP'].rolling(window=240, min_periods=1).median()\ndf_feat['VWAPday_median'] = df_feat['VWAP'].rolling(window=1440, min_periods=1).median()\n\ndf_feat['Median_return15_15'] = df_feat['Return15'].rolling(window=15, min_periods=1).median()\ndf_feat['Median_return15_15_%'] = df_feat['Return15_%'].rolling(window=15, min_periods=1).median()\ndf_feat['Median_return15_60'] = df_feat['Return15'].rolling(window=60, min_periods=1).median()\ndf_feat['Median_return15_60_%'] = df_feat['Return15_%'].rolling(window=60, min_periods=1).median()\ndf_feat['Median_return15_1440'] = df_feat['Return15'].rolling(window=1440, min_periods=1).median()\ndf_feat['Median_return15_1440_%'] = df_feat['Return15_%'].rolling(window=1440, min_periods=1).median()\ndf_feat['Median_return15_month'] = df_feat['Return15'].rolling(window=1440*30, min_periods=1).median()\ndf_feat['Median_return15_month_%'] = df_feat['Return15_%'].rolling(window=1440*30, min_periods=1).median()\ndf_feat['Median_return15_year'] = df_feat['Return15'].rolling(window=1440*30*12, min_periods=1).median()\ndf_feat['Median_return15_year_%'] = df_feat['Return15_%'].rolling(window=1440*30*12, min_periods=1).median()\ndf_feat['Median_return15_3years_%'] = df_feat['Return15_%'].rolling(window=1440*30*12*3, min_periods=1).median()\ndf_feat['ROC'] =  ((df_feat['Close'] - df_feat['Close'].shift(15)) / df_feat['Close'].shift(15))*100\n</code></pre>\n<h1>Seasonality features</h1>\n<pre><code>df_feat[\"hour\"] = df_feat[\"datetime\"].dt.hour\ndf_feat[\"day of week\"] = df_feat[\"datetime\"].dt.dayofweek \ndf_feat[\"day\"] = df_feat[\"datetime\"].dt.day\n</code></pre>",
      "rawMarkdown": "Hi,\n\nI created a notebook with a big list of features I was using during this competition.\nThey are quite an improvement from the basic ones. I chose 8 features for the model and it is scoring 0.0356 during training on a basic LGBM model.\nHave a look and give feedback\n\nI am a professional trader, but new to Kaggle. So coming up with features was something more familiar to me, but the coding part was the hard part. Unfortunately I did not manage to submit my notebook properly, but you might still find the input useful.\n\nhttps://www.kaggle.com/rafalradwanski/g-research-features-to-increase-your-score\n\n8 features selected by me:\n\n    df_feat['Z-score_15_Close'] = zscore(df_feat['Close'], 15)\n    df_feat['lower_shadow_15'] = zscore(lower_shadow_15(df_feat), 1440*30*12)\n    df_feat['upper_shadow_15'] = zscore(upper_shadow_15(df_feat), 1440*30*12)\n    df_feat['Return15_%'] = (df_feat['Close'] / df_feat['Close'].shift(15)) - 1\n    df_feat['Return60_%'] = (df_feat['Close'] / df_feat['Close'].shift(60)) - 1\n    df_feat['Candle_body_%'] = (df_feat['Close'] / df_feat['Open']) - 1\n    df_feat['ATR_15_%'] = ((df_feat['High'].rolling(window=15, min_periods=1).max())/ (df_feat['Low'].rolling(window=15, min_periods=1).min()) - 1)\n    df_feat['Z-score_return15_60_%'] = zscore(df_feat['Return15_%'], 60)\n\n\nAll the features created by me or found in other people's notebooks    \n    \n    df_feat['Volume / Count'] = df_feat['Volume'] / df_feat['Count'] \n    df_feat['Close - VWAP'] = df_feat['Close'] - df_feat['VWAP'] \n    df_feat['(Close / VWAP_15) - 1'] = (df_feat['Close'] / df_feat['VWAP'].rolling(window=15, min_periods=1).mean()) - 1\n    df_feat['Close - VWAP_15'] = (df_feat['Close'] - df_feat['VWAP'].rolling(window=15, min_periods=1).mean())\n    df_feat['sma15'] = df_feat['Close'] / df_feat['Close'].rolling(15).mean() -1\n    df_feat['sma60'] = df_feat['Close'] / df_feat['Close'].rolling(60).mean() -1\n    df_feat['sma240'] = df_feat['Close'] / df_feat['Close'].rolling(240).mean() -1\n    \n    df_feat['Bar_Range'] = df_feat['High'] - df_feat['Low']  \n    df_feat['Bar_Range_%'] = (df_feat['High'] / df_feat['Low']) -1\n    \n    df_feat['lower_Shadow'] = lower_shadow(df_feat)\n    df_feat['upper_Shadow'] = upper_shadow(df_feat)\n    df_feat['lower_shadow_percent'] = lower_shadow_percent(df_feat)\n    df_feat['upper_Shadow_percent'] = upper_shadow_percent(df_feat)\n    df_feat['lower_shadow_15'] = lower_shadow_15(df_feat)\n    df_feat['upper_shadow_15'] = upper_shadow_15(df_feat)\n    df_feat['lower_shadow_15_perc'] = lower_shadow_15_perc(df_feat)\n    df_feat['upper_shadow_15_perc'] = upper_shadow_15_perc(df_feat)\n    \n    df_feat['Return1'] = df_feat['Close'] - df_feat['Close'].shift(1) \n    df_feat['Return1_%'] = (df_feat['Close'] / df_feat['Close'].shift(1)) - 1\n    df_feat['Return15'] = df_feat['Close'] - df_feat['Close'].shift(15)\n    df_feat['Return15_%'] = (df_feat['Close'] / df_feat['Close'].shift(15)) - 1\n    df_feat['Return60_%'] = (df_feat['Close'] / df_feat['Close'].shift(60)) - 1\n    df_feat['Return240_%'] = (df_feat['Close'] / df_feat['Close'].shift(240)) - 1\n    df_feat['Candle_body'] = df_feat['Close'] - df_feat['Open'] \n    df_feat['Candle_body_%'] = (df_feat['Close'] / df_feat['Open']) - 1\n    df_feat['Candle_body15'] = df_feat['Close'] - df_feat['Open'].shift(14)\n    df_feat['Candle_body15_%'] = (df_feat['Close'] / df_feat['Open'].shift(14)) -1\n    \n    df_feat['Z-score_15_Close'] = zscore(df_feat['Close'], 15)\n    df_feat['Z-score_60_Volume'] = zscore(df_feat['Volume'], 60)\n    \n    df_feat['Mean_15_Close'] = df_feat['Close'].rolling(window=15, min_periods=1).mean()\n    df_feat['Mean_60_Volume'] = df_feat['Volume'].rolling(window=60, min_periods=1).mean()\n    df_feat['Mean_60_Count'] = df_feat['Count'].rolling(window=60, min_periods=1).mean()\n    df_feat['Z-score_60_Count'] = zscore(df_feat['Count'], 60)\n    df_feat['ATR_15'] = (df_feat['High'].rolling(window=15, min_periods=1).max() - df_feat['Low'].rolling(window=15, min_periods=1).min())\n    df_feat['ATR_15_%'] = ((df_feat['High'].rolling(window=15, min_periods=1).max())/ (df_feat['Low'].rolling(window=15, min_periods=1).min()) - 1)\n    \n    df_feat['Z-score_return15_15'] = zscore(df_feat['Return15'], 15)\n    df_feat['Z-score_return15_15_%'] = zscore(df_feat['Return15_%'], 15)\n    df_feat['Z-score_return15_60'] = zscore(df_feat['Return15'], 60)\n    df_feat['Z-score_return15_60_%'] = zscore(df_feat['Return15_%'], 60)\n    df_feat['Z-score_return15_1440'] = zscore(df_feat['Return15'], 1440)\n    df_feat['Z-score_return15_1440_%'] = zscore(df_feat['Return15_%'], 1440)\n    df_feat['Z-score_return15_month'] = zscore(df_feat['Return15'], 1440*30)\n    df_feat['Z-score_return15_month_%'] = zscore(df_feat['Return15_%'], 1440*30)\n    \n    df_feat['VWAP15_median'] = df_feat['VWAP'].rolling(window=15, min_periods=1).median()\n    df_feat['VWAP60_median'] = df_feat['VWAP'].rolling(window=60, min_periods=1).median()\n    df_feat['VWAP240_median'] = df_feat['VWAP'].rolling(window=240, min_periods=1).median()\n    df_feat['VWAPday_median'] = df_feat['VWAP'].rolling(window=1440, min_periods=1).median()\n    \n    df_feat['Median_return15_15'] = df_feat['Return15'].rolling(window=15, min_periods=1).median()\n    df_feat['Median_return15_15_%'] = df_feat['Return15_%'].rolling(window=15, min_periods=1).median()\n    df_feat['Median_return15_60'] = df_feat['Return15'].rolling(window=60, min_periods=1).median()\n    df_feat['Median_return15_60_%'] = df_feat['Return15_%'].rolling(window=60, min_periods=1).median()\n    df_feat['Median_return15_1440'] = df_feat['Return15'].rolling(window=1440, min_periods=1).median()\n    df_feat['Median_return15_1440_%'] = df_feat['Return15_%'].rolling(window=1440, min_periods=1).median()\n    df_feat['Median_return15_month'] = df_feat['Return15'].rolling(window=1440*30, min_periods=1).median()\n    df_feat['Median_return15_month_%'] = df_feat['Return15_%'].rolling(window=1440*30, min_periods=1).median()\n    df_feat['Median_return15_year'] = df_feat['Return15'].rolling(window=1440*30*12, min_periods=1).median()\n    df_feat['Median_return15_year_%'] = df_feat['Return15_%'].rolling(window=1440*30*12, min_periods=1).median()\n    df_feat['Median_return15_3years_%'] = df_feat['Return15_%'].rolling(window=1440*30*12*3, min_periods=1).median()\n    df_feat['ROC'] =  ((df_feat['Close'] - df_feat['Close'].shift(15)) / df_feat['Close'].shift(15))*100\n\n#Seasonality features\n    df_feat[\"hour\"] = df_feat[\"datetime\"].dt.hour\n    df_feat[\"day of week\"] = df_feat[\"datetime\"].dt.dayofweek \n    df_feat[\"day\"] = df_feat[\"datetime\"].dt.day",
      "votes": 10
    },
    {
      "id": 1694293,
      "postDate": "2022-02-17T10:23:37.043Z",
      "content": "<p><a href=\"https://www.kaggle.com/julianmukaj\" target=\"_blank\">@julianmukaj</a> i tested his features on my model which ais based on the same notebook he is using .I get 0.023 on PLB maybe little more with more hyperparametertuning</p>",
      "rawMarkdown": "@julianmukaj i tested his features on my model which ais based on the same notebook he is using .I get 0.023 on PLB maybe little more with more hyperparametertuning",
      "votes": 2,
      "replies": [
        {
          "id": 1694317,
          "postDate": "2022-02-17T10:44:29.240Z",
          "content": "<p>Pretty good score for only having 8 features I think! 🙂</p>",
          "rawMarkdown": "Pretty good score for only having 8 features I think! 🙂",
          "votes": 1
        },
        {
          "id": 1694401,
          "postDate": "2022-02-17T12:04:09.457Z",
          "content": "<p>Did you deal with the overlap of train and test data before submitting?<br>\nAlso I think the score would improve if the data was cleaned in a better way with 'bfill' method instead of replacing NaN with 0.<br>\nAnd recreating the 3% of missing values in Target with a method from notebook Recreating Target.</p>",
          "rawMarkdown": "Did you deal with the overlap of train and test data before submitting?\nAlso I think the score would improve if the data was cleaned in a better way with 'bfill' method instead of replacing NaN with 0.\nAnd recreating the 3% of missing values in Target with a method from notebook Recreating Target.\n",
          "votes": 1
        },
        {
          "id": 1694435,
          "postDate": "2022-02-17T12:31:01.843Z",
          "content": "<p><a href=\"https://www.kaggle.com/rafalradwanski\" target=\"_blank\">@rafalradwanski</a> yes .In your code in the get_features function you can see (valid_window = [totimestamp(\"12/03/2021\")]<br>\nthat mean that your model is validating from 12/03/2021 to 21/09/2021 .if you change it to 12/06/2021 .which is the begining date of the PLB , you can easly reproduce the PLB score without submiting (no overlap also). I test it on my notebook and i submit and it give me the same score .</p>",
          "rawMarkdown": "@rafalradwanski yes .In your code in the get_features function you can see (valid_window = [totimestamp(\"12/03/2021\")]\nthat mean that your model is validating from 12/03/2021 to 21/09/2021 .if you change it to 12/06/2021 .which is the begining date of the PLB , you can easly reproduce the PLB score without submiting (no overlap also). I test it on my notebook and i submit and it give me the same score .",
          "votes": 1
        }
      ]
    },
    {
      "id": 1694249,
      "postDate": "2022-02-17T09:42:37.190Z",
      "content": "<p>These are good features, I had similar ones in my models but roughly 10x more as I created some to match some assets volatility profile better (doge vs btc vs bch all trade very differently ex) and added regularisation to control for overfitting among some other ensemble mixing with different training sets..</p>\n<p>If you submit this what do you get on public LB? (If it lets do you do late submission)</p>",
      "rawMarkdown": "These are good features, I had similar ones in my models but roughly 10x more as I created some to match some assets volatility profile better (doge vs btc vs bch all trade very differently ex) and added regularisation to control for overfitting among some other ensemble mixing with different training sets..\n\nIf you submit this what do you get on public LB? (If it lets do you do late submission)",
      "votes": 1,
      "replies": [
        {
          "id": 1694397,
          "postDate": "2022-02-17T11:57:34.343Z",
          "content": "<p>I did not submit it, but the notebook does not deal with the train and test overlap in data, so I would expect it to score too high in the current state.</p>\n<p>I would probably use more features (from the ones the I left commented out), but there was a memory overload when using 10+ features.</p>\n<p>I think it's a great idea to make the model learn each currency separately, as they are very different in their behaviour. But I see you went a step further that you even used different features for each one, that's very interesting. How would you decide which features to use for different currencies?</p>",
          "rawMarkdown": "I did not submit it, but the notebook does not deal with the train and test overlap in data, so I would expect it to score too high in the current state.\n\nI would probably use more features (from the ones the I left commented out), but there was a memory overload when using 10+ features.\n\nI think it's a great idea to make the model learn each currency separately, as they are very different in their behaviour. But I see you went a step further that you even used different features for each one, that's very interesting. How would you decide which features to use for different currencies?"
        },
        {
          "id": 1694788,
          "postDate": "2022-02-17T17:52:49.900Z",
          "content": "<p>I too had memory issues so switched to using the Polars library instead of Pandas, it was much faster too as a bonus. </p>\n<p>For the features I did some manual inspection of the feature importance for each model on each asset so used that as a base for engineering more relevant ones. Tried to capture some market regime info in there too, nearest S/R levels, lots of volatility measures etc.. L1/L2 helped a lot on overfitting reduction.</p>\n<p>I done 3 models for each asset, each trained and evaluated on different time periods, more or less cross validation but my intention was to get the 3 models to perform well in trending markets and sideways so I chose the “folds” myself, then weighted the score between them at run time for the predictions. GPU was key for this.</p>",
          "rawMarkdown": "I too had memory issues so switched to using the Polars library instead of Pandas, it was much faster too as a bonus. \n\nFor the features I did some manual inspection of the feature importance for each model on each asset so used that as a base for engineering more relevant ones. Tried to capture some market regime info in there too, nearest S/R levels, lots of volatility measures etc.. L1/L2 helped a lot on overfitting reduction.\n\nI done 3 models for each asset, each trained and evaluated on different time periods, more or less cross validation but my intention was to get the 3 models to perform well in trending markets and sideways so I chose the “folds” myself, then weighted the score between them at run time for the predictions. GPU was key for this.\n\n\n\n",
          "votes": 1
        },
        {
          "id": 1695056,
          "postDate": "2022-02-17T23:16:38.587Z",
          "content": "<p>I  once heard from and experienced quant trader that he doesn't use pandas as it is too slow. He uses numpy arrays instead. It requires more effort to do some things, but pandas is too problematic in terms of memory use and speed. <br>\nI will check Polars.</p>\n<p>Could you share some materials on how you identify the importance of features?<br>\nRight now my approach was to create a correlation matrix with features and target. I would look for features that correlate the most with the target, while the least with each other. It seems though that this approach has some flaw, which I don't understand.  The feature Z-score_15_Close has very small negative correlation with the target, looking insignificant. And it also has 70% correlation with another feature. I decided to drop it, but when I did, the score in training went down significantly. So now I wonder why is that the case.  </p>\n<p>Looks like you did a lot of interesting things on the model level :)</p>",
          "rawMarkdown": "I  once heard from and experienced quant trader that he doesn't use pandas as it is too slow. He uses numpy arrays instead. It requires more effort to do some things, but pandas is too problematic in terms of memory use and speed. \nI will check Polars.\n\nCould you share some materials on how you identify the importance of features?\nRight now my approach was to create a correlation matrix with features and target. I would look for features that correlate the most with the target, while the least with each other. It seems though that this approach has some flaw, which I don't understand.  The feature Z-score_15_Close has very small negative correlation with the target, looking insignificant. And it also has 70% correlation with another feature. I decided to drop it, but when I did, the score in training went down significantly. So now I wonder why is that the case.  \n\nLooks like you did a lot of interesting things on the model level :)"
        }
      ]
    },
    {
      "id": 1694424,
      "postDate": "2022-02-17T12:21:37.260Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 1694293,
      "author_name": "Ghassen Alaya",
      "author_url": "",
      "post_date": "2022-02-17T10:23:37.043000",
      "content": "<p><a href=\"https://www.kaggle.com/julianmukaj\" target=\"_blank\">@julianmukaj</a> i tested his features on my model which ais based on the same notebook he is using .I get 0.023 on PLB maybe little more with more hyperparametertuning</p>",
      "votes": 2,
      "replies": [
        {
          "id": 1694317,
          "author_name": "JM",
          "author_url": "",
          "post_date": "2022-02-17T10:44:29.240000",
          "content": "<p>Pretty good score for only having 8 features I think! 🙂</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1694401,
          "author_name": "Ugibugibu",
          "author_url": "",
          "post_date": "2022-02-17T12:04:09.457000",
          "content": "<p>Did you deal with the overlap of train and test data before submitting?<br>\nAlso I think the score would improve if the data was cleaned in a better way with 'bfill' method instead of replacing NaN with 0.<br>\nAnd recreating the 3% of missing values in Target with a method from notebook Recreating Target.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1694435,
          "author_name": "Ghassen Alaya",
          "author_url": "",
          "post_date": "2022-02-17T12:31:01.843000",
          "content": "<p><a href=\"https://www.kaggle.com/rafalradwanski\" target=\"_blank\">@rafalradwanski</a> yes .In your code in the get_features function you can see (valid_window = [totimestamp(\"12/03/2021\")]<br>\nthat mean that your model is validating from 12/03/2021 to 21/09/2021 .if you change it to 12/06/2021 .which is the begining date of the PLB , you can easly reproduce the PLB score without submiting (no overlap also). I test it on my notebook and i submit and it give me the same score .</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1694249,
      "author_name": "JM",
      "author_url": "",
      "post_date": "2022-02-17T09:42:37.190000",
      "content": "<p>These are good features, I had similar ones in my models but roughly 10x more as I created some to match some assets volatility profile better (doge vs btc vs bch all trade very differently ex) and added regularisation to control for overfitting among some other ensemble mixing with different training sets..</p>\n<p>If you submit this what do you get on public LB? (If it lets do you do late submission)</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1694397,
          "author_name": "Ugibugibu",
          "author_url": "",
          "post_date": "2022-02-17T11:57:34.343000",
          "content": "<p>I did not submit it, but the notebook does not deal with the train and test overlap in data, so I would expect it to score too high in the current state.</p>\n<p>I would probably use more features (from the ones the I left commented out), but there was a memory overload when using 10+ features.</p>\n<p>I think it's a great idea to make the model learn each currency separately, as they are very different in their behaviour. But I see you went a step further that you even used different features for each one, that's very interesting. How would you decide which features to use for different currencies?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1694788,
          "author_name": "JM",
          "author_url": "",
          "post_date": "2022-02-17T17:52:49.900000",
          "content": "<p>I too had memory issues so switched to using the Polars library instead of Pandas, it was much faster too as a bonus. </p>\n<p>For the features I did some manual inspection of the feature importance for each model on each asset so used that as a base for engineering more relevant ones. Tried to capture some market regime info in there too, nearest S/R levels, lots of volatility measures etc.. L1/L2 helped a lot on overfitting reduction.</p>\n<p>I done 3 models for each asset, each trained and evaluated on different time periods, more or less cross validation but my intention was to get the 3 models to perform well in trending markets and sideways so I chose the “folds” myself, then weighted the score between them at run time for the predictions. GPU was key for this.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1695056,
          "author_name": "Ugibugibu",
          "author_url": "",
          "post_date": "2022-02-17T23:16:38.587000",
          "content": "<p>I  once heard from and experienced quant trader that he doesn't use pandas as it is too slow. He uses numpy arrays instead. It requires more effort to do some things, but pandas is too problematic in terms of memory use and speed. <br>\nI will check Polars.</p>\n<p>Could you share some materials on how you identify the importance of features?<br>\nRight now my approach was to create a correlation matrix with features and target. I would look for features that correlate the most with the target, while the least with each other. It seems though that this approach has some flaw, which I don't understand.  The feature Z-score_15_Close has very small negative correlation with the target, looking insignificant. And it also has 70% correlation with another feature. I decided to drop it, but when I did, the score in training went down significantly. So now I wonder why is that the case.  </p>\n<p>Looks like you did a lot of interesting things on the model level :)</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1694424,
      "author_name": "",
      "author_url": "",
      "post_date": "2022-02-17T12:21:37.260000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1693949": "Hi,\n\nI created a notebook with a big list of features I was using during this competition.\nThey are quite an improvement from the basic ones. I chose 8 features for the model and it is scoring 0.0356 during training on a basic LGBM model.\nHave a look and give feedback\n\nI am a professional trader, but new to Kaggle. So coming up with features was something more familiar to me, but the coding part was the hard part. Unfortunately I did not manage to submit my notebook properly, but you might still find the input useful.\n\nhttps://www.kaggle.com/rafalradwanski/g-research-features-to-increase-your-score\n\n8 features selected by me:\n\n    df_feat['Z-score_15_Close'] = zscore(df_feat['Close'], 15)\n    df_feat['lower_shadow_15'] = zscore(lower_shadow_15(df_feat), 1440*30*12)\n    df_feat['upper_shadow_15'] = zscore(upper_shadow_15(df_feat), 1440*30*12)\n    df_feat['Return15_%'] = (df_feat['Close'] / df_feat['Close'].shift(15)) - 1\n    df_feat['Return60_%'] = (df_feat['Close'] / df_feat['Close'].shift(60)) - 1\n    df_feat['Candle_body_%'] = (df_feat['Close'] / df_feat['Open']) - 1\n    df_feat['ATR_15_%'] = ((df_feat['High'].rolling(window=15, min_periods=1).max())/ (df_feat['Low'].rolling(window=15, min_periods=1).min()) - 1)\n    df_feat['Z-score_return15_60_%'] = zscore(df_feat['Return15_%'], 60)\n\n\nAll the features created by me or found in other people's notebooks    \n    \n    df_feat['Volume / Count'] = df_feat['Volume'] / df_feat['Count'] \n    df_feat['Close - VWAP'] = df_feat['Close'] - df_feat['VWAP'] \n    df_feat['(Close / VWAP_15) - 1'] = (df_feat['Close'] / df_feat['VWAP'].rolling(window=15, min_periods=1).mean()) - 1\n    df_feat['Close - VWAP_15'] = (df_feat['Close'] - df_feat['VWAP'].rolling(window=15, min_periods=1).mean())\n    df_feat['sma15'] = df_feat['Close'] / df_feat['Close'].rolling(15).mean() -1\n    df_feat['sma60'] = df_feat['Close'] / df_feat['Close'].rolling(60).mean() -1\n    df_feat['sma240'] = df_feat['Close'] / df_feat['Close'].rolling(240).mean() -1\n    \n    df_feat['Bar_Range'] = df_feat['High'] - df_feat['Low']  \n    df_feat['Bar_Range_%'] = (df_feat['High'] / df_feat['Low']) -1\n    \n    df_feat['lower_Shadow'] = lower_shadow(df_feat)\n    df_feat['upper_Shadow'] = upper_shadow(df_feat)\n    df_feat['lower_shadow_percent'] = lower_shadow_percent(df_feat)\n    df_feat['upper_Shadow_percent'] = upper_shadow_percent(df_feat)\n    df_feat['lower_shadow_15'] = lower_shadow_15(df_feat)\n    df_feat['upper_shadow_15'] = upper_shadow_15(df_feat)\n    df_feat['lower_shadow_15_perc'] = lower_shadow_15_perc(df_feat)\n    df_feat['upper_shadow_15_perc'] = upper_shadow_15_perc(df_feat)\n    \n    df_feat['Return1'] = df_feat['Close'] - df_feat['Close'].shift(1) \n    df_feat['Return1_%'] = (df_feat['Close'] / df_feat['Close'].shift(1)) - 1\n    df_feat['Return15'] = df_feat['Close'] - df_feat['Close'].shift(15)\n    df_feat['Return15_%'] = (df_feat['Close'] / df_feat['Close'].shift(15)) - 1\n    df_feat['Return60_%'] = (df_feat['Close'] / df_feat['Close'].shift(60)) - 1\n    df_feat['Return240_%'] = (df_feat['Close'] / df_feat['Close'].shift(240)) - 1\n    df_feat['Candle_body'] = df_feat['Close'] - df_feat['Open'] \n    df_feat['Candle_body_%'] = (df_feat['Close'] / df_feat['Open']) - 1\n    df_feat['Candle_body15'] = df_feat['Close'] - df_feat['Open'].shift(14)\n    df_feat['Candle_body15_%'] = (df_feat['Close'] / df_feat['Open'].shift(14)) -1\n    \n    df_feat['Z-score_15_Close'] = zscore(df_feat['Close'], 15)\n    df_feat['Z-score_60_Volume'] = zscore(df_feat['Volume'], 60)\n    \n    df_feat['Mean_15_Close'] = df_feat['Close'].rolling(window=15, min_periods=1).mean()\n    df_feat['Mean_60_Volume'] = df_feat['Volume'].rolling(window=60, min_periods=1).mean()\n    df_feat['Mean_60_Count'] = df_feat['Count'].rolling(window=60, min_periods=1).mean()\n    df_feat['Z-score_60_Count'] = zscore(df_feat['Count'], 60)\n    df_feat['ATR_15'] = (df_feat['High'].rolling(window=15, min_periods=1).max() - df_feat['Low'].rolling(window=15, min_periods=1).min())\n    df_feat['ATR_15_%'] = ((df_feat['High'].rolling(window=15, min_periods=1).max())/ (df_feat['Low'].rolling(window=15, min_periods=1).min()) - 1)\n    \n    df_feat['Z-score_return15_15'] = zscore(df_feat['Return15'], 15)\n    df_feat['Z-score_return15_15_%'] = zscore(df_feat['Return15_%'], 15)\n    df_feat['Z-score_return15_60'] = zscore(df_feat['Return15'], 60)\n    df_feat['Z-score_return15_60_%'] = zscore(df_feat['Return15_%'], 60)\n    df_feat['Z-score_return15_1440'] = zscore(df_feat['Return15'], 1440)\n    df_feat['Z-score_return15_1440_%'] = zscore(df_feat['Return15_%'], 1440)\n    df_feat['Z-score_return15_month'] = zscore(df_feat['Return15'], 1440*30)\n    df_feat['Z-score_return15_month_%'] = zscore(df_feat['Return15_%'], 1440*30)\n    \n    df_feat['VWAP15_median'] = df_feat['VWAP'].rolling(window=15, min_periods=1).median()\n    df_feat['VWAP60_median'] = df_feat['VWAP'].rolling(window=60, min_periods=1).median()\n    df_feat['VWAP240_median'] = df_feat['VWAP'].rolling(window=240, min_periods=1).median()\n    df_feat['VWAPday_median'] = df_feat['VWAP'].rolling(window=1440, min_periods=1).median()\n    \n    df_feat['Median_return15_15'] = df_feat['Return15'].rolling(window=15, min_periods=1).median()\n    df_feat['Median_return15_15_%'] = df_feat['Return15_%'].rolling(window=15, min_periods=1).median()\n    df_feat['Median_return15_60'] = df_feat['Return15'].rolling(window=60, min_periods=1).median()\n    df_feat['Median_return15_60_%'] = df_feat['Return15_%'].rolling(window=60, min_periods=1).median()\n    df_feat['Median_return15_1440'] = df_feat['Return15'].rolling(window=1440, min_periods=1).median()\n    df_feat['Median_return15_1440_%'] = df_feat['Return15_%'].rolling(window=1440, min_periods=1).median()\n    df_feat['Median_return15_month'] = df_feat['Return15'].rolling(window=1440*30, min_periods=1).median()\n    df_feat['Median_return15_month_%'] = df_feat['Return15_%'].rolling(window=1440*30, min_periods=1).median()\n    df_feat['Median_return15_year'] = df_feat['Return15'].rolling(window=1440*30*12, min_periods=1).median()\n    df_feat['Median_return15_year_%'] = df_feat['Return15_%'].rolling(window=1440*30*12, min_periods=1).median()\n    df_feat['Median_return15_3years_%'] = df_feat['Return15_%'].rolling(window=1440*30*12*3, min_periods=1).median()\n    df_feat['ROC'] =  ((df_feat['Close'] - df_feat['Close'].shift(15)) / df_feat['Close'].shift(15))*100\n\n#Seasonality features\n    df_feat[\"hour\"] = df_feat[\"datetime\"].dt.hour\n    df_feat[\"day of week\"] = df_feat[\"datetime\"].dt.dayofweek \n    df_feat[\"day\"] = df_feat[\"datetime\"].dt.day",
    "1694293": "@julianmukaj i tested his features on my model which ais based on the same notebook he is using .I get 0.023 on PLB maybe little more with more hyperparametertuning",
    "1694249": "These are good features, I had similar ones in my models but roughly 10x more as I created some to match some assets volatility profile better (doge vs btc vs bch all trade very differently ex) and added regularisation to control for overfitting among some other ensemble mixing with different training sets..\n\nIf you submit this what do you get on public LB? (If it lets do you do late submission)",
    "1694424": ""
  }
}