{
  "id": 286389,
  "title": "Ideas for Feature Engineering",
  "url": "/competitions/g-research-crypto-forecasting/discussion/286389",
  "author_name": "Abdelghani Belgaid",
  "post_date": "2021-11-09T00:47:46.874000",
  "votes": 52,
  "comment_count": 19,
  "views": 0,
  "content": "<p>Start by generating new <strong>technical indicators</strong>, such as:</p>\n<ul>\n<li>Realized Volatility</li>\n<li>Bollinger Bands</li>\n<li>Moving Average</li>\n<li>Moving Average Convergence Divergence</li>\n<li>Cumulative Returns of Price</li>\n<li>Average True Range</li>\n<li>Volume Weighted Average Price (VWAP)<ul>\n<li>VWAP</li>\n<li>Trend Strength of the VWAP</li>\n<li>VWAP Returns</li></ul></li>\n<li>Exponentially Weighted Moving Average</li>\n<li>Momentum</li>\n<li>Rate of Change</li>\n<li>Relative Strength Index</li>\n<li>Stochastic Oscillators</li>\n<li>Ichimoku Cloud</li>\n<li>Money Flow Index</li>\n<li>On-Balance Volume</li>\n<li>Linear Regression Slope</li>\n<li>Log of Return</li>\n<li>Log Transformation</li>\n</ul>\n<p>Next, we need to perform <strong>Fourier or Wavelets transforms</strong> to generalize long and short-term trends. Using these transforms, we will eliminate noise and create approximations of the real cryptocurrency movement that will improve trends prediction more accurately.</p>\n<p>Finally, after generating plenty of features, we need to perform some important steps:</p>\n<ul>\n<li>Statistical checks for the quality of the data, such as heteroscedasticity, multicollinearity, and serial correlation ;</li>\n<li>Feature importance ;</li>\n<li>Stress-test ;</li>\n<li>Reduce the dimensionality of the features.</li>\n</ul>",
  "messages": [
    {
      "id": 1576170,
      "postDate": "2021-11-09T00:47:46.873Z",
      "content": "<p>Start by generating new <strong>technical indicators</strong>, such as:</p>\n<ul>\n<li>Realized Volatility</li>\n<li>Bollinger Bands</li>\n<li>Moving Average</li>\n<li>Moving Average Convergence Divergence</li>\n<li>Cumulative Returns of Price</li>\n<li>Average True Range</li>\n<li>Volume Weighted Average Price (VWAP)<ul>\n<li>VWAP</li>\n<li>Trend Strength of the VWAP</li>\n<li>VWAP Returns</li></ul></li>\n<li>Exponentially Weighted Moving Average</li>\n<li>Momentum</li>\n<li>Rate of Change</li>\n<li>Relative Strength Index</li>\n<li>Stochastic Oscillators</li>\n<li>Ichimoku Cloud</li>\n<li>Money Flow Index</li>\n<li>On-Balance Volume</li>\n<li>Linear Regression Slope</li>\n<li>Log of Return</li>\n<li>Log Transformation</li>\n</ul>\n<p>Next, we need to perform <strong>Fourier or Wavelets transforms</strong> to generalize long and short-term trends. Using these transforms, we will eliminate noise and create approximations of the real cryptocurrency movement that will improve trends prediction more accurately.</p>\n<p>Finally, after generating plenty of features, we need to perform some important steps:</p>\n<ul>\n<li>Statistical checks for the quality of the data, such as heteroscedasticity, multicollinearity, and serial correlation ;</li>\n<li>Feature importance ;</li>\n<li>Stress-test ;</li>\n<li>Reduce the dimensionality of the features.</li>\n</ul>",
      "rawMarkdown": "Start by generating new **technical indicators**, such as:\n- Realized Volatility\n- Bollinger Bands\n- Moving Average\n- Moving Average Convergence Divergence\n- Cumulative Returns of Price\n- Average True Range\n- Volume Weighted Average Price (VWAP)\n  - VWAP\n   - Trend Strength of the VWAP\n   - VWAP Returns\n- Exponentially Weighted Moving Average\n- Momentum\n- Rate of Change\n- Relative Strength Index\n- Stochastic Oscillators\n- Ichimoku Cloud\n- Money Flow Index\n- On-Balance Volume\n- Linear Regression Slope\n- Log of Return\n- Log Transformation\n\nNext, we need to perform **Fourier or Wavelets transforms** to generalize long and short-term trends. Using these transforms, we will eliminate noise and create approximations of the real cryptocurrency movement that will improve trends prediction more accurately.\n\nFinally, after generating plenty of features, we need to perform some important steps:\n- Statistical checks for the quality of the data, such as heteroscedasticity, multicollinearity, and serial correlation ;\n- Feature importance ;\n- Stress-test ;\n- Reduce the dimensionality of the features.",
      "votes": 52
    },
    {
      "id": 1585855,
      "postDate": "2021-11-17T15:53:38.443Z",
      "content": "<p>Just finished implementing many of your ideas <a href=\"https://www.kaggle.com/yamqwe/crypto-prediction-technical-analysis-features\" target=\"_blank\">here</a>, <a href=\"https://www.kaggle.com/yamqwe/crypto-prediction-technical-analysis-feats-2\" target=\"_blank\">here</a>, and <a href=\"https://www.kaggle.com/yamqwe/feature-engineering-rolling-aggregations\" target=\"_blank\">here</a></p>",
      "rawMarkdown": "Just finished implementing many of your ideas [here](https://www.kaggle.com/yamqwe/crypto-prediction-technical-analysis-features), [here](https://www.kaggle.com/yamqwe/crypto-prediction-technical-analysis-feats-2), and [here](https://www.kaggle.com/yamqwe/feature-engineering-rolling-aggregations)",
      "votes": 6,
      "replies": [
        {
          "id": 1588527,
          "postDate": "2021-11-19T12:50:38.427Z",
          "rawMarkdown": "",
          "isDeleted": true
        }
      ]
    },
    {
      "id": 1576759,
      "postDate": "2021-11-09T12:49:27.180Z",
      "content": "<p>You can't do that for the code environment because it's loaded row by row? </p>",
      "rawMarkdown": "You can't do that for the code environment because it's loaded row by row? ",
      "votes": 4,
      "replies": [
        {
          "id": 1576787,
          "postDate": "2021-11-09T13:12:52.747Z",
          "rawMarkdown": "",
          "votes": 2,
          "isDeleted": true
        },
        {
          "id": 1576900,
          "postDate": "2021-11-09T14:38:31.010Z",
          "rawMarkdown": "",
          "votes": 1,
          "isDeleted": true
        },
        {
          "id": 1577130,
          "postDate": "2021-11-09T19:32:01.330Z",
          "content": "<p>Here's a clue..</p>\n<pre><code>last_row_arr = deque([0]*ta_indicator_length)\n\nfor idx, row in df_test.iterrows():\n    last_row_arr.append(row)\n</code></pre>",
          "rawMarkdown": "Here's a clue..\n\n```\nlast_row_arr = deque([0]*ta_indicator_length)\n\nfor idx, row in df_test.iterrows():\n    last_row_arr.append(row)\n\n```",
          "votes": 2
        },
        {
          "id": 1577160,
          "postDate": "2021-11-09T20:25:52.510Z",
          "content": "<p>Do we know if the iterator in the gresearch environment is iterating through the samples in timewise ascending order, like the nth iteration will give the nth timestep of the test set? I think this would be required for making successive appending work. Actually, while writing this I figure it has to be in ascending order, otherwise this would open doors for leakage. Posting anyways, maybe someone else wonders the same. </p>",
          "rawMarkdown": "Do we know if the iterator in the gresearch environment is iterating through the samples in timewise ascending order, like the nth iteration will give the nth timestep of the test set? I think this would be required for making successive appending work. Actually, while writing this I figure it has to be in ascending order, otherwise this would open doors for leakage. Posting anyways, maybe someone else wonders the same. "
        },
        {
          "id": 1577166,
          "postDate": "2021-11-09T20:41:44.187Z",
          "content": "<p>I am pretty sure that this is the case, otherwise the entire competition would be useless.</p>",
          "rawMarkdown": "I am pretty sure that this is the case, otherwise the entire competition would be useless."
        },
        {
          "id": 1591080,
          "postDate": "2021-11-21T23:41:37.850Z",
          "content": "<p>You're right pointing out that it requires multiple rows. Though, it doesn't take much to implement some sort of a circular buffer yourself that will save needed values every step and give you access to the history of last <em>_</em> values. Pretty much any real algo-trading system does exactly that. </p>",
          "rawMarkdown": "You're right pointing out that it requires multiple rows. Though, it doesn't take much to implement some sort of a circular buffer yourself that will save needed values every step and give you access to the history of last ___ values. Pretty much any real algo-trading system does exactly that. "
        }
      ]
    },
    {
      "id": 1595706,
      "postDate": "2021-11-25T23:20:55.787Z",
      "content": "<p>Technical indicator, is hard to use, because , you not only should implement them, you should also decide what strategy to use and at what timeframe, with what optimal parameters, and if you have events TI not working</p>",
      "rawMarkdown": "Technical indicator, is hard to use, because , you not only should implement them, you should also decide what strategy to use and at what timeframe, with what optimal parameters, and if you have events TI not working",
      "votes": 3,
      "replies": [
        {
          "id": 1615267,
          "postDate": "2021-12-12T04:48:10.510Z",
          "content": "<p>Hard to agree… There are events when almost nothing works would it be technical or fundamental. Moreover, it depends on the indicator you use. For example, moving averages almost always work (despite the condition of the dull market) as it is only a smoothing factor (buy when we are above MA, sell when we are below MA), Bollinger Bands combined with volatility analysis almost always work (buy when we penetrate the 2std upper bound with volume confirmation and opposite). Still, there exist indicators that have hard times at different market conditions (like momentum in 2007-2009) but it is not necessary to use them in feature engineering. </p>\n<p>With regard to decisions. Financial markets have a fractal element to their price fluctuations, as the same types of patterns form on monthly charts, daily charts, and 1-minute and tick charts, and the larger patterns and trends are built out of the aggregation of compounded smaller patterns and trends. The micro-scale influences the intermediate scale, which impacts the macro scale. This means that whether you use MA or BB or any other, indicators are independent of time and you can catch a trend in a daily chart using MA as well as you can catch a trend in a minute chart(though there is more noise) . So, it is just preferences. If you are more experienced you probably prefer smaller time frames as there are much more opportunities otherwise if you are 'slightly less experienced' you may prefer longer time frames. The parameters are not as hard to derive. Like, if you use shorter time frames it means you don't want to use very large('long') parameters thus you have a choice between, for example, 2-15 (hint, as you go longer the difference goes to negligible, like the difference between 14 and 15 is less than nothing) which also shrinks your choice. You can optimize parameters using a simple scipy optimizer though there is a high probability of overfitting still not necessary. The last is strategy. I just want to say that the majority of indicators give overlapping signals, for example, if you use BB and you see that the upper bound of BB was penetrated(upper bound located 2std away) you can expect your MA, momentum, MACD had already given the signal.</p>\n<p>Everything is hard to use, even though you picked something, you have to decide how to optimize it. The crypto market is a technical market btw. Tried to keep it short😄😄😄. </p>",
          "rawMarkdown": "Hard to agree... There are events when almost nothing works would it be technical or fundamental. Moreover, it depends on the indicator you use. For example, moving averages almost always work (despite the condition of the dull market) as it is only a smoothing factor (buy when we are above MA, sell when we are below MA), Bollinger Bands combined with volatility analysis almost always work (buy when we penetrate the 2std upper bound with volume confirmation and opposite). Still, there exist indicators that have hard times at different market conditions (like momentum in 2007-2009) but it is not necessary to use them in feature engineering. \n\nWith regard to decisions. Financial markets have a fractal element to their price fluctuations, as the same types of patterns form on monthly charts, daily charts, and 1-minute and tick charts, and the larger patterns and trends are built out of the aggregation of compounded smaller patterns and trends. The micro-scale influences the intermediate scale, which impacts the macro scale. This means that whether you use MA or BB or any other, indicators are independent of time and you can catch a trend in a daily chart using MA as well as you can catch a trend in a minute chart(though there is more noise) . So, it is just preferences. If you are more experienced you probably prefer smaller time frames as there are much more opportunities otherwise if you are 'slightly less experienced' you may prefer longer time frames. The parameters are not as hard to derive. Like, if you use shorter time frames it means you don't want to use very large('long') parameters thus you have a choice between, for example, 2-15 (hint, as you go longer the difference goes to negligible, like the difference between 14 and 15 is less than nothing) which also shrinks your choice. You can optimize parameters using a simple scipy optimizer though there is a high probability of overfitting still not necessary. The last is strategy. I just want to say that the majority of indicators give overlapping signals, for example, if you use BB and you see that the upper bound of BB was penetrated(upper bound located 2std away) you can expect your MA, momentum, MACD had already given the signal.\n\nEverything is hard to use, even though you picked something, you have to decide how to optimize it. The crypto market is a technical market btw. Tried to keep it short😄😄😄. ",
          "votes": 2
        },
        {
          "id": 1630589,
          "postDate": "2021-12-27T13:28:22.360Z",
          "content": "<p>In the begging you wrote<br>\n<em>\"There are events when almost nothing works would it be technical or fundamental\"</em><br>\nthen <br>\n<em>\"moving averages almost always work\"</em></p>\n<p>you make unfounded statements, very general without going into details<br>\nyou wrote <em>\"moving averages almost always work\"</em> what kind of lag you use for what timeframe, what type of MA. <br>\nif you will start playing by MA, you will lose.<br>\n\"(buy when we are above MA, sell when we are below MA), \" 😄😄😄.  You do not mention trading fees or trends.<br>\n<em>\"Financial markets have a fractal element to their price fluctuations</em>\" it's just Theory, not a fact.</p>\n<p><em>\" The micro-scale influences the intermediate scale, which impacts the macro scale.</em>\" No, when big money comes, all previous patterns crash, big investors even create cases when traders like you which use TI, start to lose money</p>\n<p>You just describe several STRATEGIES, even not checking them practically. TI as independent not working. <br>\nIf as you told TI working, what are you doing here, you should be millioner 😄😄😄.</p>",
          "rawMarkdown": "In the begging you wrote\n*\"There are events when almost nothing works would it be technical or fundamental\"*\nthen \n*\"moving averages almost always work\"*\n\nyou make unfounded statements, very general without going into details\nyou wrote *\"moving averages almost always work\"* what kind of lag you use for what timeframe, what type of MA. \nif you will start playing by MA, you will lose.\n\"(buy when we are above MA, sell when we are below MA), \" 😄😄😄.  You do not mention trading fees or trends.\n*\"Financial markets have a fractal element to their price fluctuations*\" it's just Theory, not a fact.\n\n*\" The micro-scale influences the intermediate scale, which impacts the macro scale.*\" No, when big money comes, all previous patterns crash, big investors even create cases when traders like you which use TI, start to lose money\n\nYou just describe several STRATEGIES, even not checking them practically. TI as independent not working. \nIf as you told TI working, what are you doing here, you should be millioner 😄😄😄.",
          "votes": 2
        }
      ]
    },
    {
      "id": 1586533,
      "postDate": "2021-11-18T05:48:21.577Z",
      "content": "<p>Nice list u got there 👍</p>",
      "rawMarkdown": "Nice list u got there 👍",
      "votes": 1
    },
    {
      "id": 1586194,
      "postDate": "2021-11-17T21:50:12.583Z",
      "content": "<p>Great list! As you implement these features, if you want to estimate the highest performance they can yield without overfitting, check out this post I just wrote: <a href=\"https://www.kaggle.com/leanboosting/automated-feature-engineering-part-i\" target=\"_blank\">https://www.kaggle.com/leanboosting/automated-feature-engineering-part-i</a></p>",
      "rawMarkdown": "Great list! As you implement these features, if you want to estimate the highest performance they can yield without overfitting, check out this post I just wrote: https://www.kaggle.com/leanboosting/automated-feature-engineering-part-i",
      "votes": 1
    },
    {
      "id": 1590471,
      "postDate": "2021-11-21T10:11:50.210Z",
      "content": "<p>Nice list </p>",
      "rawMarkdown": "Nice list "
    },
    {
      "id": 1590186,
      "postDate": "2021-11-21T01:42:24.393Z",
      "content": "<p>Anyone interested in pruning non-informative or redundant features from this list, check out this post I recently wrote: <a href=\"https://www.kaggle.com/leanboosting/automating-feature-engineering-part-ii\" target=\"_blank\">https://www.kaggle.com/leanboosting/automating-feature-engineering-part-ii</a></p>",
      "rawMarkdown": "Anyone interested in pruning non-informative or redundant features from this list, check out this post I recently wrote: https://www.kaggle.com/leanboosting/automating-feature-engineering-part-ii"
    },
    {
      "id": 1576248,
      "postDate": "2021-11-09T02:57:05.537Z",
      "content": "<p>Any codes for all of these factors/features.</p>",
      "rawMarkdown": "Any codes for all of these factors/features.",
      "replies": [
        {
          "id": 1576724,
          "postDate": "2021-11-09T12:17:37.073Z",
          "rawMarkdown": "",
          "isDeleted": true
        }
      ]
    },
    {
      "id": 1660875,
      "postDate": "2022-01-23T04:01:38.047Z",
      "content": "<p>Good Posts! Thanks!</p>",
      "rawMarkdown": "Good Posts! Thanks!",
      "votes": 1
    }
  ],
  "comments": [
    {
      "id": 1585855,
      "author_name": "",
      "author_url": "",
      "post_date": "2021-11-17T15:53:38.443000",
      "content": "<p>Just finished implementing many of your ideas <a href=\"https://www.kaggle.com/yamqwe/crypto-prediction-technical-analysis-features\" target=\"_blank\">here</a>, <a href=\"https://www.kaggle.com/yamqwe/crypto-prediction-technical-analysis-feats-2\" target=\"_blank\">here</a>, and <a href=\"https://www.kaggle.com/yamqwe/feature-engineering-rolling-aggregations\" target=\"_blank\">here</a></p>",
      "votes": 6,
      "replies": [
        {
          "id": 1588527,
          "author_name": "",
          "author_url": "",
          "post_date": "2021-11-19T12:50:38.427000",
          "content": "",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1576759,
      "author_name": "ambank1",
      "author_url": "",
      "post_date": "2021-11-09T12:49:27.180000",
      "content": "<p>You can't do that for the code environment because it's loaded row by row? </p>",
      "votes": 4,
      "replies": [
        {
          "id": 1576787,
          "author_name": "",
          "author_url": "",
          "post_date": "2021-11-09T13:12:52.747000",
          "content": "",
          "votes": 2,
          "replies": []
        },
        {
          "id": 1576900,
          "author_name": "",
          "author_url": "",
          "post_date": "2021-11-09T14:38:31.010000",
          "content": "",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1577130,
          "author_name": "JM",
          "author_url": "",
          "post_date": "2021-11-09T19:32:01.330000",
          "content": "<p>Here's a clue..</p>\n<pre><code>last_row_arr = deque([0]*ta_indicator_length)\n\nfor idx, row in df_test.iterrows():\n    last_row_arr.append(row)\n</code></pre>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 1577160,
          "author_name": "Wolfgang Spindeler",
          "author_url": "",
          "post_date": "2021-11-09T20:25:52.510000",
          "content": "<p>Do we know if the iterator in the gresearch environment is iterating through the samples in timewise ascending order, like the nth iteration will give the nth timestep of the test set? I think this would be required for making successive appending work. Actually, while writing this I figure it has to be in ascending order, otherwise this would open doors for leakage. Posting anyways, maybe someone else wonders the same. </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1577166,
          "author_name": "Fritz Cremer",
          "author_url": "",
          "post_date": "2021-11-09T20:41:44.187000",
          "content": "<p>I am pretty sure that this is the case, otherwise the entire competition would be useless.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1591080,
          "author_name": "pAckmAn4000",
          "author_url": "",
          "post_date": "2021-11-21T23:41:37.850000",
          "content": "<p>You're right pointing out that it requires multiple rows. Though, it doesn't take much to implement some sort of a circular buffer yourself that will save needed values every step and give you access to the history of last <em>_</em> values. Pretty much any real algo-trading system does exactly that. </p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1595706,
      "author_name": "Vladimir Shebunyaev",
      "author_url": "",
      "post_date": "2021-11-25T23:20:55.787000",
      "content": "<p>Technical indicator, is hard to use, because , you not only should implement them, you should also decide what strategy to use and at what timeframe, with what optimal parameters, and if you have events TI not working</p>",
      "votes": 3,
      "replies": [
        {
          "id": 1615267,
          "author_name": "AmbrosovDenis",
          "author_url": "",
          "post_date": "2021-12-12T04:48:10.510000",
          "content": "<p>Hard to agree… There are events when almost nothing works would it be technical or fundamental. Moreover, it depends on the indicator you use. For example, moving averages almost always work (despite the condition of the dull market) as it is only a smoothing factor (buy when we are above MA, sell when we are below MA), Bollinger Bands combined with volatility analysis almost always work (buy when we penetrate the 2std upper bound with volume confirmation and opposite). Still, there exist indicators that have hard times at different market conditions (like momentum in 2007-2009) but it is not necessary to use them in feature engineering. </p>\n<p>With regard to decisions. Financial markets have a fractal element to their price fluctuations, as the same types of patterns form on monthly charts, daily charts, and 1-minute and tick charts, and the larger patterns and trends are built out of the aggregation of compounded smaller patterns and trends. The micro-scale influences the intermediate scale, which impacts the macro scale. This means that whether you use MA or BB or any other, indicators are independent of time and you can catch a trend in a daily chart using MA as well as you can catch a trend in a minute chart(though there is more noise) . So, it is just preferences. If you are more experienced you probably prefer smaller time frames as there are much more opportunities otherwise if you are 'slightly less experienced' you may prefer longer time frames. The parameters are not as hard to derive. Like, if you use shorter time frames it means you don't want to use very large('long') parameters thus you have a choice between, for example, 2-15 (hint, as you go longer the difference goes to negligible, like the difference between 14 and 15 is less than nothing) which also shrinks your choice. You can optimize parameters using a simple scipy optimizer though there is a high probability of overfitting still not necessary. The last is strategy. I just want to say that the majority of indicators give overlapping signals, for example, if you use BB and you see that the upper bound of BB was penetrated(upper bound located 2std away) you can expect your MA, momentum, MACD had already given the signal.</p>\n<p>Everything is hard to use, even though you picked something, you have to decide how to optimize it. The crypto market is a technical market btw. Tried to keep it short😄😄😄. </p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 1630589,
          "author_name": "Vladimir Shebunyaev",
          "author_url": "",
          "post_date": "2021-12-27T13:28:22.360000",
          "content": "<p>In the begging you wrote<br>\n<em>\"There are events when almost nothing works would it be technical or fundamental\"</em><br>\nthen <br>\n<em>\"moving averages almost always work\"</em></p>\n<p>you make unfounded statements, very general without going into details<br>\nyou wrote <em>\"moving averages almost always work\"</em> what kind of lag you use for what timeframe, what type of MA. <br>\nif you will start playing by MA, you will lose.<br>\n\"(buy when we are above MA, sell when we are below MA), \" 😄😄😄.  You do not mention trading fees or trends.<br>\n<em>\"Financial markets have a fractal element to their price fluctuations</em>\" it's just Theory, not a fact.</p>\n<p><em>\" The micro-scale influences the intermediate scale, which impacts the macro scale.</em>\" No, when big money comes, all previous patterns crash, big investors even create cases when traders like you which use TI, start to lose money</p>\n<p>You just describe several STRATEGIES, even not checking them practically. TI as independent not working. <br>\nIf as you told TI working, what are you doing here, you should be millioner 😄😄😄.</p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 1586533,
      "author_name": "Anant Dashpute",
      "author_url": "",
      "post_date": "2021-11-18T05:48:21.577000",
      "content": "<p>Nice list u got there 👍</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1586194,
      "author_name": "Yves-Laurent Kom Samo, PhD",
      "author_url": "",
      "post_date": "2021-11-17T21:50:12.583000",
      "content": "<p>Great list! As you implement these features, if you want to estimate the highest performance they can yield without overfitting, check out this post I just wrote: <a href=\"https://www.kaggle.com/leanboosting/automated-feature-engineering-part-i\" target=\"_blank\">https://www.kaggle.com/leanboosting/automated-feature-engineering-part-i</a></p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1590471,
      "author_name": "Abdelrahman A.",
      "author_url": "",
      "post_date": "2021-11-21T10:11:50.210000",
      "content": "<p>Nice list </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1590186,
      "author_name": "Yves-Laurent Kom Samo, PhD",
      "author_url": "",
      "post_date": "2021-11-21T01:42:24.393000",
      "content": "<p>Anyone interested in pruning non-informative or redundant features from this list, check out this post I recently wrote: <a href=\"https://www.kaggle.com/leanboosting/automating-feature-engineering-part-ii\" target=\"_blank\">https://www.kaggle.com/leanboosting/automating-feature-engineering-part-ii</a></p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1576248,
      "author_name": "angel_never_cry",
      "author_url": "",
      "post_date": "2021-11-09T02:57:05.537000",
      "content": "<p>Any codes for all of these factors/features.</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1576724,
          "author_name": "",
          "author_url": "",
          "post_date": "2021-11-09T12:17:37.073000",
          "content": "",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1660875,
      "author_name": "myth",
      "author_url": "",
      "post_date": "2022-01-23T04:01:38.047000",
      "content": "<p>Good Posts! Thanks!</p>",
      "votes": 1,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1576170": "Start by generating new **technical indicators**, such as:\n- Realized Volatility\n- Bollinger Bands\n- Moving Average\n- Moving Average Convergence Divergence\n- Cumulative Returns of Price\n- Average True Range\n- Volume Weighted Average Price (VWAP)\n  - VWAP\n   - Trend Strength of the VWAP\n   - VWAP Returns\n- Exponentially Weighted Moving Average\n- Momentum\n- Rate of Change\n- Relative Strength Index\n- Stochastic Oscillators\n- Ichimoku Cloud\n- Money Flow Index\n- On-Balance Volume\n- Linear Regression Slope\n- Log of Return\n- Log Transformation\n\nNext, we need to perform **Fourier or Wavelets transforms** to generalize long and short-term trends. Using these transforms, we will eliminate noise and create approximations of the real cryptocurrency movement that will improve trends prediction more accurately.\n\nFinally, after generating plenty of features, we need to perform some important steps:\n- Statistical checks for the quality of the data, such as heteroscedasticity, multicollinearity, and serial correlation ;\n- Feature importance ;\n- Stress-test ;\n- Reduce the dimensionality of the features.",
    "1585855": "Just finished implementing many of your ideas [here](https://www.kaggle.com/yamqwe/crypto-prediction-technical-analysis-features), [here](https://www.kaggle.com/yamqwe/crypto-prediction-technical-analysis-feats-2), and [here](https://www.kaggle.com/yamqwe/feature-engineering-rolling-aggregations)",
    "1576759": "You can't do that for the code environment because it's loaded row by row? ",
    "1595706": "Technical indicator, is hard to use, because , you not only should implement them, you should also decide what strategy to use and at what timeframe, with what optimal parameters, and if you have events TI not working",
    "1586533": "Nice list u got there 👍",
    "1586194": "Great list! As you implement these features, if you want to estimate the highest performance they can yield without overfitting, check out this post I just wrote: https://www.kaggle.com/leanboosting/automated-feature-engineering-part-i",
    "1590471": "Nice list ",
    "1590186": "Anyone interested in pruning non-informative or redundant features from this list, check out this post I recently wrote: https://www.kaggle.com/leanboosting/automating-feature-engineering-part-ii",
    "1576248": "Any codes for all of these factors/features.",
    "1660875": "Good Posts! Thanks!"
  }
}