![\"\"](\"https://i.ibb.co/mHgnBPC/close-close.png\")
'
\n\n\n\nTL;DR: Code
\n
Hi guys, I published a notebook implementing multiple different volatility estimators that can be used as features for your models.
\nWhat is so important about volatility?
\nVolatility is basically the \"speed\" of the asset.
\nHowever, predicting volatility is basically nearly as hard as predicting the price itself. Simply due to the fact that the future volatility IS the price just without the sign.
\nFortunately, in contrast to predicting future prices, volatility has some interesting properties which we can use for estimating future volatility.
\nThis is an established field and there are many different types of volatility models that are described
\nin what is extremely broad literature.
In the notebook above I implemented multiple volatility metrics that can be used as features for your models.
\n
Close-to-Close volatility is a classic and most commonly used volatility measure, sometimes referred to as historical volatility.
\nVolatility is an indicator of the speed of a stock price change. A stock with high volatility is one where the price changes rapidly and with a bigger amplitude. The more volatile a stock is, the riskier it is.
\nClose-to-close historical volatility calculated using only stock's closing prices. It is the simplest volatility estimator. But in many cases, it is not precise enough. Stock prices could jump considerably during a trading session, and return to the open value at the end. That means that a big amount of price information is not taken into account by close-to-close volatility.
\nDespite its drawbacks, Close-to-Close volatility is still useful in cases where the instrument doesn't have intraday prices. For example, mutual funds calculate their net asset values daily or weekly, and thus their prices are not suitable for more sophisticated volatility estimators.
\n\nDefined as:
\n$$\\sigma_{realized} = \\sqrt{ \\frac{N}{n-2} \\sum\\limits_{i=1} ^{n-1} (r_t-\\bar r)^2 }$$
\nWhere the Rate of return:
\n$$r_t =\\log\\frac{C_t}{C_{t-1}}$$
and the Average rate of return:
\n$$ \\bar r =\\frac{1}{n} \\sum\\limits_{n}^{t=1}r_t$$
![\"\"](\"https://i.ibb.co/8rSV4wP/parkinson.png\")
Parkinson volatility is a volatility measure that uses the stock’s high and low price of the day.
\nThe main difference between regular volatility and Parkinson volatility is that the latter uses high and low prices for a day, rather than only the closing price. That is useful as close to close prices could show little difference while large price movements could have happened during the day. Thus Parkinson's volatility is considered to be more precise and requires less data for calculation than the close-close volatility.
\nOne drawback of this estimator is that it doesn't take into account price movements after market close. Hence it systematically undervalues volatility.
\n\nDefined as:
\n$$\\sigma_{parkinson} = \\sqrt{ {\\frac{1}{4*\\ln{2}} * \\frac{252}{n} * \\sum\\limits_{t=1}^{n} { \\ln{(\\frac{H_t}{L_t})}}^2}}$$
\n![\"\"](\"https://i.ibb.co/N154bst/german-klass.png\")
Garman Klass is a volatility estimator that incorporates open, low, high, and close prices of a security.
\nGarman-Klass volatility extends Parkinson's volatility by taking into account the opening and closing price. As markets are most active during the opening and closing of a trading session, it makes volatility estimation more accurate.
\nGarman and Klass also assumed that the process of price change is a process of continuous diffusion (geometric Brownian motion). However, this assumption has several drawbacks. The method is not robust for opening jumps in price and trend movements.
\nDespite its drawbacks, the Garman-Klass estimator is still more effective than the basic formula since it takes into account not only the price at the beginning and end of the time interval but also intraday price extremums.
\n\nDefined as:
\n$$\\sigma_{garman-klass} = \\sqrt{\\frac{N}{n} \\sum\\limits_{i=1}^{N} \\lbrack {\\frac{1}{2} * (\\log{ \\frac{H_i}{L_i}})^2 -(2*\\log2 -1) * (\\log\\frac{C_i}{O_i})^2\\rbrack}}$$
\n![\"\"](\"https://i.ibb.co/4MdyGM0/roger-sachel.png\")
Rogers-Satchell is an estimator for measuring the volatility of securities with an average return not equal to zero.
\nUnlike Parkinson and Garman-Klass estimators, Rogers-Satchell incorporates drift term (mean return not equal to zero). As a result, it provides a better volatility estimation when the underlying is trending.
\nThe main disadvantage of this method is that it does not take into account price movements between trading sessions. It means an underestimation of volatility since price jumps periodically occur in the market precisely at the moments between sessions.
\n\n$$\\sigma_{roger-satchel} = \\sqrt{ \\frac{N}{n} \\sum\\limits_{i=1}^{n} \\lbrack \\log \\frac{H_i}{L_i} * \\log \\ frac{H_i}{O_i} + \\log \\frac{HL_i}{L_i} * \\log \\frac{L_i}{O_i} \\rbrack }$$
\n![\"\"](\"https://i.ibb.co/4m08FGH/yang-zhang.png\")
Yang Zhang is a historical volatility estimator that handles both opening jumps and the drift and has a minimum estimation error.
\nWe can think of the Yang-Zhang volatility as the combination of the overnight (close-to-open volatility) and a weighted average of the Rogers-Satchell volatility and the day’s open-to-close volatility. It considered being 14 times more efficient than the close-to-close estimator.
\n\nDefined as:
\n$$\\sigma_{yang-zhang} = \\sqrt {\\sigma_o^2 + k * \\sigma_c^2 + (1-k) * \\sigma_{rs}^2}$$
\n$$\\mu_o = \\frac{1}{n} \\sum\\limits_{i=1}^{n} \\log \\frac {O_i}{C_{i-1}}$$
\n$$\\sigma_o^2 = \\frac{N}{n-1} \\sum\\limits_{i=1}^{n} (\\log \\frac {O_i}{C_{i-1}}-\\mu_o)^ 2$$
\n(Open-Close Volatility or Overnight Volatility)
\n$$\\mu_c = \\frac{1}{n} \\sum\\limits_{i=1}^{n} \\log \\frac {C_i}{O_i}$$
\n(Close-Open Volatility)
\n$$\\sigma_c^2 = \\frac{N}{n-1} \\sum\\limits_{i=1}^{n} (\\log \\frac {C_i}{O_i}-\\mu_c)^2$$
\n$$\\sigma_{rs}^2 = \\sigma_{roger-satchel}^2$$
\n$$k^* = \\frac {\\alpha} {1+ \\alpha + \\frac{n+1}{n-1}}, \\alpha$$
\ncan also be interpreted as a weighted average of the Roger-Satchell estimator, the Close-Open Volatility and the Open-Close Volatility.
\n![\"\"](\"https://i.ibb.co/Z277CMb/german-klass-yang-zhang.png\")
A modified version of Garman-Klass estimator that allows for opening spikes.
\n$$\\sigma_{garkla-yangzh} = \\sqrt {\\frac{N}{n} \\sum\\limits_{i=1}^{n} \\lbrack (\\log \\frac{O_i}{C_{i-1 }})^2 + {\\frac{1}{2} * (\\log{\\frac{H_i}{L_i}})^2 -(2*\\log2 -1) * (\\log\\frac{C_i} {O_i})^2}\\rbrack }$$
\nIn the cases where the return on assets is not zero, the volatility will be overestimated.
", "messages": [ { "id": "1645943", "postDate": "01/11/2022 11:08:30", "content": "'
\n\n\n\nTL;DR: Code
\n
Hi guys, I published a notebook implementing multiple different volatility estimators that can be used as features for your models.
\nWhat is so important about volatility?
\nVolatility is basically the \"speed\" of the asset.
\nHowever, predicting volatility is basically nearly as hard as predicting the price itself. Simply due to the fact that the future volatility IS the price just without the sign.
\nFortunately, in contrast to predicting future prices, volatility has some interesting properties which we can use for estimating future volatility.
\nThis is an established field and there are many different types of volatility models that are described
\nin what is extremely broad literature.
In the notebook above I implemented multiple volatility metrics that can be used as features for your models.
\n
Close-to-Close volatility is a classic and most commonly used volatility measure, sometimes referred to as historical volatility.
\nVolatility is an indicator of the speed of a stock price change. A stock with high volatility is one where the price changes rapidly and with a bigger amplitude. The more volatile a stock is, the riskier it is.
\nClose-to-close historical volatility calculated using only stock's closing prices. It is the simplest volatility estimator. But in many cases, it is not precise enough. Stock prices could jump considerably during a trading session, and return to the open value at the end. That means that a big amount of price information is not taken into account by close-to-close volatility.
\nDespite its drawbacks, Close-to-Close volatility is still useful in cases where the instrument doesn't have intraday prices. For example, mutual funds calculate their net asset values daily or weekly, and thus their prices are not suitable for more sophisticated volatility estimators.
\n\nDefined as:
\n$$\\sigma_{realized} = \\sqrt{ \\frac{N}{n-2} \\sum\\limits_{i=1} ^{n-1} (r_t-\\bar r)^2 }$$
\nWhere the Rate of return:
\n$$r_t =\\log\\frac{C_t}{C_{t-1}}$$
and the Average rate of return:
\n$$ \\bar r =\\frac{1}{n} \\sum\\limits_{n}^{t=1}r_t$$
Parkinson volatility is a volatility measure that uses the stock’s high and low price of the day.
\nThe main difference between regular volatility and Parkinson volatility is that the latter uses high and low prices for a day, rather than only the closing price. That is useful as close to close prices could show little difference while large price movements could have happened during the day. Thus Parkinson's volatility is considered to be more precise and requires less data for calculation than the close-close volatility.
\nOne drawback of this estimator is that it doesn't take into account price movements after market close. Hence it systematically undervalues volatility.
\n\nDefined as:
\n$$\\sigma_{parkinson} = \\sqrt{ {\\frac{1}{4*\\ln{2}} * \\frac{252}{n} * \\sum\\limits_{t=1}^{n} { \\ln{(\\frac{H_t}{L_t})}}^2}}$$
\n
Garman Klass is a volatility estimator that incorporates open, low, high, and close prices of a security.
\nGarman-Klass volatility extends Parkinson's volatility by taking into account the opening and closing price. As markets are most active during the opening and closing of a trading session, it makes volatility estimation more accurate.
\nGarman and Klass also assumed that the process of price change is a process of continuous diffusion (geometric Brownian motion). However, this assumption has several drawbacks. The method is not robust for opening jumps in price and trend movements.
\nDespite its drawbacks, the Garman-Klass estimator is still more effective than the basic formula since it takes into account not only the price at the beginning and end of the time interval but also intraday price extremums.
\n\nDefined as:
\n$$\\sigma_{garman-klass} = \\sqrt{\\frac{N}{n} \\sum\\limits_{i=1}^{N} \\lbrack {\\frac{1}{2} * (\\log{ \\frac{H_i}{L_i}})^2 -(2*\\log2 -1) * (\\log\\frac{C_i}{O_i})^2\\rbrack}}$$
\n
Rogers-Satchell is an estimator for measuring the volatility of securities with an average return not equal to zero.
\nUnlike Parkinson and Garman-Klass estimators, Rogers-Satchell incorporates drift term (mean return not equal to zero). As a result, it provides a better volatility estimation when the underlying is trending.
\nThe main disadvantage of this method is that it does not take into account price movements between trading sessions. It means an underestimation of volatility since price jumps periodically occur in the market precisely at the moments between sessions.
\n\n$$\\sigma_{roger-satchel} = \\sqrt{ \\frac{N}{n} \\sum\\limits_{i=1}^{n} \\lbrack \\log \\frac{H_i}{L_i} * \\log \\ frac{H_i}{O_i} + \\log \\frac{HL_i}{L_i} * \\log \\frac{L_i}{O_i} \\rbrack }$$
\n
Yang Zhang is a historical volatility estimator that handles both opening jumps and the drift and has a minimum estimation error.
\nWe can think of the Yang-Zhang volatility as the combination of the overnight (close-to-open volatility) and a weighted average of the Rogers-Satchell volatility and the day’s open-to-close volatility. It considered being 14 times more efficient than the close-to-close estimator.
\n\nDefined as:
\n$$\\sigma_{yang-zhang} = \\sqrt {\\sigma_o^2 + k * \\sigma_c^2 + (1-k) * \\sigma_{rs}^2}$$
\n$$\\mu_o = \\frac{1}{n} \\sum\\limits_{i=1}^{n} \\log \\frac {O_i}{C_{i-1}}$$
\n$$\\sigma_o^2 = \\frac{N}{n-1} \\sum\\limits_{i=1}^{n} (\\log \\frac {O_i}{C_{i-1}}-\\mu_o)^ 2$$
\n(Open-Close Volatility or Overnight Volatility)
\n$$\\mu_c = \\frac{1}{n} \\sum\\limits_{i=1}^{n} \\log \\frac {C_i}{O_i}$$
\n(Close-Open Volatility)
\n$$\\sigma_c^2 = \\frac{N}{n-1} \\sum\\limits_{i=1}^{n} (\\log \\frac {C_i}{O_i}-\\mu_c)^2$$
\n$$\\sigma_{rs}^2 = \\sigma_{roger-satchel}^2$$
\n$$k^* = \\frac {\\alpha} {1+ \\alpha + \\frac{n+1}{n-1}}, \\alpha$$
\ncan also be interpreted as a weighted average of the Roger-Satchell estimator, the Close-Open Volatility and the Open-Close Volatility.
\n
A modified version of Garman-Klass estimator that allows for opening spikes.
\n$$\\sigma_{garkla-yangzh} = \\sqrt {\\frac{N}{n} \\sum\\limits_{i=1}^{n} \\lbrack (\\log \\frac{O_i}{C_{i-1 }})^2 + {\\frac{1}{2} * (\\log{\\frac{H_i}{L_i}})^2 -(2*\\log2 -1) * (\\log\\frac{C_i} {O_i})^2}\\rbrack }$$
\nIn the cases where the return on assets is not zero, the volatility will be overestimated.
", "rawMarkdown": "'\n# Using volatility estimates as features\n\n> **TL;DR:** [Code](http://www.kaggle.com/yamqwe/crypto-prediction-volatility-features)\n\nFYI crypto markets don't close.
", "rawMarkdown": "FYI crypto markets don't close.", "votes": null }, { "id": "1647342", "postDate": "01/12/2022 14:08:51", "content": "I appreciate your prolific notebooks. Have you been able to actually use any of these estimators during the prediction time (not in CV, but using the API to iterate through minutes)? I ask because my (limited) experience with this competition indicates that complex feature engineering is not possible due to compute time constraints.
", "rawMarkdown": "I appreciate your prolific notebooks. Have you been able to actually use any of these estimators during the prediction time (not in CV, but using the API to iterate through minutes)? I ask because my (limited) experience with this competition indicates that complex feature engineering is not possible due to compute time constraints.", "votes": null }, { "id": "1650015", "postDate": "01/14/2022 18:46:11", "content": "It will be a problem to include these volatilities. The solution offered is too slow. If we want to use it, we need to figure out a much faster way, or only include some of them.
", "rawMarkdown": "It will be a problem to include these volatilities. The solution offered is too slow. If we want to use it, we need to figure out a much faster way, or only include some of them.", "votes": null }, { "id": "1650049", "postDate": "01/14/2022 19:16:21", "content": "Yes- that was the subtext of my question. These features are interesting, but if they can't fit in the prediction time, then ultimately not helpful.
", "rawMarkdown": "Yes- that was the subtext of my question. These features are interesting, but if they can't fit in the prediction time, then ultimately not helpful.", "votes": null }, { "id": "1652843", "postDate": "01/17/2022 03:42:17", "content": "yes, that makes them more volatile
", "rawMarkdown": "yes, that makes them more volatile", "votes": null }, { "id": "3430184", "postDate": "03/28/2026 03:10:49", "content": "Have you considered how you can use your forecasts to trade? If your forecast for realized volatility is under the implied volatility perhaps options are 'cheap'.
", "rawMarkdown": "Have you considered how you can use your forecasts to trade? If your forecast for realized volatility is under the implied volatility perhaps options are 'cheap'.", "votes": null } ], "comments": [ { "id": 1646459, "author_name": "lucasmorin", "author_url": "", "post_date": "01/11/2022 18:40:17", "content": "FYI crypto markets don't close.
", "votes": null, "replies": [ { "id": 1652843, "author_name": "jonathansilverman", "author_url": "", "post_date": "01/17/2022 03:42:17", "content": "yes, that makes them more volatile
", "votes": null, "replies": [] } ] }, { "id": 1647342, "author_name": "marketneutral", "author_url": "", "post_date": "01/12/2022 14:08:51", "content": "I appreciate your prolific notebooks. Have you been able to actually use any of these estimators during the prediction time (not in CV, but using the API to iterate through minutes)? I ask because my (limited) experience with this competition indicates that complex feature engineering is not possible due to compute time constraints.
", "votes": null, "replies": [ { "id": 1650015, "author_name": "sihead", "author_url": "", "post_date": "01/14/2022 18:46:11", "content": "It will be a problem to include these volatilities. The solution offered is too slow. If we want to use it, we need to figure out a much faster way, or only include some of them.
", "votes": null, "replies": [] }, { "id": 1650049, "author_name": "marketneutral", "author_url": "", "post_date": "01/14/2022 19:16:21", "content": "Yes- that was the subtext of my question. These features are interesting, but if they can't fit in the prediction time, then ultimately not helpful.
", "votes": null, "replies": [] } ] }, { "id": 3430184, "author_name": "fourtwentyshuvit", "author_url": "", "post_date": "03/28/2026 03:10:49", "content": "Have you considered how you can use your forecasts to trade? If your forecast for realized volatility is under the implied volatility perhaps options are 'cheap'.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1645943": "'\n# Using volatility estimates as features\n\n> **TL;DR:** [Code](http://www.kaggle.com/yamqwe/crypto-prediction-volatility-features)\n\n