I want to try to fill the gaps in the training data by collecting the information that it's missing (edit1: which is almost done: https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/285913), but first I want to understand all the features.
\nThere is an ongoing notebook that is trying to recreate Target as best as possible.
\nI don't understand how VWAP is obtained/calculated.
\nHere is some information. We have all this data in the train:
\ntrain.csv - The training set
\nEdit2: In this notebook I'm trying to answer these questions.
\nHere some information about VWAP and how it's calculated:
\nFeel free to correct me if I'm wrong or point out if I'm missing something.
", "messages": [ { "id": "1576717", "postDate": "11/09/2021 12:10:30", "content": "I want to try to fill the gaps in the training data by collecting the information that it's missing (edit1: which is almost done: https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/285913), but first I want to understand all the features.
\nThere is an ongoing notebook that is trying to recreate Target as best as possible.
\nI don't understand how VWAP is obtained/calculated.
\nHere is some information. We have all this data in the train:
\ntrain.csv - The training set
\nEdit2: In this notebook I'm trying to answer these questions.
\nHere some information about VWAP and how it's calculated:
\nFeel free to correct me if I'm wrong or point out if I'm missing something.
", "rawMarkdown": "I want to try to fill the gaps in the training data by collecting the information that it's missing (edit1: which is almost done: https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/285913), but first I want to understand all the features. \n\nThere is an ongoing notebook that is trying to [recreate Target ](https://www.kaggle.com/alexfir/recreating-target/notebook) as best as possible. \n\nI don't understand how VWAP is obtained/calculated.\n\nHere is some information. We have all this data in the train: \n\ntrain.csv - The training set \n- timestamp - A timestamp for the minute covered by the row.\n- Asset_ID - An ID code for the cryptoasset.\n- Count - The number of trades that took place this minute.\n- Open - The USD price at the beginning of the minute.\n- High - The highest USD price during the minute.\n- Low - The lowest USD price during the minute.\n- Close - The USD price at the end of the minute.\n- Volume - The number of cryptoasset units traded during the minute.\n- VWAP - The volume weighted average price for the minute.\n- Target - 15 minute residualized returns. See the 'Prediction and Evaluation' section of this notebook for details of how the target is calculated.\n\n\nEdit2: In this [notebook ](https://www.kaggle.com/ajpass/strict-trying-to-understand-vwap/notebook)I'm trying to answer these questions.\n\nHere some information about VWAP and how it's calculated:\n- [Article](https://www.investopedia.com/terms/v/vwap.asp)\n- [Youtube Video](https://www.youtube.com/watch?v=ZMp-nvV5Kw0)\n\n\n\nFeel free to correct me if I'm wrong or point out if I'm missing something.", "votes": null }, { "id": "1576793", "postDate": "11/09/2021 13:16:33", "content": "VWAP measures the average trading price for the minute. I'm sharing the code to better understand how it's computed:
def VWAP(df, period=1):\n df['VWAP_Pre'] = ((df['High']+df['Low']+ df['Close'])/3) * df['Volume']\n df['VWAP_Source'] = pd.Series(df['VWAP_Pre'].rolling(period,min_periods=period).mean())\n df['VolumeSMA'] = pd.Series(df['Volume'].rolling(period, min_periods=period).mean())\n df['VWAP'] = df['VWAP_Source']/df['VolumeSMA']\n return df['VWAP']\nThank you for your comment @abdelghanibelgaid. I tried calculting VWAP with your function with asset=bitcoin, and there is quite big difference between VWAP feature from data and your function output. I shared my notebook to you, so you can see what I'm talking about.
", "rawMarkdown": "Thank you for your comment @abdelghanibelgaid. I tried calculting VWAP with your function with asset=bitcoin, and there is quite big difference between VWAP feature from data and your function output. I shared my notebook to you, so you can see what I'm talking about.", "votes": null }, { "id": "1576949", "postDate": "11/09/2021 15:32:36", "content": "Indeed, there is a small difference between the VWAP feature from data and this function output. In the case of Bitcoin, the mean absolute deviation is about ~2.65. That might be the result of rounding numbers during computing the VWAP.
It can be. One thing I tried is modifying the first line of the function
\ndf['VWAP_Pre'] = ((df['High']+df['Low']+ df['Close'])/3) * df['Volume']
\nto
\ndf['VWAP_Pre'] = ((df['Open']+df['High']+df['Low']+ df['Close'])/4) * df['Volume'] and the MAE reduced.
Edit: Which makes no sense for calculating VWAP.
", "rawMarkdown": "It can be. One thing I tried is modifying the first line of the function\n\n`df['VWAP_Pre'] = ((df['High']+df['Low']+ df['Close'])/3) * df['Volume']`\nto\n`df['VWAP_Pre'] = ((df['Open']+df['High']+df['Low']+ df['Close'])/4) * df['Volume']` and the MAE reduced. \n\nEdit: Which makes no sense for calculating VWAP.", "votes": null }, { "id": "1577652", "postDate": "11/10/2021 09:23:58", "content": "I updated my notebook to visualize this:
\nIf the period=1, the VWAP value it's the same with function VWAP you provided and the next line:
\nbitcoin['VWAP3'] = (df['High']+df['Low']+ df['Close'])/3
", "rawMarkdown": "I updated my notebook to visualize this:\n\nIf the period=1, the VWAP value it's the same with function VWAP you provided and the next line:\n\nbitcoin['VWAP3'] = (df['High']+df['Low']+ df['Close'])/3", "votes": null }, { "id": "1578604", "postDate": "11/11/2021 07:52:25", "content": "In the last competition(https://www.kaggle.com/c/optiver-realized-volatility-prediction/data), optiver revealed the size/price of bid/ask, and it was able to figure out the VWAP value.
\nPerhaps the reason for including VWAP itself in the dataset in this competition is to prevent overfitting due to excessive feature engineering? My very personal opinion :)
Thanks @yuuniekiri for your answer, makes sense to me.
", "rawMarkdown": "Thanks @yuuniekiri for your answer, makes sense to me.", "votes": null } ], "comments": [ { "id": 1576793, "author_name": "abdelghanibelgaid", "author_url": "", "post_date": "11/09/2021 13:16:33", "content": "VWAP measures the average trading price for the minute. I'm sharing the code to better understand how it's computed:
def VWAP(df, period=1):\n df['VWAP_Pre'] = ((df['High']+df['Low']+ df['Close'])/3) * df['Volume']\n df['VWAP_Source'] = pd.Series(df['VWAP_Pre'].rolling(period,min_periods=period).mean())\n df['VolumeSMA'] = pd.Series(df['Volume'].rolling(period, min_periods=period).mean())\n df['VWAP'] = df['VWAP_Source']/df['VolumeSMA']\n return df['VWAP']\nThank you for your comment @abdelghanibelgaid. I tried calculting VWAP with your function with asset=bitcoin, and there is quite big difference between VWAP feature from data and your function output. I shared my notebook to you, so you can see what I'm talking about.
", "votes": null, "replies": [] }, { "id": 1576949, "author_name": "abdelghanibelgaid", "author_url": "", "post_date": "11/09/2021 15:32:36", "content": "Indeed, there is a small difference between the VWAP feature from data and this function output. In the case of Bitcoin, the mean absolute deviation is about ~2.65. That might be the result of rounding numbers during computing the VWAP.
It can be. One thing I tried is modifying the first line of the function
\ndf['VWAP_Pre'] = ((df['High']+df['Low']+ df['Close'])/3) * df['Volume']
\nto
\ndf['VWAP_Pre'] = ((df['Open']+df['High']+df['Low']+ df['Close'])/4) * df['Volume'] and the MAE reduced.
Edit: Which makes no sense for calculating VWAP.
", "votes": null, "replies": [] }, { "id": 1577652, "author_name": "ajpass", "author_url": "", "post_date": "11/10/2021 09:23:58", "content": "I updated my notebook to visualize this:
\nIf the period=1, the VWAP value it's the same with function VWAP you provided and the next line:
\nbitcoin['VWAP3'] = (df['High']+df['Low']+ df['Close'])/3
", "votes": null, "replies": [] } ] }, { "id": 1578604, "author_name": "yuuniekiri", "author_url": "", "post_date": "11/11/2021 07:52:25", "content": "In the last competition(https://www.kaggle.com/c/optiver-realized-volatility-prediction/data), optiver revealed the size/price of bid/ask, and it was able to figure out the VWAP value.
\nPerhaps the reason for including VWAP itself in the dataset in this competition is to prevent overfitting due to excessive feature engineering? My very personal opinion :)
Thanks @yuuniekiri for your answer, makes sense to me.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1576717": "I want to try to fill the gaps in the training data by collecting the information that it's missing (edit1: which is almost done: https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/285913), but first I want to understand all the features. \n\nThere is an ongoing notebook that is trying to [recreate Target ](https://www.kaggle.com/alexfir/recreating-target/notebook) as best as possible. \n\nI don't understand how VWAP is obtained/calculated.\n\nHere is some information. We have all this data in the train: \n\ntrain.csv - The training set \n- timestamp - A timestamp for the minute covered by the row.\n- Asset_ID - An ID code for the cryptoasset.\n- Count - The number of trades that took place this minute.\n- Open - The USD price at the beginning of the minute.\n- High - The highest USD price during the minute.\n- Low - The lowest USD price during the minute.\n- Close - The USD price at the end of the minute.\n- Volume - The number of cryptoasset units traded during the minute.\n- VWAP - The volume weighted average price for the minute.\n- Target - 15 minute residualized returns. See the 'Prediction and Evaluation' section of this notebook for details of how the target is calculated.\n\n\nEdit2: In this [notebook ](https://www.kaggle.com/ajpass/strict-trying-to-understand-vwap/notebook)I'm trying to answer these questions.\n\nHere some information about VWAP and how it's calculated:\n- [Article](https://www.investopedia.com/terms/v/vwap.asp)\n- [Youtube Video](https://www.youtube.com/watch?v=ZMp-nvV5Kw0)\n\n\n\nFeel free to correct me if I'm wrong or point out if I'm missing something.", "1576793": "`VWAP` measures the average trading price for the minute. I'm sharing the code to better understand how it's computed:\n```\ndef VWAP(df, period=1):\n df['VWAP_Pre'] = ((df['High']+df['Low']+ df['Close'])/3) * df['Volume']\n df['VWAP_Source'] = pd.Series(df['VWAP_Pre'].rolling(period,min_periods=period).mean())\n df['VolumeSMA'] = pd.Series(df['Volume'].rolling(period, min_periods=period).mean())\n df['VWAP'] = df['VWAP_Source']/df['VolumeSMA']\n return df['VWAP']\n```", "1576921": "Thank you for your comment @abdelghanibelgaid. I tried calculting VWAP with your function with asset=bitcoin, and there is quite big difference between VWAP feature from data and your function output. I shared my notebook to you, so you can see what I'm talking about.", "1576949": "Indeed, there is a small difference between the `VWAP` feature from data and this function output. In the case of `Bitcoin`, the mean absolute deviation is about ~2.65. That might be the result of rounding numbers during computing the `VWAP`.", "1577034": "It can be. One thing I tried is modifying the first line of the function\n\n`df['VWAP_Pre'] = ((df['High']+df['Low']+ df['Close'])/3) * df['Volume']`\nto\n`df['VWAP_Pre'] = ((df['Open']+df['High']+df['Low']+ df['Close'])/4) * df['Volume']` and the MAE reduced. \n\nEdit: Which makes no sense for calculating VWAP.", "1577652": "I updated my notebook to visualize this:\n\nIf the period=1, the VWAP value it's the same with function VWAP you provided and the next line:\n\nbitcoin['VWAP3'] = (df['High']+df['Low']+ df['Close'])/3", "1578604": "In the last competition(https://www.kaggle.com/c/optiver-realized-volatility-prediction/data), optiver revealed the size/price of bid/ask, and it was able to figure out the VWAP value.\nPerhaps the reason for including VWAP itself in the dataset in this competition is to prevent overfitting due to excessive feature engineering? My very personal opinion :)", "1578719": "Thanks @yuuniekiri for your answer, makes sense to me." }, "source": "meta" }