{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## G-Research Cryptocurrencies Forecasting : Data Analysis","metadata":{}},{"cell_type":"markdown","source":"\n\nCryptocurrencies are very popular but have the disavantage to be more volatile than classical stock prices. The first cryptocurrency, Bitcoin, was created in 2008 by Satoshi Nakamoto. Bitcoin is created as a decentralized system based on the blockchain technology. However, in the next years, creation of many other cryptocurrencies has been exploded. The expansion and democratization of these methods have opened new horizons for stock market.\n\nIn this Kaggle competition, the aim is to predict trends about cryptocurrencies by using the most well-known currencies. It is important to keep in mind that forecasting stock market is a hard task and thus, forecasting trends in cryptocurrencies is harder.\n","metadata":{}},{"cell_type":"markdown","source":"#### Datasets","metadata":{}},{"cell_type":"markdown","source":"5 files are available in this competition : \n- a Train set\n- a supplementary train set\n- an asset set with information about cryptos\n- an example test\n- an submission test example\n\nWe will only use the first 3.","metadata":{}},{"cell_type":"markdown","source":"### 1. Imports & Preprocessings","metadata":{}},{"cell_type":"code","source":"# system libraries\nimport warnings\nwarnings.filterwarnings('ignore')\nwarnings.simplefilter(action='ignore', category=FutureWarning)\nimport logging\nlogging.disable(logging.CRITICAL)\nimport gc\n\n# data manipulation libraries\nimport pandas as pd\nimport numpy as np\n\n# graphical libraries\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport plotly.express as px\nimport plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\n\n# financial analysis\nfrom finta import TA\n\nfrom functools import reduce\nfrom datetime import datetime\nimport time","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def crypto_df(asset_id, data):\n    # Copied from https://www.kaggle.com/odins0n/g-research-plots-eda\n    df = data[data[\"asset_name\"] == asset_id].reset_index(drop=True)\n    df['timestamp'] = pd.to_datetime(df['timestamp'], unit='s')\n    df = df.set_index('timestamp')\n    return df\n\n\ndef candelstick_chart(data, title):\n    # Copied from https://www.kaggle.com/odins0n/g-research-plots-eda\n    candlestick = go.Figure(data=[go.Candlestick(x=data.index,\n                                                 open=data[('open')],\n                                                 high=data[('high')],\n                                                 low=data[('low')],\n                                                 close=data[('close')])])\n    candlestick.update_xaxes(title_text='Time',\n                             rangeslider_visible=False)\n\n    candlestick.update_layout(\n        title={\n            'text': '{:} Candelstick Chart'.format(title),\n            'y': 0.90,\n            'x': 0.5,\n            'xanchor': 'center',\n            'yanchor': 'top'})\n\n    candlestick.update_yaxes(title_text='Price in USD', ticksuffix='$')\n    return candlestick\n\n\ndef vol_traded(data, title, color):\n    # Copied from https://www.kaggle.com/odins0n/g-research-plots-eda\n    area = px.area(data_frame=data,\n                   x=data.index,\n                   y=\"volume\")\n    area.update_traces(line_color=color)\n    area.update_xaxes(\n        title_text='Time',\n        rangeslider_visible=False)\n    area.update_yaxes(title_text='Number of trades every minute')\n    area.update_layout(showlegend=True,\n                       title={\n                           'text': '{:} Volume Traded'.format(title),\n                           'y': 0.94,\n                           'x': 0.5,\n                           'xanchor': 'center',\n                           'yanchor': 'top'})\n    return area","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"DATA_PATH = 'g-research-crypto-forecasting/'\nTRAIN_PATH = DATA_PATH + 'train.csv'\nASSETS_PATH = DATA_PATH + 'asset_details.csv'","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = pd.read_csv(TRAIN_PATH, \n                       dtype={'Asset_ID': 'int8', 'Count': 'int32', 'row_id': 'int32', 'Count': 'int32', \n                              'Open': 'float64', 'High': 'float64', 'Low': 'float64', 'Close': 'float64', \n                              'Volume': 'float64', 'VWAP': 'float64'\n                             }\n                      )\ntrain.head(3)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\nColumns of the dataset :\n\n   - timestamp - A timestamp for the minute covered by the row.\n   - Asset_ID - An ID code for the cryptoasset.\n   - Count - The number of trades that took place this minute.\n   - Open - The USD price at the beginning of the minute.\n   - High - The highest USD price during the minute.\n   - Low - The lowest USD price during the minute.\n   - Close - The USD price at the end of the minute.\n   - Volume - The number of cryptoasset units traded during the minute.\n   - VWAP - The volume weighted average price for the minute.\n   - Target - 15 minute residualized returns.\n\nTarget column contains values to predict (Information about target calculus).\n","metadata":{}},{"cell_type":"code","source":"dict_asset = {\n    0 : \"Binance Coin\",\n    1 : \"Bitcoin\",\n    2 : \"Bitcoin Cash\",\n    3 : \"Cardano\",\n    4 : \"Dogecoin\",\n    5 : \"EOS.IO\",\n    6 : \"Ethereum\",\n    7 : \"Ethereum Classic\",\n    8 : \"IOTA\",\n    9 : \"Litecoin\",\n    10 : \"Maker\",\n    11 : \"Monero\",\n    12 : \"Stellar\",\n    13 : \"TRON\"\n}\n\ntrain[\"Asset_ID\"] = train[\"Asset_ID\"].map(dict_asset)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.columns = [\"timestamp\", \"asset_name\", \"count\",\n                 \"open\", \"high\", \"low\",\n                 \"close\", \"volume\", \"vwap\",\n                 \"target\"]","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = train.dropna()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head(3)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2. Exploratory Analysis","metadata":{}},{"cell_type":"markdown","source":"#### 2.1 Candlesticks","metadata":{}},{"cell_type":"code","source":"binance = crypto_df(\"Binance Coin\", data=train)\nbtc = crypto_df(\"Bitcoin\", data=train)\nbtc_cash = crypto_df(\"Bitcoin Cash\", data=train)\ncardano = crypto_df(\"Cardano\", data=train)\ndogecoin = crypto_df(\"Dogecoin\", data=train)\neos = crypto_df(\"EOS.IO\", data=train)\neth = crypto_df(\"Ethereum\", data=train)\neth_classic = crypto_df(\"Ethereum Classic\", data=train)\niota = crypto_df(\"IOTA\", data=train)\nlitecoin = crypto_df(\"Litecoin\", data=train)\nmaker = crypto_df(\"Maker\", data=train)\nmonero = crypto_df(\"Monero\", data=train)\nstellar = crypto_df(\"Stellar\", data=train)\ntron = crypto_df(\"TRON\", data=train)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"binance_plot = candelstick_chart(binance[-120:], title=\"Binance Coin\")\nbinance_plot.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"btc_plot = candelstick_chart(btc[-120:], title=\"Bitcoin\")\nbtc_plot.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"btc_cash_plot = candelstick_chart(btc_cash[-120:], title=\"Bitcoin Cash\")\nbtc_cash_plot.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cardano_plot = candelstick_chart(cardano[-120:], title=\"Cardano\")\ncardano_plot.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dogecoin_plot = candelstick_chart(dogecoin[-120:], title=\"Dogecoin\")\ndogecoin_plot.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"eos_plot = candelstick_chart(eos[-120:], title=\"EOS.IO\")\neos_plot.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"eth_plot = candelstick_chart(eth[-120:], title=\"Ethereum\")\neth_plot.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"eth_classic_plot = candelstick_chart(eth_classic[-120:], title=\"Ethereum Classic\")\neth_classic_plot.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"iota_plot = candelstick_chart(iota[-120:], title=\"IOTA\")\niota_plot.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"litecoin_plot = candelstick_chart(litecoin[-120:], title=\"Litecoin\")\nlitecoin_plot.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"maker_plot = candelstick_chart(maker[-120:], title=\"Maker\")\nmaker_plot.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"monero_plot = candelstick_chart(monero[-120:], title=\"Monero\")\nmonero_plot.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"stellar_plot = candelstick_chart(stellar[-120:], title=\"Stellar\")\nstellar_plot.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tron_plot = candelstick_chart(tron[-120:], title=\"TRON\")\ntron_plot.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 2.2 Volumes","metadata":{}},{"cell_type":"code","source":"vol_traded(binance[-60:], \"Binance Coin\", color=\"Orange\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vol_traded(btc[-60:], \"Bitcoin\", color=\"Purple\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vol_traded(btc_cash[-60:], \"Bitcoin Cash\", color=\"Orange\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vol_traded(cardano[-60:], \"Cardano\", color=\"Purple\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vol_traded(dogecoin[-60:], \"Dogecoin\", color=\"Orange\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vol_traded(eos[-60:], \"EOS.IO\", color=\"Purple\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vol_traded(eth[-60:], \"Ethereum\", color=\"Orange\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vol_traded(eth_classic[-60:], \"Ethereum Classic\", color=\"Purple\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vol_traded(iota[-60:], \"IOTA\", color=\"Orange\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vol_traded(litecoin[-60:], \"Litecoin\", color=\"Purple\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vol_traded(maker[-60:], \"Maker\", color=\"Orange\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vol_traded(monero[-60:], \"Monero\", color=\"Purple\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vol_traded(stellar[-60:], \"Stellar\", color=\"Orange\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vol_traded(tron[-60:], \"TRON\", color=\"Purple\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del binance_plot, btc_plot, btc_cash_plot,\\\n    cardano_plot, dogecoin_plot, eos_plot,\\\n    eth_plot, eth_classic_plot, iota_plot,\\\n    litecoin_plot, maker_plot, monero_plot,\\\n    stellar_plot, tron_plot\ngc.collect()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 2.3 Central Tendency","metadata":{}},{"cell_type":"code","source":"train.groupby(\"asset_name\").mean().head(14)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.groupby(\"asset_name\").std().head(14)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 2.4 Correlations","metadata":{}},{"cell_type":"code","source":"df_close = train.copy()\ndf_close = df_close[[\"timestamp\", \"asset_name\", \"close\"]]\ndf_close[\"timestamp\"] = df_close[\"timestamp\"].apply(\n    lambda x: datetime.fromtimestamp(x))\ndf_close[\"date\"] = df_close[\"timestamp\"].astype('datetime64[s]')\ndf_close = df_close.set_index(\"date\")\ndf_close = df_close.drop(\"timestamp\", axis=1)\ndf_close = df_close[[\"asset_name\", \"close\"]]\ndf_close[\"close\"] = df_close[\"close\"].apply(lambda x: np.log(x))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del train\ngc.collect()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_close_zoom = df_close.loc['01/06/2020':'01/06/2021']\ndf_close_zoom = df_close_zoom.reset_index()\ndf_close_zoom = pd.pivot_table(\n    df_close_zoom, values=\"close\", columns=\"asset_name\", index=\"date\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(15, 15))\nsns.heatmap(df_close_zoom.diff().corr(),\n            linewidths=.5,\n            annot=True,\n            square=True,\n            cmap=\"viridis\")\nplt.xlabel(\"Asset Name\")\nplt.ylabel(\"Asset Name\")\nplt.title(\"Correlation matrix\")\nplt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del df_close,\\\n    df_close_zoom\ngc.collect()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 3. Feature Engineering","metadata":{}},{"cell_type":"code","source":"dataset = [binance, btc, btc_cash,\n           cardano, dogecoin, eos,\n           eth, eth_classic, iota,\n           litecoin, maker, monero,\n           stellar, tron]\n\n\ncols = ['open', 'close', \"high\", \"low\"]\n\nfor i in dataset:\n    for j in cols:\n        i[\"log_\"+j] = i[j].apply(lambda x: np.log(x))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have created 6 new features : \n   - Stock High minus Low price (H-L)\n   - Stock Close minus Open price (O-C)\n   - Stock price’s seven days’ moving average (7 DAYS MA)\n   - Stock price’s fourteen days’ moving average (14 DAYS MA)\n   - Stock price’s twenty one days’ moving average (21 DAYS MA)\n   - Stock price’s standard deviation for the past seven days (7 DAYS STD DEV) \n","metadata":{}},{"cell_type":"code","source":"for i in dataset:\n    i[\"H-L\"] = i[\"log_high\"] - i[\"log_low\"]\n    i[\"O-C\"] = i[\"log_open\"] - i[\"log_close\"]\n    i[\"MA_7d\"] = i[\"log_close\"].rolling(10080).mean() # 7 days = 10080 min.\n    i[\"MA_14d\"] = i[\"log_close\"].rolling(20160).mean() # 14 days = 20160 min.\n    i[\"MA_21d\"] = i[\"log_close\"].rolling(30240).mean() # 21 days = 30240 min.\n    i[\"STD_7d\"] = i[\"log_close\"].rolling(10080).std()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dataset","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Resampling minute data to hour\n\nbinance = binance.reset_index()\nbinance = binance.set_index('timestamp').resample('60min').mean()\nbinance[\"asset_name\"] = \"Binance Coin\"\n\nbtc = btc.reset_index()\nbtc = btc.set_index('timestamp').resample('60min').mean()\nbtc[\"asset_name\"] = \"Bitcoin\"\n\nbtc_cash = btc_cash.reset_index()\nbtc_cash = btc_cash.set_index('timestamp').resample('60min').mean()\nbtc_cash[\"asset_name\"] = \"Bitcoin Cash\"\n\ncardano = cardano.reset_index()\ncardano = cardano.set_index('timestamp').resample('60min').mean()\ncardano[\"asset_name\"] = \"Cardano\"\n\ndogecoin = dogecoin.reset_index()\ndogecoin = dogecoin.set_index('timestamp').resample('60min').mean()\ndogecoin[\"asset_name\"] = \"Dogecoin\"\n\neos = eos.reset_index()\neos = eos.set_index('timestamp').resample('60min').mean()\neos[\"asset_name\"] = \"EOS.IO\"\n\neth = eth.reset_index()\neth = eth.set_index('timestamp').resample('60min').mean()\neth[\"asset_name\"] = \"Ethereum\"\n\neth_classic = eth_classic.reset_index()\neth_classic = eth_classic.set_index('timestamp').resample('60min').mean()\neth_classic[\"asset_name\"] = \"Ethereum Classic\"\n\niota = iota.reset_index()\niota = iota.set_index('timestamp').resample('60min').mean()\niota[\"asset_name\"] = \"IOTA\"\n\nlitecoin = litecoin.reset_index()\nlitecoin = litecoin.set_index('timestamp').resample('60min').mean()\nlitecoin[\"asset_name\"] = \"Litecoin\"\n\nmaker = maker.reset_index()\nmaker = maker.set_index('timestamp').resample('60min').mean()\nmaker[\"asset_name\"] = \"Maker\"\n\nmonero = monero.reset_index()\nmonero = monero.set_index('timestamp').resample('60min').mean()\nmonero[\"asset_name\"] = \"Monero\"\n\nstellar = stellar.reset_index()\nstellar = stellar.set_index('timestamp').resample('60min').mean()\nstellar[\"asset_name\"] = \"Stellar\"\n\ntron = tron.reset_index()\ntron = tron.set_index('timestamp').resample('60min').mean()\ntron[\"asset_name\"] = \"TRON\"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"binance.head(5)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"binance.to_parquet(DATA_PATH+\"prep/binance.parquet.gzip\",\n                   compression=\"gzip\",\n                   index=True)\n\nbtc.to_parquet(DATA_PATH+\"prep/bitcoin.parquet.gzip\",\n               compression=\"gzip\",\n               index=True)\n\nbtc_cash.to_parquet(DATA_PATH+\"prep/bitcoin_cash.parquet.gzip\",\n                    compression=\"gzip\",\n                    index=True)\n\ncardano.to_parquet(DATA_PATH+\"prep/cardano.parquet.gzip\",\n                   compression=\"gzip\",\n                   index=True)\n\ndogecoin.to_parquet(DATA_PATH+\"prep/dogecoin.parquet.gzip\",\n                    compression=\"gzip\",\n                    index=True)\n\neos.to_parquet(DATA_PATH+\"prep/EOS.parquet.gzip\",\n               compression=\"gzip\",\n               index=True)\n\neth.to_parquet(DATA_PATH+\"prep/Ethereum.parquet.gzip\",\n               compression=\"gzip\",\n               index=True)\n\neth_classic.to_parquet(DATA_PATH+\"prep/Ethereum_classic.parquet.gzip\",\n                       compression=\"gzip\",\n                       index=True)\n\niota.to_parquet(DATA_PATH+\"prep/IOTA.parquet.gzip\",\n                compression=\"gzip\",\n                index=True)\n\nlitecoin.to_parquet(DATA_PATH+\"prep/Litecoin.parquet.gzip\",\n                    compression=\"gzip\",\n                    index=True)\n\nmaker.to_parquet(DATA_PATH+\"prep/Maker.parquet.gzip\",\n                 compression=\"gzip\",\n                 index=True)\n\nmonero.to_parquet(DATA_PATH+\"prep/Monero.parquet.gzip\",\n                  compression=\"gzip\",\n                  index=True)\n\nstellar.to_parquet(DATA_PATH+\"prep/Stellar.parquet.gzip\",\n                   compression=\"gzip\",\n                   index=True)\n\ntron.to_parquet(DATA_PATH+\"prep/TRON.parquet.gzip\",\n                compression=\"gzip\",\n                index=True)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}