{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Data Visualization and Crypto Forecasting\n# Taha Habib\n**Crypto Forecasting**","metadata":{}},{"cell_type":"code","source":"#library\nimport os\nfrom datetime import datetime\nimport time\nfrom sklearn.preprocessing import StandardScaler\nimport plotly.express as px\nimport plotly.graph_objs as go\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport math\n\nimport statsmodels.api as sm\nfrom statsmodels.tsa.ar_model import AutoReg\nfrom statsmodels.tsa.arima_model import ARMA\nfrom statsmodels.tsa.stattools import adfuller\nfrom statsmodels.graphics.tsaplots import plot_acf\nfrom statsmodels.graphics.tsaplots import plot_pacf\nfrom statsmodels.tsa.arima_model import ARIMA\nfrom statsmodels.tsa.statespace.sarimax import SARIMAX\nfrom statsmodels.tsa.seasonal import seasonal_decompose\nfrom scipy import stats\nfrom itertools import product\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\ninfo = pd.read_csv(\"/kaggle/input/g-research-crypto-forecasting/asset_details.csv\")\nctrain = pd.read_csv(\"/kaggle/input/g-research-crypto-forecasting/train.csv\")","metadata":{"execution":{"iopub.status.busy":"2023-01-04T13:22:50.442434Z","iopub.execute_input":"2023-01-04T13:22:50.442839Z","iopub.status.idle":"2023-01-04T13:23:33.539443Z","shell.execute_reply.started":"2023-01-04T13:22:50.442806Z","shell.execute_reply":"2023-01-04T13:23:33.538314Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.color_palette(\"YlOrRd\", 10)","metadata":{"execution":{"iopub.status.busy":"2023-01-04T13:23:40.351274Z","iopub.execute_input":"2023-01-04T13:23:40.352183Z","iopub.status.idle":"2023-01-04T13:23:40.366353Z","shell.execute_reply.started":"2023-01-04T13:23:40.352143Z","shell.execute_reply":"2023-01-04T13:23:40.365171Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ctrain.head()","metadata":{"execution":{"iopub.status.busy":"2023-01-04T13:23:43.510079Z","iopub.execute_input":"2023-01-04T13:23:43.510567Z","iopub.status.idle":"2023-01-04T13:23:43.543559Z","shell.execute_reply.started":"2023-01-04T13:23:43.510531Z","shell.execute_reply":"2023-01-04T13:23:43.542111Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# What is Crypto?\nCrypto, a form of digital token or “coins” that exist on a distributed and decentralized ledger called a blockchain, is a popular investment nowadays.\n\nA crypto is a collection of binary data which is designed to work as a medium of exchange. Individual coin ownership records are stored in a ledger, which is a computerized database using strong cryptography to secure transaction records, to control the creation of additional coins, and to verify the transfer of coin ownership.\n\n***KEY idea: Decentralization, Blockchain**\n\nThis project consisite of three parts:\n\n1. Data visualtion for 14 popular cryptocurrency\n2. Price History for selected individual cryptocurrency\n3. Basic Arima Model for price prediction\n\n# Popular Crypto Coins:\n\n1. Bitcoin (BTC): launched in January 2009. It introduced a novel idea set out in a white paper by the mysterious Satoshi Nakamoto—bitcoin offers the promise of an online currency that is secured without any central authority, unlike government-issued currencies.\n\n2. Ethereum (ETH): The goal behind Ethereum is to create a decentralized suite of financial products that anyone in the world can freely access, regardless of nationality, ethnicity, or fait.\n\n3. Litecoin(LTC): known as “silver to Bitcoin’s gold.” It is Based on an open-source global payment network that is not controlled by any central authority and uses “scrypt” as a proof of work, which can be decoded with the help of consumer-grade CPUs.\n\n4. Cardano (ADA)：an “Ouroboros proof-of-stake” cryptocurrency that was created with a research-based approach by engineers, mathematicians, and cryptography experts.\n\n5. Bitcoin Cash (BCH): an important place in the history of altcoins because it is one of the earliest and most successful hard forks of the original Bitcoin\n\n6. Stella(XLM): an open blockchain network designed to provide enterprise solutions by connecting financial institutions for the purpose of large transactions.(Transcations between banks)\n\n7. Dogecoin(DOGE): as the original \"memecoin\" caused a stir in 2021 as the price of the coin skyrocketed. (Popular because of Elon Musk)\n\n8. Binance Coin(BNB): a utility cryptocurrency that operates as a payment method for the fees associated with trading on the Binance Exchange. It is the third-largest cryptocurrency by market capitalization. (initiated from ETH)\n\n9. Monero(XMR): a secure, private, and untraceable currency.The development of this cryptocurrency is completely donation-based and community-driven.\n\n10. TRON: Founded in 2017 by a Singapore non-profit organization, aims to host a global entertainment system for the cost-effective sharing of digital content.\n\n11. IOTA: a distributed ledger designed to record and execute transactions between machines and devices in the Internet of Things (IoT) ecosystem.\n\n12. EOS.IO: a blockchain-based, decentralized platform used to develop, host, and run business applications, or dApps.\n\n13. Ethererum Classic: an open-source, decentralized, blockchain-based distributed cryptocurrency platform that runs smart contracts. It is as a split version of ETH.","metadata":{}},{"cell_type":"code","source":"%matplotlib inline\ninfo_s=info.sort_values(\"Weight\")\nfig_index=px.bar(info_s,x=\"Asset_Name\" , y=\"Weight\", color=\"Weight\", title=\"Popular Cryptocurrency Weight Distribution\",\n                color_continuous_scale=px.colors.sequential.YlOrRd)\nfig_index.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-04T13:23:48.839618Z","iopub.execute_input":"2023-01-04T13:23:48.840094Z","iopub.status.idle":"2023-01-04T13:23:50.056850Z","shell.execute_reply.started":"2023-01-04T13:23:48.840049Z","shell.execute_reply":"2023-01-04T13:23:50.056009Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Cryptocurrency Log Return Correlation Plot for 2021\nFrom the Correlation plot we can see Bitcoin Cash is highly correlated with EOS.IO.\n\nBinance Coin is correlated with many crypto currencies, such as , Bitcoin, Bitcoin Cash, Cardano, Ethereum.\n\nWe can have a roughly conclude that Binance Coin, Bitcoin Cash prices have a strong relation with other cryptocurrencies. Whereas Dogecoin and Monero prices are quite independent. Ealier we mentioned Monero is based a donation, which makes sense it is not correlated so much with other coins.","metadata":{}},{"cell_type":"code","source":"# time frame selection: from datetime to timestamp\ntotimestamp= lambda s: np.int32(time.mktime(datetime.strptime(s,\"%d/%m/%Y\").timetuple()))\n\n# Log Return \ndef log_return(series, periods=1):\n    return np.log(series).diff(periods=periods)\n\nall2021=pd.DataFrame([])\nfor asset_id, asset_name in zip(info.Asset_ID, info.Asset_Name):\n    asset=ctrain[ctrain[\"Asset_ID\"]==asset_id].set_index(\"timestamp\")\n    asset=asset.loc[totimestamp(\"01/01/2021\"):totimestamp(\"21/09/2021\")]\n    asset=asset.reindex(range(asset.index[0],asset.index[-1]+60,60), method=\"pad\")\n    lret=log_return(asset.Close.fillna(0))[1:]\n    all2021=all2021.join(lret,rsuffix=asset_name,how=\"outer\")\n    \nplt.imshow(all2021.corr());\nplt.yticks(info.Asset_ID, info.Asset_Name.values)\nplt.xticks(info.Asset_ID, info.Asset_Name.values,rotation=\"vertical\");\nplt.colorbar(cmap=\"coolwarm\")","metadata":{"execution":{"iopub.status.busy":"2023-01-04T13:24:12.023717Z","iopub.execute_input":"2023-01-04T13:24:12.024178Z","iopub.status.idle":"2023-01-04T13:24:17.837463Z","shell.execute_reply.started":"2023-01-04T13:24:12.024139Z","shell.execute_reply":"2023-01-04T13:24:17.836161Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Closing price trend in 2021\nWe will use the Closing Price and Candlestick graph to capture the trends of the crypto currency.\n\nWe will impute the missing time value using the \"pad\" method and present the overall closing price for Bitcoin, Etherum, Carnado","metadata":{}},{"cell_type":"code","source":"# Impute Missing Time Value\ndef c_time_sub(asset_id, data=ctrain):\n    df=data[ctrain[\"Asset_ID\"]==asset_id].set_index(\"timestamp\")\n    df=df.reindex(range(df.index[0],df.index[-1]+60,60), method=\"pad\")\n    return df","metadata":{"execution":{"iopub.status.busy":"2023-01-04T13:24:24.768640Z","iopub.execute_input":"2023-01-04T13:24:24.769085Z","iopub.status.idle":"2023-01-04T13:24:24.775388Z","shell.execute_reply.started":"2023-01-04T13:24:24.769039Z","shell.execute_reply":"2023-01-04T13:24:24.774304Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#1. Subgroup Bitcoin\nbtc=c_time_sub(asset_id=1)\n(btc.index[1:] -btc.index[:-1]).value_counts().head()\nbtc.head()","metadata":{"execution":{"iopub.status.busy":"2023-01-04T13:24:27.122862Z","iopub.execute_input":"2023-01-04T13:24:27.123399Z","iopub.status.idle":"2023-01-04T13:24:27.909144Z","shell.execute_reply.started":"2023-01-04T13:24:27.123361Z","shell.execute_reply":"2023-01-04T13:24:27.907934Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#2. Subgroup Ethereum\neth=c_time_sub(asset_id=6)\n(btc.index[1:] -btc.index[:-1]).value_counts().head()\neth.head()","metadata":{"execution":{"iopub.status.busy":"2023-01-04T13:24:30.720597Z","iopub.execute_input":"2023-01-04T13:24:30.721071Z","iopub.status.idle":"2023-01-04T13:24:31.552788Z","shell.execute_reply.started":"2023-01-04T13:24:30.721030Z","shell.execute_reply":"2023-01-04T13:24:31.551492Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#3. Subgroup Cardano\nada=c_time_sub(asset_id=3)\n(ada.index[1:] -ada.index[:-1]).value_counts().head()\nada.head()","metadata":{"execution":{"iopub.status.busy":"2023-01-04T13:24:33.689744Z","iopub.execute_input":"2023-01-04T13:24:33.690190Z","iopub.status.idle":"2023-01-04T13:24:34.430014Z","shell.execute_reply.started":"2023-01-04T13:24:33.690151Z","shell.execute_reply":"2023-01-04T13:24:34.428801Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create time interval for 2021\ndef dur(start,end,data):\n    df=data.loc[totimestamp(start): totimestamp(end)]\n    return df\n\nbtc2021= dur(start=\"01/01/2021\", end=\"21/09/2021\", data=btc)\neth2021= dur(start=\"01/01/2021\", end=\"21/09/2021\", data=eth)\nada2021= dur(start=\"01/01/2021\", end=\"21/09/2021\", data=ada)","metadata":{"execution":{"iopub.status.busy":"2023-01-04T13:24:46.433823Z","iopub.execute_input":"2023-01-04T13:24:46.434301Z","iopub.status.idle":"2023-01-04T13:24:46.442885Z","shell.execute_reply.started":"2023-01-04T13:24:46.434264Z","shell.execute_reply":"2023-01-04T13:24:46.441562Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot the Closing Price for BTC, ETH, ADA\nf= plt.figure(figsize=(10,12))  \n\ndef gplot(no , data, price, label, ylabel, color):\n    ax=f.add_subplot(no)\n    plt.plot(data[price], label=label, color=color)\n    plt.legend()\n    plt.xlabel(\"Time\")\n    plt.ylabel(ylabel)\n    return plt\n\ngplot(no=311, data=btc, price=\"Close\" , label=\"BTC 2021 Overall Performance\", ylabel=\"BTC Closing Price\", color=\"#00ABB3\")\ngplot(no=312, data=eth, price=\"Close\" ,label=\"ETH 2021 Overall Performance\", ylabel=\"ETH Closing Price\", color=\"#3C4048\")\ngplot(no=313, data=ada, price=\"Close\" ,label=\"Cardano 2021 Overall Performance\", ylabel=\"ADA Closing Price\", color=\"#B2B2B2\")\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-04T13:24:55.822831Z","iopub.execute_input":"2023-01-04T13:24:55.823253Z","iopub.status.idle":"2023-01-04T13:25:03.033323Z","shell.execute_reply.started":"2023-01-04T13:24:55.823220Z","shell.execute_reply":"2023-01-04T13:25:03.032452Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Data Selection\ndef crypto_sub(asset_id ,data= ctrain ):\n    df = data[data[\"Asset_ID\"]==asset_id].reset_index(drop = True)\n    df['timestamp'] = pd.to_datetime(df['timestamp'], unit='s')\n    df = df.set_index('timestamp')\n    return df\n\n#Subgroup Bitcoin\nbtc1=crypto_sub(asset_id=1)\n\n#Subgroup Eth\neth1=crypto_sub(asset_id=6)\n\n#Subgroup Ada\nada1=crypto_sub(asset_id=3)","metadata":{"execution":{"iopub.status.busy":"2023-01-04T13:25:08.189588Z","iopub.execute_input":"2023-01-04T13:25:08.190024Z","iopub.status.idle":"2023-01-04T13:25:09.438705Z","shell.execute_reply.started":"2023-01-04T13:25:08.189986Z","shell.execute_reply":"2023-01-04T13:25:09.437592Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def c_chart(data,label):\n    candlestick = go.Figure(data = [go.Candlestick(x =data.index, \n                                               open = data[('Open')], \n                                               high = data[('High')], \n                                               low = data[('Low')], \n                                               close = data[('Close')])])\n    candlestick.update_xaxes(title_text = 'Time',\n                             rangeslider_visible = True)\n\n    candlestick.update_layout(\n    title = {\n        'text': '{:} Candelstick Chart'.format(label),\n        \"y\":0.8,\n        \"x\":0.5,\n        'xanchor': 'center',\n        'yanchor': 'top'})\n\n    candlestick.update_yaxes(title_text = 'Price in USD', ticksuffix = '$')\n    return candlestick\n\nbtc1.head()","metadata":{"execution":{"iopub.status.busy":"2023-01-04T13:25:14.552093Z","iopub.execute_input":"2023-01-04T13:25:14.552886Z","iopub.status.idle":"2023-01-04T13:25:14.576657Z","shell.execute_reply.started":"2023-01-04T13:25:14.552849Z","shell.execute_reply":"2023-01-04T13:25:14.575690Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"During the time interval 22:40 to 22:50, there is a series of downward red candle, which represent Oversold. Theoratically, the price would go up followed by a **Oversold** interval.\n\n**Oversold** means it's trading below what would be a typical or average trading range(sometimes the investors also call it as true value). Normally, oversold draw buyers in the market.","metadata":{}},{"cell_type":"code","source":"%matplotlib inline\nbtc_candle=c_chart(btc1[-90:], label=\"BTC Price\")\nbtc_candle.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-04T13:25:18.573012Z","iopub.execute_input":"2023-01-04T13:25:18.573827Z","iopub.status.idle":"2023-01-04T13:25:18.622011Z","shell.execute_reply.started":"2023-01-04T13:25:18.573787Z","shell.execute_reply":"2023-01-04T13:25:18.620734Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# ETH Candlestick\nWe randomly pick a time interval for the ETh price, we see an over sold around 1:50am, and an overbuy around 02:05 am.","metadata":{}},{"cell_type":"code","source":"%matplotlib inline\nada_candle=c_chart(eth1[-500:-400], label=\"ETH Price \")\nada_candle.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-04T13:25:27.111163Z","iopub.execute_input":"2023-01-04T13:25:27.111575Z","iopub.status.idle":"2023-01-04T13:25:27.138491Z","shell.execute_reply.started":"2023-01-04T13:25:27.111543Z","shell.execute_reply":"2023-01-04T13:25:27.137200Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"eth_candle=c_chart(eth1[100888:100988], label=\"ETH Price \")\neth_candle.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-04T13:25:34.539339Z","iopub.execute_input":"2023-01-04T13:25:34.539758Z","iopub.status.idle":"2023-01-04T13:25:34.564171Z","shell.execute_reply.started":"2023-01-04T13:25:34.539725Z","shell.execute_reply":"2023-01-04T13:25:34.562761Z"},"trusted":true},"execution_count":null,"outputs":[]}]}