{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Loading external data with yfinance, exploration with PCA / T-SNE / UMAP\n\nIn this notebook I Load and explore external data. \n\nIt was initially built for the Optiver Volatility Forecasting Competition, following the discussion in [The Leak](https://www.kaggle.com/c/optiver-realized-volatility-prediction/discussion/256725) discussion and reusing most of the embedding (PCA / T-sne / UMAP) code from : https://www.kaggle.com/stassl/exploring-time-id-relationships. The goal was to download and visualize embeddings based on historical daily prices of the top 100 sp500 stocks. My contribution were: Installation and usage of yfinance, Collecting and ploting data from a list of tickers, building a GARMAN-KLASS volatility estimator from OLHC data. Discussion with @stassl brought numerous updates: loading tickers from wikipedia, downloading data in bulk, downloading data at 1h intervalls. The idea was to discuss a possible use of time embeddings. I didn't figure how to use them myself but it was at the core of some of the current top results.\n\nRegarding the G-research competition, I just figured that most cryptocurrencies are on yahoo-finance nowadays. So I show how to use yfinance to download relevant crypto data. Those data might be usefull to:\n - extend the training data\n - use additional data for designing CV\n - use additional data for neutralisation\n\nFeel free to upvote the discussion and the notebooks if you find the content interesting.","metadata":{}},{"cell_type":"markdown","source":"# Packages installation and import","metadata":{}},{"cell_type":"code","source":"%%capture\n!pip install umap-learn[plot]\n!pip install yfinance","metadata":{"execution":{"iopub.status.busy":"2021-11-06T09:20:57.180077Z","iopub.execute_input":"2021-11-06T09:20:57.180822Z","iopub.status.idle":"2021-11-06T09:21:12.076267Z","shell.execute_reply.started":"2021-11-06T09:20:57.180782Z","shell.execute_reply":"2021-11-06T09:21:12.075029Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import yfinance as yf\nimport glob\nimport umap\nimport umap.plot\nimport pandas as pd\nimport numpy as np\nimport seaborn as sns\nimport matplotlib as mpl\nimport matplotlib.pyplot as plt\n\nfrom joblib import Parallel, delayed\nfrom sklearn.manifold import TSNE\nfrom sklearn.preprocessing import minmax_scale\nfrom sklearn.decomposition import PCA\n\n%config InlineBackend.figure_format = 'retina'","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-11-06T09:21:12.078296Z","iopub.execute_input":"2021-11-06T09:21:12.078690Z","iopub.status.idle":"2021-11-06T09:21:12.101925Z","shell.execute_reply.started":"2021-11-06T09:21:12.078647Z","shell.execute_reply":"2021-11-06T09:21:12.100630Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Stock info ","metadata":{}},{"cell_type":"code","source":"SPY = yf.Ticker(\"SPY\")\nSPY.info","metadata":{"execution":{"iopub.status.busy":"2021-11-06T09:21:12.104777Z","iopub.execute_input":"2021-11-06T09:21:12.105509Z","iopub.status.idle":"2021-11-06T09:21:15.643457Z","shell.execute_reply.started":"2021-11-06T09:21:12.105463Z","shell.execute_reply":"2021-11-06T09:21:15.642463Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Stock Values - Exemple","metadata":{}},{"cell_type":"code","source":"import datetime\n\n# get historical market data\nSPY_histo = SPY.history(start=\"2017-01-01\", end=\"2021-07-31\")\n\nSPY_histo.index\n\nplt.figure(figsize=(10,10))\nplt.plot(SPY_histo.index, SPY_histo['Close'])\nplt.xlabel(\"date\")\nplt.ylabel(\"$ price\")\nplt.title(\"Stock Price\")","metadata":{"execution":{"iopub.status.busy":"2021-11-06T09:21:15.645173Z","iopub.execute_input":"2021-11-06T09:21:15.645468Z","iopub.status.idle":"2021-11-06T09:21:16.406045Z","shell.execute_reply.started":"2021-11-06T09:21:15.645440Z","shell.execute_reply":"2021-11-06T09:21:16.404862Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Aggregate price\n\nLoading data in bulks.","metadata":{}},{"cell_type":"code","source":"tickers = pd.read_html('https://en.wikipedia.org/wiki/S%26P_100')[2].Symbol","metadata":{"execution":{"iopub.status.busy":"2021-11-06T09:21:16.407685Z","iopub.execute_input":"2021-11-06T09:21:16.408302Z","iopub.status.idle":"2021-11-06T09:21:16.951462Z","shell.execute_reply.started":"2021-11-06T09:21:16.408255Z","shell.execute_reply":"2021-11-06T09:21:16.950484Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_prices_all = yf.download(tickers.to_list(), start='2020-01-01', interval='1h')","metadata":{"execution":{"iopub.status.busy":"2021-11-06T09:21:16.953896Z","iopub.execute_input":"2021-11-06T09:21:16.954497Z","iopub.status.idle":"2021-11-06T09:21:34.942668Z","shell.execute_reply.started":"2021-11-06T09:21:16.954451Z","shell.execute_reply":"2021-11-06T09:21:34.941788Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_prices_all.head()","metadata":{"execution":{"iopub.status.busy":"2021-11-06T09:21:34.944052Z","iopub.execute_input":"2021-11-06T09:21:34.944534Z","iopub.status.idle":"2021-11-06T09:21:34.982005Z","shell.execute_reply.started":"2021-11-06T09:21:34.944484Z","shell.execute_reply":"2021-11-06T09:21:34.981129Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"o = df_prices_all.Open\nh = df_prices_all.High\nl = df_prices_all.Low\nc = df_prices_all.Close\n\n#GARMAN-KLASS rv estimator from OLHC data\nvol = 1/2 * np.square(np.log(h/l)) - (2*np.log(2)-1)*np.square(np.log(c/o))","metadata":{"execution":{"iopub.status.busy":"2021-11-06T09:21:34.983221Z","iopub.execute_input":"2021-11-06T09:21:34.983668Z","iopub.status.idle":"2021-11-06T09:21:35.027786Z","shell.execute_reply.started":"2021-11-06T09:21:34.983635Z","shell.execute_reply":"2021-11-06T09:21:35.026632Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_prices = df_prices_all['Adj Close']","metadata":{"execution":{"iopub.status.busy":"2021-11-06T09:21:35.030490Z","iopub.execute_input":"2021-11-06T09:21:35.030838Z","iopub.status.idle":"2021-11-06T09:21:35.041269Z","shell.execute_reply.started":"2021-11-06T09:21:35.030796Z","shell.execute_reply":"2021-11-06T09:21:35.040383Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.preprocessing import MinMaxScaler\nscaler = MinMaxScaler()\n\ncolnames = df_prices.columns\ndates = df_prices.index\n\ndf_prices = pd.DataFrame(scaler.fit_transform(df_prices),index = dates, columns=colnames)\n\ndf_prices = df_prices.fillna(df_prices.mean())\ndf_prices = df_prices.dropna(axis=1, how='any')","metadata":{"execution":{"iopub.status.busy":"2021-11-06T09:21:35.043491Z","iopub.execute_input":"2021-11-06T09:21:35.044205Z","iopub.status.idle":"2021-11-06T09:21:35.097056Z","shell.execute_reply.started":"2021-11-06T09:21:35.044160Z","shell.execute_reply":"2021-11-06T09:21:35.096284Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# market volatility","metadata":{}},{"cell_type":"code","source":"df_target = (vol/vol.mean()).mean(axis=1)","metadata":{"execution":{"iopub.status.busy":"2021-11-06T09:21:35.098430Z","iopub.execute_input":"2021-11-06T09:21:35.098759Z","iopub.status.idle":"2021-11-06T09:21:35.108948Z","shell.execute_reply.started":"2021-11-06T09:21:35.098727Z","shell.execute_reply":"2021-11-06T09:21:35.107880Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_target ","metadata":{"execution":{"iopub.status.busy":"2021-11-06T09:21:35.110573Z","iopub.execute_input":"2021-11-06T09:21:35.111018Z","iopub.status.idle":"2021-11-06T09:21:35.122409Z","shell.execute_reply.started":"2021-11-06T09:21:35.110974Z","shell.execute_reply":"2021-11-06T09:21:35.121632Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(df_target)","metadata":{"execution":{"iopub.status.busy":"2021-11-06T09:21:35.123775Z","iopub.execute_input":"2021-11-06T09:21:35.124365Z","iopub.status.idle":"2021-11-06T09:21:35.381958Z","shell.execute_reply.started":"2021-11-06T09:21:35.124319Z","shell.execute_reply":"2021-11-06T09:21:35.380938Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# PCA","metadata":{}},{"cell_type":"code","source":"%%time\nemb = PCA(n_components=2).fit_transform(df_prices)","metadata":{"execution":{"iopub.status.busy":"2021-11-06T09:21:35.383485Z","iopub.execute_input":"2021-11-06T09:21:35.383934Z","iopub.status.idle":"2021-11-06T09:21:35.448159Z","shell.execute_reply.started":"2021-11-06T09:21:35.383892Z","shell.execute_reply":"2021-11-06T09:21:35.447100Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(10, 8))\nplt.scatter(emb[:, 0], emb[:, 1], s=3, c=df_target, edgecolors='none', cmap='jet', norm=mpl.colors.LogNorm());\ncb = plt.colorbar(label='realized volatility', format=mpl.ticker.ScalarFormatter(),\n                  ticks=mpl.ticker.LogLocator(10))\ncb.ax.yaxis.set_minor_formatter(mpl.ticker.ScalarFormatter())\nplt.title('PCA time_id embeddings');","metadata":{"execution":{"iopub.status.busy":"2021-11-06T09:21:35.449771Z","iopub.execute_input":"2021-11-06T09:21:35.450396Z","iopub.status.idle":"2021-11-06T09:21:36.124924Z","shell.execute_reply.started":"2021-11-06T09:21:35.450350Z","shell.execute_reply":"2021-11-06T09:21:36.123847Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# TSNE","metadata":{}},{"cell_type":"code","source":"%%time\nemb = TSNE(n_components=2, perplexity=40, learning_rate=50, verbose=1, init='pca', n_iter=2000,\n           early_exaggeration=12).fit_transform(df_prices)","metadata":{"execution":{"iopub.status.busy":"2021-11-06T09:21:36.126363Z","iopub.execute_input":"2021-11-06T09:21:36.126716Z","iopub.status.idle":"2021-11-06T09:22:04.092528Z","shell.execute_reply.started":"2021-11-06T09:21:36.126682Z","shell.execute_reply":"2021-11-06T09:22:04.091646Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(10, 8))\nplt.scatter(emb[:, 0], emb[:, 1], s=3, c=df_target, edgecolors='none', cmap='jet', norm=mpl.colors.LogNorm());\ncb = plt.colorbar(label='realized volatility', format=mpl.ticker.ScalarFormatter(),\n                  ticks=mpl.ticker.LogLocator(10))\ncb.ax.yaxis.set_minor_formatter(mpl.ticker.ScalarFormatter())\nplt.title('TSNE time_id embeddings');","metadata":{"execution":{"iopub.status.busy":"2021-11-06T09:22:04.094101Z","iopub.execute_input":"2021-11-06T09:22:04.094694Z","iopub.status.idle":"2021-11-06T09:22:04.806412Z","shell.execute_reply.started":"2021-11-06T09:22:04.094654Z","shell.execute_reply":"2021-11-06T09:22:04.805598Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# UMAP","metadata":{}},{"cell_type":"code","source":"%%time\nemb = umap.UMAP(n_neighbors=60, min_dist=0.1, target_metric='euclidean', init='spectral', \n                low_memory=False, verbose=True, spread=0.5, local_connectivity=1, \n                repulsion_strength=1, negative_sample_rate=5).fit_transform(df_prices)","metadata":{"execution":{"iopub.status.busy":"2021-11-06T09:22:04.810033Z","iopub.execute_input":"2021-11-06T09:22:04.811904Z","iopub.status.idle":"2021-11-06T09:22:28.262102Z","shell.execute_reply.started":"2021-11-06T09:22:04.811861Z","shell.execute_reply":"2021-11-06T09:22:28.260781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(10, 8))\nplt.scatter(emb[:, 0], emb[:, 1], s=3, c=df_target, edgecolors='none', cmap='jet', norm=mpl.colors.LogNorm());\ncb = plt.colorbar(label='realized volatility', format=mpl.ticker.ScalarFormatter(),\n                  ticks=mpl.ticker.LogLocator(10))\ncb.ax.yaxis.set_minor_formatter(mpl.ticker.ScalarFormatter())\nplt.title('UMAP time_id embeddings');","metadata":{"execution":{"iopub.status.busy":"2021-11-06T09:22:28.263658Z","iopub.execute_input":"2021-11-06T09:22:28.263964Z","iopub.status.idle":"2021-11-06T09:22:28.828841Z","shell.execute_reply.started":"2021-11-06T09:22:28.263935Z","shell.execute_reply":"2021-11-06T09:22:28.827841Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# High volatility days","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(10, 8))\nplt.scatter(emb[:, 0], emb[:, 1], s=3, c=df_target>df_target.quantile(0.9), edgecolors='none', cmap='jet', norm=mpl.colors.LogNorm());\ncb = plt.colorbar(label='realized volatility', format=mpl.ticker.ScalarFormatter(),\n                  ticks=mpl.ticker.LogLocator(10))\ncb.ax.yaxis.set_minor_formatter(mpl.ticker.ScalarFormatter())\nplt.title('UMAP time_id embeddings');","metadata":{"execution":{"iopub.status.busy":"2021-11-06T09:22:28.830321Z","iopub.execute_input":"2021-11-06T09:22:28.830830Z","iopub.status.idle":"2021-11-06T09:22:29.250555Z","shell.execute_reply.started":"2021-11-06T09:22:28.830793Z","shell.execute_reply":"2021-11-06T09:22:29.249527Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Plotting times","metadata":{}},{"cell_type":"code","source":"import matplotlib.dates as mdates\n\nplt.figure(figsize=(10, 8))\nplt.scatter(emb[:, 0], emb[:, 1], s=3, c=[mdates.date2num(i) for i in df_prices.index], edgecolors='none', cmap='jet');\n\ncb = plt.colorbar(label='date', format=mdates.AutoDateFormatter(mdates.MonthLocator(interval=6)))\ncb.ax.yaxis.set_minor_formatter(mpl.ticker.ScalarFormatter())\n\nplt.title('UMAP time_id embeddings');","metadata":{"execution":{"iopub.status.busy":"2021-11-06T09:22:29.251797Z","iopub.execute_input":"2021-11-06T09:22:29.252075Z","iopub.status.idle":"2021-11-06T09:22:29.847100Z","shell.execute_reply.started":"2021-11-06T09:22:29.252048Z","shell.execute_reply":"2021-11-06T09:22:29.846071Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Optiver - Stock 31 \nStock 31 has been problematic from the beggining of the optiver competition. From deanonimizing stock price it appeared to be GE.","metadata":{}},{"cell_type":"code","source":"GE = yf.Ticker(\"GE\")\nGE.info","metadata":{"execution":{"iopub.status.busy":"2021-11-06T09:22:29.848535Z","iopub.execute_input":"2021-11-06T09:22:29.848846Z","iopub.status.idle":"2021-11-06T09:22:33.552925Z","shell.execute_reply.started":"2021-11-06T09:22:29.848815Z","shell.execute_reply":"2021-11-06T09:22:33.551691Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see there is : \n    \n'lastSplitDate': 1627862400,\n'lastSplitFactor': '1:8',","metadata":{}},{"cell_type":"code","source":"date = datetime.datetime.fromtimestamp(1627862400)\nprint(\"d2 =\", date.strftime(\"%B %d, %Y\"))","metadata":{"execution":{"iopub.status.busy":"2021-11-06T09:22:33.554597Z","iopub.execute_input":"2021-11-06T09:22:33.555020Z","iopub.status.idle":"2021-11-06T09:22:33.561587Z","shell.execute_reply.started":"2021-11-06T09:22:33.554978Z","shell.execute_reply":"2021-11-06T09:22:33.560221Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"GE splitted between training and testing set. Which is a supplementary problem. ","metadata":{}},{"cell_type":"code","source":"# get historical market data\nGE_histo = GE.history(start=\"2017-01-01\", end=\"2021-08-31\")\n\nGE_histo.index\n\nplt.figure(figsize=(10,10))\nplt.plot(GE_histo.index, GE_histo['Close'])\nplt.xlabel(\"date\")\nplt.ylabel(\"$ price\")\nplt.title(\"Stock Price\")","metadata":{"execution":{"iopub.status.busy":"2021-11-06T09:22:33.562949Z","iopub.execute_input":"2021-11-06T09:22:33.563245Z","iopub.status.idle":"2021-11-06T09:22:34.387487Z","shell.execute_reply.started":"2021-11-06T09:22:33.563216Z","shell.execute_reply":"2021-11-06T09:22:34.386747Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"GE.calendar","metadata":{"execution":{"iopub.status.busy":"2021-11-06T09:22:34.388572Z","iopub.execute_input":"2021-11-06T09:22:34.388968Z","iopub.status.idle":"2021-11-06T09:22:34.399026Z","shell.execute_reply.started":"2021-11-06T09:22:34.388938Z","shell.execute_reply":"2021-11-06T09:22:34.398147Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"GE.actions","metadata":{"execution":{"iopub.status.busy":"2021-11-06T09:22:34.400495Z","iopub.execute_input":"2021-11-06T09:22:34.400813Z","iopub.status.idle":"2021-11-06T09:22:34.426069Z","shell.execute_reply.started":"2021-11-06T09:22:34.400783Z","shell.execute_reply":"2021-11-06T09:22:34.425293Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We see the last split that wrecked the LB. \nNext step: other stocks ?","metadata":{}},{"cell_type":"markdown","source":"# G-research CryptoCurrencies","metadata":{}},{"cell_type":"markdown","source":"It appears that crypto are now on yahoo finance. What a time to be alive.\n\nMissing for now :\n\nBinance Coin\nEOS.IO\nEthereum Classic\n","metadata":{}},{"cell_type":"code","source":"list_crypto = ['BCH-USD',\n'BTC-USD',\n'ETH-USD',\n'LTC-USD',\n'XMR-USD',\n'TRX-USD',\n'XLM-USD',\n'ADA-USD',\n'MIOTA-USD',\n'MKR-USD',\n'DOGE-USD']","metadata":{"execution":{"iopub.status.busy":"2021-11-06T09:22:34.429634Z","iopub.execute_input":"2021-11-06T09:22:34.430166Z","iopub.status.idle":"2021-11-06T09:22:34.434468Z","shell.execute_reply.started":"2021-11-06T09:22:34.430133Z","shell.execute_reply":"2021-11-06T09:22:34.433551Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# get crypto infos","metadata":{}},{"cell_type":"code","source":"BTC = yf.Ticker(\"BTC-USD\")\nBTC.info","metadata":{"execution":{"iopub.status.busy":"2021-11-06T09:22:34.436010Z","iopub.execute_input":"2021-11-06T09:22:34.436572Z","iopub.status.idle":"2021-11-06T09:22:37.932130Z","shell.execute_reply.started":"2021-11-06T09:22:34.436511Z","shell.execute_reply":"2021-11-06T09:22:37.930850Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# get prices in bulk","metadata":{}},{"cell_type":"code","source":"df_prices_crypto = yf.download(list_crypto, start=\"2021-11-04\", end=\"2021-11-06\", interval='1m')","metadata":{"execution":{"iopub.status.busy":"2021-11-06T09:22:37.934008Z","iopub.execute_input":"2021-11-06T09:22:37.934441Z","iopub.status.idle":"2021-11-06T09:22:40.034745Z","shell.execute_reply.started":"2021-11-06T09:22:37.934398Z","shell.execute_reply":"2021-11-06T09:22:40.033359Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_prices_crypto","metadata":{"execution":{"iopub.status.busy":"2021-11-06T09:22:40.036510Z","iopub.execute_input":"2021-11-06T09:22:40.036967Z","iopub.status.idle":"2021-11-06T09:22:40.093784Z","shell.execute_reply.started":"2021-11-06T09:22:40.036923Z","shell.execute_reply":"2021-11-06T09:22:40.092657Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# standardised prices","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(10,10))\nplt.plot(df_prices_crypto.index, df_prices_crypto['Close']/df_prices_crypto['Close'].iloc[0])\nplt.xlabel(\"date\")\nplt.ylabel(\"$ price\")\nplt.title(\"Stock Price\")","metadata":{"execution":{"iopub.status.busy":"2021-11-06T09:22:40.095165Z","iopub.execute_input":"2021-11-06T09:22:40.095466Z","iopub.status.idle":"2021-11-06T09:22:41.055089Z","shell.execute_reply.started":"2021-11-06T09:22:40.095437Z","shell.execute_reply":"2021-11-06T09:22:41.054299Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# log returns","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(10,10))\nplt.plot(df_prices_crypto.index, np.log(df_prices_crypto['Close']/df_prices_crypto['Close'].shift()))\nplt.xlabel(\"date\")\nplt.ylabel(\"$ price\")\nplt.title(\"Stock Price\")","metadata":{"execution":{"iopub.status.busy":"2021-11-06T09:22:41.056152Z","iopub.execute_input":"2021-11-06T09:22:41.056416Z","iopub.status.idle":"2021-11-06T09:22:42.156233Z","shell.execute_reply.started":"2021-11-06T09:22:41.056389Z","shell.execute_reply":"2021-11-06T09:22:42.155160Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# crypto exploration with UMAP","metadata":{}},{"cell_type":"code","source":"scaler = MinMaxScaler()\n\ndf_prices_crypto = df_prices_crypto['Adj Close']\n\ncolnames = df_prices_crypto.columns\ndates = df_prices_crypto.index\n\ndf_prices_crypto = pd.DataFrame(scaler.fit_transform(df_prices_crypto),index = dates, columns=colnames)\n\ndf_prices_crypto = df_prices_crypto.fillna(df_prices_crypto.mean())\ndf_prices_crypto = df_prices_crypto.dropna(axis=1, how='any')\n\nemb = umap.UMAP(n_neighbors=60, min_dist=0.1, target_metric='euclidean', init='spectral', \n                low_memory=False, verbose=True, spread=0.5, local_connectivity=1, \n                repulsion_strength=1, negative_sample_rate=5).fit_transform(df_prices_crypto)\n\nplt.figure(figsize=(10, 8))\nplt.scatter(emb[:, 0], emb[:, 1], s=3, c=df_prices_crypto.mean(axis=1), edgecolors='none', cmap='jet', norm=mpl.colors.LogNorm());\ncb = plt.colorbar(label='avg standardized price', format=mpl.ticker.ScalarFormatter(),\n                  ticks=mpl.ticker.LogLocator(10))\ncb.ax.yaxis.set_minor_formatter(mpl.ticker.ScalarFormatter())\nplt.title('UMAP time_id embeddings');","metadata":{"execution":{"iopub.status.busy":"2021-11-06T09:22:43.134241Z","iopub.execute_input":"2021-11-06T09:22:43.134790Z","iopub.status.idle":"2021-11-06T09:23:02.100432Z","shell.execute_reply.started":"2021-11-06T09:22:43.134729Z","shell.execute_reply":"2021-11-06T09:23:02.099203Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(10, 8))\nplt.scatter(emb[:, 0], emb[:, 1], s=3, c=dates, edgecolors='none', cmap='jet', norm=mpl.colors.LogNorm());\ncb = plt.colorbar(label='time', format=mpl.ticker.ScalarFormatter())\ncb.ax.yaxis.set_minor_formatter(mpl.ticker.ScalarFormatter())\nplt.title('UMAP time_id embeddings');","metadata":{"execution":{"iopub.status.busy":"2021-11-06T09:24:55.203669Z","iopub.execute_input":"2021-11-06T09:24:55.204234Z","iopub.status.idle":"2021-11-06T09:24:55.641987Z","shell.execute_reply.started":"2021-11-06T09:24:55.204199Z","shell.execute_reply":"2021-11-06T09:24:55.640782Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scaler = MinMaxScaler()\n\ndf_ret_crypto = np.log(df_prices_crypto/df_prices_crypto.shift())\ndf_ret_crypto = df_ret_crypto[~np.isinf(df_ret_crypto).any(axis=1)]\ndf_ret_crypto = df_ret_crypto.fillna(df_ret_crypto.mean(skipna=True))\n\ncolnames = df_ret_crypto.columns\ndates = df_ret_crypto.index\n\ndf_ret_crypto = pd.DataFrame(scaler.fit_transform(df_ret_crypto),index = dates, columns=colnames)\n\nemb = umap.UMAP(n_neighbors=60, min_dist=0.1, target_metric='euclidean', init='spectral', \n                low_memory=False, verbose=True, spread=0.5, local_connectivity=1, \n                repulsion_strength=1, negative_sample_rate=5).fit_transform(df_ret_crypto)\n\nplt.figure(figsize=(10, 8))\nplt.scatter(emb[:, 0], emb[:, 1], s=3, c=df_ret_crypto.mean(axis=1), edgecolors='none', cmap='jet', norm=mpl.colors.LogNorm());\ncb = plt.colorbar(label='avg standardized returns', format=mpl.ticker.ScalarFormatter(),\n                  ticks=mpl.ticker.LogLocator(10))\ncb.ax.yaxis.set_minor_formatter(mpl.ticker.ScalarFormatter())\nplt.title('UMAP time_id embeddings');","metadata":{"execution":{"iopub.status.busy":"2021-11-06T09:23:02.557895Z","iopub.execute_input":"2021-11-06T09:23:02.558191Z","iopub.status.idle":"2021-11-06T09:23:23.571178Z","shell.execute_reply.started":"2021-11-06T09:23:02.558160Z","shell.execute_reply":"2021-11-06T09:23:23.569877Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(10, 8))\nplt.scatter(emb[:, 0], emb[:, 1], s=3, c=dates, edgecolors='none', cmap='jet', norm=mpl.colors.LogNorm());\ncb = plt.colorbar(label='time', format=mpl.ticker.ScalarFormatter())\ncb.ax.yaxis.set_minor_formatter(mpl.ticker.ScalarFormatter())\nplt.title('UMAP time_id embeddings');","metadata":{"execution":{"iopub.status.busy":"2021-11-06T09:24:29.017234Z","iopub.execute_input":"2021-11-06T09:24:29.017691Z","iopub.status.idle":"2021-11-06T09:24:29.456638Z","shell.execute_reply.started":"2021-11-06T09:24:29.017656Z","shell.execute_reply":"2021-11-06T09:24:29.455048Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}