{"cells":[{"metadata":{"_uuid":"573041e5ba337d9cdbc972473d1af4b69054e985","_cell_guid":"55901724-7c89-4d32-8ecf-247f37965ba2"},"cell_type":"markdown","source":"## Table of Content:\n"},{"metadata":{"_uuid":"feaf5f77881a33c7fb250a7675a52f03e6a2be03","_cell_guid":"d95099e0-7a39-40a8-bacf-2e4c3cfcfb95"},"cell_type":"markdown","source":"# **// Part 0 // First Kaggle Competition - 'Hello World'**\n\nThis is the first competition I enter and the first real-world problem that I am publishing on. I hope to learn a lot myself and why not help some of my fellow learners along the way. Here is the table of content:\n\n* [Problem Description](#Part1)\n* [Initial Analysis](#Part2)\n* [Further Analysis](#Part3)\n* [Building a new classification](#Part4)\n* [A few more points](#Part5)\n\n# **// Part 1 // Problem Description**<a class=\"anchor\" id=\"Part1\"></a>\n\nThe competition is called : **Corporacion Favorita Grocery Sales Forecasting**.\n\nThe task is to predict sales in the stores of an Ecuadorian supermarket chain so that they can avoid overstocking, which would reduce waste and loss, and minimize understocking, which induces opportunity cost and lower customer satisfaction. For the sake of both the buisiness and the environment, better predictions are highly desirable. This increased efficiency could also result in higher profits for the stakeholders and/or a better pricepoint for customers, depending on the choices made by the chain.\n\nThe given data is a table with the following variables/features: date, store id, item id, sales volume, promotion.\nWhereas store id and item id are integers, promotion is a boolean and sales volume is a float (integers for discrete items, float for volume/weight). We can see the data as N time series, one per (store, item) combination. Many of these time series are most likely correlated to each other and some sort of dimensional reduction will be most welcome here. \n\nThe company also offers some other data sets, such as a list of stores with their location, a time series of daily transactions per store, a list of holidays and events, a list of products by category, and the price of oil, of which a good chunk of the ecuadorian economy is allegedly tied to. These are additional tools to simplify and/or enhance the predictions, and some other external data could also be used in this regard. We will most likely do so in future kernels.\n\nThe primary task is therefore forecasting. For the sake of simplicity, and as it is our first dip in time series, this kernel will focus on the daily transactions data of a single store instead of the sales data for all stores.\n"},{"metadata":{"_uuid":"89210ac8934aeaeeaaa2b194439c712aded9e688","_cell_guid":"ff2fd268-839a-4483-8681-3c26914f7d9b"},"cell_type":"markdown","source":"# // Set-up\n\n  Kaggle allows for the data to be used from their server directly and has most libraries available, it is therefore a great place to work on your project instead of using you own machine. Combined with Jupyter, it becomes an extremely intuitive tool for exploratory work, doing things step-by-step and tuning our approach as we go. Moreover it allows readers to learn, fork the project and get their hands dirty. We are really looking forward to publishing!\n  \n  Let's then start by loading the libraries and methods we need:  "},{"metadata":{"_kg_hide-output":true,"_uuid":"54cda7088daf8d27c5627cb80f9bef15fa30f6e7","collapsed":true,"_cell_guid":"75fd2f9a-7297-4203-b8f7-1502f66e0c9e","trusted":false},"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\")\n\n# DATA MANIPULATION\nimport numpy as np # linear algebra\nimport random as rd # generating random numbers\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport datetime # manipulating date formats\nfrom operator import add # elementwise addition\n\n# VIZUALIZATION\nimport matplotlib.pyplot as plt # basic plotting\nimport seaborn # for prettier plots\nimport folium # plotting data on interactive maps\n\n# UNSUPERVISED LEARNING\nfrom sklearn.cluster import AgglomerativeClustering as AggClust # Hierarchical Clustering\nfrom scipy.cluster.hierarchy import ward,dendrogram # Hierarchical Clustering + Dendograms\n\n# TIME SERIES\nfrom statsmodels.tsa.arima_model import ARIMA\nfrom statsmodels.tsa.statespace.sarimax import SARIMAX\nfrom pandas.plotting import autocorrelation_plot\nfrom statsmodels.tsa.stattools import adfuller, acf, pacf,arma_order_select_ic","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0eb792941c43ec9306fa07c35a55611101b4e052","_cell_guid":"e15e0b3e-2e6f-4be8-a8c5-879f3228656c"},"cell_type":"markdown","source":"We can also import the data. First let us look at a simple version of the problem and see if it can guide us. One of the files is a record of the number of transactions per store per day. This can be seen as a simpler problem - predicting the traffic in a given store."},{"metadata":{"_uuid":"2d3b84468f9166e001f7ddf20700ac61030101ee","collapsed":true,"_cell_guid":"6eccd50b-c197-4fbb-8f21-dac64e7165d4","trusted":false},"cell_type":"code","source":"# Reading daily transfers per store\nsales=pd.read_csv('../input/transactions.csv')\n\n# Reading store list\nstores=pd.read_csv('../input/stores.csv')\nstores.type=stores.type.astype('category')\n\n# Adding information about the stores\nsales=pd.merge(sales,stores,how='left')\n\n# Reading the holiday and events schedule\nholidays=pd.read_csv('../input/holidays_events.csv')\n\n# Formatting the dates properly\nsales['date']=sales.date.apply(lambda x:datetime.datetime.strptime(x, '%Y-%m-%d'))\nholidays['date']=holidays.date.apply(lambda x:datetime.datetime.strptime(x, '%Y-%m-%d'))\n\n# Isolating events that do not correspond to holidays\nevents=holidays.loc[holidays.type=='Event']\nholidays=holidays.loc[holidays.type!='Event']\n\n# Extracting year, week and day\nsales['year'],sales['week'],sales['day']=list(zip(*sales.date.apply(lambda x: x.isocalendar())))\n\n# Creating a categorical variable showing weekends\nsales['dayoff']=[x in [6,7] for x in sales.day]\n\n# Adjuusting this variable to show all holidays\nfor (d,t,l,n) in zip(holidays.date,holidays.type,holidays.locale,holidays.locale_name):\n  if t!='Work Day':\n    if l=='National':\n      sales.loc[sales.date==d,'dayoff']=True\n    elif l=='Regional':\n      sales.loc[(sales.date==d)&(sales.state==n),'dayoff']=True\n    else:\n      sales.loc[(sales.date==d)&(sales.city==n),'dayoff']=True\n  else:\n    sales.loc[(sales.date==d),'dayoff']=False\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f6258c70c12aa2f48b482ed1abd5a96f708cd39a","_cell_guid":"9896392a-e519-40de-8979-9d383cbd492c"},"cell_type":"markdown","source":"# // Exploratory Analysis\n\nLet us start by seeing what our tables look like. We'll use head() and info() on the **sales** table:"},{"metadata":{"_kg_hide-input":true,"_uuid":"0d50f51e8c9a07edbdb89bac0ba7a8927cb257e9","collapsed":true,"_cell_guid":"1c6a9def-31c1-48ea-96bf-b44753df47e4","trusted":false},"cell_type":"code","source":"sales.info()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_uuid":"07fc58e6032faade715096ec74b15bd4769674e0","collapsed":true,"_cell_guid":"129726cb-2a42-4b05-bfb1-79a5b110e497","trusted":false},"cell_type":"code","source":"sales.head(20)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3cffaf345ed32fb7ec7d904ca37e59ebfa387879","_cell_guid":"e01a22ed-afa5-4525-97c8-49f83d8f1248"},"cell_type":"markdown","source":"### **Store #47**\nThe chain established itselft in Quito in 1952 ([Wikipedia](https://es.wikipedia.org/wiki/Corporaci%C3%B3n_Favorita)), so let's pick a shop in Quito as a starting point, as the brand is well established there. Let us pick #47 and plot the corresponding transactions time series. \nWith a well established store, we can predict that the time series will be almost stationary. High seasonality is expected too, as people consume more during celebration periods. Let's see what we get:"},{"metadata":{"_kg_hide-input":true,"_uuid":"ad0de98e86f562c47958c3d9e4f3e609e016aea7","collapsed":true,"_cell_guid":"e71aaaa4-27a3-4ff3-8a50-76577d059749","trusted":false},"cell_type":"code","source":"ts=sales.loc[sales['store_nbr']==47,['date','transactions']].set_index('date')\nts=ts.transactions.astype('float')\nplt.figure(figsize=(12,12))\nplt.title('Daily transactions in store #47')\nplt.xlabel('time')\nplt.ylabel('Number of transactions')\nplt.plot(ts);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e72886f2f0262e8b872b59ebc9a643bf76a6a108","_cell_guid":"5f8d67a5-ee54-4035-9893-fed92f7e47b8"},"cell_type":"markdown","source":"### ** Seasonality and Outliers **\nWe can easily identify a peak around Christmas, a very low volume day mid 2015 and a fairly stable behavior in between. The surge in volume around Christmas was expected, but what happened mid 2015? As it turns out it corresponds to a giant national protest in Ecuador, which will most likely impact other stores as well. [Wikipedia](https://en.wikipedia.org/wiki/2015_Ecuadorian_protests#24_June)\n\n  We can also see that outside of this holiday period, the transactions have quite a lot of variance, the volume oscillating between 3000 and 5000. Let's check the rolling mean and standard deviation with a window of a month::"},{"metadata":{"_kg_hide-input":true,"_uuid":"f9953c2056b9b6a292c47919f492874510866689","collapsed":true,"_cell_guid":"53fd7336-5259-4f3b-831e-9b5eb606f41e","trusted":false},"cell_type":"code","source":"plt.figure(figsize=(12,12))\nplt.plot(ts.rolling(window=30,center=False).mean(),label='Rolling Mean');\nplt.plot(ts.rolling(window=30,center=False).std(),label='Rolling sd');\nplt.legend();","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1a7ef912f8ff3742bc155428fcfb4eabfea09f3e","collapsed":true,"_cell_guid":"5644678e-5195-487c-bf19-41a7e9ae297e","trusted":false},"cell_type":"code","source":"def test_stationarity(timeseries):\n    \n    #Perform Dickey-Fuller test:\n    print('Results of Dickey-Fuller Test:')\n    dftest = adfuller(timeseries, autolag='AIC')\n    dfoutput = pd.Series(dftest[0:4], index=['Test Statistic','p-value','#Lags Used','Number of Observations Used'])\n    for key,value in dftest[4].items():\n        dfoutput['Critical Value (%s)'%key] = value\n    print (dfoutput)\n\ntest_stationarity(ts)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"59cbdb5b50503eb160b1b28f88a65b11a21eb5cc","_cell_guid":"3d210bdf-01a2-4839-a334-5d619107c40f"},"cell_type":"markdown","source":"The Dickey-Fuller test has for null-hypothesis that the time-series is not stationary. With a p-value of 0 up to the 9th decimal we can safely reject it and conclude that our time-series is indeed stationary. Let us then proceed with the forecasting."},{"metadata":{"_uuid":"38fc0f0573915646c65c1385fd6292a2d2daa357","_cell_guid":"d056da9d-817a-4241-89c6-5dff30265d65"},"cell_type":"markdown","source":"# **// Part 2 // Initial Analysis** <a class=\"anchor\" id=\"Part1\"></a>\n\n  We will start by using an ARIMA model on this time series and see how successful we are. ARIMA stands for AutoRegressive Integrated Moving Average. I based my work on [**this page**](https://www.analyticsvidhya.com/blog/2016/02/time-series-forecasting-codes-python/) and [**this page**](https://machinelearningmastery.com/arima-for-time-series-forecasting-with-python/), that I found quite helpful. Penn State also offers the content of its Applied Time Series Analysis [**online**](https://onlinecourses.science.psu.edu/stat510/node/33)."},{"metadata":{"_uuid":"8a1e3e9bde3cab3ad67de68f68062c7f5294d80e","_cell_guid":"31bba401-a5c6-442b-a8c0-295f005b1354"},"cell_type":"markdown","source":"The ARIMA models have three parameters, corresponging to its AR, I and MA components:\n\n- The number of autoregressive (AR) terms (p): AR terms are just lags of the dependent variable.\n- The number of differences (d): These are the number of non-seasonal differences.\n- The number of moving average (MA) terms (q): MA terms are lagged forecast errors in the prediction equation.\n\nWe will set d to zero and not use the Integrated part of the model, effectivefly using an ARMA model. Let's start with checking for the auto-correlation of our time series, so as to understand it better and gauge what the parameters of our model should be."},{"metadata":{"_uuid":"02606a3c60899563444970ccc290834dd91bcbed","collapsed":true,"_cell_guid":"0cf2b0d6-d342-4b42-a9bb-4b74326a56b8","trusted":false},"cell_type":"code","source":"plt.figure(figsize=(12,6))\nautocorrelation_plot(ts);\nplt.figure(figsize=(12,6))\nautocorrelation_plot(ts);\nplt.xlim(xmax=100);\nplt.figure(figsize=(12,6))\nautocorrelation_plot(ts);\nplt.xlim(xmax=10);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d85e458bf4d97d7196a98b4625c2ca9a39f51950","_cell_guid":"1501117b-aa51-41ea-982e-9817db457245"},"cell_type":"markdown","source":"The highest auto-correclation peaks are every ~365 day and oscillates with a frequence of about 7 days. This corresponds with our intuitive ideas about shopping - that it is a weekly chore. Even though ecuadorians are most likely paid monthly, there is no significant periodicity visible on this scale.\n\nFrom the last diagram it seems that the autocorrelation is significant (above the dashed line) for a period of maximum 2 days. All the references then suggest to use 2 for the p parameter. But the stats.model library contains a nifty tool for coefficient selection : arma_order_select_ic(). It performs a grid search with the p and q parameters. Let's see what it says for max parameters 10 and the [Baysian Information Criterion](https://en.wikipedia.org/wiki/Bayesian_information_criterion) for estimator:"},{"metadata":{"_kg_hide-output":false,"_uuid":"e792a7fb117e2ecb36ac744c57fb6d640e9d1eb5","collapsed":true,"_cell_guid":"2708583e-2237-491d-b00d-af4d2bf18cb9","trusted":false},"cell_type":"code","source":"\nresult = arma_order_select_ic(ts,max_ar=10, max_ma=10, ic=['aic','bic'], trend='c', fit_kw=dict(method='css',maxiter=500))\nprint('The bic prescribes these (p,q) parameters : {}'.format(result.bic_min_order))\nprint('The aic prescribes these (p,q) parameters : {}'.format(result.aic_min_order))\nplt.figure(figsize=(12,6))\nplt.subplot(1,2,1)\nplt.title('bic results')\nseaborn.heatmap(result.bic);\nplt.subplot(1,2,2)\nplt.title('aic results')\nseaborn.heatmap(result.aic);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a3a0a42aab9ebfde645b4d1ca187ee967b6d0689","_cell_guid":"4201db55-83da-4393-98c5-c238ac4eac7f"},"cell_type":"markdown","source":"With the initial parameters, we are getting many many convergence warnings, which may show that the model does not fit well the time series. It may be important to factor out seasonality first, which we shall do in a later part of the analysis. To reduce the amount of warnings, we raised the number of iterations to a maximum of 500, in case of a very slow converging rate. We also plotted heatmaps of the results for both the aic and bic, for comparison.\n\n  It is clear from the heatmap that we can afford to run the model with parameters (5,0,5)"},{"metadata":{"_uuid":"c139aea82db3078122b330e25d256b53372c53f6","collapsed":true,"_cell_guid":"707e9243-cad1-4496-bcb7-7fae65d2c402","trusted":false},"cell_type":"code","source":"pdq=(5,0,5)\nmodel = ARIMA(ts, order = pdq, freq='W')\nmodel_fit = model.fit(disp=False,method='css',maxiter=100)\n# plot residual errors\nresiduals = pd.DataFrame(model_fit.resid)\nfig,axes = plt.subplots(nrows=1, ncols=2,figsize=(12,6))\nresiduals.plot(ax=axes[0])\nresiduals.plot(kind='kde',ax=axes[1]);\nresiduals.describe().T","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3c082032c303e5764f1b37d87409465f0ac8ea64","_cell_guid":"3717928e-41a2-49b8-a9b6-ef319c35f31c"},"cell_type":"markdown","source":"We can check on the residuals that the model found, which will for sure contain our outlier of mid-2015. We can plot the residuals and their distribution. We can see that we are getting a Gaussian, centered at 0 with  as tandard deviation of about 380 transactions."},{"metadata":{"_uuid":"29c4aba5b05e1ed3697a02b9d5ba9b7415e30594","collapsed":true,"_cell_guid":"04d5db31-f67c-433f-9659-2aea33c86860","trusted":false},"cell_type":"code","source":"plt.figure(figsize=(12,6))\nplt.subplot\nplt.plot(ts);\nplt.plot(model_fit.fittedvalues,alpha=.7);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"07ac2cf3b9153142286efd8d87c7a26249caccd7","_cell_guid":"b9897597-c395-40cd-9e93-47d8b8455fd2"},"cell_type":"markdown","source":"That seems rather good. We can note that the outlier is fitted though so that is a sign we are overfitting. \nLet us carry on nonetheless and split the data into training and testing so as to see the quality of predictions it gives..\n"},{"metadata":{"_kg_hide-output":true,"_uuid":"07e8712fd2466eb8b90f106a91711bfc3d2e5869","collapsed":true,"_cell_guid":"507bb602-875f-4047-8096-9784896ef337","trusted":false},"cell_type":"code","source":"forecast_len=30\nsize = int(len(ts)-forecast_len)\ntrain, test = ts[0:size], ts[size:len(ts)]\nhistory = [x for x in train]\npredictions = list()\n\nprint('Starting the ARIMA predictions...')\nprint('\\n')\nfor t in range(len(test)):\n    model = ARIMA(history, order = pdq, freq='W');\n    model_fit = model.fit(disp=0);\n    output = model_fit.forecast()\n    yhat = output[0]\n    predictions.append(float(yhat))\n    obs = test[t]\n    history.append(obs)\nprint('Predictions finished.\\n')\n    \n\npredictions_series = pd.Series(predictions, index = test.index)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"09ce918d2fc53d36138a7c26b5a694f8f64f908c","collapsed":true,"_cell_guid":"8066de4b-825a-48b4-a6f4-5d32c3949754","trusted":false},"cell_type":"code","source":"plt.figure(figsize=(12,12))\nplt.title('Store 47 : Transactions')\nplt.xlabel('Date')\nplt.ylabel('Transactions')\nplt.plot(ts[-2*forecast_len:], 'o', label='observed');\nplt.plot(predictions_series, '-o',label='rolling one-step out-of-sample forecast');\nplt.legend(loc='upper right');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dc36efdb74d22f16391b586ccb5ebd5a2de3ea9a","_cell_guid":"a09158d7-11ff-411f-864b-21603ebae72f"},"cell_type":"markdown","source":"The results seem rather good. Below is a histogram of the difference in predictions to see how good a forecast we made. We see that 2/3 of errors are below 200 transactions, which is about a 5% error. Not bad for a frist try and a month long window."},{"metadata":{"_uuid":"a1091002078d5c8dc478a4abd835cd6dc9e81e45","collapsed":true,"_cell_guid":"d8f5efc9-5c44-4d38-aa42-e3ec27f58865","trusted":false},"cell_type":"code","source":"plt.figure(figsize=(12,12))\nx=abs(ts[-forecast_len:]-predictions_series)\nseaborn.distplot(x,norm_hist=False,rug=True,kde=False);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"919a5fbc34a9d6b37fb6847f924e7f3e499b91af","_cell_guid":"b1e9585f-a8eb-4f80-a194-98412c93cc67"},"cell_type":"markdown","source":"# **// Part 3 // Further Analysis** <a class=\"anchor\" id=\"Part3\"></a>\n\nWe'd like to improve our forecast and generalize it for all stores. Note that the forecast was rather long to run due to a lot of non-convergence in the model. We would like to fix that. But before diving in more complex statistical models, we may want to understand a little more the nature of the data we are manipulating and forecasting.\n\n### ** Two time Series ?**\n\nThe fact that holidays transactions volume is higher than the rest of the year is giving us a hint that this behavior may be more generalized. Afterall people have more time to shop when they don't work. With this data we have a solid basis to verify this assumption. Let's plot working days and days off with two different colors to see if there is a trend.\nAs you can see below, there is a solid horizontal trend line at ~4700 transactions for days off and at ~3400 transactions for working days, repeated year after year. This could be good news for our model. Indeed, being able to separate 'apples' from 'oranges' will remove a lot of the apparent randomness that is contained in the time series as it stands. We may want to build a proper clustering engine for our days before forecasting and split the time series so as to improve our predictive power."},{"metadata":{"_kg_hide-input":true,"_uuid":"8c392b66dc8a1d264c1d182995fb20c53f1f16ce","collapsed":true,"_cell_guid":"1bc76208-5e8d-4f5e-809f-4a957a3bede6","trusted":false},"cell_type":"code","source":"# The function takes a store number, an integer for the rolling means window and a bool\n# for whether or not to split the working days and off days\ndef plot_store_transactions(store_viz,n=30,split=False):\n    temp=sales.loc[sales.store_nbr==store_viz].set_index('date')\n    plt.figure(figsize=(12,6))\n    if split:\n        ax1=plt.subplot(1,2,1)\n        plt.scatter(temp.loc[~temp.dayoff].index,\n                    temp.loc[~temp.dayoff].transactions,label='working days')\n        plt.scatter(temp.loc[temp.dayoff].index,\n                    temp.loc[temp.dayoff].transactions,label='off days')\n        plt.legend()\n        plt.title('Daily transactions. Store {}, Type {}, Cluster {}'.format(store_viz,\n                                                                        list(stores.loc[stores.store_nbr==store_viz,'type'])[0],\n                                                                        list(stores.loc[stores.store_nbr==store_viz,'cluster'])[0])\n                 )\n        ax2=plt.subplot(1,2,2,sharey=ax1,sharex=ax1)\n        plt.plot(temp.loc[~temp.dayoff,'transactions'].rolling(window=n).mean(),label='working days')\n        plt.plot(temp.loc[temp.dayoff,'transactions'].rolling(window=n).mean(),label='off days')\n        plt.legend()\n        plt.title('Store {}: {} day rolling means'.format(store_viz,n))\n        plt.setp(ax2.get_yticklabels(), visible=False)\n    else:\n        ax1=plt.subplot(1,2,1)\n        plt.scatter(temp.index,temp.transactions)\n        plt.title('Daily transactions. Store {}, Type {}, Cluster {}'.format(store_viz,\n                                                                        list(stores.loc[stores.store_nbr==store_viz,'type'])[0],\n                                                                        list(stores.loc[stores.store_nbr==store_viz,'cluster'])[0])\n                 )\n        ax2=plt.subplot(1,2,2,sharey=ax1)\n        plt.plot(temp.transactions.rolling(window=n).mean())\n        plt.title('Store {}: {} day rolling means'.format(store_viz,n))\n        plt.setp(ax2.get_yticklabels(), visible=False)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false,"_uuid":"40df40bb95e0a1c9ae8f75aca1d15f9f8d84bebe","collapsed":true,"_cell_guid":"a3df06c6-1257-4359-98ea-abb5fadf08d9","trusted":false},"cell_type":"code","source":"plot_store_transactions(47,30,True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e439691669a9c32f5444fcedddb1b2a31ec573b9","_cell_guid":"24157c70-5890-46ba-9232-bcd475585a9a"},"cell_type":"markdown","source":"### ** Is 'type' a good classifier? **\nThe plot for store #47 did confirm our assumption, but it would be a mistake to assume that it is therefore true for all stores.\nThis is the biggest chain in the country and there are therefore many stores to analyse, 54 exactly in this dataset. Are they all behaving like store #47? The company provided two different classifications of their stores without any indications of what they correspond to: type and cluster. Would they by chance correspond to patterns in the transaction volume time series? \n\nLet's plot them grouped by type to see if a pattern emerges or if we ought to seek our own classification. The following function creates a grid of plots for all stores of a given type."},{"metadata":{"_kg_hide-input":false,"_uuid":"0fae2c6291f0059b88a235cb37a5514a77001d19","collapsed":true,"_cell_guid":"817de8b1-452d-4a79-8011-6f6be9810dc5","trusted":false},"cell_type":"code","source":"def plot_store_transactions_type(typ):\n    typ_stores=stores.loc[stores.type==typ,'store_nbr']\n    n=len(typ_stores)\n    m=1\n    for x in range(1,6):\n        if (n-1) in range((x-1)**2,x**2):\n            m=x\n    plt.figure(figsize=(15,15))\n    for x in range(n):\n        nbr=typ_stores.iloc[x]\n        ax1 = plt.subplot(m,m,x+1)\n        ax1.scatter(sales.loc[(~sales.dayoff)&(sales.store_nbr==nbr),'date'].values,\n                sales.loc[(~sales.dayoff)&(sales.store_nbr==nbr),'transactions'])\n        ax1.scatter(sales.loc[(sales.dayoff)&(sales.store_nbr==nbr),'date'].values,\n                sales.loc[(sales.dayoff)&(sales.store_nbr==nbr),'transactions'])\n        plt.title('Store {}, Type {}, Cluster {}'.format(nbr,\n                                                         list(stores.loc[stores.store_nbr==nbr,'type'])[0],\n                                                         list(stores.loc[stores.store_nbr==nbr,'cluster'])[0])\n             )\n        plt.suptitle(' Type {} stores'.format(typ),fontsize=25)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_uuid":"f63bb3a52dfb6748fe17706147d66012835defbd","collapsed":true,"_cell_guid":"15e916aa-2ced-4c89-9704-a47f0456478f","trusted":false},"cell_type":"code","source":"plot_store_transactions_type('A')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"40052fa78ae0a413f95215f14eafea4d90f13262","_cell_guid":"22a7bc76-8151-4d04-81cd-2a442a65d1bd"},"cell_type":"markdown","source":"It seems that all show a similar pattern to store #47, which is part of this group. This is encouraging. Note some slight difference though: Store #49 shows a distinctive upward trend, while store #51 seems to barely have any difference between working days and off days. Store 52 is brand new and has very few data points.\n\nLet us now see type 'B'."},{"metadata":{"_kg_hide-input":true,"_uuid":"2867b5062c913e22aa0315f4660594328d3fb74b","collapsed":true,"_cell_guid":"d76dbc7b-6fff-4650-9908-70953c9f361a","trusted":false},"cell_type":"code","source":"plot_store_transactions_type('B')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e89a82131f5b4e10c56e86ecca38944b620afb3b","_cell_guid":"4a01ee1d-a7cd-458d-b2ec-433222fd679d"},"cell_type":"markdown","source":"The stores with a clear difference between working days and off days are the minority here. Note that store #20 and #21 show a surge of volume at opening before stabilizing after a few months. There must have entered a market with a lack of service. Note that store #18 has a 'hole' in the data for about a 4 months late 2016. Probably some renovation.\n\nLet's check type 'C'."},{"metadata":{"_kg_hide-input":true,"_uuid":"05ab8fdc9bc416e332fb10c75f78ec14e3c7171f","collapsed":true,"_cell_guid":"5a1f2da9-f3f8-450d-a1b0-6a0c0fb2adb3","trusted":false},"cell_type":"code","source":"plot_store_transactions_type('C')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a5b6a6d8bff9beafb2abc91987cd743eb1a3daee","_cell_guid":"2411d348-5680-44f6-b111-1f2b439e866e"},"cell_type":"markdown","source":"Type 'C' is even less consistant as the previous ones. Stores #13, #14 and #19 have the off days split into two distinct groups. Stores #15 and #40 have slightly less transactions on off days than working days. There must be a geographical reason for these behaviors, such as stores situatied in industrial zones. It is something worth investigating later."},{"metadata":{"_kg_hide-input":true,"_uuid":"a907d08532cc76aa9ba5eafd7a6406f5162c35a3","collapsed":true,"_cell_guid":"eb96074b-714f-46b4-9fb5-7bded4dc0825","trusted":false},"cell_type":"code","source":"plot_store_transactions_type('D')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"963a4f05499a48c0072180b9d8d76ef751544e34","_cell_guid":"02b9e7a3-c3c0-4b98-9569-599a11524606"},"cell_type":"markdown","source":"These show a mix of behaviors previously seen. Store #1 shows its off days both split and below working days. Store #24 has a 'hole' of 3 months mid 2015 while store #53 has a clear discontinuity in April 2016.\nThe discontinuity was explained by the company, as the consequence of the relief efforts from the population after an earthquate of magnitude 7.8 in the region."},{"metadata":{"_kg_hide-input":true,"_uuid":"dfe05495eaad1c14548f246c48bd7c12f7550db2","collapsed":true,"_cell_guid":"274f4458-5b25-4b9c-9f3f-3558386962c7","trusted":false},"cell_type":"code","source":"plot_store_transactions_type('E')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"508e92b2199887c8aea3db44c2d6caa318f68f68","_cell_guid":"49b4421c-6350-4bc2-b4b9-69b5cb752067"},"cell_type":"markdown","source":"This group is a little more homogenius with common transactions volume. Note that stores #36 and #43 both have discontinuities.\n\n## ** Clusters are no better **\n\n  Another classification provided by the company is what they called 'clusters'. The fonction below does the same as previously but grouping by clusters."},{"metadata":{"_uuid":"2ad9c92130025636eac3d2104e526b91a8f5a6c7","collapsed":true,"_cell_guid":"f66a99b5-60b9-4497-bb1d-29e323b428df","trusted":false},"cell_type":"code","source":"def plot_store_transactions_cluster(clust):\n    clust_stores=stores.loc[stores.cluster==clust,'store_nbr']\n    n=len(clust_stores)\n    m=1\n    for x in range(1,6):\n        if (n-1) in range((x-1)**2,x**2):\n            m=x\n    plt.figure(figsize=(15,15))\n    for x in range(n):\n        nbr=clust_stores.iloc[x]\n        ax1 = plt.subplot(m,m,x+1)\n        ax1.scatter(sales.loc[(~sales.dayoff)&(sales.cluster==clust)&(sales.store_nbr==nbr),'date'].values,\n                sales.loc[(~sales.dayoff)&(sales.cluster==clust)&(sales.store_nbr==nbr),'transactions'])\n        ax1.scatter(sales.loc[(sales.dayoff)&(sales.cluster==clust)&(sales.store_nbr==nbr),'date'].values,\n                sales.loc[(sales.dayoff)&(sales.cluster==clust)&(sales.store_nbr==nbr),'transactions'])\n        plt.title('Store {}, Type {}, Cluster {}'.format(nbr,\n                                                         list(stores.loc[stores.store_nbr==nbr,'type'])[0],\n                                                         list(stores.loc[stores.store_nbr==nbr,'cluster'])[0])\n             )\n        plt.suptitle(' Cluster {} stores'.format(clust),fontsize=25)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0163a9b5597bd6887967e033f96f34928f7ea93e","_cell_guid":"8d425401-9599-4533-b13b-7c0c9df7d161"},"cell_type":"markdown","source":"  The gaphs below belong to stores of the cluster #13, which show 4 very different behaviors. Clusters are therefore not a good classification for forcasting models either"},{"metadata":{"_kg_hide-input":true,"_uuid":"80d700c4b543dd896015a23a968510658e00f6f4","collapsed":true,"_cell_guid":"6e27c5aa-6484-4c5f-b64a-ad454effc61a","trusted":false},"cell_type":"code","source":"plot_store_transactions_cluster(13)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"20e8fc53c995592afddb361932cd44d405e9f655","_cell_guid":"dc48ef88-2519-4349-8405-a804fb7113f4"},"cell_type":"markdown","source":""},{"metadata":{"_uuid":"4015942b7abb27a5c1c9640f58236254cc2c60b8","_cell_guid":"7aa5c102-06f2-4b5f-b4d2-d7f1c7039709"},"cell_type":"markdown","source":"# **// Part 4 // Building a new classification** <a class=\"anchor\" id=\"Part4\"></a>\n\nWe saw that there are a lot of different stores in the dataset and their clients motivations to shop are very varied. It is crucial to understand that before diving deeper into statistical analysis. Getting a better classification of the stores so as to adapt the models to them will require some unsupervised learning and we will start with getting clusters with these simple features:\n- Mean and standard deviation of working day transactions\n- Mean and standard deviation of holiday transactions\n- Mean and standard deviation of week day transactions\n\nLet us build the table with the data:\n"},{"metadata":{"scrolled":true,"_uuid":"919c40d20782745030edbcb94d76b2841cd76c3f","collapsed":true,"_cell_guid":"725eea58-aaca-4069-be5d-30d3a42c5bb9","trusted":false},"cell_type":"code","source":"#creating a table with means and std of transaction volume per type of day per store\nMeans1=sales.groupby(['store_nbr','dayoff']).transactions.agg(['mean','std']).unstack(level=1)\n\n#creating a table with means and std of transaction volume per day of the week per store\nMeans2=sales.groupby(['store_nbr','day']).transactions.agg(['mean','std']).unstack(level=1)\n\n# Creating a table  with the daily average of transaction volume per store \nsales_by_store=sales.groupby(['store_nbr']).transactions.sum()/sales.groupby(['store_nbr']).transactions.count()\n# Creating a new columns with ratio of transactions of the day / daily average\nsales['normalized']=[v/sales_by_store[s] for (s,v) in zip(sales.store_nbr,sales.transactions)]\n\n#creating a table with means and std of normalized transaction volume per type of day per store\nMeans1_norm=sales.groupby(['store_nbr','dayoff']).normalized.agg(['mean','std']).unstack(level=1)\n#creating a table with means and std of normalized transaction volume per day of the week per store\nMeans2_norm=sales.groupby(['store_nbr','day']).normalized.agg(['mean','std']).unstack(level=1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4eb01cae200fc84eefb047b6fc82594d6203605b","_cell_guid":"938011eb-9c9d-47cd-ae10-88124ec95fc3"},"cell_type":"markdown","source":"Let us plot the new features to see if any clustering appear obvious:"},{"metadata":{"_kg_hide-input":true,"_uuid":"4a1e447b9abc31fec8a65cbe41613dc2b3812dc5","collapsed":true,"_cell_guid":"c4223140-bf41-4907-a279-3dd01e7416d6","trusted":false},"cell_type":"code","source":"plt.figure(figsize=(12,16))\nplt.subplot(2,2,1)\nplt.scatter(Means1.iloc[:,0],Means1.iloc[:,1])\nplt.xlabel('Means of working days')\nplt.ylabel('Means of holidays')\nplt.plot([0,5000],[0,5000])\nplt.title('Comparing mean by type of day')\nplt.subplot(2,2,2)\nplt.scatter(Means1.iloc[:,2],Means1.iloc[:,3])\nplt.xlabel('Standard dev. of working days')\nplt.ylabel('Standard dev. of holidays')\nplt.plot([0,1000],[0,1000])\nplt.title('Comparing std by type of day');\nplt.subplot(2,2,3)\nplt.scatter(Means1.iloc[:,0],Means1.iloc[:,2])\nplt.xlabel('Means of working days')\nplt.ylabel('Standard dev. of working days')\nplt.plot([0,5000],[0,500])\nplt.title('Comparing mand and std for working days')\nplt.subplot(2,2,4)\nplt.scatter(Means1.iloc[:,1],Means1.iloc[:,3])\nplt.xlabel('Means of holidays days')\nplt.ylabel('Standard dev. of holidays')\nplt.plot([0,5000],[0,1000]);\nplt.title('Comparing mand and std for holidays');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3d8c907bf9d99fcd848a0c9cd950b2db92a73fff","collapsed":true,"_cell_guid":"2193cea7-684b-4b60-95cd-a8755e174dc2","trusted":false},"cell_type":"code","source":"plt.figure(figsize=(12,6))\nplt.subplot(1,2,1)\nseaborn.heatmap(Means2.iloc[:,0:7],cmap='Oranges');\nplt.subplot(1,2,2)\nseaborn.heatmap(Means2.iloc[:,7:14],cmap='Oranges');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a43430a0288b8c84ef0929254bc321fbe4c4ce53","_cell_guid":"3875c939-2704-4c9f-b01d-49406188debe"},"cell_type":"markdown","source":"The plots don't show any obvious clustering, though the heatmaps seems to have some patterns in them. But dur to difference in average tansactions volume, stores that follow a similar pattern but are of different size may not be similar at all with the raw data. Let's look at the normalized data now:"},{"metadata":{"_uuid":"62c2545fb2ac8bccc5662b0df9d6aba397aa29ce","collapsed":true,"_cell_guid":"f01c39d7-071d-4e16-87a6-3493918c5949","trusted":false},"cell_type":"code","source":"plt.figure(figsize=(12,16))\nplt.subplot(2,2,1)\nplt.scatter(Means1_norm.iloc[:,0],Means1_norm.iloc[:,1])\nplt.xlabel('Means of working days')\nplt.ylabel('Means of holidays')\nplt.title('Comparing mean by type of day')\nplt.subplot(2,2,2)\nplt.scatter(Means1_norm.iloc[:,2],Means1_norm.iloc[:,3])\nplt.xlabel('Standard dev. of working days')\nplt.ylabel('Standard dev. of holidays')\nplt.title('Comparing std by type of day');\nplt.subplot(2,2,3)\nplt.scatter(Means1_norm.iloc[:,0],Means1_norm.iloc[:,2])\nplt.xlabel('Means of working days')\nplt.ylabel('Standard dev. of working days')\nplt.title('Comparing mean and std for working days')\nplt.subplot(2,2,4)\nplt.scatter(Means1_norm.iloc[:,1],Means1_norm.iloc[:,3])\nplt.xlabel('Means of holidays days')\nplt.ylabel('Standard dev. of holidays')\nplt.title('Comparing mean and std for holidays');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9483188c804ab9ee1cf4ffd071b726d94dd24f30","_cell_guid":"1dbef1f0-4247-49a1-99a8-24728bf90f20"},"cell_type":"markdown","source":"Note that the fact that the means are linearly correlated should be obvious the slope should correspond to the ratio of days off / working days. It gives us a very simple clustering of the stores, there is the one in the bottom right and the rest. We expect that lonely store to be store #1. Let us check the heatmaps now."},{"metadata":{"_uuid":"8cdc7278160c067f32cf0814892689283348bf4a","collapsed":true,"_cell_guid":"d21df8e1-bb7f-4acb-9192-e08b176f736b","trusted":false},"cell_type":"code","source":"plt.figure(figsize=(12,6))\nplt.subplot(1,2,1)\nseaborn.heatmap(Means2_norm.iloc[:,0:7],cmap='Oranges');\nplt.subplot(1,2,2)\nseaborn.heatmap(Means2_norm.iloc[:,7:14],cmap='Oranges');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ccd6811905851dd74330a5d5a2223c34cbc470f1","_cell_guid":"1c53f933-d161-4f7e-97e2-f7d736f6866f"},"cell_type":"markdown","source":"### **Hierarchical clustering**\nSome real patterns appear now, such as the very distinct standard deviation pattern of stores #25 and  #26, as well as the highlighted Saturday shopping frenzy at store 19 and 25. This is a good starting point to train a ML algorithm for clustering.\n\n  We will run a hierachical clustering machine based on the AgglomerativeClustering algorithm from the sklearn library. But that means we need to decide how many clusters we want to see. Instead of counting by hand the number if patterns we can see in the 54 stores, we can use the ward clustering and dendogram features of the scipy library to get an idea of how many clusters we may want:"},{"metadata":{"_uuid":"5521ab8917537632abf0a5a813cdc415e4b78b8c","collapsed":true,"_cell_guid":"c5d57dc7-2d28-4f93-8cbb-603907b89128","trusted":false},"cell_type":"code","source":"fig=plt.figure(figsize=(15,15))\nax = fig.add_subplot(1, 1, 1)\ndendrogram(ward(Means2_norm),ax=ax)\nax.tick_params(axis='x', which='major', labelsize=15)\nax.tick_params(axis='y', which='major', labelsize=8)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"34fc96d2913e23e46f9ec655d26db702f81f6603","_cell_guid":"c6f4b599-7b08-4188-81c5-9e136748c3dc"},"cell_type":"markdown","source":"It seems that 5 clusters would be a good number. Note that the ward clustering is different from the agglomerative clustering, so we don't have to take exactly 5. We decided to pick 6 clusters."},{"metadata":{"_uuid":"fe6bab6bad05f9efbd766b3b11f1fd823f147b8f","collapsed":true,"_cell_guid":"9646833f-7aa3-4215-bc19-a40c5c3c584e","trusted":false},"cell_type":"code","source":"clustering=AggClust(n_clusters=6)\ncluster=clustering.fit_predict(Means2_norm)\nstores['new_cluster']=cluster\n\ndef plot_store_transactions_new_cluster(clust):\n    clust_stores=stores.loc[stores['new_cluster']==clust,'store_nbr']\n    n=len(clust_stores)\n    m=1\n    for x in range(1,10):\n        if (n-1) in range((abs(x-1))**2,x**2):\n            m=x\n    plt.figure(figsize=(15,15))\n    for x in range(n):\n        nbr=clust_stores.iloc[x]\n        ax1 = plt.subplot(m,m,x+1)\n        ax1.scatter(sales.loc[(~sales.dayoff)&(sales.store_nbr==nbr),'date'].values,\n                sales.loc[(~sales.dayoff)&(sales.store_nbr==nbr),'transactions'])\n        ax1.scatter(sales.loc[(sales.dayoff)&(sales.store_nbr==nbr),'date'].values,\n                sales.loc[(sales.dayoff)&(sales.store_nbr==nbr),'transactions'])\n        plt.title('Store {}, Type {}, Cluster {}'.format(nbr,\n                                                         list(stores.loc[stores.store_nbr==nbr,'type'])[0],\n                                                         list(stores.loc[stores.store_nbr==nbr,'cluster'])[0])\n             )\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cc8b1a5e9626ccd66ce9aab0ea3f60cafdd7fa88","collapsed":true,"_cell_guid":"87eb87c8-90a3-4107-b4ae-cdcf50f86cd5","trusted":false},"cell_type":"code","source":"plot_store_transactions_new_cluster(1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b4a78bd94fa4d63214898893e1b418bafa0f0950","collapsed":true,"_cell_guid":"8fc20eb1-7846-4ff6-a2dc-42288c075c84"},"cell_type":"markdown","source":"This group is very homogenous. It seems to be stores with very little difference between the days of the week and apart from store #16, with a constant mean. Judging by this group, the clustering seems quite good. Let's check the second cluster."},{"metadata":{"_uuid":"3837f53554b8b99f571723a32bf5a6337c4c5c37","collapsed":true,"_cell_guid":"d0ba7cf5-8b5b-487a-8870-0e6da60ce752","trusted":false},"cell_type":"code","source":"plot_store_transactions_new_cluster(2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b732cd3eea5af692f19e6c08ca5d5c4888e4d49b","_cell_guid":"53bbdbef-1b6b-4f2f-af28-d64ac19c4e26"},"cell_type":"markdown","source":"This again is a very homogenous cluster, with a Saturdays and Sundays showing very diferent volumes of transactions."},{"metadata":{"_uuid":"168b46bced1985c440f729b2abdca3cfcea8b661","collapsed":true,"_cell_guid":"83b27780-5758-42e6-8914-5e597249d617","trusted":false},"cell_type":"code","source":"plot_store_transactions_new_cluster(3)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"002b8f2e822bf2be13a9cf7e1ec6dfac18092605","_cell_guid":"eaa9b9bf-b9d0-466e-a8e2-dce6bf896909"},"cell_type":"markdown","source":"This group is again very homogenous, with volumes of sales on days off distinctively higher than on working days. You are welcome to explore the remaining 3 groups on your own. Group 4 is a group where there is absolutely no difference between the days of the week and Group 5 contains Store 1 alone, where sales during weekends are way below weekday levels."},{"metadata":{"_uuid":"cf582790609da088bb803758233866a7ac7ca32e","_cell_guid":"a20bcb9a-fafa-4fcf-afaa-c90beb6f7a49"},"cell_type":"markdown","source":"# **// Part 5 // A few more points**<a class=\"anchor\" id=\"Part5\"></a>"},{"metadata":{"_uuid":"05866e1dd72b138b84e8bfbb3d634baf6c9b7ae3","_cell_guid":"4e8667a0-fafe-4b70-bb7f-2b9877ad2ecf"},"cell_type":"markdown","source":"Some other data could help us improve our analysis and predictive models by factoring in some other factors.\n\n##  ** Geography **\n\n  The stores may be linked in ways that are rendered obvious by seeing their geographical locations. This could also help us understand the grouping provided by the store chain. Let's therefore plot all stores using their location to see if we can find more hints about what we see. Unfortunately the locations are not provided at a granularity below city and states. We will therefore use the city names and **add some noise** to the GPS coordinates so as to separate the icons for visibility.\n  As it turns out, the data is far from representing all stores owned by the company so we cannot use a simple method as a google search to identify the type of neighborhood/environment the stores are located in.\n\n  To plot the map, we will be using Folium, which is really nice and allows very interactive content."},{"metadata":{"_uuid":"6529845e8dd89baef62b0c300436ed460bcd7d7f","collapsed":true,"_cell_guid":"fc233951-59e2-47b5-b10e-f736e03207b0","trusted":false},"cell_type":"code","source":"store_locations={\n 'Ambato' : [-1.2543408,-78.6228504],\n 'Babahoyo' : [-1.801926,-79.53464589999999],\n 'Cayambe' : [0.025,-77.98916659999998],\n 'Cuenca' : [-2.9001285,-79.0058965],\n 'Daule' : [-1.86218,-79.97766899999999],\n 'El Carmen' : [-0.266667, -79.4333],\n 'Esmeraldas' : [0.9681788999999998,-79.6517202],\n 'Guaranda' : [-1.5904721,-78.9995154],\n 'Guayaquil' : [-2.1709979,-79.92235920000002],\n 'Ibarra' : [0.3391763,-78.12223360000002],\n 'Latacunga' : [-0.7754954,-78.52064999999999],\n 'Libertad' : [-2.2344458,-79.91122430000001],\n 'Loja' : [-4.0078909,-79.21127690000003],\n 'Machala' : [-3.2581112,-79.9553924],\n 'Manta' : [-0.9676533,-80.70891010000003],\n 'Playas' : [-2.6284683,-80.38958860000002],\n 'Puyo' : [-1.4923925,-78.00241340000002],\n 'Quevedo' : [-1.0225124,-79.46040349999998],\n 'Quito' : [-0.1806532,-78.46783820000002],\n 'Riobamba' : [-1.6635508,-78.65464600000001],\n 'Salinas' : [-2.2233633,-80.958462],\n 'Santo Domingo' : [-0.2389045,-79.17742679999998]\n}\n\n# Defining a color dictionary\ncol={'A':'red','B':'blue','C':'green','D':'pink','E':'beige',\n     0:'red',1:'blue',2:'green',3:'darkblue',4:'pink',5:'beige'}\n\n#\ndef add_city_map(name,typ):\n    folium.Marker(\n         location=list(map(add,store_locations.get(name),[(0.5-rd.random())/20,(0.5-rd.random())/20])),\n         icon=folium.Icon(color=col.get(typ), icon='shopping-cart'),\n    ).add_to(map_Ecuador)\n\nmap_Ecuador=folium.Map(location=[-1.233333, -78.516667],zoom_start=7)\n\n# Enabling clustering (also replace map_ecuador by store_cluster in the add_city_map function)\n# from folium.plugins import MarkerCluster\n#store_cluster=MarkerCluster().add_to(map_Ecuador)\n\n[add_city_map(x,y) for x,y in zip(stores.city,stores.type)]\nmap_Ecuador","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cbb9fdbbe82671426292269fad84cb31244eaef7","_cell_guid":"ea517bd1-9b6e-4271-8854-ab743c9ae06a"},"cell_type":"markdown","source":"The map does not teach us anything about the 'type' classification and based on what we have seen so far, I am very tempted to drop it altogether. Note that it could be interesting to map the correlations between the stores to see if there is some seasonal component that spreads from the capital out for examples or phenomenons of the same type. In further analysis we may build networks around the stores and see if any interesting information can be extracted from it.\n\nNote that to change the icon to a intuitive one, we checked the documentation for the icon method (see below), and followed the link provided, sending me to the github page of the Leaflet plugin that Folium is built upon. "},{"metadata":{"_uuid":"dc7342dc6e30b033faba5dec0f55647a4db6bd6f","collapsed":true,"_cell_guid":"0bc23d1a-71c8-4f80-9d91-e395ea9c0062","trusted":false},"cell_type":"code","source":"help(folium.Icon)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ee9530a4473f2fc557b2d3abcdd8d8206a929145","_cell_guid":"3e3b364e-26f6-443f-aee7-98070fd30ba1"},"cell_type":"markdown","source":"Let's plot the new clusters to see if we can quickly make georgraphical sense of the classification:"},{"metadata":{"_uuid":"5e7139fbc58f2e478d531d2fb3e60a5afc6ee552","collapsed":true,"_cell_guid":"afa42ee2-6717-4732-bf90-6b065450c075","trusted":false},"cell_type":"code","source":"map_Ecuador=folium.Map(location=[-1.233333, -78.516667],zoom_start=7)\n\n# Enabling clustering (also replace map_ecuador by store_cluster in the add_city_map function)\n# from folium.plugins import MarkerCluster\n#store_cluster=MarkerCluster().add_to(map_Ecuador)\n\n[add_city_map(x,y) for x,y in zip(stores.city,stores.new_cluster)]\nmap_Ecuador","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a730e96418cb3afc59ab587eacbd7310ad556c26","_cell_guid":"a4eacf2e-345d-4c8d-b67e-d0faa01076a3"},"cell_type":"markdown","source":"## ** Weather **\n\nWeather could also have a major impact on sales, with people chosing to stay home and delay their purchases during heavy rain or extreme heat/cold. Unfortunately we am currently unable to find free and public weather data for ecuadorian cities."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.5","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}