{"metadata": {"kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"}, "nteract": {"version": "0.2.0"}, "language_info": {"codemirror_mode": {"version": 3, "name": "ipython"}, "version": "3.6.3", "mimetype": "text/x-python", "name": "python", "file_extension": ".py", "pygments_lexer": "ipython3", "nbconvert_exporter": "python"}}, "nbformat": 4, "nbformat_minor": 1, "cells": [{"metadata": {"_uuid": "bebee09f6ba3e3094872bd22e7def05d9629666a", "_cell_guid": "ca8d9581-d7ab-418e-ae17-f3ff7d0b2a6f", "collapsed": true}, "source": ["# Explory data analysis for Favorita Grocery Sales Forecasting\n", "\n", "(If you found this EDA useful don't forget to upvote it, thanks a lot!)\n", "\n", "Note:  The Kaggle docker image for kernel is not up to date for bokeh so during the interactive plots of timeseries you may see a weird number for dates, that shouldn't be a problem but I didn't find any alternative to show the date properly on an older version of bokeh.\n"], "cell_type": "markdown"}, {"metadata": {"_uuid": "09d0621d2dc933c1228ccdf60c21638a347225e4", "_cell_guid": "6cbba35c-0bf3-4c4b-9596-f3e05ccd8655"}, "source": ["## 1. Intro & global analysis\n", "\n", "The data comes in the shape of multiple csv files which, given their structure, seems to be extracted from a relational database. The 2 main files: `train.csv` & `test.csv` contains the sales by date, store, and item. The test data contains the same features without the sales information, which we are tasked to predict. The train vs test split is based on the date. In addition, **some test items are not included in the train data**.\n", "\n", "Furthermore, there are 5 additional data files that provide the following information:\n", "\n", " - `stores.csv`: Details about the stores, such as location and type.\n", "\n", " - `items.csv`: Item metadata, such as class and whether they are perishable.\n", "\n", " - `transactions.csv`: Count of sales transactions for the training data\n", "\n", " - `oil.csv`: Daily oil price. This is relevant, because \u201cEcuador is an oil-dependent country and its economical health is highly vulnerable to shocks in oil prices.\u201d\n", "\n", " - `holidays_events.csv`: Holidays in Ecuador. Some holidays can be transferred to another day (possibly from weekend to weekday).\n", "\n", "In addition, the [data description](https://www.kaggle.com/c/favorita-grocery-sales-forecasting/data) notes the importance of public sector pay days (on the 15th and the end of the month) as well as the impact of **a major earthquake on April 16 2016 which greatly affected supermarket sales for several weeks after the earthquake**."], "cell_type": "markdown"}, {"metadata": {"_uuid": "04961462b3ca86aa417cb860cd0f88939bbabac7", "_cell_guid": "fcde58e5-b99c-403e-a2ba-6685c2c6520c"}, "source": ["Import the necessary libs"], "cell_type": "markdown"}, {"metadata": {"_uuid": "f72ec3929885541cbd18dec23880a6dc8011d777", "_cell_guid": "f9ef4841-400e-4916-a6f5-c779d9859303", "collapsed": true}, "execution_count": null, "outputs": [], "source": ["import os\n", "import numpy as np\n", "import pandas as pd\n", "import seaborn as sns\n", "from bokeh.io import show, output_notebook\n", "from bokeh.models import ColumnDataSource, FactorRange, HoverTool\n", "from bokeh.plotting import figure\n", "import matplotlib.pyplot as plt\n", "import matplotlib.ticker as ticker\n", "from tqdm import tqdm\n", "from IPython.display import display\n", "from random import randint\n", "\n", "output_notebook()\n", "\n", "%matplotlib inline\n", "%config InlineBackend.figure_format = 'retina'"], "cell_type": "code"}, {"metadata": {"_uuid": "a6453bada55c1148c4b4ebe61be33a78a44be6ae", "_cell_guid": "e9b5ca64-91d0-4158-84a3-a1fe772321f6", "collapsed": true}, "execution_count": null, "outputs": [], "source": ["input_files_path = '../input/'\n", "\n", "files = [\"train\", \"test\", \"transactions\", \"stores\", \"oil\", \"items\", \"holidays_events\"]\n", "datasets_path = [input_files_path + f + \".csv\" for f in files]"], "cell_type": "code"}, {"metadata": {"_uuid": "b3435bb1f2d1cbb8f50dbf6fb7635c9b0d883e63", "_cell_guid": "c7160c2d-c508-414e-9e73-6a233ef0b284"}, "source": ["Now lets take a look at the data files:"], "cell_type": "markdown"}, {"metadata": {"_uuid": "432db94bb14135e3db560f114960c2079b24dd0d", "_cell_guid": "459ccffb-7926-4a24-ad10-47d3aacf46e7", "collapsed": true}, "execution_count": null, "outputs": [], "source": ["print('# File sizes')\n", "for f in datasets_path:\n", "    print(f.ljust(30) + str(round(os.path.getsize(input_files_path + f) / 1000000, 2)) + 'MB')"], "cell_type": "code"}, {"metadata": {"_uuid": "17afc3942133f240ca03f32b089bfba38e5308b9", "_cell_guid": "66a987b2-691d-4ea4-a301-def952199872"}, "source": ["The `train.csv` is 5gb in size so we won't load all the data in pandas dataframes but instead we will conduct this EDA on ~20% of the data (By reading by chunks, put `None` in `chunksize` if you want to read the whole data). Be careful the whole datasets take ~10gb of RAM. **The training set has 125 497 040 entries and the the testing set has 3 370 464 entries.**"], "cell_type": "markdown"}, {"metadata": {"_uuid": "fcccf858df8cad05d2aa2829b6ba219af3a2ade9", "_cell_guid": "df5ce7e5-d4c4-4e67-bc66-a8ef269fb1e4", "collapsed": true}, "execution_count": null, "outputs": [], "source": ["chunksize = 25_000_000 # ~20% of the training set\n", "data_df = {}\n", "for filename, filepath in tqdm(zip(files, datasets_path), total=len(files)):\n", "    if chunksize:\n", "        data_df[filename] = pd.read_csv(filepath, chunksize=chunksize, low_memory=False).get_chunk()\n", "    else:\n", "        data_df[filename] = pd.read_csv(filepath, low_memory=False)"], "cell_type": "code"}, {"metadata": {"_uuid": "7f65642b9b99d63012efdd5caf9178c13f4c0a50", "_cell_guid": "24dc9b44-7058-4e4b-ae0e-704c558361d3"}, "source": ["Get a sense of the [dataset](https://www.kaggle.com/c/favorita-grocery-sales-forecasting/data):"], "cell_type": "markdown"}, {"metadata": {"scrolled": false, "_uuid": "5f26bb57ebd05a5043b3b6a5e6ec645ddbd55e9d", "_cell_guid": "fdebc5fe-eee5-4bd8-b0d4-7653c74d7221", "collapsed": true}, "execution_count": null, "outputs": [], "source": ["for chunk, df in data_df.items():\n", "    print(\"Dataset {} of size {} with fields:\\n {}\".format(chunk, len(df), df.dtypes))\n", "    display(df.tail())\n", "    print(df.describe(), end='\\n\\n\\n')"], "cell_type": "code"}, {"metadata": {"_uuid": "3a3e57ede8b7a9bc1242963915eefb62ed6be4f8", "_cell_guid": "3ba2ebb3-7644-4b48-b4fa-4b60fd0ac0c2"}, "source": ["\n", "Lets now detail what every field is about and how do they relate to each other across the different csv files:\n", " - `train` & `test`:\n", "     - `id`: The entry id (_non null_)\n", "     - `date`: The date relative to each `unit_sales` (/!\\ The training data does not include rows for items that had zero `unit_sales` for a store/date combination) (_non null_)\n", "     - `store_nbr`: The id of the store selling that item on the specified date (to find it on `stores.csv`)\n", "     - `item_nbr`: The id of the item (to find it on `items.csv`)\n", "     - `unit_sales`: The **target** variable expressed as integer (e.g. a bag of chips) or float (e.g. 1.5 kg of cheese). Negative values of unit_sales represent returns of that particular item.\n", "     - `onpromotion`: If the item was on promotion on that day (/!\\ Pandas automatically infer columns types and will changes `NaN` values to `False` if the Dataframe was created on the whole train/test set. In our case we won't consider it a problem and will consider all `NaN` values to be `False`.)\n", "     \n", " - stores:\n", "     - `store_nbr`: Unique id for the store\n", "     - `city`: City in which the store is located\n", "     - `state`: State in which the store is located\n", "     - `type`: (guess) Type of stores?\n", "     - `cluster`: A [\"grouping of similar stores\"](https://www.kaggle.com/c/favorita-grocery-sales-forecasting/data)\n", " - items: \n", "     - `item_nbr`: Unique id for the item\n", "     - `family`: Kind of item regrouped by a broad category such as `BEVERAGES` or `GROCERY`\n", "     - `class`: ?\n", "     - `perishable`: `0` or `1`, wether the item is perishable or not\n", " - transactions:\n", "     - `date`: The date at which the transaction was done (serves as the unique identifier)\n", "     - `store_nbr`: The store id on which the transaction was made\n", "     - `transactions`: The transaction count\n", " - oil: \n", "     - `date`: The date, act as an unique identifier\n", "     - `dcoilwtico`: The oil price index\n", " - holidays_events:\n", "     - `date`: The date of the event (serves as the unique identifier)\n", "     - `type`: Type of event \n", "     - `locale`: Wether the event was regional, national or local\n", "     - `locale_name`: The region in which the event applies\n", "     - `description`: The name of the holiday\n", "     - `transferred`: Boolean, indicates if the holiday was transfered to another day. If this field is `True` then the entry can be considered as a normal day and thus we need to find the next day marked as `Transfered` which is the actual day when the event was celebrated."], "cell_type": "markdown"}, {"metadata": {"_uuid": "ba58a8a45164f08b12dc1894c16b729f9b3e0272", "_cell_guid": "f5d79c0c-3d72-4111-b38f-554c00eaa386"}, "source": ["Lets retrieve and summarize some useful properties of few specific fields:"], "cell_type": "markdown"}, {"metadata": {"scrolled": false, "_uuid": "d10ca11483a05eccfe1461c6cf3c571d541f1a4f", "_cell_guid": "994e86f4-63da-427b-ac8b-63fc3a5ddb76", "collapsed": true}, "execution_count": null, "outputs": [], "source": ["train_df = data_df[\"train\"]\n", "train_df[\"date\"] = pd.to_datetime(train_df[\"date\"])\n", "test_df = data_df[\"test\"]\n", "test_df[\"date\"] = pd.to_datetime(test_df[\"date\"])\n", "oil_df = data_df[\"oil\"]\n", "oil_df[\"date\"] = pd.to_datetime(oil_df[\"date\"])\n", "items_df = data_df[\"items\"]\n", "stores_df = data_df[\"stores\"]\n", "transactions_df = data_df[\"transactions\"]\n", "transactions_df[\"date\"] = pd.to_datetime(transactions_df[\"date\"])\n", "holidays_events_df = data_df[\"holidays_events\"]\n", "holidays_events_df[\"date\"] = pd.to_datetime(holidays_events_df[\"date\"])\n", "\n", "print(\"Train set date range: {} to {}\".format(train_df[\"date\"].min(), train_df[\"date\"].max()))\n", "print(\"Test set date range: {} to {}\".format(test_df[\"date\"].min(), test_df[\"date\"].max()))\n", "print(\"Transactions date range: {} to {}\".format(transactions_df[\"date\"].min(), transactions_df[\"date\"].max()))\n", "print(\"Promotions count on train set: {}\".format(len(train_df[\"onpromotion\"]) - \n", "                                                 train_df[\"onpromotion\"].isnull().sum()))\n", "print(\"Promotions count on test set: {}\".format(len(test_df[\"onpromotion\"]) - \n", "                                                len(test_df[test_df[\"onpromotion\"] == True])))\n", "print(\"Number of different stores: {}\".format(len(stores_df)))\n", "print(\"Number of different stores clusters: {}\".format(stores_df[\"cluster\"].max()))\n", "print(\"Oil price index range: {} - {}\".format(oil_df[\"dcoilwtico\"].min(), oil_df[\"dcoilwtico\"].max()))\n", "print(\"Oil unknown price count: {} \".format(oil_df[\"dcoilwtico\"].isnull().sum()))\n", "print(\"Items {} uniques families: {}\\n\".format(len(items_df[\"family\"].unique()), items_df[\"family\"].unique()))\n", "print(\"Different types of holiday events: {}\\n\".format(holidays_events_df[\"type\"].unique()))\n", "print(\"Different holiday locales: {}\\n\".format(holidays_events_df[\"locale\"].unique()))\n", "print(\"Region list where the holidays applies: {}\\n\".format(holidays_events_df[\"locale_name\"].unique()))\n", "print(\"All different kind of holidays: {}\".format(holidays_events_df[\"description\"].unique()))\n", "print(\"Example of transfered holidays: \")\n", "display(holidays_events_df[holidays_events_df[\"transferred\"] == True].head())"], "cell_type": "code"}, {"metadata": {"_uuid": "739c09166ef0d3c6df5f9168b2148b0ecbd0e190", "_cell_guid": "302b4d0d-7af7-4e9d-824d-150d85e8d5aa"}, "source": ["Now we can draw few observations:\n", " - `train` & `test`: We will plot the sales from April 16 2016 (the earthquake) and onwards to see how the data changes on that period. We may want to exclude that period from our dataset for few ML models.\n", " - `stores`: `store_nbr` is not a useful feature for machine learning models. We will need to find a way to encode each store so that our machine learning models could differentiate them even if they belongs to the same city/state pair.\n", " - `items`: There are 4100 different items and `item_nbr` is not a useful feature for machine learning models.\n", " - `transactions`: The transactions are only available for the training set\n", " - `oil`: Few dates are missing. We may want to complete the missing wholes by averaging the price index based on the dates before and after the one we are looking for.\n", " - `holidays_events`: The day when the holiday was reported is irrelevant, we need to reshape this dataset so that we keep only the effective date when the event happened as holidays may be correlated to higher sales. We also need to be careful to what region the holiday applies\n", "\n", "As we said before the shape of the data let us believe that these datasets come from a relational database (linked together via their `*_nbr` field). That may be a good idea to fit them back into one (SQLlite) so we can query and cobine the data easily with SQL without loading everything in memory at once."], "cell_type": "markdown"}, {"metadata": {"_uuid": "0af6f2abf60f4d81830110d3d8ef439b83ccc1bd", "_cell_guid": "8473dc2b-7404-4c0e-99c4-844369350c66"}, "source": ["Before going further lets merge train and test sets, so all operations can be operated onto 1 Dataframe, we can split them back later as we know the train set range from **2013-01-01 to 2017-08-15** (on the whole train set) and the test set from **2017-08-16 to 2017-08-31**. We will also replace our missing values on the `onpromotion` to `False`."], "cell_type": "markdown"}, {"metadata": {"_uuid": "57f8c36a2fed75b8855db44b994e7802065a2abb", "_cell_guid": "ca2a1e4d-12dc-43b9-8fe7-62a3983eee6f", "collapsed": true}, "execution_count": null, "outputs": [], "source": ["train_df['onpromotion'].fillna(False, inplace=True)\n", "test_df['onpromotion'].fillna(False, inplace=True)\n", "\n", "final_df = train_df.append(test_df)\n", "assert len(final_df) == len(train_df) + len(test_df)\n", "final_df['onpromotion'] = final_df['onpromotion'].astype(bool)\n", "print(\"Final df size: {}\".format(len(final_df)))\n", "#assert len(final_df) == len(final_df[\"item_nbr\"].unique())\n", "final_df.head()"], "cell_type": "code"}, {"metadata": {"_uuid": "73d2d22166b81614664298ca1dd0f97e3a307b4a", "_cell_guid": "ad67b3ff-51ab-4d3a-aea7-2b65d43ac6ba"}, "source": ["## 2. Features visualization\n", "\n", "Now we will plot few variables to get a sense of their individual repartition."], "cell_type": "markdown"}, {"metadata": {"_uuid": "4652c00a0006e0d8447d2c216b35cad9e51182e9", "_cell_guid": "984b2de1-cd3b-4c44-8a6b-d6e55a702b3f"}, "source": ["### 2.1 Train & Test"], "cell_type": "markdown"}, {"metadata": {"_uuid": "4176e769276c8f8895af5877ddc5c491220f115b", "_cell_guid": "d6642624-fe4b-4366-8887-7aa1b05f2162"}, "source": ["The repartition of the `onpromotion` field"], "cell_type": "markdown"}, {"metadata": {"_uuid": "028ea3ffff54b3e58147418ce4cf58ca55abf52b", "_cell_guid": "64707d0a-3439-4841-b4a1-600350c31b07", "collapsed": true}, "execution_count": null, "outputs": [], "source": ["# id\tdate\tstore_nbr\titem_nbr\tunit_sales\tonpromotion\n", "sns.countplot(x=\"onpromotion\", data=final_df);"], "cell_type": "code"}, {"metadata": {"_uuid": "86382ec214815120685ade7de4289e467a79789a", "_cell_guid": "2c187a5e-43c4-4047-a27c-390483512cea", "collapsed": true}, "execution_count": null, "outputs": [], "source": ["unit_df = train_df.groupby(\"date\").sum().reset_index()\n", "\n", "source = ColumnDataSource(unit_df)\n", "hover = HoverTool(\n", "    tooltips=[\n", "        (\"date\", \"@date{%F}\"),\n", "        (\"unit_sales\", \"@unit_sales{0.00 a}\"),\n", "    ], \n", "# Kaggle docker image is not up to date for bokeh\n", "#     formatters={\n", "#         'date': 'datetime'\n", "#     },\n", ")\n", "\n", "\n", "p = figure(x_axis_type=\"datetime\", tools=[hover, 'pan', 'box_zoom', 'wheel_zoom', 'reset'], \n", "           title=\"Unit sales by date\", plot_width=900, plot_height=400)\n", "p.xgrid.grid_line_color=None\n", "p.ygrid.grid_line_alpha=0.5\n", "p.xaxis.axis_label = 'Time'\n", "p.yaxis.axis_label = 'Value'\n", "\n", "p.line(x=\"date\", y=\"unit_sales\", line_color=\"gray\", source=source)\n", "\n", "show(p)"], "cell_type": "code"}, {"metadata": {"_uuid": "f0c1d5a666c0eb30e546e61f256e8bf0d4b1b717", "_cell_guid": "4057cab3-f29a-4f8f-8392-3f5b7e0bb5a6"}, "source": ["Unit sales on promotion vs not on promotion"], "cell_type": "markdown"}, {"metadata": {"_uuid": "f552a7448046cd5c302046f2e3e647b2cd0be3d5", "_cell_guid": "5afeff29-1a7b-4233-83d2-9af0bf7706ee", "collapsed": true}, "execution_count": null, "outputs": [], "source": ["on_prom_df = train_df[train_df[\"onpromotion\"] == True].groupby(\"date\").sum()\n", "on_prom_df.sort_values(\"onpromotion\", inplace=True)\n", "ax = sns.lmplot(x=\"onpromotion\", y=\"unit_sales\", data=on_prom_df)\n", "ax.set(xlabel='onpromotion count by date', ylabel='unit_sales count');"], "cell_type": "code"}, {"metadata": {"_uuid": "4f99c5bcaa80d125b996559b730c132d4391679d", "_cell_guid": "8937a9a2-2376-46b9-a760-863032eab512", "collapsed": true}, "execution_count": null, "outputs": [], "source": ["train_df['weekday'] = pd.DatetimeIndex(train_df['date']).weekday\n", "train_df['month'] = pd.DatetimeIndex(train_df['date']).month\n", "r = train_df.pivot_table(values='unit_sales', index='weekday', columns='month', aggfunc=np.mean)\n", "r.columns = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'June', 'July', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']\n", "r.index = ['Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat', 'Sun']\n", "f, ax = plt.subplots(figsize=(15, 6))\n", "sns.heatmap(r, linewidths=.5, ax=ax, cmap='rainbow')"], "cell_type": "code"}, {"metadata": {"_uuid": "51b1c876ae3dbda4fef5b138c1cbc7b8a5eed6d4", "_cell_guid": "5c5c58d6-872b-40d5-a8cf-5237e493b859"}, "source": ["As we can see very few items are in promotion but the sales are very correlated to the promotions. As we approach the end of the year the sales are increasing. We can also see from our interactive graph that sales decrease dramatically on 2014-01-01 which mark the end of end of the feast season then go up quickly after. We didn't use all our data but this is probably a pattern occuring every year.\n", "\n", "We can also see a jump in sales between **2014-02-27 and 2014-03-01**, a 2 days interval. We may want to check if that pattern repeats on the whole dataset and why it happens.\n", "\n", "We can also observe that most of the sales happen during weekends."], "cell_type": "markdown"}, {"metadata": {"_uuid": "f2165669d991dbc75c6fb7ff3826ccb445c7874f", "_cell_guid": "425bcf2b-2dba-41c0-9fc3-f1034e912a8b"}, "source": ["### 2.2 Stores"], "cell_type": "markdown"}, {"metadata": {"_uuid": "68f954f0c75069b2686f32d77c1383307791bbe2", "_cell_guid": "e9e0929b-a693-4ca6-a10c-2f546fcf1cb8", "collapsed": true}, "execution_count": null, "outputs": [], "source": ["# store_nbr\tcity\tstate\ttype\tcluster\n", "f, (ax1, ax2, ax3) = plt.subplots(3, 1, figsize=(14, 12))\n", "\n", "sns.countplot(x=\"city\", data=stores_df, ax=ax1)\n", "sns.countplot(x=\"state\", data=stores_df, ax=ax2)\n", "sns.countplot(x=\"cluster\", data=stores_df, ax=ax3)\n", "\n", "plt.setp(ax1.xaxis.get_majorticklabels(), rotation=70)\n", "plt.setp(ax2.xaxis.get_majorticklabels(), rotation=70)\n", "plt.tight_layout()"], "cell_type": "code"}, {"metadata": {"_uuid": "1a3297f3ec9542c7f2ee8b7a0a4dacc7f2630681", "_cell_guid": "8d2928b9-7ae9-4152-aed9-892959155156"}, "source": ["Most of the stores are in Quito (from the Pichincha region) and Guayaquil (from the Guayas region) with Quito being the capital of Ecuador. As for the clusters they regroup stores of similar \"type\"."], "cell_type": "markdown"}, {"metadata": {"_uuid": "641421014378e38edff6cb5648f743d502d73145", "_cell_guid": "d7cf4395-d01a-4d42-9d02-d20a825dac09"}, "source": ["### 2.3 Items"], "cell_type": "markdown"}, {"metadata": {"scrolled": false, "_uuid": "505278b089116424841397a71cf2689e714c461a", "_cell_guid": "97d74f35-b13b-451e-927e-ebfa5e85da3c", "collapsed": true}, "execution_count": null, "outputs": [], "source": ["perish_items = final_df.merge(items_df, on='item_nbr')\n", "\n", "f, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 6))\n", "ax1.set_title(\"Perishable items relative to all different items\")\n", "ax2.set_title(\"Perishable items relative to all items in train/test set\")\n", "sns.countplot(x=\"perishable\", data=items_df, ax=ax1)\n", "sns.countplot(x=\"perishable\", data=perish_items, ax=ax2);"], "cell_type": "code"}, {"metadata": {"_uuid": "738f115c762f8b69e4ee77f65476caad1f3a88e1", "_cell_guid": "eb4c818a-a285-46f0-baa8-94b0a597dae1", "collapsed": true}, "execution_count": null, "outputs": [], "source": ["f, ax = plt.subplots(figsize=(16, 8))\n", "sns.countplot(x=\"family\", data=items_df, ax=ax)\n", "plt.setp(ax.xaxis.get_majorticklabels(), rotation=80);"], "cell_type": "code"}, {"metadata": {"_uuid": "eed0eabaab296921cba0ff7d4c9c883e6cd1a1d3", "_cell_guid": "78662c6a-d5bf-4455-ba10-1c1cda304932"}, "source": ["We can see that ~1/3 of the items are marked as perishable and the same can be said to all perishable objects from our test/train sets. The most recurring categories are GROCERY I and BEVERAGES which is to be expected."], "cell_type": "markdown"}, {"metadata": {"_uuid": "2d50b3d37685338641713a65d43982d0d3ffbd67", "_cell_guid": "411a4edd-0e65-4d49-af4c-1e73027d1156"}, "source": ["### 2.4 Transactions"], "cell_type": "markdown"}, {"metadata": {"_uuid": "d23c7c3daf6f72c7bac5f0f09faa2cf3e40ad18b", "_cell_guid": "f9818f7c-d79a-4e85-858d-2da0f33d13d6", "collapsed": true}, "execution_count": null, "outputs": [], "source": ["trans_per_date = transactions_df.groupby([\"date\"], as_index=False).sum()\n", "\n", "source = ColumnDataSource(trans_per_date)\n", "hover = HoverTool(\n", "    tooltips=[\n", "        (\"date\", \"@date{%F}\"),\n", "        (\"transactions\", \"@transactions{0.00 a}\"),\n", "    ], \n", "# Kaggle docker image is not up to date for bokeh\n", "#     formatters={\n", "#         'date': 'datetime'\n", "#     },\n", ")\n", "\n", "\n", "p = figure(x_axis_type=\"datetime\", tools=[hover, 'pan', 'box_zoom', 'wheel_zoom', 'reset'], \n", "           title=\"Transactions\", plot_width=900, plot_height=400)\n", "p.xgrid.grid_line_color=None\n", "p.ygrid.grid_line_alpha=0.5\n", "p.xaxis.axis_label = 'Time'\n", "p.yaxis.axis_label = 'Transactions'\n", "\n", "p.line(x=\"date\", y=\"transactions\", line_color=\"gray\", source=source)\n", "\n", "show(p)"], "cell_type": "code"}, {"metadata": {"_uuid": "3fd5634bfa00cfe08a258a35fa1bb2a963e693c6", "_cell_guid": "ac65d590-5278-4190-8f05-aa690a5fb2de"}, "source": ["We can see that the earthquake on April 16 2016 didn't affect the sales overall. But if we look at the individual items or cities which were the most affected by the earthquake we may find more relevant informations."], "cell_type": "markdown"}, {"metadata": {"_uuid": "220c7d3e634c7c50e098ae959009c52d25863047", "_cell_guid": "fc4780b0-c6c4-4bb4-a23f-eded1c2237df"}, "source": ["### 2.5 Oil\n", "\n", "Here we will display the change of oil in price over time and see if it has any correlation with the sales on the same period as the competition states: \"Ecuador is an oil-dependent country and it's economical health is highly vulnerable to shocks in oil prices\""], "cell_type": "markdown"}, {"metadata": {"_uuid": "58c4ddc27b726c3060ad38ab6e3b0713804f17ea", "_cell_guid": "f8408515-5eb4-4976-b581-f56da7cfd402", "collapsed": true}, "execution_count": null, "outputs": [], "source": ["source = ColumnDataSource(oil_df)\n", "hover = HoverTool(\n", "    tooltips=[\n", "        (\"date\", \"@date{%F}\"),\n", "        (\"dcoilwtico\", \"@dcoilwtico{0.00 a}\"),\n", "    ], \n", "# Kaggle docker image is not up to date for bokeh\n", "#     formatters={\n", "#         'date': 'datetime'\n", "#     },\n", ")\n", "\n", "\n", "p = figure(x_axis_type=\"datetime\", tools=[hover, 'pan', 'box_zoom', 'wheel_zoom', 'reset'], \n", "           title=\"Oil price index\", plot_width=900, plot_height=400)\n", "p.xgrid.grid_line_color=None\n", "p.ygrid.grid_line_alpha=0.5\n", "p.xaxis.axis_label = 'Time'\n", "p.yaxis.axis_label = 'Index'\n", "\n", "p.line(x=\"date\", y=\"dcoilwtico\", line_color=\"gray\", source=source)\n", "\n", "show(p)"], "cell_type": "code"}, {"metadata": {"_uuid": "d9f96e7e3214a1055dcb2cef437795279528662e", "_cell_guid": "8799cd02-07e5-483e-8270-6868560a1555", "collapsed": true}, "execution_count": null, "outputs": [], "source": ["transac = transactions_df.groupby(\"date\").sum().reset_index()\n", "transac = transac.merge(oil_df, on='date')\n", "sns.jointplot(x='dcoilwtico', y='transactions', data=transac);"], "cell_type": "code"}, {"metadata": {"_uuid": "eb817f9dd9a151781dde77eb3db88dcf7b03a31e", "_cell_guid": "0ab9be7b-05ed-49b5-9c1d-75918f4db9c0"}, "source": ["The oil price doesn't seem to be very correlated to the number of transactions. Maybe somehow it affect the grocery prices but it's not an information relevant to us."], "cell_type": "markdown"}, {"metadata": {"_uuid": "709c2750e47ae1f8ce0ab0034520dcd7e378b371", "_cell_guid": "c5ae3c9c-8ad1-4aac-912c-3f33e2b64e7e"}, "source": ["### 2.6 Holidays events"], "cell_type": "markdown"}, {"metadata": {"_uuid": "8785861d51f4b5fda068656905069f0db49e9704", "_cell_guid": "d26cb994-2b32-4be2-b0f1-98993eb716d5", "collapsed": true}, "execution_count": null, "outputs": [], "source": ["holidays_events_df.head()"], "cell_type": "code"}, {"metadata": {"_uuid": "bc41d15f001c75ada233022a411c548f81467600", "_cell_guid": "44413031-7576-42f4-b3d4-85f3000e6d1a", "collapsed": true}, "execution_count": null, "outputs": [], "source": ["f, ax = plt.subplots(1, 3, figsize=(16, 4))\n", "ax = ax.ravel()\n", "\n", "sns.countplot(x=\"type\", data=holidays_events_df, ax=ax[0])\n", "sns.countplot(x=\"locale\", data=holidays_events_df, ax=ax[1])\n", "sns.countplot(x=\"transferred\", data=holidays_events_df, ax=ax[2])\n", "plt.tight_layout()\n", "\n", "f, axf = plt.subplots(figsize=(14, 8))\n", "sns.countplot(x=\"locale_name\", data=holidays_events_df, ax=axf)\n", "plt.setp(axf.xaxis.get_majorticklabels(), rotation=80)\n", "plt.tight_layout()"], "cell_type": "code"}, {"metadata": {"_uuid": "a8fd82d36c7786ff8ba9af37a757b207648e9084", "_cell_guid": "d9463faa-9cd5-43b1-92c5-cc918479a0c5"}, "source": ["Most of the type of holidays are actually holidays followed by \"Additional\" and \"Event\" and take place in Ecuador most of the time (which explains why we see so few events being regional). \"Additional\" denote a regular calendar holiday for example Christmas Eve. "], "cell_type": "markdown"}, {"metadata": {"collapsed": true}, "execution_count": null, "outputs": [], "source": [], "cell_type": "code"}]}