{"metadata": {"kernelspec": {"language": "python", "display_name": "Python 3", "name": "python3"}, "language_info": {"pygments_lexer": "ipython3", "version": "3.6.3", "name": "python", "codemirror_mode": {"name": "ipython", "version": 3}, "mimetype": "text/x-python", "nbconvert_exporter": "python", "file_extension": ".py"}}, "cells": [{"metadata": {"collapsed": true, "_cell_guid": "720926e1-5657-48f6-bb4c-4dfd369f4424", "_kg_hide-input": true, "_uuid": "3ce1d9107f9a7c6340f67be60c4fa2f3828ed1c1"}, "source": ["import numpy as np\n", "import pandas as pd\n", "\n", "import geojson\n", "\n", "import matplotlib.pyplot as plt\n", "from mpl_toolkits.basemap import Basemap\n", "import matplotlib.cm\n", "from matplotlib.patches import Polygon\n", "from matplotlib.collections import PatchCollection\n", "import matplotlib.colors as colors\n", "\n", "import seaborn as sns\n", "\n", "cmap = sns.diverging_palette(220, 15, as_cmap=True)"], "cell_type": "code", "outputs": [], "execution_count": null}, {"metadata": {"_cell_guid": "1011dc53-742e-4e6f-9225-d67393ac73b1", "_kg_hide-input": true, "_uuid": "a2c9565ee83f580187b2ad2529708e15d51cde93", "_kg_hide-output": true}, "source": ["# Reading Data\n", "df_train = pd.read_csv('../input/favorita-grocery-sales-forecasting/train.csv')\n", "df_items = pd.read_csv('../input/favorita-grocery-sales-forecasting/items.csv')\n", "df_stores = pd.read_csv('../input/favorita-grocery-sales-forecasting/stores.csv')\n", "df_train['date'] = pd.to_datetime(df_train['date'])\n", "df_train.set_index('date', inplace=True)"], "cell_type": "code", "outputs": [], "execution_count": null}, {"metadata": {"collapsed": true, "_cell_guid": "a9f3ba80-ef50-4e63-bd93-6931889044b7", "_kg_hide-input": true, "_uuid": "41c7cc8a04699bc3ce64530fbc5c5a709eecb4de"}, "source": ["# To make it more manageable, I'm just taking the data from the current year for the a further analysis\n", "df_2017 = df_train[df_train.index>'2017-1-1']\n", "df_2017 = pd.merge(df_2017, df_items, on='item_nbr', right_index=True)\n", "df_2017 = pd.merge(df_2017, df_stores, on='store_nbr', right_index=True)"], "cell_type": "code", "outputs": [], "execution_count": null}, {"metadata": {"_cell_guid": "31108df7-c53a-4f9f-830a-7ead2d130a91", "_uuid": "6d04aea5a4b161ee28174a794ae301b4134aabee"}, "source": ["## Introduction"], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "859e5b0a-66d2-49ba-a032-b52612b1d1df", "_uuid": "7e94f77ecbd9b0544ea496f9cec8f7772c2df5d0"}, "source": ["This kernel is not intended to perform data analysis from the perspective of its numerical nature, but attempts to get insights and information about the data by observing the Corporaci\u00f3n Favorita business. As the title might suggest, here I'm trying to use all the senses (at least the most part) to create intuition about the data. My only focus is to foment new ideias, discussion and try to create a more rich atmosphere for the Feature Engineering part.\n", "\n", "Because of the size of the training set, I performed a simplification and considered only the portion of the data related to 2017. I thought that this would be the correct approach for two main  reasons:\n", "* The year 2017 has the most complete and largest portfolio of products;\n", "* The most recent year can tell us more about the consumption habits of clients.\n", "\n", "To plot the map of Ecuador, I used the file ecuardor_geojson ([https://www.kaggle.com/victorgrobberio/ecuardorgeojson](http://))."], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "57e30e6b-a7f3-488b-b6c8-d0986a7e9aa5", "_uuid": "ec913d7efb8bc75b542f4f0f23c510479ca151eb"}, "source": ["## Business Analysis"], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "22c4ccfb-ab34-443d-ab4e-50d577118e2f", "_uuid": "296db62d4405603df9d4f216fba4135b1e9ba693"}, "source": ["Check the portfolio and number of stores over time. This is a very interesting proxy for the size of the series. Newer products or products from newer stores are shorter than the oldest ones. This \"lack of information\" might have a great impact to our modeling algorithms.\n", "The major part of products has its beginning dating 2015-2016, which implies that we may have the most part of the series with \"2 years long\"."], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "8add1ee4-42a1-432c-8070-78450eb9a21b", "_kg_hide-input": true, "_uuid": "197de7414bb32cc3a0c23501bfb5f6f43caac81e"}, "source": ["sns.set_style(\"whitegrid\")\n", "\n", "func = lambda df_grouped: len(df_grouped.unique())\n", "df_monthly = df_train.groupby(pd.TimeGrouper('MS')).agg({'item_nbr': func,\n", "                                                         'store_nbr': func})\n", "\n", "ax = df_monthly.plot(subplots=True, layout=(1,2), figsize=(13,5), legend=False, linewidth=3, colormap=cmap)\n", "ax[0][0].set_title(\"Favorita's Portfolio size Evolution\")\n", "ax[0][0].set_ylabel(\"Qtd. of different SKUs\")\n", "ax[0][1].set_title(\"Favorita's Growth\")\n", "ax[0][1].set_ylabel(\"Number of stores\")\n", "plt.show()"], "cell_type": "code", "outputs": [], "execution_count": null}, {"metadata": {"_cell_guid": "dc18c96c-1d2a-4ead-a5db-cca525838efe", "_uuid": "e5b91795752eba0dd51f3f8043dab56e629e4528"}, "source": ["To see the characteristics of sales data, here follows to histograms. The first one indicates the length os the series of each \"product-store\". Its possible to observe that the major part of the \"product-store\" has long series. Actually many product-store have almost 5 years of data records.\n", "\n", "Another import point is how intermitent are the series. This is illustrated by the second histogram. Note that the major part of the series is quite complete (which means that the major part of the \"product-store\" sells everyday and we have this information). There are just a few intermitent series."], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "91b6a4d1-ebaf-44c0-8f6b-b03aa1baa48f", "_kg_hide-input": true, "_uuid": "9ea273341c8f907469d5aaed6efc7de47e07be34"}, "source": ["def func_tag(df):\n", "    try:\n", "        last_day = df.index.max()\n", "        first_day = df.index.min()\n", "\n", "        period_len = (last_day - first_day).days\n", "        fill_rate = len(df) / period_len\n", "\n", "        return pd.Series({'period_len': period_len, 'fill_rate': fill_rate})\n", "    except:\n", "        return pd.Series({'period_len': np.nan, 'fill_rate': np.nan})\n", "    \n", "df_train.drop(['id', 'unit_sales', 'onpromotion'], axis=1, inplace=True)\n", "df_train = df_train.groupby(['store_nbr', 'item_nbr']).apply(func_tag).reset_index()\n", "df_train['fill_rate'] = 100 * df_train['fill_rate']\n", "df_train[df_train['fill_rate']>100] = 100\n", "\n", "sns.set_style(\"whitegrid\")\n", "fig, axes = plt.subplots(nrows=2, figsize=(7, 12), sharex=False)\n", "\n", "df_train[['period_len']].hist(ec='black', ax=axes[0])\n", "axes[0].set_title('Distribution of the series length')\n", "axes[0].set_xlabel('Days')\n", "\n", "df_train[['fill_rate']].hist(ec='black', ax=axes[1])\n", "axes[1].set_title('Distribution of the series fill rate')\n", "axes[1].set_xlabel('Fill rate (%)')\n", "plt.show()\n", "\n", "del df_train"], "cell_type": "code", "outputs": [], "execution_count": null}, {"metadata": {"_cell_guid": "d714af94-0993-4d0d-87dc-eb88a00a6388", "_uuid": "923f01b5f31a1a2418c8c96d560fc726f12ec1ae"}, "source": ["Altought it's a big corporation, the Corporaci\u00f3n Favorita isn't present all over the country. Here we can see it's total unit sales per state. It's mandatory to say that this is a very simplified analysis, because I'm not considering negative sales and I'm using the unit sales ignoring the fact that some products aren't sold in \"units\" but in kilograms (for example). \n", "Because of the sales from the capital (Quito) were much higher than the other cities, I used the logarithm to create a more heterogeneous heatmap."], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "4d611026-3f70-41c1-88fa-d0d36fbd0d4e", "_kg_hide-input": true, "_uuid": "09c819e3a77b73f89e5860e3804e7c248c6aa9af"}, "source": ["sns.set_style(\"white\")\n", "df_grouped_state = df_2017.groupby([pd.TimeGrouper('D'), 'state']).agg({'unit_sales':sum}).reset_index()\n", "\n", "x = df_grouped_state.groupby('state').agg({'unit_sales': sum}).reset_index()\n", "x['state'] = x['state'].apply(lambda state_name: state_name.upper())\n", "\n", "with open(\"../input/ecuardorgeojson/ecuador.geojson\") as json_file:\n", "    json_data = geojson.load(json_file)\n", "\n", "fig, ax = plt.subplots(figsize=(10,9))\n", "\n", "patches = []\n", "for feature in json_data['features']:\n", "    state = feature['properties']['DPA_DESPRO'].upper()\n", "    city = feature['properties']['DPA_DESCAN'].upper()\n", "    poly = Polygon(np.array(feature['geometry']['coordinates'][0][0]), closed=True)\n", "    patches.append({'state': state, 'city': city, 'poly': poly})\n", "df_patches = pd.DataFrame.from_records(patches)\n", "df_patches = pd.merge(df_patches, x, on='state', how='left')\n", "df_patches = df_patches.fillna(1)\n", "\n", "# cmap=matplotlib.cm.RdBu_r\n", "p = PatchCollection(df_patches['poly'])#, cmap=matplotlib.cm.RdBu_r)\n", "\n", "norm = colors.Normalize()\n", "p.set_facecolor(cmap(norm(np.log(df_patches['unit_sales']))))\n", "\n", "mapper = matplotlib.cm.ScalarMappable(norm=norm, cmap=cmap)\n", "mapper.set_array(df_patches['unit_sales'])\n", "plt.colorbar(mapper, shrink=0.4, label=\"2017's Sales Units (log)\")\n", "\n", "ax.add_collection(p)\n", "ax.set_xlim(-81.2, -75)\n", "ax.set_ylim(-5.1, 1.3)\n", "plt.title('Heatmap - Total unit sales (Log Scale) per State', fontsize=14)\n", "plt.tick_params(axis='both', which='both', bottom='off', top='off', labelleft='off', labelbottom='off')\n", "plt.show()"], "cell_type": "code", "outputs": [], "execution_count": null}, {"metadata": {"_cell_guid": "f160d549-b252-4a35-90fe-d523f5506448", "_uuid": "bfec74ae6c8d67a7b335eba86b4831366e265f2c"}, "source": ["Here we can see how the sales was distributed in the states and cities. This might indicate that Corporaci\u00f3n Favorita is trying to compete for the big markets and could have a good market share there. To validate this idea, we have to check the most economic data from the cities from Ecuador."], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "8293428d-013e-4b6d-b86f-5a9e128f2c03", "_kg_hide-input": true, "_uuid": "81e69df7996f2f84af3c0ddbb82c0e7d68b59819"}, "source": ["sns.set_style(\"whitegrid\")\n", "fig, axes = plt.subplots(nrows=2, figsize=(7, 12), sharex=False)\n", "\n", "df = df_grouped_state.pivot(index='date', columns='state', values='unit_sales').sum()\n", "df.sort_values(ascending=True).plot(kind='barh', colormap=cmap, ax=axes[0])\n", "axes[0].set_title('Total sales in 2017 - per State')\n", "plt.xlabel('Total units sold')\n", "\n", "df_grouped_city = df_2017.groupby('city').agg({'unit_sales':sum}).sort_values('unit_sales', ascending=True)\n", "df_grouped_city.plot(kind='barh', colormap=cmap, ax=axes[1], legend=False)\n", "axes[1].set_title('Total sales in 2017 - per City')\n", "plt.xlabel('Total units sold')\n", "plt.show()"], "cell_type": "code", "outputs": [], "execution_count": null}, {"metadata": {"_cell_guid": "2a1817d5-9636-4399-879c-e0f1ebee9e1f", "_uuid": "1d31460fd51d5eeb3d9e3593e438f8b9a6900fa1"}, "source": ["To give us a more intuitive feeling about the sales behavior, I'm comparing the sales behavior of the corporation per state. Two interesting insights come out: \n", "* The corporation isn't growing their market share or improving sales force in 2017. The total sales units have a static behavior all over the year (for the engineers, the sales are in their steady state);\n", "* Visually one might realize that sales have a very interesing seasonality."], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "8636fce5-f339-4384-81a8-31f70ea73daf", "_kg_hide-input": true, "_uuid": "578855d37ad6656bac0460be367b9f0ae17d4bfd"}, "source": ["ax = np.log(df_grouped_state.pivot(index='date', columns='state', values='unit_sales')).plot(figsize=(14,7), colormap=cmap, linewidth=2)\n", "lgd = plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0., fontsize=12)\n", "plt.title('Sales evolution - 2017', fontsize=14)\n", "plt.ylabel('Total units sold')\n", "plt.grid(which='minor')\n", "plt.show()"], "cell_type": "code", "outputs": [], "execution_count": null}, {"metadata": {"_cell_guid": "31a29d90-1c46-4b31-be40-e794f56be464", "_uuid": "7e6cb0df58851dbc3f562dd475852cce9caedccf"}, "source": ["In the next figure we can verify how the observed seasonality affects the sales throughout the week. Here is very clear that:\n", "* Saturday and Sunday are the most important days for the major part of the stores.\n", "\n", "Two questions that we need to answer now are:\n", "* Why is Thursday the less significant day for sales for almost all stores?\n", "* What do the stores with big sales on the beginning of the week (Mon, Tue, Wed) have in common?\n"], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "f4698106-721c-4c6d-8dc1-789b0a7228f9", "_kg_hide-input": true, "_uuid": "9010b9792df3c5d6b6d7d1a840a1dca6b83281ff"}, "source": ["df_2017['weekday'] = [item.weekday() for item in df_2017.index]\n", "df_stores_weekday = df_2017.groupby(['store_nbr', 'weekday']).sum().reset_index()\n", "df_stores_weekday = df_stores_weekday.pivot(index='weekday', columns='store_nbr', values='unit_sales')\n", "\n", "plt.figure(figsize=(15,4))\n", "ax = sns.heatmap(df_stores_weekday.apply(lambda col: (col-min(col))/(max(col)-min(col)), axis=0), \n", "                 cmap=cmap, cbar_kws={'label': 'Normalized Sale'})\n", "ax.set_ylabel('Weekday')\n", "ax.set_xlabel('Store number')\n", "ax.set_yticklabels(['MON', 'TUE', 'WED', 'THU', 'FRI', 'SAT', 'SUN'])\n", "ax.set_title('Sales intensity of each store - per weekday')\n", "plt.yticks(rotation=0)\n", "plt.show()"], "cell_type": "code", "outputs": [], "execution_count": null}, {"metadata": {"_cell_guid": "9b820ee6-2826-4cc7-b33c-0b6402dcfb06", "_uuid": "53f7ffd88ba5708e7ff83949fdd6e470f6814e9b"}, "source": ["I've selected the stores that have the most significant day on Monday, Tuesday or Wednesday. (I've got no insight here)"], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "a61c68ee-b093-424f-b514-804457d56213", "_kg_hide-input": true, "_uuid": "6a139acf6cb1830feb2d413f48664f43db08eaa0"}, "source": ["different_stores = df_stores_weekday.apply(lambda col: col.idxmax() if col.idxmax()<3 else None, axis=0).dropna().index\n", "df_stores[df_stores['store_nbr'].isin(different_stores)].set_index('store_nbr')"], "cell_type": "code", "outputs": [], "execution_count": null}, {"metadata": {"_cell_guid": "339e9365-74fb-436b-9bf3-f82f6281e30f", "_uuid": "eb01f849550d7c53834ddb4f54e2576e276c2111"}, "source": ["The same approach is applied to verify the seasonality of products (in this case, data from 2016 was also used)"], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "9ff6cc4c-c0e0-495d-a7fc-c0e62c11e5f0", "_kg_hide-input": true, "_uuid": "03be51393357e2e3dd2926c3350535ac1075dfda", "_kg_hide-output": false}, "source": ["df_train = pd.read_csv('../input/favorita-grocery-sales-forecasting/train.csv');\n", "df_train['date'] = pd.to_datetime(df_train['date'])\n", "df_train.set_index('date', inplace=True)\n", "\n", "df = df_train[df_train.index>='2016-1-1'].drop(['id', 'store_nbr', 'onpromotion'], axis=1)\n", "df = pd.merge(df, df_items[['item_nbr', 'class']], on='item_nbr', right_index=True).drop(['item_nbr'], axis=1)\n", "df['month'] = [item.month for item in df.index]\n", "df_item_month = df.groupby(['class', 'month']).mean().reset_index()\n", "df_item_month = df_item_month.pivot(index='month', columns='class', values='unit_sales')\n", "\n", "plt.figure(figsize=(10,15))\n", "ax = sns.heatmap(df_item_month.apply(lambda col: (col-min(col))/(max(col)-min(col)), axis=0).T, \n", "                 cmap=cmap, cbar_kws={'label': 'Normalized Sale'})\n", "ax.set_ylabel('Product Class')\n", "ax.set_xlabel('Month')\n", "ax.set_title(\"Sales intensity of each product's class - per month\")\n", "plt.show()"], "cell_type": "code", "outputs": [], "execution_count": null}, {"metadata": {"_cell_guid": "c3718d03-7015-4674-9be1-03604c3992c6", "_uuid": "254a53e71722a53fbbad243493bf91720658dc0c"}, "source": ["Product classes and family market share can show us information about offer and demand. In other words: what does the corporation is good at selling. Observe that \"food\" (this could be used as a feature in the future) represent the most part of the units sold by the corp. Personal care articles might have their importance, but doesn't represent a major share."], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "18652cc3-4e66-4762-979d-df69631fbe2d", "_kg_hide-input": true, "_uuid": "e4812533a1611f586660be16e6354deb24019737"}, "source": ["sns.set_style(\"whitegrid\")\n", "df = df_2017.groupby('family').agg({'unit_sales': sum}).sort_values('unit_sales', ascending=False)\n", "\n", "OTHERS = df.iloc[10:].sum()\n", "df.drop(df.iloc[10:].index.tolist(), inplace=True)\n", "df.loc['OTHERS'] = OTHERS\n", "\n", "ax = df.plot.pie(y='unit_sales', figsize=(6, 6), colormap=cmap)\n", "handles, labels = ax.get_legend_handles_labels()\n", "lgd = ax.legend(handles, labels, bbox_to_anchor=(1.3, 0.8), loc=2, borderaxespad=0., fontsize=12)\n", "plt.ylabel(' ')\n", "plt.title('Market share per product family (2017)', fontsize=14)\n", "plt.show()"], "cell_type": "code", "outputs": [], "execution_count": null}, {"metadata": {"_cell_guid": "10e02337-632f-4939-8c01-c18e3d897f74", "_kg_hide-input": true, "_uuid": "3fdb7a3dd94a3fe9d86088d6401d79953e0b5ecc"}, "source": ["sns.set_style(\"whitegrid\")\n", "df = df_2017.groupby('class').agg({'unit_sales': sum}).sort_values('unit_sales', ascending=False)\n", "\n", "OTHERS = df.iloc[10:].sum()\n", "df.drop(df.iloc[10:].index.tolist(), inplace=True)\n", "df.loc['OTHERS'] = OTHERS\n", "\n", "ax = df.plot.pie(y='unit_sales', figsize=(6, 6), colormap=cmap)\n", "handles, labels = ax.get_legend_handles_labels()\n", "lgd = ax.legend(handles, labels, bbox_to_anchor=(1.3, 0.8), loc=2, borderaxespad=0., fontsize=12)\n", "plt.ylabel(' ')\n", "plt.title('Market share per product class (2017)', fontsize=14)\n", "plt.show()"], "cell_type": "code", "outputs": [], "execution_count": null}, {"metadata": {"_cell_guid": "ef8a7157-3bc5-4865-956c-620b43a8949f", "_uuid": "e6f4f15a60eeea6fa453ab2e25f249920f7d25ee"}, "source": ["To check the influence of promotion actions on sales, I've first normalized the sales data for each \"product-store\", fitting the values on the [0, 1] range (no negative value was considered here). For this analysis I've also considered just the \"product-store\"s that were on promotion at some moment."], "cell_type": "markdown"}, {"metadata": {"_cell_guid": "a5c80983-c720-488a-a7dc-97a7ee783d6e", "_kg_hide-input": true, "_uuid": "1c7e78be7a7945267ae44d3bf4eb6692ce7ab9c3", "_kg_hide-output": false}, "source": ["func_norm = lambda df: (df['unit_sales'] - df['unit_sales'].min())/(df['unit_sales'].max() - df['unit_sales'].min())\n", "df_2017 = df_2017[['unit_sales', 'item_nbr', 'store_nbr', 'onpromotion']].set_index(['onpromotion'])\n", "df_2017 = df_2017[df_2017['unit_sales']>0] # Take just positive sales\n", "df_2017 =  df_2017.groupby(['store_nbr', 'item_nbr']).apply(func_norm).reset_index() # Normalize sales item-store\n", "\n", "def func(df):\n", "    if len(df['onpromotion'].unique())==2:\n", "        return df\n", "    else:\n", "        pass\n", "    \n", "teste = df_2017.groupby(['item_nbr', 'store_nbr']).apply(func).dropna()\n", "\n", "sns.boxplot(x=\"onpromotion\", y=\"unit_sales\", data=teste)\n", "plt.ylabel('unit_sales normalized')\n", "plt.show()"], "cell_type": "code", "outputs": [], "execution_count": null}], "nbformat": 4, "nbformat_minor": 1}