{"metadata": {"language_info": {"version": "3.6.3", "codemirror_mode": {"version": 3, "name": "ipython"}, "mimetype": "text/x-python", "nbconvert_exporter": "python", "file_extension": ".py", "pygments_lexer": "ipython3", "name": "python"}, "kernelspec": {"language": "python", "name": "python3", "display_name": "Python 3"}}, "nbformat": 4, "cells": [{"cell_type": "markdown", "metadata": {"_cell_guid": "20852b3b-d8b5-43e6-9643-531e8589f6d2", "_uuid": "22a30a2394d50c42a92a254646ba9c0c9d0b53f2"}, "source": ["# Introduction\n", "In this exploratory data analysis, I plan on analyzing the various data sets provided to us. I started with the training and test sets, sample submission, and store, item, and transaction metadata but plan on including the other data sets as well at a later point. Below, I will investigate the distribution of observations with respect to different features. I also inspected the distribution of the target variable, which is unit sales, for outliers. I found some interesting differences between the distribution of observations in the training and tests sets that are worth considering before training models. I also found that, within the store metadata, that store cluster is a subdivision of store type and that one state can have many different store types. Please upvote if you find this useful! Enjoy."]}, {"cell_type": "code", "metadata": {"_cell_guid": "c7559664-49d4-42ec-a869-ac99e106d344", "_uuid": "e0d10f96858d10b74e3ca6dee4255f9a671e6414", "collapsed": true}, "execution_count": null, "source": ["# import necessary modules\n", "import numpy as np\n", "import pandas as pd\n", "import seaborn as sns\n", "import matplotlib.pyplot as plt\n", "import gc"], "outputs": []}, {"cell_type": "code", "metadata": {"_cell_guid": "b24626c7-2572-4cd9-907a-c92e07e58b4c", "_uuid": "dde3bd832d87db3c4549e232c47fe414f7067728"}, "execution_count": null, "source": ["# read data sets (this may take a few minutes)\n", "df_train = pd.read_csv(\"../input/train.csv\")\n", "df_test = pd.read_csv(\"../input/test.csv\")\n", "df_sub = pd.read_csv(\"../input/sample_submission.csv\")\n", "df_stores = pd.read_csv(\"../input/stores.csv\")\n", "df_items = pd.read_csv(\"../input/items.csv\")\n", "df_trans = pd.read_csv(\"../input/transactions.csv\")\n", "df_oil = pd.read_csv(\"../input/oil.csv\")\n", "df_holiday = pd.read_csv(\"../input/holidays_events.csv\")"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "36ef8bd6-3d8b-4cd1-a1ab-c5c6e570fba7", "_uuid": "81e200a78c31959804c229a8671e38412cc6565e"}, "source": ["# Training set"]}, {"cell_type": "code", "metadata": {"_cell_guid": "f6743ca5-45ef-41b0-904a-7f1720db60eb", "_uuid": "3e7d062850b8edaa808593f32677209748649bf0"}, "execution_count": null, "source": ["# inspect training set\n", "print(df_train.shape)\n", "df_train.head()"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "e5c60fa1-f36f-4dec-8fbf-0f2693c73100", "_uuid": "edb9b8a51f0723041c7a10b79c5175f08b5e431c"}, "source": ["The training set has 125,497,040 rows and 6 columns: row id, date, store number, item number, unit sales (keep in mind that this can be an integer, float, or -1, which represents a returned item), and whether there was a promotion for a particular item. "]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "f9ab0024-2ba1-4599-b9fa-6fbfd6be8e1f", "_uuid": "ba77d545047b8fb67791d59195d0db4df72f3602"}, "source": ["## Dates\n", "\n", "Now, let's get a sense of the time range for which the data was collected."]}, {"cell_type": "code", "metadata": {"_cell_guid": "1a2e5bb9-7b8e-4d79-be60-8c93a64f2953", "_uuid": "b3c5c9250a057ab25c54ba329aaba3239165abef", "collapsed": true}, "execution_count": null, "source": ["# convert date to datetime\n", "df_train[\"date\"] =  pd.to_datetime(df_train[\"date\"])"], "outputs": []}, {"cell_type": "code", "metadata": {"_cell_guid": "68e27f2f-e3da-4d6e-99f5-af36fb329d2a", "_uuid": "1d7c5af3ad97eb77c11e81aa1fac8a76e7fbcff5"}, "execution_count": null, "source": ["df_train[\"date\"].dt.year.value_counts(sort = False).plot.bar()"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "a67676b1-d145-4dc7-8d30-2618002ca4eb", "_uuid": "a5e919ccdde5cc6f00b31fd6d8d059356a0ac8a3"}, "source": ["They have collected data from 2013 to 2017. There is an increase in the number of observations for each year except 2017 but this is probably because it is not yet over. Note that the training set is quite large so I will focus my analysis hereafter on 2016 data."]}, {"cell_type": "code", "metadata": {"_cell_guid": "0489bea8-ee16-4851-95bc-abe5c096ec3f", "_uuid": "4ce72ae0363eee58967d20ed9a8374c3a1469dd3", "_kg_hide-output": true}, "execution_count": null, "source": ["df_train_2016 = df_train[df_train[\"date\"].dt.year == 2016]\n", "del df_train; gc.collect() # free up some memory"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "8eb0fce6-e323-4d0f-8a00-48abe2f42d6a", "_uuid": "4ef27b6883b7892454d43a8fdf185989af153016"}, "source": ["Let's take a look at how the data is distributed by month."]}, {"cell_type": "code", "metadata": {"_cell_guid": "e193f5a0-61d0-40eb-80b5-f7fe03f00798", "_uuid": "d44ae64327a3f450258c8cb1b282a92e7ed06650"}, "execution_count": null, "source": ["df_train_2016[\"date\"].dt.month.value_counts(sort = False).plot.bar()"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "76046412-9f09-464f-aebf-17b5b061c891", "_uuid": "c00fd16619a46f7694c15e89e3ea8945d3d502c5"}, "source": ["The observations are almost uniformly distributed by month. The maximum occurs in December and the minimum in February. How about by day of the month?"]}, {"cell_type": "code", "metadata": {"_cell_guid": "994bd093-25b4-4d07-beca-cc4bdf5eecca", "_uuid": "0e95b794bfb5d8aa6c4e97989118d1c825a7f1b3"}, "execution_count": null, "source": ["df_train_2016[\"date\"].dt.day.value_counts(sort = False).plot.bar()"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "bc9bb697-ad38-4bc6-8498-5b911925e727", "_uuid": "780b0d57a24d738a0f77b4ea083e86cd648699fc", "collapsed": true}, "source": ["Again, the observations are almost uniformly distributed by day.\n", "\n", "## Stores\n", "Now, let's determine how many stores there are and the distribution of observations for each store."]}, {"cell_type": "code", "metadata": {"_cell_guid": "9012e384-5e4d-4f1d-ba62-50516ac64ee5", "_uuid": "f5a92cd9103f50fd5151d9efb562343d6c5511df"}, "execution_count": null, "source": ["df_train_2016[\"store_nbr\"].unique()"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "56dd5c3d-29fb-48aa-a718-ceb8ed06e51c", "_uuid": "a9f6eed8000f760258fc9d38efb55fdda65780a9"}, "source": ["There is data for 54 stores in 2016. At a later point, I will analyze the stores data also provided. Now, how about the distibution of observations for each store?"]}, {"cell_type": "code", "metadata": {"_cell_guid": "f2af3723-b3c2-4f13-a870-1cf976de0e32", "_uuid": "752a6b70758ee7386f7d30eb8e603fb659c45e35"}, "execution_count": null, "source": ["df_train_2016[\"store_nbr\"].value_counts(sort = False).plot.bar()"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "d8a9f283-7eed-4517-9fb5-70ed579bbb86", "_uuid": "96f96bda0d29e4ac3c3b8650d425a59e9919b2c1"}, "source": ["The interpretation of this plot is as follows. The y-axis is the number of observations corresponding to a particular store. A store with more observations does not necessarily outperform a store with fewer observations. This is because some stores may offer a wider range of products but push less volume. This would inflate their numbers in this plot. We can interpret the y-axis as a measure of the store's variety.\n", "\n", "## Items\n", "Now, let's determine how many unique items there are overall and in each store."]}, {"cell_type": "code", "metadata": {"_cell_guid": "4b8ed10f-d6c6-4456-b1a8-d9e804ab735c", "_uuid": "53af816b1a638d3bca0e2c9cabeb145cfb78a6cd"}, "execution_count": null, "source": ["df_train_2016[\"item_nbr\"].unique().shape[0]"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "535fe8d4-d859-4536-88ec-231d0b00f856", "_uuid": "25e186ae857824c606192c6720f8e9811c958062"}, "source": ["There were 3886 different types of items either sold or returned during 2016."]}, {"cell_type": "code", "metadata": {"_cell_guid": "eea8dc4f-bb52-4b7a-b685-fc19dc584f06", "_uuid": "70b18c63533a953929823137a5e9d4c113c8cf46"}, "execution_count": null, "source": ["stores = np.arange(1, 55)\n", "items_store = np.zeros((54, ))\n", "for i, store in enumerate(stores) :\n", "    items_store[i] = df_train_2016[\"item_nbr\"][df_train_2016[\"store_nbr\"] \\\n", "                                               == store].unique().shape[0]\n", "sns.barplot(stores, items_store)"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "2ce3c2b3-6269-47c2-aee0-473909d22d93", "_uuid": "194662ae3b08e509756533ed386dbc09c95fef67"}, "source": ["As expected, this is very similar to the last bar plot because it measures the variety of items in each store. Interestingly, store 52 has 0 unique items. This is because there is no store number 52.\n", "\n", "## Item sales\n", "This is our target variable so it is very important to get a sense of its distribution."]}, {"cell_type": "code", "metadata": {"_cell_guid": "b27b2655-d4cb-42d1-be8e-81a39a3bcfaa", "_uuid": "b30bdb7eba55227c2539a054afd4cf05153e73ab"}, "execution_count": null, "source": ["df_train_2016[\"unit_sales\"].describe()"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "ca978544-1226-4c57-840f-7a3e15a23a00", "_uuid": "6f5a43d1cac45bc945c779c408348f142d4188ac"}, "source": ["It is probably a good thing that the mean and median for unit sales is positive, otherwise the company would be losing money. Suprisingly, on one day, 4673 items were returned. I wonder if this corresponds to some sort of outbreak or health concern for a particular product. On the other hand, on another day, 89440 items were purchased. Perhaps this was before some sort of natural disaster (*e.g.* a hurricane).\n", "\n", "## Promotions\n", "Now, let's find out how many items were purchased by coupon clippers."]}, {"cell_type": "code", "metadata": {"_cell_guid": "23314911-26d0-401d-b768-33124b612aea", "_uuid": "07b5525ba59fad3c0a012da603785682002a518b"}, "execution_count": null, "source": ["df_train_2016[\"onpromotion\"].value_counts()"], "outputs": []}, {"cell_type": "code", "metadata": {"_cell_guid": "ca5e7d8c-ed09-4e46-a2eb-9920c31c50e1", "_uuid": "3ac930e577b5a0ee1857a2380984346ee0ce5b18"}, "execution_count": null, "source": ["3514584 / (3514584 + 31715287) * 100"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "06879ae6-2b0f-4a98-bd1a-3393aa37e43d", "_uuid": "9d88bdc083b79e8e7058181f19eb7f91c5578810"}, "source": ["About 10% of items are purchased on promotion.\n", "\n", "## Missing data and outliers\n", "A less exciting but very important step in this analysis is to determine if any data is missing and if there are any outliers in the target data."]}, {"cell_type": "code", "metadata": {"_cell_guid": "974688bd-f5a3-4112-ba57-882d67db006e", "_uuid": "80f90d9f7d65cf9e019ea809f894492263d73dcc"}, "execution_count": null, "source": ["df_train_2016.isnull().sum()"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "5a2b565a-92a7-4bf3-8034-3b11dd52f26c", "_uuid": "da398c1a6df29ee8fe3bff8f6ffddb61298225b5"}, "source": ["Yay! There is no missing data in the training set. How about outliers in the target variable?"]}, {"cell_type": "code", "metadata": {"_cell_guid": "e6f3c430-2674-4972-adb1-f9f1d70b847b", "_uuid": "892962b25c99d2a117fc210e319d96e4997af96c", "_kg_hide-output": true}, "execution_count": null, "source": ["unit_sales = df_train_2016[\"unit_sales\"].values\n", "del df_train_2016; gc.collect()"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "34896988-99d4-4575-b47a-b9a564af7532", "_uuid": "f9cac7e699b17049b86aa0f5c67a219d111ac82f"}, "source": ["Again, I had to remove some data to free up memory for plots."]}, {"cell_type": "code", "metadata": {"_cell_guid": "39cc4e83-c6bd-4c8d-b1b8-a27f8715738e", "_uuid": "cb80fb71662895dd3078e1895f2117018ffdb6fa"}, "execution_count": null, "source": ["plt.scatter(x = range(unit_sales.shape[0]), y = np.sort(unit_sales))"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "a60732e6-05c0-40bc-b6a8-dd26ded7cb74", "_uuid": "5e8195ac6f3f6398f4a4cd361ea8672166f732be"}, "source": ["This plot isn't super informative but does show us that quite a few outliers exist on the sales side."]}, {"cell_type": "code", "metadata": {"_cell_guid": "efc7ec2b-2795-4c39-947e-fa60c225456d", "_uuid": "c0431b38f7c1cb549eb76730a012b3bac3793568", "_kg_hide-output": true}, "execution_count": null, "source": ["del unit_sales; gc.collect()"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "79ef4b67-fbe2-47c3-a3cd-d66b1dd3a606", "_uuid": "01a3d5b725ee19fea13edb05c6cc1ae4cf264644"}, "source": ["# Test set"]}, {"cell_type": "code", "metadata": {"_cell_guid": "9624d401-8066-4e10-b042-672334565814", "_uuid": "c7a13ed3e313776d9ff0b73b78d4baacb15e1b77"}, "execution_count": null, "source": ["# inspect test set\n", "print(df_test.shape)\n", "df_test.head()"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "fa717c47-e4db-4c4d-b2f6-a58c42f828c5", "_uuid": "ece017c8bfccaef2b477ca99e88219a7115479f1"}, "source": ["There are 3,370,464 rows in the test set, which is approximately 1.5 orders of magnitude smaller than the training set. Since the training set is so large, we can afford to reserve more of this data for validation, which will promote more robust predictions. The columns in the test set are similar to those in the training set except the unit sales column is missing. Now, let's determine the range of dates present in the test set and compare to the training set."]}, {"cell_type": "code", "metadata": {"_cell_guid": "c857b956-bfe1-4745-97dd-a7536192889d", "_uuid": "68e07dc7872522b35da5fc122b51f9799fd8e4df"}, "execution_count": null, "source": ["# convert date to datetime\n", "df_test[\"date\"] =  pd.to_datetime(df_test[\"date\"])\n", "df_test[\"date\"].dt.year.value_counts(sort = False).plot.bar()"], "outputs": []}, {"cell_type": "code", "metadata": {"_cell_guid": "1801636c-1306-4265-a6ba-7f42812d42fe", "_uuid": "101b71f8544c164815fd1618ad187c5eeb605495"}, "execution_count": null, "source": ["df_test[\"date\"].dt.month.value_counts(sort = False).plot.bar()"], "outputs": []}, {"cell_type": "code", "metadata": {"_cell_guid": "de3a1422-00d2-41a4-8a6f-a9349f7726e4", "_uuid": "ae147b11235965a4c01d4415979256f95f1d2ea8"}, "execution_count": null, "source": ["df_test[\"date\"].dt.day.value_counts(sort = False).plot.bar()"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "aff62df7-a808-4dc7-ab1c-f16bc5034591", "_uuid": "3864d3f50ea5dbbdee41916fa7116037a0ffdbcd"}, "source": ["The test set only samples observations from July 16-31, 2017.\n", "\n", "## Stores\n", "Now, let's see if the same store numbers appear in the test set as do in the training set."]}, {"cell_type": "code", "metadata": {"_cell_guid": "28bfbc31-4f4d-427b-89e5-5f6b67bfc4b1", "_uuid": "05232b72666119d9e9126b7801ce5bab0c1b7160"}, "execution_count": null, "source": ["df_test[\"store_nbr\"].value_counts(sort = False).plot.bar()"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "d65d607c-d1b9-4754-b467-23aabc5266ef", "_uuid": "476f85450d063e21d514490c1ca64d6a994fe2bc"}, "source": ["The stores are sampled uniformly and, unlike the training set, store 52 has observations.\n", "\n", "## Items\n", "It's also useful to know how many unique items appear in the test set."]}, {"cell_type": "code", "metadata": {"_cell_guid": "8f963c18-4215-4aab-92f6-80992d3f5873", "_uuid": "a5e6b354326c5d70c2233769c15d4b52c6a6c94b"}, "execution_count": null, "source": ["df_test[\"item_nbr\"].unique().shape[0]"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "16f5bb5d-e66b-476f-85b1-af10f277ff87", "_uuid": "794c4a5340ecda8f09e14273397cc9b7a6fe9906"}, "source": ["There are actually more unique items in the test set (3901) than in the training set (3886). This makes sense because the supermarket adapts over times to meet the diverse needs of its customers."]}, {"cell_type": "code", "metadata": {"_cell_guid": "d9669752-cc2e-40fa-b6fb-801c8c16b6f3", "_uuid": "698cdf31f4b234a239ef3f31e29e7c496b2df850"}, "execution_count": null, "source": ["stores = np.arange(1, 55)\n", "items_store = np.zeros((54, ))\n", "for i, store in enumerate(stores) :\n", "    items_store[i] = df_test[\"item_nbr\"][df_test[\"store_nbr\"] \\\n", "                                         == store].unique().shape[0]\n", "sns.barplot(stores, items_store)"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "b65217ae-8129-47f8-b7d0-cc3ec8ba8d4c", "_uuid": "5142aaad0a86489c6615b5ca334934cc76eeabef"}, "source": ["For the test set, the number of unique items in each store (a measure of that stores variety) are exactly the same. This is suprising because we would expect that each store caters to slightly different communities and their needs and therefore the variety should not be uniform. However, it is also possible that their itention was to provide a test set with balanced observations.\n", "\n", "## Promotions\n", "Last for the test set, let's see if the percentage of items on sale matches that of the training set."]}, {"cell_type": "code", "metadata": {"_cell_guid": "1b25f589-c9d6-4f54-9988-3cf94d21a02d", "_uuid": "10401a8edb78c5f4f0d09818e6b1d313fd3873fa"}, "execution_count": null, "source": ["df_test[\"onpromotion\"].value_counts()"], "outputs": []}, {"cell_type": "code", "metadata": {"_cell_guid": "97daedce-091e-4957-80ca-01eb148bd2bd", "_uuid": "55ea85beae4f6a98f23a3e4667083abfbeda641e"}, "execution_count": null, "source": ["198597/(198597 + 3171867) * 100"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "dcab9bb1-d4f3-43ca-b7e0-67a3f0feafa3", "_uuid": "da85b1277a6d05f10a6642083ebdfad39926054d"}, "source": ["The percentage of items on sale for the test set is a little more than half of that for the training set ($\\approx$10%). From this analysis of the test set, we find that while it has a similar structure to that of the training set, it's observations correspond to a time period not sampled by the training set. Additionally, the number of unique items, each stores' variety of items, and the number of observations corresponding to promotional purchases are different than the training set. Therefore, it may be beneficial to sample the training set in a way that more closely matches the distribution of items in the test set."]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "348dfbf5-3512-451e-97dc-e8a10fe060b0", "_uuid": "e4c9f0af98017053078edec4ffed299cccbef82b"}, "source": ["# Sample submission\n", "Next, let's take a look at what they would like us to submit."]}, {"cell_type": "code", "metadata": {"_cell_guid": "36f37433-8fdc-40a5-b624-658f37de169a", "_uuid": "b7db580ea105f05bfc38136627ac75fa9c6157bf"}, "execution_count": null, "source": ["print(df_sub.shape)\n", "df_sub.head()"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "6789637d-690a-42df-89f3-c26dd134bfe1", "_uuid": "e1a460efdb60c3e9d3f5fb47c00ed8622029bd21"}, "source": ["Just like the test set, there are 3,370,464 rows in the sample submission. The submission file has two columns, the row id and the unit sales. Let's see if the row ids in the sample submission match that of the test set."]}, {"cell_type": "code", "metadata": {"_cell_guid": "280d40b3-7429-4e90-a7d9-95ebcddae2f1", "_uuid": "dd0ce9c5b3f35ebad6020d23c4f9eb59b8316535"}, "execution_count": null, "source": ["(df_sub[\"id\"] - df_test[\"id\"]).sum()"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "0c481bb8-9f78-410d-80a3-3a8507f195d3", "_uuid": "30bf155f44b8698e6d2c9391a6bf4d7b6f758ead"}, "source": ["Yes, they do! So, at the very least, our task boils down to predicting the number of unit sales given the date, store number, item number, and promotional status. But we are also provided with other information about the stores, items, transactions, economic health, and holidays. Let's clear some memory before looking at this metadata."]}, {"cell_type": "code", "metadata": {"_cell_guid": "988cf73e-a290-4c71-8076-5c8eccfcea21", "_uuid": "44e305ae8e76e4ea79e01b9019bd84d1de0d69ac", "_kg_hide-output": true}, "execution_count": null, "source": ["del df_test, df_sub; gc.collect()"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "374249f5-67dc-4fa1-8ea7-83b987969028", "_uuid": "ef24011d3d72ee2af5d7050f350e9bc1d306eb6a"}, "source": ["# Stores\n", "First, let's inspect the contents of the data set."]}, {"cell_type": "code", "metadata": {"_cell_guid": "f36110af-30cd-4f9f-a595-c9f397249f9a", "_uuid": "a28750141667edd69aba9d0866d956deae4f2ea1"}, "execution_count": null, "source": ["print(df_stores.shape)\n", "df_stores.head()"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "ea0797a5-4dd0-4b12-a27d-483ee4de68c6", "_uuid": "5fa58813c6ae81b8667d99114c5777e304992367"}, "source": ["Store metadata includes the store number, the city in which the store is located, the state in which the city is located, type, and cluster. A cluster is a similar grouping of stores perhaps in the type of products they sell, their size, or their locations. It is not clear how the different types are defined or how they differ from clusters. Are there any missing values in the stores data set?"]}, {"cell_type": "code", "metadata": {"_cell_guid": "10cc39a2-5e59-4fd2-b93f-c73d147fc62b", "_uuid": "36f309e130cf9a1ed9fa18b49b619d7e30565469"}, "execution_count": null, "source": ["df_stores.isnull().sum()"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "db3849be-5954-4548-9ed3-59a33e207c26", "_uuid": "05d7b5809fc29222b9bd1791c7f42d9adfa20551"}, "source": ["Nope!\n", "\n", "## Cities"]}, {"cell_type": "code", "metadata": {"_cell_guid": "4efc264c-33e2-415f-a68f-ff76ec865fb3", "_uuid": "1d38f5340a17e667c47ba74b2b7c5b56c9192ee3"}, "execution_count": null, "source": ["df_stores[\"city\"].unique().shape[0]"], "outputs": []}, {"cell_type": "code", "metadata": {"_cell_guid": "c868563e-0d75-45ea-a495-da9e54a2d779", "_uuid": "6979d22de7f636b78009aa28f4681403298bd47d"}, "execution_count": null, "source": ["df_stores[\"city\"].value_counts(sort = False).plot.bar()"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "ea32a11f-4d22-410f-9676-1a45f39ba508", "_uuid": "f8849ba26e17287f8789047b5994ef27ce4dcf35"}, "source": ["The 54 stores are located in 22 different cities. The city with the most stores is Quinto and there are a number of cities that only have one store."]}, {"cell_type": "markdown", "metadata": {"_cell_guid": "56e0ddec-377d-49ec-8e95-5380b1832f15", "_uuid": "3c39d79ddaad5caaf7bd07d5b5e17d154ff108ea"}, "source": ["## State"]}, {"cell_type": "code", "metadata": {"_cell_guid": "d32e1a8a-c880-46e3-bb68-d87e9a7dd59f", "_uuid": "c9c41a3e1dbf3804175c2ac200f596bf20741a66"}, "execution_count": null, "source": ["print(df_stores[\"state\"].unique().shape[0])\n", "df_stores[\"state\"].value_counts(sort = False).plot.bar()"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "38273333-25a9-43af-89d0-76fa13d28755", "_uuid": "397928aa9698783bb3858d3dfa400b1f73ac7545"}, "source": ["Of the 24 provinces in Ecuador (https://en.wikipedia.org/wiki/Provinces_of_Ecuador), 16 of them have supermarkets.\n", "\n", "## Type and cluster\n", "Since we don't know what type means and the cluster definition is a little bit vague, let's compare them."]}, {"cell_type": "code", "metadata": {"_cell_guid": "00031db6-a5ff-4129-9c32-7856aaf2860f", "_uuid": "260e4010cb7e926840bb508a829b19efdcda9e35"}, "execution_count": null, "source": ["print(df_stores[\"type\"].unique().shape[0])\n", "df_stores[\"type\"].value_counts(sort = False).plot.bar()"], "outputs": []}, {"cell_type": "code", "metadata": {"_cell_guid": "c4746b3c-e2c5-4da7-8eea-6bc0fb2d6c01", "_uuid": "5ae06eae3130a3f5fd12235f9c13f2ddfb4ead6e"}, "execution_count": null, "source": ["print(df_stores[\"cluster\"].unique().shape[0])\n", "df_stores[\"cluster\"].value_counts(sort = False).plot.bar()"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "901670d0-d7af-430a-a121-08cf41779b2c", "_uuid": "03dc9ab53e550dc0fa24d70747d72e2c5dcbece2"}, "source": ["From these plots, we can see a few differences between type and cluster. First, there are fewer types (5, A through E) than clusters (1-17). Second, for the most part, there are more stores per type than stores per cluster. Let's see how each cluster is represented by type."]}, {"cell_type": "code", "metadata": {"_cell_guid": "68564728-2308-4bc1-8fe4-c47a99f93053", "_uuid": "505dc8dad229fa66342b2e1589621bde35178e62"}, "execution_count": null, "source": ["df_stores.groupby([\"type\", \"cluster\"]).size()"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "1226fbcc-c6ed-4524-9074-98f74604f6a9", "_uuid": "5fb7d3e8cc28d7cf61087a67b40c068785ee7b85"}, "source": ["For the most part, stores in a particular cluster have one type. However, there is one exception. Cluster 10 can be type B, D, and E. How about state by type?"]}, {"cell_type": "code", "metadata": {"_cell_guid": "9e87ac2e-708f-47b4-9ea3-b602e9f81129", "_uuid": "333ac24aa75d1de4e6bf0a21dc026e0ffc73630b"}, "execution_count": null, "source": ["df_stores.groupby([\"type\", \"state\"]).size()"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "b6772377-f289-4810-8be2-8d3abc481971", "_uuid": "d47d58f5e46e3930747118a854eb535045fd5f68"}, "source": ["There are many states that have different types of stores. For example, Guayas has stores of type A, B, D, and E. This tells us that clusters are, for the most part, a subdivision of store type."]}, {"cell_type": "code", "metadata": {"_cell_guid": "1b225d08-251c-468f-bb4a-0dab3f1f4253", "_uuid": "370c3bed3f3033e4ea1c6ae7df105ec68d97ce40"}, "execution_count": null, "source": ["del df_stores; gc.collect()"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "2185558e-c35c-456d-a36a-78ef831324dc", "_uuid": "3d7634327016bd4693372343da3e4b9473a2a9dc"}, "source": ["# Items\n", "Now let's take a look at the items metadata."]}, {"cell_type": "code", "metadata": {"_cell_guid": "ea01134f-2aaa-46c3-97ef-d64c5a2065d6", "_uuid": "28fc8c72e58fb179276f537f315d1dff4e07dd19"}, "execution_count": null, "source": ["print(df_items.shape)\n", "df_items.head()"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "f289f454-2a55-4ad8-80a5-0c89dbe5ff10", "_uuid": "c7ea5354e37dcd03051c4bfccac5c7197708daee"}, "source": ["The items data has 4100 rows and 4 columns: item number, family, class, and whether or not the item is perishable. Now, let's see how many different item families there are."]}, {"cell_type": "code", "metadata": {"_cell_guid": "084479cf-0e6c-4d39-8de9-93e29c9fe8d2", "_uuid": "974fe139f1af6a2e59401244a54bbab56db3d15b"}, "execution_count": null, "source": ["print(df_items[\"family\"].unique().shape[0])\n", "df_items[\"family\"].value_counts(sort = False).plot.bar()"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "5c9d6443-8fdb-4726-8b36-813580df21a8", "_uuid": "894b25471eeb09e19833604f329185faf6bee05f"}, "source": ["There are 33 differenet item families ranging from school and office supplies to poulty. The most popular family is called \"grocery 1\". This must be a generic groceries column. How about item classes?"]}, {"cell_type": "code", "metadata": {"_cell_guid": "913d265a-0208-4363-8b5a-97b5e8cf22ce", "_uuid": "fb997f44912b8120e4c5bf489be1ba9ec79afaa5"}, "execution_count": null, "source": ["print(df_items[\"class\"].unique().shape[0])\n", "print(df_items[\"class\"].value_counts()[0:5])\n", "df_items[\"class\"].plot.hist(bins = 50)"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "462a0646-d2cb-4e2d-a937-e8b13bf73377", "_uuid": "fa41ef73adef17bce134e1217a5fc3eef4306324"}, "source": ["There are 337 different item classes and the number of items in each class is plotted above. There is an abundance of items in classes 1016 (133), 1040 (110), 1124 (100), 1034 (98), and 1122 (81). Finally, let's find out the percentage of items that are perishable."]}, {"cell_type": "code", "metadata": {"_cell_guid": "ac359a03-03c2-49bb-96dd-c62f8d8a35bc", "_uuid": "1f1612bb86cf46f50e614d003b247fa97def0a98"}, "execution_count": null, "source": ["df_items[\"perishable\"].value_counts()"], "outputs": []}, {"cell_type": "code", "metadata": {"_cell_guid": "2bede46a-0542-4534-99a3-22b25a9cec98", "_uuid": "e2061b83bc010a32e98628ba379d0527d0f35119"}, "execution_count": null, "source": ["986 / (986 + 3114) * 100"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "e9fff3d0-5916-4eef-bd4f-d7e194f91f4c", "_uuid": "445178dce9681f150d8cbfcde70cf3024253a182"}, "source": ["24% of the of the items are perishable."]}, {"cell_type": "code", "metadata": {"_cell_guid": "d2288a9b-57f0-4412-b7bc-c66eec4cc17c", "_uuid": "268e7bf5398c991f43d2da07601fd7f0cfb8ba93"}, "execution_count": null, "source": ["del df_items; gc.collect()"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "36253073-ab0a-4fba-b3f9-5df120d8b646", "_uuid": "1051821c4fcc3c275b719397795a17a4bfade08b"}, "source": ["# Transactions\n", "We're getting there only a few more data sets to analyze. Next up is the transactions metadata."]}, {"cell_type": "code", "metadata": {"_cell_guid": "7014492b-c959-4196-8756-0c289dc3eec3", "_uuid": "9ad530f8c9e4f1ec6d9e5b18c1063c654faa8712"}, "execution_count": null, "source": ["print(df_trans.shape)\n", "print(df_trans.head())\n", "df_trans.isnull().sum()"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "90df9354-025d-402d-a8dc-755d66f88635", "_uuid": "a987f6cac840d5e24821e508667b0e56286e4ea1"}, "source": ["There are 83488 columns in the transactions data and three columns: date, store number, and the number of transactions. This data tells us how many transactions were made in each store on each business day since they started collecting data. There is also no missing data."]}, {"cell_type": "code", "metadata": {"_cell_guid": "ad65ce86-2b49-4b37-a2c8-88f5f595e955", "_uuid": "f70e30a9acfe5a641e85ee0adae9ea648a765b10", "collapsed": true}, "execution_count": null, "source": ["df_trans[\"date\"] =  pd.to_datetime(df_trans[\"date\"])"], "outputs": []}, {"cell_type": "code", "metadata": {"_cell_guid": "c5bd0ea4-d021-4830-885f-430ba0067bf3", "_uuid": "de17ac2d8422e583a98d646947c15ce02164abaf"}, "execution_count": null, "source": ["df_trans[\"date\"].dt.year.value_counts(sort = False).plot.bar()"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "ad62e924-24c1-431f-b82c-e077f39b44a6", "_uuid": "caefdee18d5f195a18c88771d199fa34e05b7e6c"}, "source": ["There is data from 2013 to 2017 with an increase in the number of observations each year except for 2017, which is not yet complete. This is similar to the training data. This tells us that the number of stores is increasing each year and perhaps they are now open on more days."]}, {"cell_type": "code", "metadata": {"_cell_guid": "8ed71bc5-4b56-4be5-9869-8f902df8a05f", "_uuid": "fc2445c0bbdc03962bcbc385abb01bef3445a1a0"}, "execution_count": null, "source": ["df_trans[\"date\"].dt.month.value_counts(sort = False).plot.bar()"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "392295cc-11d6-42e8-9c51-f2b36b84c525", "_uuid": "ad2b378d6a8c356f4d093dc3f000beef1bcc8282"}, "source": ["In general, there are more observations for January through July than the colder months. I'm not sure why this is but perhaps there are more store closures during this month for holiday or weather reasons."]}, {"cell_type": "code", "metadata": {"_cell_guid": "ddd0099f-c4cf-4ef9-bc7e-5de892d11355", "_uuid": "fc57bd86e9a7532249da65d242ffe3c5ac7cbb2d"}, "execution_count": null, "source": ["df_trans[\"date\"].dt.day.value_counts(sort = False).plot.bar()"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "88a98142-728b-47a3-b252-037eb4d50ab0", "_uuid": "b48e15c40d93a44933bfb090c6e65f3d528f5188"}, "source": ["The distribution of observations by day of the month is nearly uniform with minima at the 1st, 25th, and 31st days of the month. The dips at the 1st and 25th are likely due to New Years and Christmas whereas the dip at the 31st is likely due to the fact that not all months have 31 days. Now, let's take a look at the distribution of observations for each store."]}, {"cell_type": "code", "metadata": {"_cell_guid": "5d715fed-8d15-438b-aa1e-4685cfc113fb", "_uuid": "c9dc6140c9c7edc206418bad78fffac5898fd1fd"}, "execution_count": null, "source": ["df_trans[\"store_nbr\"].value_counts(sort = False).plot.bar()"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "3a64d5ac-e424-48d0-8893-23dba86bdb51", "_uuid": "e13e85561ffa76f22f68a9adc7c21f8b2eb7f4c4"}, "source": ["There are a lot of stores with > 1600 observations. These were all probably open prior to 2013. Stores with < 1600 observations were all probably opened after 2013, the stores with the fewest observations being opened most recently."]}, {"cell_type": "code", "metadata": {"_cell_guid": "caa38fc0-7c6b-4aef-9508-5727c362a298", "_uuid": "3766eacfabd9e1eb3ade7ad55a3e25572ea5a734"}, "execution_count": null, "source": ["df_trans[\"transactions\"].plot.hist(bins = 100)"], "outputs": []}, {"cell_type": "code", "metadata": {"_cell_guid": "cf015001-1c00-4467-bf9d-303ebeca3408", "_uuid": "7411a99656780feed11481c391c53e8580adb35a"}, "execution_count": null, "source": ["del df_trans; gc.collect()"], "outputs": []}, {"cell_type": "markdown", "metadata": {"_cell_guid": "6e4a2e07-06cb-49bc-b741-049579c66212", "_uuid": "d8bce92500f00c3ebc80deb3a29ebe95f84a9706"}, "source": ["This is a very informative plot as it tells us that the total number of transactions is approximately Poisson distributed. Perhaps the number of transactions for particular items are also Poisson distributed.\n", "\n", "# Conclusion\n", "We looked at how the observations in both the training and test sets are distributed by date, store, item, and promotions. We also got more familiar with the distribution of the target value, unit sales, which may require processing to remove outliers. We noticed that there are some differences between the training and test sets with regards to the time period sampled, the number of unique items, each stores' variety, and percentage of sale items. We also discovered that store cluster is a subdivision of store type and that one state can have many different store types.\n", "\n", "# Future directions\n", "In the future, I plan on exploring the other data sets that were provided. I hope to update this kernel soon! Thanks for reading."]}], "nbformat_minor": 1}