{"nbformat_minor": 1, "cells": [{"source": ["\n", "\n", "# What have I done?\n", "\n", "I have updated some visualiations compared to the previous version\n", "\n", "* **Explored the smaller datasets:**\n", "    * **stores.csv**\n", "    * **items.csv**\n", "    * **transactions.csv**\n", "    * **holidays_events.csv**\n", "    * **oil.csv**\n", "    \n", "* **Performed each of the following on the above csv files:**\n", "    * **Checking for missing vlaues**\n", "    * **Analyzing Datatypes**\n", "    * **Data Explorations and Visualisation**\n", "    * **One-Hot Encoding (of string type columns)**\n", "    * **Memory reduction (so as to keep minimal memory while merging all dataframes)([SEE THIS KERNEL FOR MORE](https://www.kaggle.com/jeru666/zillow-revamped-with-memory-reduction))**\n", "\n", "# What is to be done?\n", "\n", "* **Merge all files EXCEPT the train csv file for further analysis**\n", "* **Analysis of train dataset**\n", "* **Merge the above dataframes with train dataset and perform further analysis**\n", "* **Release a baseline model**"], "cell_type": "markdown", "metadata": {"_uuid": "6b08a59965fd9de0328abd7df6915fc82c3944af", "_cell_guid": "7d3cee18-00ee-408f-ae26-2b091c75127c"}}, {"source": ["import numpy as np # linear algebra\n", "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n", "from matplotlib import pyplot as plt\n", "import seaborn as sns\n", "import missingno as msno\n", "\n", "from subprocess import check_output\n", "\n", "df_train = pd.read_csv('../input/train.csv', parse_dates=['date'], dtype={'id':np.uint32, 'store_nbr':np.uint8, 'item_nbr': np.uint32, 'onpromotion': np.bool, 'unit_sales': np.float32})\n", "\n", "#df_test = pd.read_csv('../input/test.csv', parse_dates=['date'], dtype={'id':np.uint32, 'store_nbr':np.uint8, 'item_nbr': np.uint32, 'onpromotion': np.bool})\n", "\n", "#df_sample = pd.read_csv('../input/sample_submission.csv')\n", "df_stores = pd.read_csv('../input/stores.csv')\n", "df_items = pd.read_csv('../input/items.csv')\n", "df_transactions = pd.read_csv('../input/transactions.csv')\n", "df_oil = pd.read_csv('../input/oil.csv')\n", "df_holidays_events = pd.read_csv('../input/holidays_events.csv')"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "5afef2ce115c17470111545df070ebd55ed24875", "_cell_guid": "f81b550b-8806-4c1d-b758-49540049710f"}, "execution_count": null}, {"source": ["#df_test.head()\n", "#df_test.isnull().values.any()\n", "\n"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "afe3a84b70656eb93bb41ec37781e1e6dc8a6d39", "_cell_guid": "db915a68-e498-4b8b-96f6-14f8bcf1f32f"}, "execution_count": null}, {"source": ["## Memory Reduction\n", "The following is code snippet used to reduce memory wherever possible. I will be calling these functions after analyzing each and every csv file. I have done this to keep a minimal memory constraint when we finally merge all the dataframes to feed it for machine learning algorithms."], "cell_type": "markdown", "metadata": {"_uuid": "0136d3668815350ef27232b386102821f28069a4", "_cell_guid": "4b467560-926e-4b34-a433-4242120ef61e"}}, {"source": ["#--- for integer type columns ---\n", "def change_datatype(df):\n", "    float_cols = list(df.select_dtypes(include=['int']).columns)\n", "    for col in float_cols:\n", "        if ((np.max(df[col]) <= 127) and(np.min(df[col] >= -128))):\n", "            df[col] = df[col].astype(np.int8)\n", "        elif ((np.max(df[col]) <= 32767) and(np.min(df[col] >= -32768))):\n", "            df[col] = df[col].astype(np.int16)\n", "        elif ((np.max(df[col]) <= 2147483647) and(np.min(df[col] >= -2147483648))):\n", "            df[col] = df[col].astype(np.int32)\n", "        else:\n", "            df[col] = df[col].astype(np.int64)\n", "\n", "#--- for float type columns ---\n", "def change_datatype_float(df):\n", "    float_cols = list(df.select_dtypes(include=['float']).columns)\n", "    for col in float_cols:\n", "        df[col] = df[col].astype(np.float32)            "], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "03b54dc36ac7facefc05af56ddcd8bff154b2921", "_cell_guid": "3bec4210-3eac-4150-a3fd-64a8af920456"}, "execution_count": null}, {"source": ["# 1. stores.csv\n", "\n", " First let us explore the csv files apart from the train data. Since we have to merge the others to the train data, we can analyze the others first."], "cell_type": "markdown", "metadata": {"_uuid": "53c8ca048f94e52662310b2776aecb1e0933d321", "_cell_guid": "71b9c752-cd80-4710-a5c3-4bad2fb82a20"}}, {"source": ["print(df_stores.shape)\n", "df_stores.head()"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "5e9ee302177c4bb10dd9f85c1c3b1a69a7ed02a2", "_cell_guid": "cb4f3182-a669-4c73-b232-082ca97a1a21"}, "execution_count": null}, {"source": ["### Intuition:\n", "**stores.csv** file provides data for each unique store in the country:\n", "* Each store is represented by a unique ID **store_nbr**.\n", "* The **city** and **state** of its location is also provided. \n", "* Each store is of particular **type** (grocery, stationary, furniture, automotive, etc.)\n", "* Every store has a **cluster** of similar stores. These are probably branches of the same store spread across the country."], "cell_type": "markdown", "metadata": {"_uuid": "6ab1e1e055d41a1db61ef11264ceae65e264933a", "_cell_guid": "382dfb18-7adf-4527-a06c-19152a90747c"}}, {"source": ["## Checking Missing Values"], "cell_type": "markdown", "metadata": {"_uuid": "90c7977c559645461b262e8d88a1bb7a95e176ef", "_cell_guid": "aae9947c-be0c-41dc-a880-c952ef7fbff6"}}, {"source": ["df_stores.isnull().values.any()"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "b3824aa6b9d04064d67d37308933bf92fe0ce0da", "_cell_guid": "448ffc96-409a-446b-b8f9-3f2e7ddc5e95"}, "execution_count": null}, {"source": ["## Analyzing Datatypes"], "cell_type": "markdown", "metadata": {"_uuid": "b9720d5f6b77fb7b9e94e68ec82b6427cbd50ad9", "_cell_guid": "06f7a6cf-7956-45eb-8a42-eebf6378f3d2"}}, {"source": ["pp = pd.value_counts(df_stores.dtypes)\n", "fig = plt.figure(figsize=(10,4))\n", "pp.plot.bar(color='green')\n", "plt.show()\n", "\n", "print(df_stores.dtypes.unique())\n", "print(df_stores.dtypes.nunique())"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "23189b192936b71f627d2986d5a19db8057dc079", "_cell_guid": "78a32294-28c8-4bbc-ba80-ccd3bddd34c0"}, "execution_count": null}, {"source": ["## Data Exploration & Visualisation\n", "** Number of unique stores **"], "cell_type": "markdown", "metadata": {"_uuid": "d2f54511a9bb91ea81dfe56204986cef9f0acc67", "_cell_guid": "74b1edda-ae39-42c0-b8fc-b99acb95d510"}}, {"source": ["df_stores.store_nbr.nunique()"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "1ef359f28259a100e264c12fcb68bc602a61d21e", "_cell_guid": "67e55f8f-c89a-4ef3-9b3a-620c7017499f"}, "execution_count": null}, {"source": ["**Stores distribution across cities**"], "cell_type": "markdown", "metadata": {"_uuid": "d7db45d6cef00546fe3fb7576777a0740d431e50", "_cell_guid": "43988284-edb1-43eb-acf1-e2daa0480bee"}}, {"source": ["#--- Various cities distribution ---\n", "fig, ax = plt.subplots()\n", "fig.set_size_inches(8, 8)\n", "ax = sns.countplot(y=df_stores['city'], data=df_stores) "], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "7f1760c4dad19efc9522f05daf692a9870ad18fa", "_cell_guid": "4d4c9a6c-ca8b-41ef-a9a6-075eb574d3ef"}, "execution_count": null}, {"source": ["Most of the stores are located in Quito and Guayaquil.\n", "\n", "**Stores distribution across states**"], "cell_type": "markdown", "metadata": {"_uuid": "56a4b940caf36120c05eb3477d58bd8c705180b5", "_cell_guid": "b00c54a7-ef7a-4a57-8eb4-3dd44145a6af"}}, {"source": ["#--- Various states distribution ---\n", "fig, ax = plt.subplots()\n", "fig.set_size_inches(8, 8)\n", "ax = sns.countplot(y=df_stores['state'], data=df_stores) "], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "6e77b3655c8643ec79a8268e0770b7548da46d45", "_cell_guid": "dc4b9472-cadf-4826-8d7d-f4229dabb441"}, "execution_count": null}, {"source": ["**Different types of stores**"], "cell_type": "markdown", "metadata": {"_uuid": "8f220732ac8424e06fa688ab0264306a5d7d74c1", "_cell_guid": "368446f4-4106-40a8-8180-213b3856632c"}}, {"source": ["#--- Various types ---\n", "fig, ax = plt.subplots()\n", "fig.set_size_inches(10, 7)\n", "ax = sns.countplot(x=\"type\", data=df_stores, palette=\"Set3\")"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "a3e281e77606bcc41946367635d28d8c1824800b", "_cell_guid": "ecd82bf8-7f97-4436-a3d8-755479b6a39f"}, "execution_count": null}, {"source": ["** Types of stores across cities and states**"], "cell_type": "markdown", "metadata": {"_uuid": "28b082ed26356006c5b028a1e1ad3aa8ab38da57", "_cell_guid": "d3e2edd6-bb1b-4572-8c11-4b1118ab538d"}}, {"source": ["ct = pd.crosstab(df_stores.city, df_stores.type)\n", "\n", "ct.plot.bar(figsize = (12, 6), stacked=True)\n", "plt.legend(title='type')\n", "\n", "plt.show()"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "78e3dffa627187d5a793dff2ab47a0b8bcb3f378", "_cell_guid": "4457439d-6af7-4667-9952-fb2646c3aed9"}, "execution_count": null}, {"source": ["* Type C and A are more widespread across cities.\n", "* Only Guayaquil houses stores of all types!\n", "* City Quito houses a huge number of stores. It is probably a population hub!"], "cell_type": "markdown", "metadata": {"_uuid": "7eee5e10ea71c011dbb6a99080f09eb4c026fe2f", "_cell_guid": "c8eced5a-4bc7-43bc-ad4e-a2266a6f49ac"}}, {"source": ["ct = pd.crosstab(df_stores.state, df_stores.type)\n", "ct.plot.bar(figsize = (12, 6), stacked=True)\n", "plt.legend(title='type')\n", "plt.show()"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "f5a5f9a7fde4e888f665e7a5f053f12d8435aae0", "_cell_guid": "7d192813-2349-4082-a7ed-5a157fdf0c74"}, "execution_count": null}, {"source": ["* Type C is more widespread than any other typ.\n", "* State Guayas has all types of stores.\n", "* State Pichincha has a large number of stores"], "cell_type": "markdown", "metadata": {"_uuid": "c9d755a3d4151aee1c5ac5735dd7a219f435c98e", "_cell_guid": "cf809b5b-5d15-477d-8ddc-e1939afdd4cd"}}, {"source": ["** Clusters**"], "cell_type": "markdown", "metadata": {"_uuid": "36deca9d63611fa3e4bc612bca44b4da981ec2a8", "_cell_guid": "75e224b7-a39c-43bc-b896-3926662456db"}}, {"source": ["It is claimed that `cluster` is a grouping of similar stores. So let us find the total number of such grouping."], "cell_type": "markdown", "metadata": {"_uuid": "ad56552a69e6de0a53075cea6851c0358c9f8164", "_cell_guid": "57bb9a41-0a4d-4c42-ac00-ce1c75b1f454"}}, {"source": ["df_stores.cluster.sum()"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "74b72c6b328c31ac2ac674887a38d7c4e1b7e53f", "_cell_guid": "d8138d79-9f45-436e-b5fe-255b1ad4e2a1"}, "execution_count": null}, {"source": ["### Inferences:\n", "* So there are 458 stores spread across these cities. \n", "* 54 is the number of unique stores."], "cell_type": "markdown", "metadata": {"_uuid": "a5c6cc4666de19336d7b14d6aa6df199739fd851", "_cell_guid": "2c99692c-bcfb-43d4-89ea-b04bf90e5dbf"}}, {"source": ["**Distribution of different clusters**"], "cell_type": "markdown", "metadata": {"_uuid": "f0db0e1632af05adca09f30e1fd35836f32ca53f", "_cell_guid": "3bbb6a3c-314f-4282-a0e5-514764202bf0"}}, {"source": ["fig, ax = plt.subplots()\n", "fig.set_size_inches(12, 7)\n", "ax = sns.countplot(x=\"cluster\", data=df_stores)"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "c8e34bbc15bef2da7549ba22918cbd848f882b4e", "_cell_guid": "571da7bc-2165-491c-9a31-24f572c52b3b"}, "execution_count": null}, {"source": ["Stores usually prefer clusters of 3, 6, 10 or 15 predominantly.\n", "\n", "** Distribution of all stores across cities **"], "cell_type": "markdown", "metadata": {"_uuid": "c886aea10938b1c24e1f663d68e4aaf8d4a2dfb5", "_cell_guid": "b7332228-57bf-4ee3-81ea-f555c44abe27"}}, {"source": ["mm = (df_stores.groupby(['city']).sum())\n", "\n", "fig, ax = plt.subplots()\n", "fig.set_size_inches(12, 8)\n", "ax = sns.barplot(x = mm.index, y= \"cluster\", data = mm)\n", "ax.set_xticklabels(ax.get_xticklabels(), rotation = 75, fontsize = 12)"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "be62533ab33ba79ffbf2e691ef6bc017d5beee6c", "_cell_guid": "475c33e4-2622-462a-befb-1c69574249de"}, "execution_count": null}, {"source": ["**Clusters in cities**"], "cell_type": "markdown", "metadata": {"_uuid": "14508a3a1f452897901c23ab28e20b458e69807d", "_cell_guid": "c3c504db-ee7e-49f8-af6e-2e9ddd22ed19"}}, {"source": ["ct = pd.crosstab(df_stores.city, df_stores.cluster)\n", "ct.plot.bar(figsize = (12, 6), stacked=True)\n", "plt.legend(title='cluster')\n", "plt.show()"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "64ee6a1989188f3c9b181e38864bcadd9cf6d7b4", "_cell_guid": "493b0fac-5015-46c6-90f1-5eab73cd1e5a"}, "execution_count": null}, {"source": ["**Clusters in states**"], "cell_type": "markdown", "metadata": {"_uuid": "17f4964a680ee18d9ff5c3da7bfb347f13c8bee9", "_cell_guid": "115eebd2-3421-486c-992e-ef47762c9255"}}, {"source": ["ct = pd.crosstab(df_stores.state, df_stores.cluster)\n", "ct.plot.bar(figsize = (12, 6), stacked=True)\n", "plt.legend(title='cluster')\n", "plt.show()"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "5f0d760be7dded533ed7f60aff67904d4b80d2cc", "_cell_guid": "0952d243-130a-42f0-a9b5-6c6eca4ca849"}, "execution_count": null}, {"source": ["## One-Hot Encoding\n", "\n", "We have three columns of type **object**, which must be converted to a form digestible by ML algorithms."], "cell_type": "markdown", "metadata": {"_uuid": "18430196a7df3493c7daab7585da58888e12520a", "_cell_guid": "9688b122-11dc-4d72-826c-7548927b3b64"}}, {"source": ["obj_cols = list(df_stores.select_dtypes(include=['object']).columns)\n", "for col in obj_cols:\n", "    df_stores[col], _ = pd.factorize(df_stores[col])\n", "    \n", "df_stores.head(10)"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "13df4c140476e1abee0ee688799bf63396814966", "_cell_guid": "0f37de33-509a-4e7e-b84d-cc54a4601560"}, "execution_count": null}, {"source": ["## Memory Check"], "cell_type": "markdown", "metadata": {"_uuid": "5d333c3b0c982e0a8fc9b71563a83207c3ff41b9", "_cell_guid": "9d6c3e0f-d5f3-407a-a61d-978d1557900f"}}, {"source": ["mem = df_stores.memory_usage(index=True).sum()\n", "print(\"Memory consumed by stores dataframe initially  :   {} MB\" .format(mem/ 1024**2))\n", "\n", "change_datatype(df_stores)\n", "change_datatype_float(df_stores)\n", "\n", "mem = df_stores.memory_usage(index=True).sum()\n", "print(\"\\n Memory consumed by stores dataframe later  :   {} MB\" .format(mem/ 1024**2))"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "7dbffcb3cf2a1d8028d586af6de3b2ceee92eb51", "_cell_guid": "41fdee58-40f5-4906-8867-5722208f82bf"}, "execution_count": null}, {"source": ["# 2. items.csv"], "cell_type": "markdown", "metadata": {"_uuid": "5d381746bf1f5b7b17315e7a8d6633a6ee1eaa7d", "_cell_guid": "65e2e623-6977-479f-bc88-a452a5e23702"}}, {"source": ["print(df_items.shape)\n", "df_items.head()"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "0968998003106d53f8763a57ea8cdd8986a879ae", "_cell_guid": "9727f108-9f18-4918-8d0e-548fa9a9cbce"}, "execution_count": null}, {"source": ["## Checking Missing Values"], "cell_type": "markdown", "metadata": {"_uuid": "53cd52991dd7400bdfc5ada17de77e249b645b24", "_cell_guid": "47c90e5e-4d83-440c-ba56-57bff3368160"}}, {"source": ["df_items.isnull().values.any()"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "83937cf2d7fd26cba683b5ace230587a1c8b7f6a", "_cell_guid": "2c0f64df-a590-4f4c-b442-5251f71876b9"}, "execution_count": null}, {"source": ["## Analyzing Datatypes"], "cell_type": "markdown", "metadata": {"_uuid": "3ef2226ec7350ab2c968780ffd0d3dcbdc1ac5b0", "_cell_guid": "906bd162-13ff-4fa7-91e0-b5708dd772a8"}}, {"source": ["fig = plt.figure(figsize=(10,4))\n", "pp = pd.value_counts(df_items.dtypes)\n", "pp.plot.bar(color='red')\n", "plt.show()\n", "\n", "print(df_items.dtypes.unique())\n", "print(df_items.dtypes.nunique())"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "94f86828f19ff67d3c90aca5c0dcfda40140a73e", "_cell_guid": "ce79f486-0b14-4bbc-b6a4-69d0f6664522"}, "execution_count": null}, {"source": ["## Data Exploration & Visualisation"], "cell_type": "markdown", "metadata": {"_uuid": "23da3de3fb7a9c86cdf36e4d99ba99970dbff21e", "_cell_guid": "a133be78-b881-4315-ae41-73c0a9e17bd2"}}, {"source": ["df_items.item_nbr.nunique()"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "3466614d51c5978950cadb81e979cf4eaf7b3adc", "_cell_guid": "1a4ad63d-05e9-4c3a-a951-f4df2fbf2f92"}, "execution_count": null}, {"source": ["**Distribution of  various families of items**"], "cell_type": "markdown", "metadata": {"_uuid": "81d862166281e0217b44f1e3762449472a069337", "_cell_guid": "e89978d9-0772-48e5-a503-6a37b1b7bffb"}}, {"source": ["fig, ax = plt.subplots()\n", "fig.set_size_inches(12, 8)\n", "ax = sns.countplot(y = \"family\", data = df_items)"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "e29422e043b4a21901e2beee1cff356a008b5334", "_cell_guid": "d8d35eb7-f523-434b-9c96-0092b859b58d"}, "execution_count": null}, {"source": ["### Inference:\n", "Most of the unique items belong to categories:\n", "* Grocery I\n", "* Cleaning\n", "* Beverages\n", "\n"], "cell_type": "markdown", "metadata": {"_uuid": "5a6115ec9b9e23d9f24a17c0989b113e10442837", "_cell_guid": "7fa313a3-fbeb-4ab9-bc8a-849839cf442f"}}, {"source": ["**Distribution of perishable goods by family**"], "cell_type": "markdown", "metadata": {"_uuid": "3a6fbb3b98681ff6103e5e7b21bcd5be454c6986", "_cell_guid": "cbf298d5-7939-407e-8403-9164de98693c"}}, {"source": ["ct = pd.crosstab(df_items.family, df_items.perishable)\n", "ct.plot.bar(figsize = (12, 7), stacked=True)\n", "plt.legend(title='perishable')\n", "plt.show()"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "ed46169b0f4772b55ea8b37d6ba9e3de48579676", "_cell_guid": "248df911-b440-4ccf-aa19-4c9e660beb7d"}, "execution_count": null}, {"source": ["''' \n", "mc = (df_items.groupby(['family']).sum())\n", "fig, ax = plt.subplots()\n", "fig.set_size_inches(12, 8)\n", "ax = sns.barplot(x = mc.index, y= \"perishable\", data = mc)\n", "ax.set_xticklabels(ax.get_xticklabels(), rotation = 75, fontsize = 9)\n", "'''"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "3c26dc7eb1f78fc333804a9d77cb24c0a57ee428", "_cell_guid": "ad4645ee-b2ff-4775-abf0-9519dcfbb435"}, "execution_count": null}, {"source": ["**Distrbution of number of unique classes per family of items.**"], "cell_type": "markdown", "metadata": {"_uuid": "db26ce918607fff9204f79dddcd1a8287d1a8967", "_cell_guid": "6d98121d-8fbf-49b5-9883-879b096f22c7"}}, {"source": ["xc = df_items.groupby(['family'])['class'].nunique()\n", "fig, ax = plt.subplots()\n", "fig.set_size_inches(12, 6)\n", "xc.plot.bar(color='magenta')\n", "plt.show()"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "45a6d37735f801a1bd0d062805998ce1787a86e9", "_cell_guid": "a8b12d52-c174-471f-9345-1904758370ef"}, "execution_count": null}, {"source": ["### Inference:\n", "* Grocery I family has the most number of classes of items present.\n", "\n", "## One-Hot Encoding"], "cell_type": "markdown", "metadata": {"_uuid": "b16789b1fa54a334ed9dd140cc13134c050f2d1c", "_cell_guid": "b8269992-1ae4-43c8-9a2b-45e434552c66"}}, {"source": ["obj_cols = list(df_items.select_dtypes(include=['object']).columns)\n", "for col in obj_cols:\n", "    df_items[col], _ = pd.factorize(df_items[col])\n", "    \n", "df_items.head(10)"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "908dca3518db9270e6eeeca9f18ae9a43ee5efc4", "_cell_guid": "9f5621c6-52b7-4fd2-8145-e75d00db97c7"}, "execution_count": null}, {"source": ["## Memory Check"], "cell_type": "markdown", "metadata": {"_uuid": "bf751bce11ffa956dd13416ed1a78ed25474ecfc", "_cell_guid": "c72a7cff-e596-4278-8f3c-de58bc472dc7"}}, {"source": ["mem = df_items.memory_usage(index=True).sum()\n", "print(\"Memory consumed by items dataframe initially  :   {} MB\" .format(mem/ 1024**2))\n", "\n", "change_datatype(df_items)\n", "change_datatype_float(df_items)\n", "\n", "mem = df_items.memory_usage(index=True).sum()\n", "print(\"\\n Memory consumed by items dataframe later  :   {} MB\" .format(mem/ 1024**2))"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "b6d1f5c9627ec98545a399dcda957f92c7ef3909", "_cell_guid": "ada052b1-4b25-4ed0-9e45-af7943e80bc7"}, "execution_count": null}, {"source": ["# 3. transactions.csv"], "cell_type": "markdown", "metadata": {"_uuid": "2548cd4ec42b10d5663146fda8502f09e41b139b", "_cell_guid": "32c043dd-83d2-4484-8a21-86975cc4b281"}}, {"source": ["print(df_transactions.shape)\n", "df_transactions.head()"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "f42679a29f4847d7b1d8239812e8163d4376edff", "_cell_guid": "16776ec9-c23f-4651-927b-fec410bf6e57"}, "execution_count": null}, {"source": ["## Checking Missing Values"], "cell_type": "markdown", "metadata": {"_uuid": "c6309386dd3f77b7930f643e536cadb6b5b4a9bf", "_cell_guid": "6f976632-1bde-4637-9989-589103f37fd0"}}, {"source": ["df_transactions.isnull().values.any()"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "826ebff3f409ba8aad4bc50cdb01099504be6b48", "_cell_guid": "c8b3bea2-63d1-42b4-aa49-b66d1dc63a8b"}, "execution_count": null}, {"source": ["\n", "## Analyzing Datatypes"], "cell_type": "markdown", "metadata": {"_uuid": "0514b048e88bc5c0206cdd896fdaf9edd072af2d", "_cell_guid": "e3b7d4f0-ad82-42f6-a222-c7f11ff4eeee"}}, {"source": ["fig = plt.figure(figsize=(10,4))\n", "pp = pd.value_counts(df_transactions.dtypes)\n", "pp.plot.bar(color='orange')\n", "plt.show()\n", "\n", "print(df_transactions.dtypes.unique())\n", "print(df_transactions.dtypes.nunique())"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "82c915876bc4873c4b502f2d236938232b5766fd", "_cell_guid": "01ee357d-f9ff-42d6-afbc-5e10e8b5f71a"}, "execution_count": null}, {"source": ["**Distribution of total number of transactions made per individual store**"], "cell_type": "markdown", "metadata": {"_uuid": "dd1641e1d1da1c6258974d39d80c667c677a7073", "_cell_guid": "811c44a3-654f-468b-9d0a-abf916a7e639"}}, {"source": ["pc = (df_transactions.groupby(['store_nbr']).sum())\n", "fig, ax = plt.subplots()\n", "fig.set_size_inches(12, 8)\n", "ax = sns.barplot(x = pc.index, y= \"transactions\", data = pc)\n", "ax.set_xticklabels(ax.get_xticklabels(), rotation = 75, fontsize = 9)"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "71f8b9067496fedb36b9cb979bd011136daa9afb", "_cell_guid": "b1a78662-abcc-4c57-80a4-d18caa8597f4"}, "execution_count": null}, {"source": ["### Inference:\n", "* Only 4 stores have crossed the 6M transcation mark.\n", "* Few stores have transactions of less than a million.\n", "\n", "Below is another form of representing the above graph:"], "cell_type": "markdown", "metadata": {"_uuid": "d6f04118f35383bab5dbb5a3d5c100d0759a2494", "_cell_guid": "fb8a9c93-67e3-4ea5-ae10-68c9279b87b4"}}, {"source": ["g = sns.factorplot(x=\"store_nbr\", y=\"transactions\", size = 12, data=df_transactions)\n"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "7b84ea5fc37ce47be55733384485f1c8fb8c0863", "_cell_guid": "c4feed13-a9a3-4075-a853-07c931eba45f"}, "execution_count": null}, {"source": ["**Converting date column from type *object* to type *datetime* **"], "cell_type": "markdown", "metadata": {"_uuid": "c5fc90ab59080ae91091568a2d185310a65135c4", "_cell_guid": "c3c5d670-d14f-4601-9392-0c412eed70a7"}}, {"source": ["df_transactions.date = pd.to_datetime(df_transactions.date)\n", "df_transactions.date.dtype"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "32c298054effaa2ed3740b2d0b77c6651dd25ec8", "_cell_guid": "1cf5c9fa-129b-4f00-934a-fd0f9b157447"}, "execution_count": null}, {"source": ["Store numbers with **max** and **min** transcations over time"], "cell_type": "markdown", "metadata": {"_uuid": "c1a786b23979b9de8bd6d7042a056a20b53e7309", "_cell_guid": "2d3ca1b5-b9df-4780-b27d-d23cd12b2f82"}}, {"source": ["max_transaction = df_transactions['transactions'].max()\n", "min_transaction = df_transactions['transactions'].min()\n", "\n", "print(df_transactions.store_nbr[df_transactions['transactions'] == max_transaction])\n", "print(df_transactions.store_nbr[df_transactions['transactions'] == min_transaction])"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "6d08bac6ec20707ef8dcc49b5e34e3e1b5232a36", "_cell_guid": "bc0105ea-cedc-423f-9dab-89a241ef465f"}, "execution_count": null}, {"source": ["** List of stores having the top 100 transactions **"], "cell_type": "markdown", "metadata": {"_uuid": "58b6b48b66e036956f7a4a0ea1cbabc709c28c8d", "_cell_guid": "9be30aef-86bb-4567-878c-cebf840ac4a3"}}, {"source": ["top_trans = df_transactions.nlargest(100, 'transactions')\n", "print(top_trans.store_nbr.unique())"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "9851e8e7fe59882cdd71813709a1ddb564eabfe8", "_cell_guid": "5945ba42-078f-4035-a43d-c6e1d327b98f"}, "execution_count": null}, {"source": ["** List of stores having the top 500 transactions **"], "cell_type": "markdown", "metadata": {"_uuid": "7a68c1bd4d4c726c9d36c2638f78abf440569b43", "_cell_guid": "ae79e4a8-5fa1-4219-b0ee-121b863cfa20"}}, {"source": ["top_trans = df_transactions.nlargest(500, 'transactions')\n", "print(top_trans.store_nbr.unique())"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "cf7bb9f5c7d6b3f7700ee7ab8fd90e2cac17504d", "_cell_guid": "c6af816a-f02b-4d77-b82a-bd8dd0d75da6"}, "execution_count": null}, {"source": ["** List of stores having the top 1000 transactions **"], "cell_type": "markdown", "metadata": {"_uuid": "ae045723560cf613b7ac5993f4f8fb6e83ce6d5e", "_cell_guid": "2712d691-df6c-469e-b51c-4df0ab7de482"}}, {"source": ["top_trans = df_transactions.nlargest(1000, 'transactions')\n", "print(top_trans.store_nbr.unique())"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "146c3606aceffc866c288d05c57c795077116693", "_cell_guid": "a621270c-46b9-482e-ac4d-f3166d7ff825"}, "execution_count": null}, {"source": ["## Memory Reduction"], "cell_type": "markdown", "metadata": {"_uuid": "dd382d4de31de4d92007bc27e26fb3a216c18d25", "_cell_guid": "07080b3f-3d90-458e-9d22-09db89bcf025"}}, {"source": ["mem = df_transactions.memory_usage(index=True).sum()\n", "print(\"Memory consumed by transactions dataframe initially  :   {} MB\" .format(mem/ 1024**2))\n", "\n", "change_datatype(df_transactions)\n", "change_datatype_float(df_transactions)\n", "\n", "mem = df_transactions.memory_usage(index=True).sum()\n", "print(\"\\nMemory consumed by transactions dataframe later  :   {} MB\" .format(mem/ 1024**2))"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "0b3e50497b168b669d1f9f50b2a9713f809ce0cf", "_cell_guid": "9abfb52e-5bb9-4684-b433-300988d3557b"}, "execution_count": null}, {"source": ["# 4. holidays_events.csv"], "cell_type": "markdown", "metadata": {"_uuid": "d1ee6bed612f2b9b38fe66fa3e2eb9c2aa07b7b8", "_cell_guid": "7de4381e-57d0-4ade-80e3-af81b31dbfda"}}, {"source": ["print(df_holidays_events.shape)\n", "df_holidays_events.head()"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "88670730cb8cb9d851e9d3ff27b96d7c03de78be", "_cell_guid": "552f523a-d26f-4366-9478-523b04c2d05f"}, "execution_count": null}, {"source": ["## Checking Missing Values"], "cell_type": "markdown", "metadata": {"_uuid": "503eafd29281a8c8101b70795b35572219b2f4cb", "_cell_guid": "b7dc9e7f-c4bb-4ec6-aeba-4b2a8091f40b"}}, {"source": ["df_holidays_events.isnull().values.any()"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "ab2839c5f47eb71a28dba6b1de8acc9a04994352", "_cell_guid": "e77708d8-3339-48cb-a8a9-d1dcf491ca5b"}, "execution_count": null}, {"source": ["## Analyzing Datatypes"], "cell_type": "markdown", "metadata": {"_uuid": "24ee61211ac7ea2ff2197fa555b8c66dc4113c5d", "_cell_guid": "d4211bed-f6bc-4985-9d9d-3cb058ee38e4"}}, {"source": ["fig = plt.figure(figsize=(10,4))\n", "pp = pd.value_counts(df_holidays_events.dtypes)\n", "pp.plot.bar(color='violet')\n", "plt.show()\n", "\n", "print(df_holidays_events.dtypes.unique())\n", "print(df_holidays_events.dtypes.nunique())"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "3a4eb3f916815408795afb07ec3a0fc65483627b", "_cell_guid": "335f55fd-a01c-4d3c-bab7-a674377cfc98"}, "execution_count": null}, {"source": ["** Various types of holidays present **"], "cell_type": "markdown", "metadata": {"_uuid": "10134e6afb2ba16f9285c48f120eacde4fd5f14d", "_cell_guid": "3630b86b-129c-4dc8-99c8-77e43a2de1bd"}}, {"source": ["print(df_holidays_events.type.unique())\n", "df_holidays_events.type.value_counts()"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "c7e3e493d733d8b86fd53d034f8f32b73886983c", "_cell_guid": "4ad73510-cb62-40ca-af56-a55e1816c9d0"}, "execution_count": null}, {"source": ["fig, ax = plt.subplots()\n", "fig.set_size_inches(8, 6)\n", "ax = sns.countplot( y=\"type\", data=df_holidays_events, palette=\"RdBu\")"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "605c5ce4fefcf8f16f04a13a83fea8169911e6f0", "_cell_guid": "69e447a5-876d-4eed-a7c2-c5216742b43f"}, "execution_count": null}, {"source": ["**Distribution of different locales of holidays**"], "cell_type": "markdown", "metadata": {"_uuid": "ea85c7b69aeab08984a57f0d88c9880acca8b691", "_cell_guid": "81622874-afff-4aa0-b581-d48a7b7c88bc"}}, {"source": ["print(df_holidays_events.locale.unique())\n", "df_holidays_events.locale.value_counts()"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "965c73423b4e8c246819d565ec57edbb0e0ba141", "_cell_guid": "1ef39ae5-e9b5-49ab-bce4-74615f074d59"}, "execution_count": null}, {"source": ["fig, ax = plt.subplots()\n", "fig.set_size_inches(9, 5)\n", "ax = sns.countplot( x=\"locale\", data=df_holidays_events, palette=\"muted\")"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "7268196d6911a16735218fdac979d698d4e8b1c1", "_cell_guid": "e491cd41-f046-4ee3-9aec-e505f7bf4353"}, "execution_count": null}, {"source": ["**Distribtuion of types of holidays with respect to various locales**"], "cell_type": "markdown", "metadata": {"_uuid": "8bb7b3a894677c3b1e1fc54e266b4af0224cc1e4", "_cell_guid": "d3b71abc-cf8b-40e3-b144-f9c11a264a6c"}}, {"source": ["ct = pd.crosstab(df_holidays_events.type, df_holidays_events.locale)\n", "ct.plot.bar(figsize = (12, 7), stacked=True)\n", "plt.legend(title='locale')\n", "plt.show()\n", "'''\n", "fig, ax = plt.subplots()\n", "fig.set_size_inches(12, 8)\n", "sns.countplot( x=\"type\", hue=\"locale\", data=df_holidays_events, palette=\"muted\")\n", "'''"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "e30b3af31eec126bba1f43b30390092c1ce6833e", "_cell_guid": "c58337b1-a76a-4ef6-bd35-f74f256ca894"}, "execution_count": null}, {"source": ["**Distribution of holidays transferred vs non-transferred**"], "cell_type": "markdown", "metadata": {"_uuid": "c24ede7eeb2ba357d572da95a84f8d8dcd817094", "_cell_guid": "825f8784-3ef5-4326-8eb2-ddcadc75bbd5"}}, {"source": ["df_holidays_events.transferred.value_counts()"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "5a8386f78ea72dd1d69ef787201256a396b73daf", "_cell_guid": "a7224538-1b23-4f3e-8616-e5aa497b1204"}, "execution_count": null}, {"source": ["df_holidays_events.transferred.hist()"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "30a782768f4e360e72dbb444aa7ba19a3f2b9bcf", "_cell_guid": "72d5422c-28c4-4eda-b121-a6f528199965"}, "execution_count": null}, {"source": ["fig, ax = plt.subplots()\n", "fig.set_size_inches(12, 8)\n", "sns.countplot( x=\"type\", hue=\"transferred\", data=df_holidays_events)"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "166b10c6081e9825beab65859b80d0139995896b", "_cell_guid": "fc7c2664-bc5b-4b01-b79f-032eb0f4038f"}, "execution_count": null}, {"source": ["g = sns.factorplot(x=\"who\", y=\"survived\", col=\"class\",\n", "                    data=df_holidays_events, saturation=.5,\n", "                    kind=\"bar\", ci=None, aspect=.6)\n"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "09cc434d5e9b818c66dc56505dfa3a95876f2546", "_cell_guid": "8c4321fb-a617-4052-afd6-b814b8bc5d18"}, "execution_count": null}, {"source": ["### Inferences:\n", "* Holidays of type *holiday* alone were transferred.\n", "* Only 12 holidays were transferred throughout the occurrence.\n", "* *Regional* locales were of type *holiday* only.\n", "* Apart from these the above graphs are self-explanatory!"], "cell_type": "markdown", "metadata": {"_uuid": "c99ddd37bb03dde4bce6169fbec6a3bf4798945a", "_cell_guid": "77b4e28b-922f-47d3-97d0-c18bcb79d9f4"}}, {"source": ["## One-Hot Encoding"], "cell_type": "markdown", "metadata": {"_uuid": "295fcbf493ddedf18f40e55484864b7ddf8cd02d", "_cell_guid": "3c576b08-c6f9-471f-bd69-b578e6436d3f"}}, {"source": ["obj_cols = list(df_holidays_events.select_dtypes(include=['object']).columns)\n", "for col in obj_cols:\n", "    df_holidays_events[col], _ = pd.factorize(df_holidays_events[col])\n", "    \n", "df_holidays_events.head(10)"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "8a8ead7be1e8aaf7adff11cd2a3facf097a6c57a", "_cell_guid": "63a70320-fda7-421b-900d-9d23bdf54f71"}, "execution_count": null}, {"source": ["## Memory Reduction"], "cell_type": "markdown", "metadata": {"_uuid": "c55642239417a41f8ce59764c09cab5fb82befff", "_cell_guid": "34638d65-4180-454d-acdd-6d054ed2e631"}}, {"source": ["mem = df_holidays_events.memory_usage(index=True).sum()\n", "print(\"Memory consumed by transactions dataframe initially  :   {} MB\" .format(mem/ 1024**2))\n", "\n", "change_datatype(df_holidays_events)\n", "change_datatype_float(df_holidays_events)\n", "\n", "mem = df_holidays_events.memory_usage(index=True).sum()\n", "print(\"\\nMemory consumed by transactions dataframe later  :   {} MB\" .format(mem/ 1024**2))"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "86d92344ebe81dca3d9c31a295e1f8254ea4392c", "_cell_guid": "2734e6ba-4356-49c1-9266-c7893c6e9bb4"}, "execution_count": null}, {"source": ["# 5. oil.csv"], "cell_type": "markdown", "metadata": {"_uuid": "af74e566e5b5697ba7989bb8905f5dd6b4c906f6", "_cell_guid": "86fd99bd-3e41-40f1-a593-c0a95590648c"}}, {"source": ["print(df_oil.shape)\n", "df_oil.head()"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "8018632283522f3660806b6926b9191420de9de1", "_cell_guid": "ab352d24-985a-466a-89ab-bf55ceb1708e"}, "execution_count": null}, {"source": ["## Checking Missing Values"], "cell_type": "markdown", "metadata": {"_uuid": "ff60e7e7d27809cf7b38526b11054b60aafb0087", "_cell_guid": "14a631ef-830c-4b83-ac2f-4a6e81305e27"}}, {"source": ["df_oil.isnull().values.any()"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "0ac5f5b95a9737e932d485b87e60035d2b79109f", "_cell_guid": "3278100f-7273-42a8-be4f-47b42796b79c"}, "execution_count": null}, {"source": ["## Analyzing Datatypes"], "cell_type": "markdown", "metadata": {"_uuid": "ef2d77ca26d7607cbf50b857a7d9b376af183e37", "_cell_guid": "f3dacd11-4e60-40f9-9a34-f70c43f80492"}}, {"source": ["fig = plt.figure(figsize=(10,4))\n", "pp = pd.value_counts(df_oil.dtypes)\n", "pp.plot.bar(color='pink')\n", "plt.show()\n", "\n", "print(df_oil.dtypes.unique())\n", "print(df_oil.dtypes.nunique())"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "684376c4670424dc713262762d09b7c033e9b3f7", "_cell_guid": "9ad49121-de8a-4c54-be49-0a3d88021ce1"}, "execution_count": null}, {"source": ["**Converting date column from type object to type datetime**"], "cell_type": "markdown", "metadata": {"_uuid": "3de9f1fea9b20658d8e501840d2991e0e3e86d19", "_cell_guid": "9ff50acd-8c73-47c5-859a-ab1cc68f31db"}}, {"source": ["df_oil.date = pd.to_datetime(df_oil.date)\n", "df_oil.date.dtype"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "a3273a92395201310b5fc57bb5bfa41c1354e04a", "_cell_guid": "d3999388-4b85-4352-8713-924d6df34fd3"}, "execution_count": null}, {"source": ["**Dates having highest and lowest oil prices**"], "cell_type": "markdown", "metadata": {"_uuid": "0376b1f115e812a127d3dfca3ed40aa03634e840", "_cell_guid": "192c41a2-5431-40e4-bd18-ae16b08e5bea"}}, {"source": ["print('Maximum price date : ', df_oil.date[df_oil['dcoilwtico'] == df_oil['dcoilwtico'].max()])\n", "print('Minimum price date : ', df_oil.date[df_oil['dcoilwtico'] == df_oil['dcoilwtico'].min()])"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "85a4cf074d0e478d2fc209f24640914976bc505b", "_cell_guid": "89f7d081-35ef-4ad6-9f23-6c72e4e89615"}, "execution_count": null}, {"source": ["Displayed below is the boxplot of the oil prices across the given timeline comprising of:\n", "* mean\n", "* range\n"], "cell_type": "markdown", "metadata": {"_uuid": "9ce54f52a2a8b0fb2843107ef4432314b8f4ae2c", "_cell_guid": "a6ac1f9e-6632-4bcd-b567-df37649d7c51"}}, {"source": ["ax = sns.boxplot(x=df_oil[\"dcoilwtico\"])"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "28c011adc197b51b5111448c9ff76f05432ffaa6", "_cell_guid": "85680e1d-87d0-4080-90fa-a83a173a9976"}, "execution_count": null}, {"source": ["#f, ax = plt.subplots(figsize=(12, 5))\n", "#sns.tsplot(data = df_oil['dcoilwtico'], time = df_oil['date'], err_style=\"ci_bars\", interpolate=False)\n", "\n", "f, ax = plt.subplots(figsize=(15, 7))\n", "sns.tsplot(data = df_oil['dcoilwtico'], time = df_oil['date'],ci=\"sd\")\n"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "5c14af26aa8a33db6d320b2692a05d4add2c2d57", "_cell_guid": "1eff0892-a287-4b19-aa02-21d81f9b932c"}, "execution_count": null}, {"source": ["As stated we will now merge the dataframes mentioned bove to get further insights in the data."], "cell_type": "markdown", "metadata": {"_uuid": "c397849fa2c5803cdf29b56d188501434fee8d06", "_cell_guid": "bcfcae7c-e675-4e16-97ea-6da22446e4db"}}, {"source": ["# 6. train.csv"], "cell_type": "markdown", "metadata": {"_uuid": "c15ba11260b868aa40cea54b3b0a0a1e7f455a27", "_cell_guid": "00cbf03d-f343-4192-94c6-a7b1c06b26dc"}}, {"source": ["print(df_train.shape)\n", "df_train.head()"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "7e0d214a795b1397781bd818fe1d449b90d5ee4c", "_cell_guid": "30e8da64-4027-402b-8fa9-0c27773aea4f"}, "execution_count": null}, {"source": ["mem = df_train.memory_usage(index=True).sum()\n", "print(\"Memory consumed by train dataframe : {} MB\" .format(mem/ 1024**2))"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "a60d36f69006a9e0523c70ada47ed27cfd705c09", "_cell_guid": "be4547fa-7032-4efb-9ee9-7e2839e25728"}, "execution_count": null}, {"source": ["## Checking Missing Values"], "cell_type": "markdown", "metadata": {"_uuid": "26e9064c793a005ce1c8f13094755366ab3b6e05", "_cell_guid": "e7e851a1-6dbe-4d7a-bd6f-d7fcbccbc8d4"}}, {"source": ["df_train.isnull().values.any()"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "dedb1ab846d7e0936284f48b761ef94c17e2b7f7", "_cell_guid": "53bff6f8-4289-4fc7-bf3c-2fc67b9d7548"}, "execution_count": null}, {"source": ["df_train.columns[df_train.isnull().any()].tolist()"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "c18a6d568a4ba60e2309ef8621259c3d62d3eb66", "_cell_guid": "5cdfeb12-ab27-48f9-89e0-d3d6b03518f5"}, "execution_count": null}, {"source": ["Missing values are only found in column **onpromotion**.\n", "\n", "Imputing missing values with -1"], "cell_type": "markdown", "metadata": {"_uuid": "cc4657ad0f18861d1732bcd46671442b478790ce", "_cell_guid": "9fe97a04-d4d7-4d7e-ae27-d181990e56e0"}}, {"source": ["df_train[[ 'onpromotion']] = df_train[['onpromotion']].fillna(np.int(-1))"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "5030f2678e2bb5f475eb4fcd625bfb92e9768520", "_cell_guid": "4f5ae145-7aa4-4c59-ae4b-4100a7eb5a8f"}, "execution_count": null}, {"source": ["\n", "\n", "## Analyzing Datatypes"], "cell_type": "markdown", "metadata": {"_uuid": "65b378341a24ba54185cd73c46b384372447e812", "_cell_guid": "cc4694f5-9c4d-4e34-9039-0f042f33493c"}}, {"source": ["pp = pd.value_counts(df_train.dtypes)\n", "pp.plot.bar()\n", "plt.show()\n", "\n", "print(df_train.dtypes.unique())\n", "print(df_train.dtypes.nunique())"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "7d0a40ca51a18fdf2483effddf1575bf01d9d1b5", "_cell_guid": "f46c3535-bc02-4967-947e-fa1a4b6d7ed6"}, "execution_count": null}, {"source": ["## Analysis of target variable *unitsales*"], "cell_type": "markdown", "metadata": {"_uuid": "d87dc6482a39f04204ba0d51a2749f9b50e18cfa", "_cell_guid": "6fdb7c77-5838-47e1-a21c-4a01483423f9"}}, {"source": ["'''\n", "f, ax = plt.subplots(figsize=(15, 7))\n", "sns.tsplot(data = df_train['unit_sales'], time = df_train['date'], ci=\"sd\")\n", "'''"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "87252f3fbc12f94581943c0d937523df689a6209", "_cell_guid": "15941654-826b-4d0c-a473-adecbbccc12c"}, "execution_count": null}, {"source": ["'''\n", "stores_unitsales = (df_train.groupby(['store_nbr']).sum())\n", "fig, ax = plt.subplots()\n", "fig.set_size_inches(12, 8)\n", "ax = sns.barplot(x = stores_unitsales.index, y= \"unit_sales\", data = stores_unitsales)\n", "ax.set_xticklabels(ax.get_xticklabels(), rotation = 75, fontsize = 12)\n", "'''\n"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "7d1d14a3dfd64682ae27ce007663627a517b673b", "_cell_guid": "f87e91a1-c5f6-43c2-96e4-506881125c2f"}, "execution_count": null}, {"source": ["''' \n", "items_unitsales = (df_train.groupby(['item_nbr']).sum())\n", " \n", "fig, ax = plt.subplots()\n", "fig.set_size_inches(12, 8)\n", "ax = sns.barplot(x = items_unitsales.index, y= \"unit_sales\", data = items_unitsales)\n", "ax.set_xticklabels(ax.get_xticklabels(), rotation = 75, fontsize = 12)\n", "''' "], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "cf1e8949330e77f4144a491b13b4a7993ced8175", "_cell_guid": "13fdf48f-fbda-4c09-b208-2c44cdc9c3f4"}, "execution_count": null}, {"source": ["## Analysis of target variable *onpromotion*"], "cell_type": "markdown", "metadata": {"_uuid": "3b5e47d42ca6b31513285ef2d292d6a473444a51", "_cell_guid": "c17acec1-77b8-40ba-9c0a-a26ebb3f1637"}}, {"source": ["''' \n", "print(df_train.onpromotion.nunique())\n", "print(df_train.onpromotion.unique())\n", "'''"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "42b91157b23846628b25a18300997a1c5d476da2", "_cell_guid": "0f0dfef0-52fe-495d-8002-666a370cbe9c"}, "execution_count": null}, {"source": ["#--- non Nan rows ---\n", "#df_train.onpromotion.count()"], "outputs": [], "cell_type": "code", "metadata": {"collapsed": true, "_uuid": "6cd41011606c58d521af6edf47efcc52604a77b3", "_cell_guid": "0fd1d7cc-6891-4c45-baed-e94ab4e7d7d4"}, "execution_count": null}, {"source": ["## *TO BE CONTINUED......*"], "cell_type": "markdown", "metadata": {"_uuid": "1ebb7560f4d483a956f7acfe012b75f4481ed5ae", "_cell_guid": "faa9da4b-64f0-4fae-843b-3bd68cc8241c"}}, {"source": ["# What is to be done?\n", "\n", "* **Analysis of train dataset** (unfortunately cannot be done due to kernel timeout :( )\n", "* **Merge the above dataframes with train dataset and perform further analysis**\n", "* **Release a baseline model**\n", "\n", "\n", "# STAY TUNED !!!"], "cell_type": "markdown", "metadata": {"_uuid": "141744f879f3bce557b13c949f1f2142c80dd3a0", "_cell_guid": "b4101385-b44e-4a61-9b64-5dcdf7403aa0"}}], "metadata": {"language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "nbconvert_exporter": "python", "mimetype": "text/x-python", "name": "python", "pygments_lexer": "ipython3", "version": "3.6.3"}, "kernelspec": {"language": "python", "name": "python3", "display_name": "Python 3"}}, "nbformat": 4}