{"nbformat": 4, "nbformat_minor": 1, "cells": [{"cell_type": "markdown", "source": ["This are basic stpes for fitting random forest in data, I havent applied any optimization. Also, EDA is not emloyed.\n", "\n", "How oil prices affects is needed to be analysed.\n", "\n", "**Code for data size reduction is taken from [here](http://https://www.kaggle.com/jagangupta/memory-optimization-and-eda-on-entire-dataset)**"], "metadata": {"_cell_guid": "31798cae-43ae-468e-b59d-19431b0c336c", "_uuid": "c9340ed24ea7e6eef50dc28b6535d143165b1b68"}}, {"execution_count": null, "cell_type": "code", "outputs": [], "source": ["# This Python 3 environment comes with many helpful analytics libraries installed\n", "# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n", "# For example, here's several helpful packages to load in \n", "\n", "import numpy as np # linear algebra\n", "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n", "\n", "\n", "# Input data files are available in the \"../input/\" directory.\n", "# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n", "\n", "from subprocess import check_output\n", "print(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))\n", "\n", "import matplotlib.pyplot as plt\n", "\n", "# Any results you write to the current directory are saved as output."], "metadata": {"_cell_guid": "4e98e01c-8337-4653-860c-f531f83034f2", "_uuid": "cae29306733103cf248329602c4e0338d679621a"}}, {"cell_type": "markdown", "source": ["**Reading and transforming data for reducing size**"], "metadata": {"_cell_guid": "f434ca44-2142-40a5-8110-651198b7f36a", "_uuid": "b16ba627598756bbf5e452908f81a0dd764c0e6d"}}, {"execution_count": null, "cell_type": "code", "outputs": [], "source": ["dtypes = {'id':'int64', 'item_nbr':'int32', 'store_nbr':'int8', 'onpromotion':str}\n", "data = {\n", "    'tra': pd.read_csv('../input/train.csv', dtype=dtypes, parse_dates=['date']),\n", "    'tes': pd.read_csv('../input/test.csv', dtype=dtypes, parse_dates=['date']),\n", "    'ite': pd.read_csv('../input/items.csv'),\n", "    'sto': pd.read_csv('../input/stores.csv'),\n", "    'trn': pd.read_csv('../input/transactions.csv', parse_dates=['date']),\n", "    'hol': pd.read_csv('../input/holidays_events.csv', dtype={'transferred':str}, parse_dates=['date']),\n", "    'oil': pd.read_csv('../input/oil.csv', parse_dates=['date']),\n", "    }\n"], "metadata": {"collapsed": true, "_cell_guid": "f1cc037a-5e70-42a5-9361-096a4b307c8f", "_uuid": "d1eb3c8f73b0b1b19d5b9262c8140e4a199e9953"}}, {"execution_count": null, "cell_type": "code", "outputs": [], "source": ["train = data['tra'][(data['tra']['date'].dt.month == 8) & (data['tra']['date'].dt.day > 15)]"], "metadata": {"collapsed": true, "_cell_guid": "9bae44be-dccd-4c25-8071-edea26b8073c", "_uuid": "84a3d27864b4ed610ac7f15166cdcd746690492e"}}, {"execution_count": null, "cell_type": "code", "outputs": [], "source": ["train = train_all[(train_all['date'].dt.month == 8) & (train_all['date'].dt.day > 15)]\n", "train[\"onpromotion\"].fillna(0, inplace=True)\n", "train['id'] = train['id'].astype(np.uint32)\n", "train['store_nbr'] = train['store_nbr'].astype(np.uint8)\n", "train['item_nbr'] = train['item_nbr'].astype(np.uint32)\n", "train['unit_sales'] = train['unit_sales'].astype(np.uint32)\n", "train[\"onpromotion\"]=train[\"onpromotion\"].astype(np.int8)"], "metadata": {"collapsed": true, "_cell_guid": "6ef7d1cb-9dca-41a3-9744-5079a0451fc2", "_uuid": "2cf49241f6a174e1b0a8ca09657d1f787cebea29"}}, {"execution_count": null, "cell_type": "code", "outputs": [], "source": ["test = pd.read_csv(\"../input/test.csv\",parse_dates=[\"date\"])"], "metadata": {"collapsed": true, "_cell_guid": "52bf248f-a4c0-4fb5-926c-1c1ee2da5925", "_uuid": "b01dc4ed8401d553ba942c6409138922ee8fa674"}}, {"execution_count": null, "cell_type": "code", "outputs": [], "source": ["test[\"onpromotion\"].fillna(0, inplace=True)\n", "test['id'] = test['id'].astype(np.uint32)\n", "test['store_nbr'] = test['store_nbr'].astype(np.uint8)\n", "test['item_nbr'] = test['item_nbr'].astype(np.uint32)\n", "test[\"onpromotion\"]=test[\"onpromotion\"].astype(np.int8)"], "metadata": {"collapsed": true, "_cell_guid": "18530360-bd11-4139-9a56-0cacd2cfa82a", "_uuid": "0524627d7b8c4c67c47ff4df0e653003910461b3"}}, {"execution_count": null, "cell_type": "code", "outputs": [], "source": ["items = pd.read_csv('../input/items.csv')\n", "transaction = pd.read_csv('../input/transactions.csv')\n", "stores = pd.read_csv('../input/stores.csv')"], "metadata": {"collapsed": true, "_cell_guid": "37c3d27c-d0f1-41f7-85a5-6d919005e813", "_uuid": "857124153ff9ad1038af6576f61f13557bc23833"}}, {"cell_type": "markdown", "source": ["**Sampling because actual dataset is too large to fit in memory**"], "metadata": {"_cell_guid": "98574d66-5634-407f-9c42-0e1ebd15c2b0", "_uuid": "f08c71ad171d707729b5aa0a6fbc121e9e91d1dc"}}, {"execution_count": null, "cell_type": "code", "outputs": [], "source": ["print(\"Total Obsevations before sampling\",len(train))\n", "strain = train#.sample(frac=1.0,replace=True)\n", "print(\"Total Obsevations after sampling\",len(strain))"], "metadata": {"collapsed": true, "_cell_guid": "e89b6b8b-e05e-46ed-af63-58f5d21088a0", "_uuid": "59ff127dbbbc6c60ab3c067dce58f7552af18f22"}}, {"cell_type": "markdown", "source": ["**Megre data with item and store details**"], "metadata": {"_cell_guid": "8034c9c8-1667-42fa-a21f-30b9738d0497", "_uuid": "7b656135776d05da53bc75fe097514b35a58f2df"}}, {"execution_count": null, "cell_type": "code", "outputs": [], "source": ["df = strain.merge(right = items, on='item_nbr', right_index  = True)\n", "df = df.merge(right=stores,on='store_nbr', right_index  = True)"], "metadata": {"collapsed": true, "_cell_guid": "6901af14-9f74-4996-8992-91aecb13ef0a", "_uuid": "4670bf74ce0971e90d26e492e27ac4c5a26fd4ff"}}, {"execution_count": null, "cell_type": "code", "outputs": [], "source": ["df_test = test.merge(right = items, on='item_nbr', right_index  = True)\n", "df_test = df_test.merge(right=stores,on='store_nbr', right_index  = True)"], "metadata": {"collapsed": true, "_cell_guid": "6afcc94e-a14f-450e-8204-d50d7530b387", "_uuid": "056956d738defed3952d2c4d3bcacab35cbdc81d"}}, {"cell_type": "markdown", "source": ["**Converting categorical data to nomincal using onehotencoding**"], "metadata": {"_cell_guid": "ebd4a48d-18f6-4085-b072-98f3318aa4cc", "_uuid": "6a7664bc3f87a52d9685779c8a36f7c8805cc32b"}}, {"execution_count": null, "cell_type": "code", "outputs": [], "source": ["from sklearn.model_selection import GridSearchCV\n", "reduced_df = df.drop(['city','state','id'], axis=1)\n", "reduced_df_test = df_test.drop(['city','state','id'], axis=1)\n", "ohe_df = pd.get_dummies(df, columns=['family','type','city','state']) \n", "ohe_df_test = pd.get_dummies(df_test, columns=['family','type','city','state']) \n", "ohe_reduced_df = pd.get_dummies(reduced_df, columns=['family','type'])\n", "ohe_reduced_df_test = pd.get_dummies(reduced_df_test, columns=['family','type'])"], "metadata": {"collapsed": true, "_cell_guid": "cc08ff94-2094-44e6-a138-7ce0e690e493", "_uuid": "3b960507f278b2513aa3dee2ac5bfce8b0299ac4"}}, {"cell_type": "markdown", "source": ["**Divide train data in test and train set**"], "metadata": {"_cell_guid": "f6bc70d3-7a57-410b-b5c1-e4779b0fa101", "_uuid": "e64c383f663fb0305c8c733d2c384cb6f540446d"}}, {"execution_count": null, "cell_type": "code", "outputs": [], "source": ["from sklearn.model_selection   import train_test_split\n", "unitSales,reduced_unitSales = ohe_df['unit_sales'],ohe_reduced_df['unit_sales']\n", "features,reduced_features = ohe_df.drop('unit_sales', axis = 1),ohe_reduced_df.drop('unit_sales', axis = 1)\n", "\n", "X_train, X_test, y_train, y_test = train_test_split(features, unitSales, test_size=0.2, random_state=42)\n", "#X_train, X_test, y_train, y_test = train_test_split(features, unitSales, test_size=0.2, random_state=42)"], "metadata": {"collapsed": true, "_cell_guid": "14c20893-d873-46dd-be12-7206faf4d539", "_uuid": "1a9630c689f1b6c9cb3ef865cfb277443ace58a5"}}, {"cell_type": "markdown", "source": ["**Builiding RandomForestRegressor model**"], "metadata": {"_cell_guid": "7c935e96-09b9-4de8-96e7-26ea638174d8", "_uuid": "fd103d71e89a88dd81eef5ccf6fd937f8f389fb1"}}, {"execution_count": null, "cell_type": "code", "outputs": [], "source": ["from sklearn.tree import DecisionTreeRegressor\n", "from sklearn.ensemble import RandomForestRegressor\n", "from sklearn import cross_validation\n", "from sklearn.metrics import classification_report,confusion_matrix\n", "rf = RandomForestRegressor()\n", "rf.fit(X_train, y_train)"], "metadata": {"collapsed": true, "_cell_guid": "32dddd1c-a5a7-4067-b0ce-239463c946a9", "_uuid": "0308c583545029334d948a0ba4138cc0ab37ef3c"}}, {"cell_type": "markdown", "source": ["**Prediction on train set**"], "metadata": {"_cell_guid": "14139958-6fb3-4998-bd40-0508456fbef3", "_uuid": "c64a0b65158c58ff39785cd61a2c207469256856"}}, {"execution_count": null, "cell_type": "code", "outputs": [], "source": ["prediction = rf.predict(X_test)"], "metadata": {"collapsed": true, "_cell_guid": "43e7ccab-fdc3-451a-8b61-2ddcea38172e", "_uuid": "44c1e132f4a4e6e70f632c202517e7b16a11c718"}}, {"cell_type": "markdown", "source": ["**Convert prediction to Dataframe**"], "metadata": {"_cell_guid": "9a1797db-6b21-4b12-8797-4dc6e3ab333b", "_uuid": "07b6e44283d43a3f35c3041bb0e6a292b63efdb3"}}, {"execution_count": null, "cell_type": "code", "outputs": [], "source": ["prediction = pd.DataFrame(prediction, index=X_test.index,columns=['PedictedSale'])\n", "#Use below code to when predicting with test dataset\n", "#prediction = pd.DataFrame(prediction, index=test.index,columns=['PedictedSale'])\n"], "metadata": {"collapsed": true, "_cell_guid": "60cb343e-9c80-46fb-84b7-7453b5468072", "_uuid": "ad80ed5aa78cf25cb7ecff5cbe0207806bc97371"}}, {"cell_type": "markdown", "source": ["**Concate actual sale with predicted sale**"], "metadata": {"_cell_guid": "1c37d3eb-fc73-4891-85ef-30412658abe9", "_uuid": "f4fce88e381c7b4f4dce8d6e238f20f2e89dbdad"}}, {"execution_count": null, "cell_type": "code", "outputs": [], "source": ["sub = pd.concat([prediction, y_test.to_frame()], axis=1)\n", "sub = pd.concat([sub, X_test['perishable'].to_frame()], axis=1)"], "metadata": {"collapsed": true, "_cell_guid": "ca170906-c4a3-4395-bd69-38dc656670d1", "_uuid": "81d2cfcd7fc5b18ea080dcbcd178b1ed3afdc3db"}}, {"cell_type": "markdown", "source": ["**If negative sale (predicted as well as actual) convert it to zero**"], "metadata": {"_cell_guid": "0a013f31-aa5e-4c24-8d65-53e100d37fcf", "_uuid": "d3301c30db8133bb821a86fd087d34a424741bb2"}}, {"execution_count": null, "cell_type": "code", "outputs": [], "source": ["sub['newPS'] = sub.apply(lambda row: 0 if(row['PedictedSale']<0) else row['PedictedSale'] , axis=1)\n", "sub['newUS'] = sub.apply(lambda row: 0 if(row['unit_sales']<0) else row['unit_sales'] , axis=1)"], "metadata": {"collapsed": true, "_cell_guid": "e5782f97-1817-4734-adb9-94775f40c36d", "_uuid": "f301eb6d0e3518a26ed3a89dbc219cc077ddce78"}}, {"cell_type": "markdown", "source": ["Calculations for **Normalized Weighted Root Mean Squared Logarithmic Error (NWRMSLE) ** "], "metadata": {"_cell_guid": "bca1adcb-1ef9-40d9-8c8b-04db94907759", "_uuid": "7527c0831a2bbdc4d8333cc04cfd3b4e2713b6f6"}}, {"execution_count": null, "cell_type": "code", "outputs": [], "source": ["sub['newPS'] = np.log(sub.newPS + 1 )\n", "sub['newUS'] = np.log(sub.newPS + 1 )\n", "sub['yhatminusy'] = (sub['newPS']-sub['newUS'])**2\n", "sub['perishable'] = sub.perishable>0\n", "sub['perishableW'] = sub.apply(lambda row: 1.5 if(row['perishable']) else 1, axis=1)\n", "sub['comp1'] = sub.yhatminusy*sub.perishableW"], "metadata": {"collapsed": true, "_cell_guid": "fb3a126c-5fa7-459b-a7dd-2662ddb75df2", "_uuid": "267c38529e938da0e63b5293b6cbd62dec3f83cf"}}, {"cell_type": "markdown", "source": ["**Ploating Predicted Vs Actual Sale**"], "metadata": {"_cell_guid": "7bfe437c-538e-4342-ace8-dcfde0a6fa40", "_uuid": "598c7a7036109c6c1ee82af29e7a5b575e258015"}}, {"execution_count": null, "cell_type": "code", "outputs": [], "source": ["ax = sub['PedictedSale'].resample('m').mean().plot(figsize = (15, 6), color='red')\n", "fig = sub['unit_sales'].resample('m').mean().plot(ax=ax).get_figure()\n", "plt.legend(['Predicted Sale', 'actual'], loc='upper right')\n", "plt.show()"], "metadata": {"collapsed": true, "_cell_guid": "107fb258-f274-4a09-9509-b95f385b1a86", "_uuid": "3b7df57040244986d8484d62d32f4a6824ff29e7"}}, {"cell_type": "markdown", "source": ["**NWRMSLE and experiments**"], "metadata": {"_cell_guid": "c5c13886-31e0-487d-86aa-e1c64dcba9f7", "_uuid": "355997448934bbd8107dc642a6d5e4a1d659d2b7"}}, {"execution_count": null, "cell_type": "code", "outputs": [], "source": ["(sub.comp1.sum()/sub.perishableW.sum())**0.5"], "metadata": {"collapsed": true, "_cell_guid": "4a5143e8-c802-4a25-a61f-ea8a5899ba07", "_uuid": "2083c19903fd6252ba6f9bb195881d9e5b84771c"}}, {"cell_type": "markdown", "source": ["Following are results **without considering** 'city','state','id' while training model\n", "\n", "**Test Dataset** | **Train Dataset Size**  | **NWRMSLE **\n", "\n", "X_test  | 0.0001 | 1.1829906898993958\n", "\n", "X_test  | 0.001  | 1.094370399919558\n", "\n", "X_test  | 0.01    | 1.1043093902342433 \n", "\n", "____________________________________________\n", "\n", "Following are results **with considering** 'city','state','id' while training model\n", "\n", "**Test Dataset** | **Train Dataset Size**  | **NWRMSLE **\n", "\n", "X_test  | 0.0001 | 1.0789754618301821\n", "\n", "X_test  | 0.001  | 1.0688842227424471\n", "\n", "X_test  | 0.01    | 1.1023627733144485 \n", "\n", "\n", "\n", "test  | 0.001 | \n", "\n", "X_test  | 0.01 | 1.094370399919558\n", "\n"], "metadata": {"_cell_guid": "73df41d9-aa39-4068-b2e7-834bb17f42a2", "_uuid": "86331d9ab5c42b26b21505c91c7eaac9c5df8641"}}, {"cell_type": "markdown", "source": ["> **Egerly seeking comments ....Thank you..... **"], "metadata": {"_cell_guid": "29febdd6-11f3-4352-a4af-0e037ce26bf9", "_uuid": "63c887350e36f035f689b638c505e3157ed95adf"}}], "metadata": {"kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"}, "language_info": {"pygments_lexer": "ipython3", "mimetype": "text/x-python", "name": "python", "file_extension": ".py", "codemirror_mode": {"name": "ipython", "version": 3}, "nbconvert_exporter": "python", "version": "3.6.3"}}}