{"cells": [{"source": ["## [ Data Load (Train/Test) & Preprocessing to come up with factors ] ##\n", "\n", "**[ Points to be considered as in Data Analysis ]**\n", "\n", "P0) test.csv - predict for the period 2017.08.16 ~ 2017.08.20 \n", "=> hence pick only for AUG data from train.csv, concerning the seasoning impact etc.\n", "       \n", "**train.csv :**\n", "\n", "P1) Negative values of unit_sales represent returns of that particular item.\n", "\n", "P2) Approximately 16% of the onpromotion values in this file are NaN.\n", "\n", "P3) The training data does not include rows for items that had zero unit_sales for a store/date combination. There is no information as to whether or not the item was in stock for the store on the date, and teams will need to decide the best way to handle that situation. Also, there are a small number of items seen in the training data that aren't seen in the test data.\n", "\n", "**test.csv :**\n", "\n", "P4) Test data has a small number of items that are not contained in the training data. Part of the exercise will be to predict a new item sales based on similar products.\n", "\n", "**stores.csv :**\n", "\n", "P5) city, state, type, and cluster. cluster is a grouping of similar stores.\n", "\n", "**items.csv :**\n", "\n", "P6) family, class, and perishable.\n", "\n", "P7) Items marked as perishable have a score weight of 1.25; otherwise, the weight is 1.0.\n", "\n", "**transactions.csv :**\n", "\n", "P8) The count of sales transactions for each date, store_nbr combination. Only included for the training data timeframe.\n", "\n", "**oil.csv :**\n", "\n", "P9) Daily oil price. Includes values during both the train and test data timeframe. (Ecuador is an oil-dependent country and it's economical health is highly vulnerable to shocks in oil prices.)\n", "\n", "**holidays_events.csv :**\n", "\n", "P10) Pay special attention to the transferred column. A holiday that is transferred officially falls on that calendar day, but was moved to another date by the government. A transferred day is more like a normal day than a holiday. To find the day that it was actually celebrated, look for the corresponding row where type is Transfer. For example, the holiday Independencia de Guayaquil was transferred from 2012-10-09 to 2012-10-12, which means it was celebrated on 2012-10-12. Days that are type Bridge are extra days that are added to a holiday (e.g., to extend the break across a long weekend). These are frequently made up by the type Work Day which is a day not normally scheduled for work (e.g., Saturday) that is meant to payback the Bridge. Additional holidays are days added a regular calendar holiday, for example, as typically happens around Christmas (making Christmas Eve a holiday).\n", "\n", "**Additional Notes**\n", "\n", "P11) Wages in the public sector are paid every two weeks on the 15 th and on the last day of the month. Supermarket sales could be affected by this.\n", "P12) A magnitude 7.8 earthquake struck Ecuador on April 16, 2016. People rallied in relief efforts donating water and other first need products which greatly affected supermarket sales for several weeks after the earthquake.\n", "\n", "[ Data analysis and wrangle ]\n", "- REMOVE_NOISE_BY_NAN_COLUMNS\n", "- CHECK_CORRELATION_AND_KEY_FACTORS\n", "- LOG_TRANSFORMATION\n", "\n", "[ Factor analysis - too much relationships between the factors may cause inaccurate result. so proceed PCA analysis ]\n", "\n", "[ Data Model Selection ]\n", "- Apply models and get R2 score with cross-validation on KFold - (1) data  (2) dataPCA\n", "- cross validation : As test sets can provide unstable result because of sampling, the solution is to systematically perform the sampling & average the results. It is a statistical approach to observe many results and take the average of them.\n", "\n", "[ Prediction - Now get the best predict model, applying GridSearch ]\n", "- GridSearch : Searching for the optimal parameters of an algorithym to achieve the best possible predictive performance\n", "\n"], "metadata": {"_cell_guid": "d680f230-0c97-3375-f47a-16cb2cb12e77", "_uuid": "51d5a909dbcd1a2ab49f9d085d009b6b90f77ffb"}, "cell_type": "markdown"}, {"source": ["import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "import seaborn\n", "import gc; gc.enable()\n", "from sklearn import preprocessing\n", "\n", "from sklearn.decomposition import PCA\n", "from sklearn.model_selection import KFold\n", "from sklearn import linear_model\n", "from sklearn.metrics import make_scorer\n", "from sklearn.ensemble import BaggingRegressor\n", "from sklearn.ensemble import RandomForestRegressor\n", "from sklearn import svm\n", "from sklearn.metrics import r2_score\n", "from sklearn.ensemble import AdaBoostRegressor\n", "from sklearn.model_selection import cross_val_score\n", "from sklearn.tree import DecisionTreeRegressor\n", "from sklearn.model_selection import GridSearchCV\n", "\n", "dtypes = {'id':'int64', 'item_nbr':'int32', 'store_nbr':'int8', 'onpromotion':str}\n", "input = {\n", "    'train'  : pd.read_csv('../input/train.csv', dtype=dtypes, parse_dates=['date']),\n", "    'test'   : pd.read_csv('../input/test.csv', dtype=dtypes, parse_dates=['date']),\n", "    'items'  : pd.read_csv('../input/items.csv'),\n", "    'stores' : pd.read_csv('../input/stores.csv'),\n", "    'txns'   : pd.read_csv('../input/transactions.csv', parse_dates=['date']),\n", "    'holevts': pd.read_csv('../input/holidays_events.csv', dtype={'transferred':str}, parse_dates=['date']),\n", "    'oil'    : pd.read_csv('../input/oil.csv', parse_dates=['date']),\n", "    }\n", "input['train'].head()"], "metadata": {"_cell_guid": "c2d0b1a9-a6e3-7ace-ab73-6375fccb210a", "_uuid": "13736a4dae4ab0138b952b72bfd42a44614fa1fa", "scrolled": false}, "cell_type": "code", "outputs": [], "execution_count": null}, {"source": ["test = input['test']\n", "#P0) test.csv - predict for the period 2017.08.16 ~ 2017.08.20 \n", "# => hence pick only for AUG data from train.csv, concerning the seasoning impact etc.\n", "train = input['train'][(input['train']['date'].dt.month == 8) & (input['train']['date'].dt.day > 15)]\n", "#P1) Negative values of unit_sales represent returns of that particular item\n", "# => in prediction, returns means same as zero_sales - so convert to 0                       \n", "unit_sales = train['unit_sales'].values\n", "unit_sales[unit_sales < 0.] = 0.\n", "#check histogram and see if better to log-transform\n", "seaborn.distplot(train['unit_sales']);"], "metadata": {"_cell_guid": "18037571-c728-461c-bf12-85e57a65d9fb", "_uuid": "f0e34e4ea9e68b64ee718229bfcb0befe8a39ef0"}, "cell_type": "code", "outputs": [], "execution_count": null}, {"source": ["train['unit_sales'] = np.log1p(unit_sales)\n", "seaborn.distplot(np.log1p(unit_sales));"], "metadata": {"_cell_guid": "1343186d-2bdc-4459-8534-b7e700113c3d", "_uuid": "dd3dd670326554bfd37be1458bf3f95867a034d8"}, "cell_type": "code", "outputs": [], "execution_count": null}, {"source": ["def proc_object_data(df):\n", "    col = [c for c in df.columns if df[c].dtype == 'object']\n", "    df[col] = df[col].apply(preprocessing.LabelEncoder().fit_transform)\n", "    return df\n", "\n", "#P5) stores - city, state, type, and cluster. cluster is a grouping of similar stores.\n", "input['stores'] = proc_object_data(input['stores'])\n", "train = pd.merge(train, input['stores'], how='left', on=['store_nbr'])\n", "test = pd.merge(test, input['stores'], how='left', on=['store_nbr'])\n", "\n", "#P6) items - family, class, and perishable.\n", "#P7) Items marked as perishable have a score weight of 1.25; otherwise, the weight is 1.0.\n", "input['items'] = proc_object_data(input['items'])\n", "train = pd.merge(train, input['items'], how='left', on=['item_nbr'])\n", "test = pd.merge(test, input['items'], how='left', on=['item_nbr'])\n", "\n", "#P9) Daily oil price. Includes values during both the train and test data timeframe. (Ecuador is an oil-dependent country and it's economical health is highly vulnerable to shocks in oil prices.)\n", "train = pd.merge(train, input['oil'], how='left', on=['date'])\n", "test = pd.merge(test, input['oil'], how='left', on=['date'])\n", "\n", "#P8) The count of sales transactions for each date, store_nbr combination. Only included for the training data timeframe.\n", "# => consider that transaction volume may not have relation with sales qty for each product - so ignore.\n", "train.head(n=10)"], "metadata": {"_cell_guid": "595a06c8-4b2e-4311-989b-0f62d9f96676", "_uuid": "86eae491cd7f5a87661257b4498d35b2d54df0b4"}, "cell_type": "code", "outputs": [], "execution_count": null}, {"source": ["#P10) A holiday that is transferred officially falls on that calendar day, but was moved to another date by the government. A transferred day is more like a normal day than a holiday. To find the day that it was actually celebrated, look for the corresponding row where type is Transfer. \n", "# => filter out : transferred == 'TRUE'\n", "holevts = input['holevts']\n", "holevts = holevts[(holevts.transferred != 'TRUE') & (holevts.transferred != 'True')]\n", "# Days that are type Bridge are extra days that are added to a holiday (e.g., to extend the break across a long weekend). \n", "# These are frequently made up by the type Work Day which is a day not normally scheduled for work (e.g., Saturday) that is meant to payback the Bridge.\n", "# => filter out : type = 'Work Day'\n", "holevts = holevts[holevts.type != 'Work Day']\n", "#Additional holidays are days added a regular calendar holiday, for example, as typically happens around Christmas (making Christmas Eve a holiday)\n", "#type (Holiday, Transfer, Bridge, Additional) => Holiday // type (Event) => Event\n", "holevts['on_hol'] = holevts['type'].map({\"Holiday\":\"Holiday\", \"Transfer\":\"Holiday\", \"Bridge\":\"Holiday\", \"Additional\":\"Holiday\"})\n", "holevts['on_evt'] = holevts['type'].map({\"Event\":\"Event\"})\n", "col = [c for c in holevts if c in ['date', 'locale_name','on_hol','on_evt']]\n", "holevts_L = holevts[holevts.locale == 'Local'][col].rename(columns={'locale_name':'city'})\n", "holevts_R = holevts[holevts.locale == 'Regional'][col].rename(columns={'locale_name':'state'})\n", "holevts_N = holevts[holevts.locale == 'National'][col]\n", "\n", "# Actually our test data is only for 2017.08.16~20, at which there's no holiday - hene it won't impact this case. \n", "# But still proceed to prepare factors (on_hol, on_evt) as these might be one of key factor in general.\n", "train = pd.merge(train, holevts_L, how='left', on=['date','city'])\n", "train = pd.merge(train, holevts_R, how='left', on=['date','state'])\n", "train = pd.merge(train, holevts_N, how='left', on=['date'])\n", "test = pd.merge(test, holevts_L, how='left', on=['date','city'])\n", "test = pd.merge(test, holevts_R, how='left', on=['date','state'])\n", "test = pd.merge(test, holevts_N, how='left', on=['date'])\n", "train.head(n=10)"], "metadata": {"_cell_guid": "534d9942-59f2-43cb-bceb-85b0967b82f1", "_uuid": "a9ba5a59c807343a19f07fb9beae502f24e72590"}, "cell_type": "code", "outputs": [], "execution_count": null}, {"source": ["data = pd.concat([train,test],ignore_index=True)\n", "\n", "def proc_cvt_data(df):\n", "    df['year'] = df['date'].dt.year\n", "    df['month'] = df['date'].dt.month\n", "    df['day'] = df['date'].dt.day\n", "    df['date'] = df['date'].dt.dayofweek    \n", "    df['wage'] = df['day'].map({15:1, 31:1})\n", "    df['onpromotion'] = df['onpromotion'].map({'False': 0, 'True': 1})\n", "    df['perishable'] = df['perishable'].map({0:1.0, 1:1.25})\n", "    df['on_hol'] = np.where(df[[\"on_hol_x\",\"on_hol_y\",\"on_hol\"]].apply(lambda x: x.str.contains('Holiday')).any(1), 1,0)\n", "    df['on_evt'] = np.where(df[[\"on_evt_x\",\"on_evt_y\",\"on_evt\"]].apply(lambda x: x.str.contains('Event')).any(1), 1,0)\n", "    df = df.drop([\"on_hol_x\",\"on_hol_y\",\"on_evt_x\",\"on_evt_y\",\"locale_name\"], axis=1)\n", "    df = df.fillna(-1)\n", "    return df\n", "data = proc_cvt_data(data)\n", "\n", "train = data[data.id < 125497040]\n", "test = data[data.id >= 125497040]\n", "labels = train[\"unit_sales\"]\n", "ids = test[\"id\"]\n", "train.head(n=20)"], "metadata": {"_cell_guid": "dbeca86e-768c-4009-857f-13d1ca642e79", "_uuid": "8797f8cd2eeba16a7c492905480443370f5f79be"}, "cell_type": "code", "outputs": [], "execution_count": null}, {"source": [" ## [ Data analysis and wrangle ] ##\n", "\n", "- REMOVE_NOISE_BY_NAN_COLUMNS\n", "- CHECK_CORRELATION_AND_KEY_FACTORS\n", "- LOG_TRANSFORMATION"], "metadata": {"_cell_guid": "7b5e8464-f184-42db-b4cf-11fce4e5fcb4", "_uuid": "98751790c2c8f12b0028024245794a1d850a8ebf"}, "cell_type": "markdown"}, {"source": ["del input['train']; gc.collect();\n", "del input['test']; gc.collect();\n", "del input['items']; gc.collect();\n", "del input['stores']; gc.collect();\n", "del input['txns']; gc.collect();\n", "del input['holevts']; gc.collect();\n", "del input['oil']; gc.collect();\n", "\n", "# Count the number of NaNs each column has. Display columns having more than 30% NAN => skip it as we've done preprocessing\n", "#DROP_NAN_PCT = 0.3\n", "#nans=pd.isnull(data).sum()\n", "#nans[nans > data.shape[0] * DROP_NAN_PCT]"], "metadata": {"collapsed": true, "_cell_guid": "92ba1dbf-0b3b-7d19-ba9f-702c59991612", "_uuid": "dbeb656bd595f43f759e396e60095b8aeb860cdf", "scrolled": false}, "cell_type": "code", "outputs": [], "execution_count": null}, {"source": ["# CHECK_CORRELATION_AND_KEY_FACTORS\n", "corrmat = train.corr()\n", "f, ax = plt.subplots(figsize=(12, 12))\n", "k = 30  #number of variables for heatmap\n", "cols = corrmat.nlargest(k, 'unit_sales')['unit_sales'].index\n", "cm = np.corrcoef(train[cols].values.T)\n", "seaborn.set(font_scale=1)\n", "hm = seaborn.heatmap(cm, cbar=True, annot=True, square=True, fmt='.2f', annot_kws={'size': 10}, yticklabels=cols.values, xticklabels=cols.values)\n", "plt.show()"], "metadata": {"_cell_guid": "e05ed876-a14c-42ab-af33-2a5d1d7715d2", "_uuid": "be62f16fa4dfb088f5153d1b577ff7789697c14d"}, "cell_type": "code", "outputs": [], "execution_count": null}, {"source": ["## [ Factor analysis - too much relationships between the factors may cause inaccurate result. so proceed PCA analysis ]##"], "metadata": {"_cell_guid": "4affb7d5-578e-6862-243f-630f21a05934", "_uuid": "2d8ef0a7e634dc69321377aabbb817f5fa186390"}, "cell_type": "markdown"}, {"source": ["# heatmap check shows no strong relationships between the factors so skip the PCA analysis\n", "#data = data.drop(\"unit_sales\", 1)\n", "#pca = PCA(whiten=True).fit(data)\n", "#np.cumsum(pca.explained_variance_ratio_)\n", "#shows the variance is explained by N factors "], "metadata": {"collapsed": true, "_cell_guid": "6954036d-1415-4bcf-d67b-ac7a410f8a65", "_uuid": "cfe7ea02f7989c219ce62b1b13d70982757265b4"}, "cell_type": "code", "outputs": [], "execution_count": null}, {"source": ["## [ Data Model Selection ] ##\n", "\n", "Apply models and get R2 score with cross-validation on KFold -\n", "cross validation - As test sets can provide unstable result because of sampling, the solution is to systematically perform the sampling & average the results. It is a statistical approach to observe many results and take the average of them."], "metadata": {"_cell_guid": "8901b708-1465-3e16-aca4-f3ea1bac97d8", "_uuid": "751604ca33ff3584256d5237a4d9756f027166df"}, "cell_type": "markdown"}, {"source": ["def apply_models(train,labels):\n", "    results={}\n", "    def train_get_score(clf):        \n", "        cv = KFold(n_splits=2,shuffle=True,random_state=45)\n", "        r2_val_score = cross_val_score(clf, train, labels, cv=cv,scoring=make_scorer(r2_score))\n", "        return [r2_val_score.mean()]\n", "\n", "    results[\"Linear\"]=train_get_score(linear_model.LinearRegression())\n", "    results[\"Ridge\"]=train_get_score(linear_model.Ridge())\n", "    #results[\"Bayesian Ridge\"]=train_get_score(linear_model.BayesianRidge())\n", "    results[\"Hubber\"]=train_get_score(linear_model.HuberRegressor())\n", "    results[\"Lasso\"]=train_get_score(linear_model.Lasso(alpha=1e-4))\n", "    results[\"RandomForest\"]=train_get_score(RandomForestRegressor())\n", "    #results[\"SVM RBF\"]=train_get_score(svm.SVR())\n", "    #results[\"SVM Linear\"]=train_get_score(svm.SVR(kernel=\"linear\"))\n", "    \n", "    results = pd.DataFrame.from_dict(results,orient='index')\n", "    results.columns=[\"R Square Score\"] \n", "    results.plot(kind=\"bar\",title=\"Model Scores\")\n", "    axes = plt.gca()\n", "    axes.set_ylim([0.5,1])\n", "    return results\n", "\n", "apply_models(train,labels)"], "metadata": {"_cell_guid": "a85a2fa0-59ec-c055-f088-ab0a29ea68bd", "_uuid": "2330f5fce90f3425f16fe3b8584e8caeecb4b8cd"}, "cell_type": "code", "outputs": [], "execution_count": null}, {"source": ["# [ Prediction - Now get the best predict model, applying GridSearch ]\n", "\n", "GridSearch - Searching for the optimal parameters of an algorithym to achieve the best possible predictive performance"], "metadata": {"_cell_guid": "7aa307c3-7834-47b5-b5cd-9837917720c2", "_uuid": "3b7d078319b388b82e49136ddef08c9f8ca2bbbd"}, "cell_type": "markdown"}, {"source": ["def get_predict_model(clf, train, labels):\n", "    cv = KFold(n_splits=2,shuffle=True,random_state=45)\n", "    parameters = {'alpha': [1000,100,10],'epsilon' : [1.2,1.25,1.50],'tol' : [1e-10]}\n", "    grid_obj = GridSearchCV(clf, parameters, cv=cv,scoring=make_scorer(r2_score))\n", "    predict_model = grid_obj.fit(train, labels).best_estimator_\n", "    #predict_model = clf.fit(train,labels)\n", "    return predict_model\n", "\n", "predict_model = get_predict_model(linear_model.HuberRegressor(),train,labels)"], "metadata": {"_cell_guid": "77b2423f-1b69-0d57-8cc7-ebf3b6d54b23", "_uuid": "eb7103d8e739c7562439e25006abb3ad266a8adf", "scrolled": false}, "cell_type": "code", "outputs": [], "execution_count": null}, {"source": ["predictions = (np.exp(predict_model.predict(test)) - 1) # restoring unit values \n", "sub = pd.DataFrame({\"id\": ids, \"unit_sales\": predictions})\n", "print(sub)"], "metadata": {"_cell_guid": "4e591e4d-20de-4926-be69-d563310b2dc1", "_uuid": "e7edb89a9c6b4810a0af4e374ebaf917d60b2dec"}, "cell_type": "code", "outputs": [], "execution_count": null}, {"source": ["sub.to_csv(\"favorita_submission.csv\", index=False)"], "metadata": {"collapsed": true, "_cell_guid": "fcb069e8-8310-4311-8760-2be8d33dec0e", "_uuid": "5637df41ff05f5bdf1ecd8e952d801deb67d1710"}, "cell_type": "code", "outputs": [], "execution_count": null}], "nbformat": 4, "metadata": {"_change_revision": 0, "language_info": {"nbconvert_exporter": "python", "version": "3.6.3", "pygments_lexer": "ipython3", "codemirror_mode": {"name": "ipython", "version": 3}, "name": "python", "file_extension": ".py", "mimetype": "text/x-python"}, "kernelspec": {"name": "python3", "display_name": "Python 3", "language": "python"}, "_is_fork": false}, "nbformat_minor": 1}