{"nbformat": 4, "nbformat_minor": 1, "metadata": {"kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "name": "python", "version": "3.6.3", "pygments_lexer": "ipython3", "mimetype": "text/x-python", "nbconvert_exporter": "python"}}, "cells": [{"cell_type": "markdown", "source": ["This notebook explains how to make a simple prediction algorithm based on the mean of the item sales per item and stores. The code shown here has been written to understand how to organize the data. The code is not efficient in terms of computing time but is fast enough. "], "metadata": {"_cell_guid": "5470680b-8c39-4153-8385-4896906568a8", "_uuid": "ed736772741d3543f9128c3460f69690fb8f0d2d"}}, {"cell_type": "markdown", "source": ["**Organizing the data in a matrix**. Matrix format is easy to understand and manipulate (even more if you come from matlab, which is my case). The training data will be organized in a three-dimensional matrix: date - items - stores. You can understand this as a book where each page contains the dates and the sells of all the items. Each page is an store (makes sense, no?). "], "metadata": {"_cell_guid": "a5556e54-198b-41e6-9353-9780c4681b5c", "_uuid": "943dfb6b36e2a738414e69d2b0cbdaad8064b754"}}, {"cell_type": "code", "source": ["# This Python 3 environment comes with many helpful analytics libraries installed\n", "# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n", "# For example, here's several helpful packages to load in \n", "\n", "import numpy as np # linear algebra\n", "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n", "\n", "# Input data files are available in the \"../input/\" directory.\n", "# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n", "\n", "from subprocess import check_output\n", "print(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))\n", "# Any results you write to the current directory are saved as output.\n", "\n", "\n", "\n", "#%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n", "# Load data (use only data from 2017)\n", "train = pd.read_csv('../input/train.csv',skiprows=range(1, 101688780))\n", "\n", "#Remove information about returned items\n", "ind = train['unit_sales'] >= 0\n", "train = train[ind]\n", "train.reset_index(inplace=True) #reset the indexes to account for the removed rows\n", "train['date'] = pd.to_datetime(train['date']) # make the date column a valid datetime object\n", "\n", "stores = pd.read_csv('../input/stores.csv') # load stores information\n", "items = pd.read_csv('../input/items.csv') #load information about items for sale\n", "\n", "\n", "\n", "grouped = train.groupby(['item_nbr', 'store_nbr']) #group data by items and stores\n", "grouped = grouped.groups # get the groups\n", "groupedKeys = list(grouped.keys()) # save the indexes that forma the groups in a list\n", "\n", "\n", "dates = pd.date_range(train['date'].iloc[0], train['date'].iloc[-1]) # generate all dates from january 01 of 2017\n", "itemSales = np.zeros((len(dates),len(items),len(stores))) # matrix to save the item sales\n", "\n", "#here we go throgh all the groups and organize the data in the three-dimensional matrix (Future kernels will use this)\n", "for k in groupedKeys:\n", "    indItem = np.where(items['item_nbr']==k[0]) # map the item_nbr to the index in the items.csv data\n", "    indStore = k[1] # index for the store\n", "    \n", "    index = grouped[k] #indexes that form te group 'k'\n", "    ind = dates.searchsorted(np.array(train['date'].iloc[index])) #every index is associated with a date, here we map the dates to our new format in the 3D matrix\n", "    \n", "    itemSales[ind,indItem,indStore-1] = train['unit_sales'].iloc[index] #assign the value of the items.\n", "    "], "execution_count": null, "metadata": {"_cell_guid": "12b73cf7-3d5f-4789-8a08-cc9c3e87c14d", "collapsed": true, "_uuid": "a92fe9ea69f6d3e11ca1b617a52a7e963701fa68"}, "outputs": []}, {"cell_type": "markdown", "source": ["Having the data organized as a matrix we can do several analysis (more later). The simpliest thing to do is use the average sales as a predictor. However, note that in the evaluation, the cost function is the Normalized Weighted Root Mean Squared Logarithmic Error. A look to the equation show that the error is calculated as the difference of the logarithms of the predicted versus the actual values. For this reason, our predictor should use the logarithm of the values to calculate the value of any stimator we calculate. In the case of the calculation of the mean:"], "metadata": {"_cell_guid": "23964ba8-efc2-4621-8474-c32a45187ada", "_uuid": "11f37119d74917c80c2eb80160d0918c12604a6f"}}, {"cell_type": "code", "source": ["meanSales = np.expm1(np.mean(np.log1p(itemSales),axis=0))"], "execution_count": null, "metadata": {"_cell_guid": "b73e9e9b-69e5-46ec-b221-465f88e78ecb", "collapsed": true, "_uuid": "ee965e78b8c035f3003306eae8a8d51fc99ff71f"}, "outputs": []}, {"cell_type": "markdown", "source": ["Note that we calculate the mean of the log1p() of the sales and then we undo the logarithm using exponential expm1(). Now we can use the calculated mean as prediction for the test data:"], "metadata": {"_cell_guid": "1897042e-6e79-47e5-b6a7-0e5719def186", "_uuid": "b2b0f7e96ad1536d7cba438a20810b03eed97903"}}, {"cell_type": "code", "source": ["test = pd.read_csv('../input/test.csv') #load test data\n", "\n", "grouped_test = test.groupby(['item_nbr','store_nbr']) # group the data by items and stores\n", "grouped_test = grouped_test.groups\n", "testkeys = list(grouped_test.keys())\n", "\n", "predicted = np.zeros((len(test),1)) # initialize the predicted output vector\n", "for k in testkeys:\n", "    # for cada group look in the meanSales the prediction. note that meanSales is 4100X54 (items,stores)\n", "    indItem = np.where(items['item_nbr']==k[0])\n", "    indStore = k[1]\n", "    \n", "    index = grouped_test[k]\n", "    predicted[index,0] = meanSales[indItem,indStore-1]\n", "    \n", "    \n", "# now save the data\n", "\n", "submit = pd.DataFrame(np.random.randn(len(test),2), columns=['id', 'unit_sales'])    \n", "submit['id'] = test['id']\n", "submit['unit_sales'] = predicted # undo the log transform\n", " \n", "submit.to_csv('prediction_01.csv', index = False)\n"], "execution_count": null, "metadata": {"_cell_guid": "f9672860-fdc5-4fe3-a1af-4bd9e3a1c2f5", "collapsed": true, "_uuid": "cd13c423aa082aac58135864ce6cef0f0f47a4e7"}, "outputs": []}, {"cell_type": "markdown", "source": ["Once this is submitted, a performance of **0.615** is obtained. Not state of the art, but pretty good for such a simple approach."], "metadata": {"_cell_guid": "a4cbc4f3-1957-4120-8b45-f3a9c69f0298", "_uuid": "b7b3f74185495ffc7e9d7ef358ce37e6890bd455"}}, {"cell_type": "markdown", "source": ["The key points exposed here are as follow: 1. The train data does not contain the days when a particular item is not sold, therefore not organizing the data leads to over-stimation of the mean. 2 working with the log of the data leads to an estimator that is more suited to the proposed task."], "metadata": {"_cell_guid": "45a8a9cf-f77b-4c7f-86b9-8298e66f5554", "_uuid": "31b804f52bac1f0b6d976d2b4e689c7c674daf46"}}, {"cell_type": "markdown", "source": ["I will be updating this with new methods to estimate the sales and including the information about the oil price and holidays to obtain better stimators."], "metadata": {"_cell_guid": "925fc791-095d-4753-a2d5-d4e4e7d39bd7", "_uuid": "84628be507bbbcf40fd3183bdb8aaa841e27b856"}}]}