{"cells":[{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport os\nfrom IPython.display import display_markdown as mkdown # as print\n\ndef nl():\n    print('\\n')\n\nfor f in os.listdir('../input'):\n    print(f.ljust(30) + str(round(os.path.getsize('../input/' + f) / 1000000, 2)) + 'MB')"},{"cell_type":"markdown","metadata":{},"source":"#### It looks like we are given quite a few sets as an input! Let's take a look at each one, starting with train and test."},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"df_train = pd.read_csv('../input/train.csv', nrows=500000)"},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"df_train[df_train.Dev_uni_proxima == 1].tail()"},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"#df_train = pd.read_csv('../input/train.csv', nrows=500000)\ndf_test = pd.read_csv('../input/test.csv', nrows=500000)\n\nnl()\nprint('Size of training set: ' + str(df_train.shape))\nprint(' Size of testing set: ' + str(df_test.shape))\n\nnl()\nprint('Columns in train: ' + str(df_train.columns.tolist()))\nprint(' Columns in test: ' + str(df_test.columns.tolist()))\n\nnl()\nprint(df_train.describe())"},{"cell_type":"markdown","metadata":{},"source":"`Demanda_uni_equil` is the target value that we are trying to predict.\n\nLet's take a look at the distribution:"},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"target = df_train['Demanda_uni_equil'].tolist()\n\ndef label_plot(title, x, y):\n    plt.title(title)\n    plt.xlabel(x)\n    plt.ylabel(y)\n\nplt.hist(target, bins=200, color='blue')\nlabel_plot('Distribution of target values', 'Demanda_uni_equil', 'Count')\nplt.show()"},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"print(\"Looks like we have some pretty big outliers, let's zoom in and try again\")\n\nprint('Data with target values under 50: ' + str(round(len(df_train.loc[df_train['Demanda_uni_equil'] <= 50]) / 5000, 2)) + '%')\n\nplt.hist(target, bins=50, color='blue', range=(0, 50))\nlabel_plot('Distribution of target values under 50', 'Demanda_uni_equil', 'Count')\nplt.show()\n"},{"cell_type":"markdown","metadata":{},"source":"From this distribution, we can see that some target values are much more common than others.\n\nLet's find the mode of the target and make a naive submission using that!"},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"from collections import Counter\nprint(Counter(target).most_common(10))\nprint('Our most common value is 2')\n\nsub = pd.read_csv('../input/sample_submission.csv')\nsub['Demanda_uni_equil'] = 2\nsub.to_csv('mostcommon.csv', index=False)"},{"cell_type":"markdown","metadata":{},"source":"Interestingly, our script (0.96080) performs worse than submitting `6` as the predicted value. This could be for two reasons:\n\n1) Our values are incorrect since we have only read the first 500,000 values of the dataset and the set is not randomised.  \n2) Due to the [evaluation metric](https://www.kaggle.com/c/grupo-bimbo-inventory-demand/details/evaluation) predicting 6 actually gives a lower overall logarithmic error.\n\nWe will begin by investigating the first possibility, and will look at whether the time-series has any effect on data."},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"pseudo_time = df_train.loc[df_train.Demanda_uni_equil < 20].index.tolist()\ntarget = df_train.loc[df_train.Demanda_uni_equil < 20].Demanda_uni_equil.tolist()\n\nplt.hist2d(pseudo_time, target, bins=[50,20])\nlabel_plot('Histogram of target value over index', 'Index', 'Target')\nplt.show()"},{"cell_type":"markdown","metadata":{},"source":"It does not look like the time-series has much effect on the data, except for that anomaly around 200k (we may take a closer look at another time)\n\nTo test out option 2, I created a script which evaluates the RMSLE on the training set to try and find the best value to submit, and it scored 0.82735:  \nhttps://www.kaggle.com/anokas/grupo-bimbo-inventory-demand/optimised-beat-the-benchmark\n\nNow that we have found the best naive submission to make, we can go onto looking at the other columns!\n\nWe will begin by looking at the time column, semana (meaning week)"},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"semana = df_train['Semana']\nprint(semana.value_counts())\nprint('\\nIt looks like by sampling only the first 500,000 columns, we have only sampled from week 3.\\nWe will have to take a larger portion of the dataset\\n')\n\ntiming = pd.read_csv('../input/train.csv', usecols=['Semana','Demanda_uni_equil'])\nprint('Size: ' + str(timing.shape))\n\nprint(timing['Semana'].value_counts())\nplt.hist(timing['Semana'].tolist(), bins=7, color='red')\nlabel_plot('Distribution of weeks in training data', 'Semana', 'Frequency')\nplt.show()\n\ntiming_test = pd.read_csv('../input/test.csv', usecols=['Semana'])\nprint(timing_test['Semana'].value_counts())"},{"cell_type":"markdown","metadata":{},"source":"We have a different set of weeks in the testing data for us to predict - meaning that this is likely a time series prediction problem for each of the product/client/location pairs in train and test sets.\n\nSince this appears to be a time series prediction task, let's see if there are any trends in the target value over time."},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"timing = timing.sample(1000000)\ntiming = timing.loc[timing['Demanda_uni_equil'] < 15] # We only want to look at the most common values\n\nplt.hist2d(timing['Semana'].tolist(), timing['Demanda_uni_equil'].tolist(), bins=[7, 15])\nlabel_plot('Distribution of target value over time', 'Week', 'Target')\nplt.show()"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":0}