{"cells":[{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"target = df_train['Demanda_uni_equil'].tolist()\n\ndef label_plot(title, x, y):\n    plt.title(title)\n    plt.xlabel(x)\n    plt.ylabel(y)\n\nplt.hist(target, bins=200, color='blue')\nlabel_plot('Distribution of target values', 'Demanda_uni_equil', 'Count')\nplt.show()\n\nprint(\"Looks like we have some pretty big outliers, let's zoom in and try again\")\n\nprint('Data with target values under 50: ' + str(round(len(df_train.loc[df_train['Demanda_uni_equil'] <= 50]) / 5000, 2)) + '%')\n\nplt.hist(target, bins=50, color='blue', range=(0, 50))\nlabel_plot('Distribution of target values under 50', 'Demanda_uni_equil', 'Count')\nplt.show()\n"},{"cell_type":"markdown","metadata":{},"source":"From this distribution, we can see that some target values are much more common than others.\n\nLet's find the mode of the target and make a naive submission using that!"},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"from collections import Counter\nprint(Counter(target).most_common(10))\nprint('Our most common value is 2')\n\nsub = pd.read_csv('../input/sample_submission.csv')\nsub['Demanda_uni_equil'] = 2\nsub.to_csv('mostcommon.csv', index=False)"},{"cell_type":"markdown","metadata":{},"source":"Interestingly, our script (0.96080) performs worse than submitting `6` as the predicted value. This could be for two reasons:\n\n1) Our values are incorrect since we have only read the first 500,000 values of the dataset and the set is not randomised.  \n2) Due to the [evaluation metric](https://www.kaggle.com/c/grupo-bimbo-inventory-demand/details/evaluation) predicting 6 actually gives a lower overall logarithmic error.\n\nWe will begin by investigating the first possibility, and will look at whether the time-series has any effect on data."},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"pseudo_time = df_train.loc[df_train.Demanda_uni_equil < 20].index.tolist()\ntarget = df_train.loc[df_train.Demanda_uni_equil < 20].Demanda_uni_equil.tolist()\n\nplt.hist2d(pseudo_time, target, bins=[50,20])\nlabel_plot('Histogram of target value over index', 'Index', 'Target')\nplt.show()"},{"cell_type":"markdown","metadata":{},"source":"It does not look like the time-series has much effect on the data, except for that anomaly around 200k (we may take a closer look at another time)\n\nTo test out option 2, I created a script which evaluates the RMSLE on the training set to try and find the best value to submit, and it scored 0.82735:  \nhttps://www.kaggle.com/anokas/grupo-bimbo-inventory-demand/optimised-beat-the-benchmark\n\nNow that we have found the best naive submission to make, we can go onto looking at the other columns!\n\nWe will begin by looking at the time column, semana (meaning week)"},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"semana = df_train['Semana']\nprint(semana.value_counts())\nprint('\\nIt looks like by sampling only the first 500,000 columns, we have only sampled from week 3.\\nWe will have to take a larger portion of the dataset\\n')\n\ntiming = pd.read_csv('../input/train.csv', usecols=['Semana','Demanda_uni_equil'])\nprint('Size: ' + str(timing.shape))\n\nprint(timing['Semana'].value_counts())\nplt.hist(timing['Semana'].tolist(), bins=7, color='red')\nlabel_plot('Distribution of weeks in training data', 'Semana', 'Frequency')\nplt.show()\n\ntiming_test = pd.read_csv('../input/test.csv', usecols=['Semana'])\nprint(timing_test['Semana'].value_counts())"},{"cell_type":"markdown","metadata":{},"source":"We have a different set of weeks in the testing data for us to predict - meaning that this is likely a time series prediction problem for each of the product/client/location pairs in train and test sets.\n\nSince this appears to be a time series prediction task, let's see if there are any trends in the target value over time."},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"timing = timing.sample(1000000)\ntiming = timing.loc[timing['Demanda_uni_equil'] < 15] # We only want to look at the most common values\n\nplt.hist2d(timing['Semana'].tolist(), timing['Demanda_uni_equil'].tolist(), bins=[7, 15])\nlabel_plot('Distribution of target value over time', 'Week', 'Target')\nplt.show()"},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nfrom subprocess import check_output\nprint(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))\n\n# Any results you write to the current directory are saved as output."},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"# This R environment comes with all of CRAN preinstalled, as well as many other helpful packages\n# The environment is defined by the kaggle/rstats docker image: https://github.com/kaggle/docker-rstats\n# For example, here's several helpful packages to load in \n\nlibrary(ggplot2) # Data visualization\nlibrary(readr) # CSV file I/O, e.g. the read_csv function\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nsystem(\"ls ../input\")\n\n# Any results you write to the current directory are saved as output."},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"# This Julia environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/julia docker image: https://github.com/kaggle/docker-julia\n# For example, here's a helpful package to load in \n\nusing DataFrames # data processing, CSV file I/O - e.g. readtable(\"../input/MyTable.csv\")\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nrun(`ls ../input`)\n\n# Any results you write to the current directory are saved as output."},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nfrom subprocess import check_output\nprint(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))\n\n# Any results you write to the current directory are saved as output."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":0}