{"cells":[
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport os\nfrom IPython.display import display_markdown as mkdown # as print\n\ndef nl():\n    print('\\n')\n\nfor f in os.listdir('../input'):\n    print(f.ljust(30) + str(round(os.path.getsize('../input/' + f) / 1000000, 2)) + 'MB')"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "#### It looks like we are given quite a few sets as an input! Let's take a look at each one, starting with train and test."
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "df_train = pd.read_csv('../input/train.csv', nrows=500000)\ndf_test = pd.read_csv('../input/test.csv', nrows=500000)\n\nnl()\nprint('Size of training set: ' + str(df_train.shape))\nprint(' Size of testing set: ' + str(df_test.shape))\n\nnl()\nprint('Columns in train: ' + str(df_train.columns.tolist()))\nprint(' Columns in test: ' + str(df_test.columns.tolist()))\n\nnl()\nprint(df_train.describe())"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "df_train.head(10)"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "`Demanda_uni_equil` is the target value that we are trying to predict.\n\nLet's take a look at the distribution:"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "target = df_train['Demanda_uni_equil'].tolist()\n\ndef label_plot(title, x, y):\n    plt.title(title)\n    plt.xlabel(x)\n    plt.ylabel(y)\n\nplt.hist(target, bins=200, color='blue')\nlabel_plot('Distribution of target values', 'Demanda_uni_equil', 'Count')\nplt.show()\n\nprint(\"Looks like we have some pretty big outliers, let's zoom in and try again\")\n\nprint('Data with target values under 50: ' + str(round(len(df_train.loc[df_train['Demanda_uni_equil'] <= 50]) / 5000, 2)) + '%')\n\nplt.hist(target, bins=50, color='blue', range=(0, 50))\nlabel_plot('Distribution of target values under 50', 'Demanda_uni_equil', 'Count')\nplt.show()\n\n"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "From this distribution, we can see that some target values are much more common than others.\n\nLet's find the mode of the target and make a naive submission using that!"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "from collections import Counter\nprint(Counter(target).most_common(10))\nprint('Our most common value is 2')\n\nsub = pd.read_csv('../input/sample_submission.csv')\nsub['Demanda_uni_equil'] = 2\nsub.to_csv('mostcommon.csv', index=False)"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "Interestingly, our script (0.96080) performs worse than submitting `6` as the predicted value. This could be for two reasons:\n\n1) Our values are incorrect since we have only read the first 500,000 values of the dataset and the set is not randomised.  \n2) Due to the [evaluation metric](https://www.kaggle.com/c/grupo-bimbo-inventory-demand/details/evaluation) predicting 6 actually gives a lower overall logarithmic error.\n\nWe will begin by investigating the first possibility, and will look at whether the time-series has any effect on data."
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "pseudo_time = df_train.loc[df_train.Demanda_uni_equil < 20].index.tolist()\ntarget = df_train.loc[df_train.Demanda_uni_equil < 20].Demanda_uni_equil.tolist()\n\nplt.hist2d(pseudo_time, target, bins=[50,20])\nlabel_plot('Histogram of target value over index', 'Index', 'Target')\nplt.show()"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "semana = df_train['Semana']\nprint(semana.value_counts())\nprint('\\nIt looks like by sampling only the first 500,000 columns, we have only sampled from week 3.\\nWe will have to take a larger portion of the dataset\\n')\n\ntiming = pd.read_csv('../input/train.csv', usecols=['Semana','Demanda_uni_equil'])\nprint('Size: ' + str(timing.shape))\n\nprint(timing['Semana'].value_counts())\nplt.hist(timing['Semana'].tolist(), bins=7, color='red')\nlabel_plot('Distribution of weeks in training data', 'Semana', 'Frequency')\nplt.show()\n\ntiming_test = pd.read_csv('../input/test.csv', usecols=['Semana'])\nprint(timing_test['Semana'].value_counts())"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "We have a different set of weeks in the testing data for us to predict - meaning that this is likely a time series prediction problem for each of the product/client/location pairs in train and test sets.\n\nSince this appears to be a time series prediction task, let's see if there are any trends in the target value over time."
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "timing = timing.sample(1000000)\ntiming = timing.loc[timing['Demanda_uni_equil'] < 15] # We only want to look at the most common values\n\nplt.hist2d(timing['Semana'].tolist(), timing['Demanda_uni_equil'].tolist(), bins=[7, 15])\nlabel_plot('Distribution of target value over time', 'Week', 'Target')\nplt.show()"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": ""
 }
],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}}, "nbformat": 4, "nbformat_minor": 0}