{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "2bc40e98-7925-756e-717e-ca4ebc263e97"
      },
      "source": [
        "<h1> Grupo Bimbo Inventory Demand</h1>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "c63b75ba-8e9e-ec24-07b7-ade52f93a43e"
      },
      "outputs": [],
      "source": [
        "# This Python 3 environment comes with many helpful analytics libraries installed\n",
        "# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n",
        "# For example, here's several helpful packages to load in \n",
        "\n",
        "import numpy as np # linear algebra\n",
        "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
        "\n",
        "# Input data files are available in the \"../input/\" directory.\n",
        "# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n",
        "\n",
        "from subprocess import check_output\n",
        "print(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))\n",
        "\n",
        "# Any results you write to the current directory are saved as output."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "ec87d310-7b67-1281-c743-0fe24a51b034"
      },
      "source": [
        "<h1>Background</h1>\n",
        "\n",
        "In this challenge we are trying to predict the demand of a certain product on a certain week.  We are given data including the demand of each product across 9 weeks at several different Bimbo stores.  Initially it doesn't seem apparent what information we can gain from the route id information.  I would expect that we can try to cluster the products together based on product type and whether or not they peak and trough in demand at similar times."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ebc765b9-95ef-a8cf-a9a7-e85b7f697a55"
      },
      "outputs": [],
      "source": [
        "# Read in all the data\n",
        "df_train = pd.read_csv('../input/train.csv', nrows=500000)\n",
        "df_test = pd.read_csv('../input/test.csv', nrows=500000)\n",
        "cliente_tabla = pd.read_csv('../input/cliente_tabla.csv')\n",
        "producto_tabla = pd.read_csv('../input/producto_tabla.csv')\n",
        "town_state = pd.read_csv('../input/town_state.csv')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "4de95007-99a4-818e-d6d3-2b11c65fc5c8"
      },
      "outputs": [],
      "source": [
        "# What data columns do each hold?\n",
        "print('Train data columns')\n",
        "print(df_train.columns)\n",
        "\n",
        "print('Test data columns')\n",
        "print(df_test.columns)\n",
        "\n",
        "print('Client table data columns')\n",
        "print(cliente_tabla.columns)\n",
        "\n",
        "print('Prodcut table data columns')\n",
        "print(producto_tabla.columns)\n",
        "\n",
        "print('Town state columns')\n",
        "print(town_state.columns)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "98e0b7fc-a507-c783-354b-ef6dc402150f"
      },
      "outputs": [],
      "source": [
        "# Let's quickly merge the tables\n",
        "df_train = pd.merge(df_train, cliente_tabla, on=\"Cliente_ID\", how=\"left\")\n",
        "df_train = pd.merge(df_train, producto_tabla, on=\"Producto_ID\", how=\"left\")\n",
        "df_train = pd.merge(df_train, town_state, on=\"Agencia_ID\", how=\"left\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "02a555de-6b22-99ef-0854-4dbc9292e2f1"
      },
      "outputs": [],
      "source": [
        "df_train.describe()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "3b02e31a-c780-0ddb-f050-6a681e57461d"
      },
      "outputs": [],
      "source": [
        "# Let's look at the distribution of the variable we are trying\n",
        "# to describe - 'demanda_uni_equil'\n",
        "# Let's create a histogram of this column of data\n",
        "target = df_train['Demanda_uni_equil'].tolist()\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline\n",
        "plt.hist(target, bins=30)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "81fc2059-4609-0d49-ecb8-9dea51aa434c"
      },
      "outputs": [],
      "source": [
        "# Since the distribution has large outliers we can reduce the range\n",
        "plt.hist(target, bins=30, range=(0,30))\n",
        "plt.title('Distribution of target values')\n",
        "plt.xlabel('Demanda_uni_equil')\n",
        "plt.ylabel('Count')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "fbf29cff-0153-64b1-4b6e-242a34264dfc"
      },
      "outputs": [],
      "source": [
        "# Since we only have to predict the value for 'demanda_uni_equil' \n",
        "# let's try submitting a model with the mean and with the mode\n",
        "# of the training data\n",
        "\n",
        "# Two ways of printing out most common values\n",
        "# As a data frame\n",
        "print(df_train.Demanda_uni_equil.value_counts()[0:10])\n",
        "print('\\n')\n",
        "# As a list using Counter\n",
        "from collections import Counter\n",
        "print(Counter(target).most_common(10))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "768ad6b8-7927-6e02-34ce-dd1ce7714d2c"
      },
      "outputs": [],
      "source": [
        "# Generate a submission by replacing all the values in \n",
        "# 'Demanda_uni_equil' by the mode 2\n",
        "# I will pass these files into Kaggle and see the output\n",
        "\n",
        "#sub_mode = pd.read_csv('../input/sample_submission.csv')\n",
        "#sub_mode['Demanda_uni_equil'] = 2\n",
        "#sub_mode.to_csv('mode.csv', index=False)\n",
        "\n",
        "#sub_mean = pd.read_csv('../input/sample_submission.csv')\n",
        "#sub_mean['Demanda_uni_equil'] = 6.9\n",
        "#sub_mean.to_csv('mean.csv', index=False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "92d27c80-2531-9ba9-b49f-03bf77ff5305"
      },
      "outputs": [],
      "source": [
        "# Does time series have any effect on the data?\n",
        "# Let's first select all the data with a demand < 20\n",
        "psuedo_time = df_train.loc[df_train.Demanda_uni_equil < 20].index.tolist()\n",
        "target = df_train.loc[df_train.Demanda_uni_equil < 20].Demanda_uni_equil.tolist()\n",
        "\n",
        "\n",
        "plt.hist2d(psuedo_time, target, bins=[50, 20])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "9b9b7ed1-9cba-8f35-42d7-acbb8c1d5185"
      },
      "outputs": [],
      "source": [
        "# In order to avoid having to submit to Kaggle to figure out the\n",
        "# error, let's instead write our own root mean squared logarithmic\n",
        "# error function\n",
        "\n",
        "import math\n",
        "def rmsle(y, y_pred):\n",
        "    assert len(y) == len(y_pred)\n",
        "    terms_to_sum = [(math.log(y_pred[i] + 1) - math.log(y[i] + 1))\n",
        "                   ** 2.0 for i, pred in enumerate(y_pred)]\n",
        "    return ((1.0 / len(y)) * sum(terms_to_sum)) ** 0.5\n",
        "\n",
        "# Let's now pass in all values from 3-12 to find the best naive \n",
        "# estimate.\n",
        "errors = []\n",
        "for num in range(3,12):\n",
        "    length = len(df_train.Demanda_uni_equil)\n",
        "    errors += [rmsle(df_train.Demanda_uni_equil, [num for i in \n",
        "                                                range(length)])]\n",
        "print(errors)\n",
        "bestNaive = np.argmin(errors)\n",
        "print(bestNaive)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "b1c09654-a1e9-7e16-7806-f37add19a9ed"
      },
      "source": [
        "Thus we can see that if we were to predict the same demand for each value the optimal value would be to predict 1 unit of demand for each object.  This gives us an error of 0.78 when computing root mean squared logarithmic error.  \n",
        "\n",
        "What does this error actually mean?  We could try a root mean squared error as well since that is a little more intuitive."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "c8258997-e1cc-c42f-20a0-4705e63b86c8"
      },
      "outputs": [],
      "source": [
        "# Look at the other columns\n",
        "semana = df_train['Semana']\n",
        "print(semana.value_counts())\n",
        "\n",
        "print(\"Thus we see that the first 500000 rows of the data only includes data from week 3\")\n",
        "print('\\nWe want to sample from the entire distribution\\n')\n",
        "\n",
        "# Let's look at the 'Semana' column for the entire dataset\n",
        "timing = pd.read_csv('../input/train.csv', usecols=['Semana', \n",
        "                                                   'Demanda_uni_equil'])\n",
        "print('Size: ' + str(timing.shape))\n",
        "#print(timing['Semana'].value_counts())\n",
        "\n",
        "#Let's visualize this with a histogram\n",
        "#plt.hist(timing['Semana'].tolist(), bins=7)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "1570caa0-c8a1-0682-2c2f-c7e7f4d286f7"
      },
      "outputs": [],
      "source": [
        "df_train = DataFrame()\n",
        "chunksize = 10**7\n",
        "for chunk in pd.read_csv('../input/train.csv', chunksize=chunksize):\n",
        "    samplesize = len(chunk) / 10\n",
        "    df_train = df_train.append(chunk.sample(n=samplesize))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "34412343-bce0-cc6a-2104-9893236d2d2c"
      },
      "outputs": [],
      "source": [
        "# Since this is too much data to read in and process at once\n",
        "# we can instead sample randomly from the dataset\n",
        "\n",
        "import random\n",
        "\n",
        "num_rows = 74180464\n",
        "sample_size = 500000 #desired sample size\n",
        "filename = \"../input/train.csv\"\n",
        "skip = sorted(random.sample(range(num_rows),num_rows - sample_size))\n",
        "df = pd.read_csv(filename, skiprows=skip)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "a532e314-d1f6-97b6-379e-e59b267f16f4"
      },
      "source": [
        "<h2>Core Data Exploration</h2>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "e8d94df7-6ef3-a5d2-e179-dbb483bbc4d2"
      },
      "source": [
        "Which variables are the best indicators and best predict demand? \n",
        "How can we visualize this?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "1ab21cb8-74e7-3a8c-9933-92180f739e64"
      },
      "outputs": [],
      "source": [
        "# First how do agencies predict return rate?\n",
        "# How many unique agencies (distributors) are there?\n",
        "\n",
        "# Let's add in the column names again\n",
        "df.columns = ['Semana', 'Agencia_ID', 'Canal_ID', 'Ruta_SAK', 'Cliente_ID',\n",
        "       'Producto_ID', 'Venta_uni_hoy', 'Venta_hoy', 'Dev_uni_proxima',\n",
        "       'Dev_proxima', 'Demanda_uni_equil']\n",
        "\n",
        "print(len(df.Agencia_ID.unique()))\n",
        "\n",
        "# What is the distribution for how many units/ week these agencies sell?\n",
        "\n",
        "def return_rate(a,b):\n",
        "    print(a)\n",
        "    df['return_rate'] = 1\n",
        "    #print(df)\n",
        "    return df\n",
        "\n",
        "agencia_groups = df.groupby(by='Agencia_ID')\n",
        "#print(agencia_groups.count()) # This tells us the number of rows for each agency\n",
        "agencia_sums = agencia_groups.sum() # This sums all of the units sold and returned per agency\n",
        "#print(agencia_sums)\n",
        "#for d, r in agencia_sums['Venta_uni_hoy']:\n",
        "#    print(agencia_sums[row])\n",
        "    #agencia_sums[row]['return_rate'] = agencia_sums[row]['Dev_uni_proxima'] / agencia_sums[row]['Venta_uni_hoy']\n",
        "\n",
        "print(agencia_sums)\n",
        "test = agencia_sums.aggregate(np.sum)\n",
        "print(test)\n",
        "#agencia_sums = agencia_sums['Venta_uni_hoy', 'Dev_uni_proxima'].map(return_rate)\n",
        "#print(agencia_sums)\n",
        "\n",
        "# What are the top agencies in terms of number of units sold?\n",
        "\n",
        "\n",
        "# How can we use this information to visualize the correlation between agency and \n",
        "# return rate?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "c3bf6b82-6f33-8bfb-8fc2-a2e46567e9b6"
      },
      "outputs": [],
      "source": [
        "df_train = pd.read_csv('../input/train.csv', nrows=1000000)\n",
        "df_train.Semana.unique()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "be191ac9-26df-2f6f-ff0f-43d2b3f4d3dc"
      },
      "source": [
        "Now I'm going to learn from the Exploring Products script https://www.kaggle.com/vykhand/grupo-bimbo-inventory-demand/exploring-products/comments\n",
        "\n",
        "The main focus of this script is as follows\n",
        "1) \n",
        "2) String processing\n",
        "3) Plotting out the demand?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "b0e35b87-3709-5821-99a5-9e24aa7a5c01"
      },
      "outputs": [],
      "source": [
        "# Why do we aggregate the products based on week and product ID? What does that\n",
        "# do for us?\n",
        "# Why did we do all the stemming of the words?\n",
        "# Why do we create a bag of words and vectorize?\n",
        "# What do we do once we have all the word roots and the number of times they appear?\n",
        "# How would I create the top demand plot?\n",
        "# Aggregate the products by name and count. Leaves me with those groups. \n",
        "# How do I sum? Get that column and call sum().  Then sort and graph\n",
        "# Why did we do np.log1p(df)? What does this graph tell us?\n",
        "# What are the actual units of the x axis?\n",
        "# How did they come to the conlucsion that some products have no sales but have returns?\n",
        "# Group by product name and sum the columns\n",
        "# How could we statistically check if there is a difference of demand from week to week?\n",
        "# What have I learned from this...my goal is to predict the output based on input"
      ]
    }
  ],
  "metadata": {
    "_change_revision": 0,
    "_is_fork": false,
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.5.2"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}