{ "cells": [ { "cell_type": "markdown", "metadata": { "_cell_guid": "4b750b64-3560-f4ce-8d70-a0383820c281" }, "source": [ "## Bimbo - Bag of Words" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "c62a5521-706b-64ed-e8b7-2e980c11f6b9" }, "outputs": [], "source": [ "# This Python 3 environment comes with many helpful analytics libraries installed\n", "# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n", "# For example, here's several helpful packages to load in \n", "\n", "import numpy as np # linear algebra\n", "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n", "\n", "# Input data files are available in the \"../input/\" directory.\n", "# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n", "\n", "from subprocess import check_output\n", "print(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))\n", "\n", "# Any results you write to the current directory are saved as output." ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "b9ef303b-41e0-e557-7e30-254e5b881d26" }, "source": [ "## How can we use a Bag of Words approach to predict the demand of a product based on the product's name?" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "fd0fb442-e5bd-2615-aa86-baa5e598f3b5" }, "source": [ "What are we asking? We want to find a way to predict demand for products that \n", "we haven't seen before. Thus we want to extract the stems of the words instead of \n", "predicting based on the entire product name.\n", "\n", "For example we want 'tort' to represent all products with names such as 'Tortilla\n", "de Tomate', 'Tortilla con Huevos', 'Papa's Tortillas' as all tortilla products\n", "may have a similar demand cycle. \n", "\n", "Thus we will train a bag of words model using these product name stems and further\n", "build a Random Forest that will model this process.\n", "\n", "How does the Random Forest work? How does it accurately deal with matching word \n", "similarities?" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "58cc22c1-1228-5d03-ea69-197c37ff1a8b" }, "outputs": [], "source": [ "# We are focusing on the products first so let's read in the product file\n", "\n", "products = pd.read_csv('../input/producto_tabla.csv')" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "87b5d5a6-d2da-9a74-b393-7768310a0d1d" }, "outputs": [], "source": [ "# What does this file look like?\n", "print(products.head())" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "114dd53a-e593-2633-5057-180f7384851a" }, "outputs": [], "source": [ "# How many unique products are there?\n", "products.NombreProducto.nunique()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "1525b182-ac5a-719c-14e4-f81d03b48c56" }, "source": [ "As we can see there is extra information included in the product name such as weight (750g), brand (BIM), and number of pieces (6p).\n", "\n", "Would our bag of words work better with or without this information? With this information seems reasonable, however we need to clean this out first before we can pass it into our text cleaning functions.\n", "\n", "Let's add columns for brand, short_name, weight and pieces" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "ffdf9c84-17aa-ba64-df2a-3e905a999f13" }, "outputs": [], "source": [ "\n", "products['short_name'] = products.NombreProducto.str.extract('^(\\D*)', expand = False)\n", "products['brand'] = products.NombreProducto.str.extract('^.+\\s(\\D+) \\d+$', \n", " expand=False)\n", "w = products.NombreProducto.str.extract('(\\d+)(Kg|g)', expand=True)\n", "products['weight'] = w[0].astype('float')*w[1].map({'Kg':1000, 'g':1})\n", "products['pieces'] = products.NombreProducto.str.extract('(\\d+)p ', expand=False).astype('float')\n", "products.head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "2e8f8108-4e37-3264-fa90-7e24a998f3a7" }, "outputs": [], "source": [ "# Has this changed the number of unique words we have?\n", "products.short_name.nunique()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "8646459b-e15f-a57e-4039-2cb97919c0d1" }, "source": [ "We can see that 2592 - 1014 products had the same name but different pieces,\n", "weight or brand.\n", "An interesting question would be - how much do products with the same name but \n", "different values for these other categories differ with regard to demand?" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "6077238e-5ebe-0f72-bb7e-cc7d2bfc9801" }, "outputs": [], "source": [ "# Now that we have only the product names in one column.\n", "# How do we extract the key parts of each word?\n", "\n", "# First let's remove words with little meaning.\n", "from nltk.corpus import stopwords\n", "stopwords = set(stopwords.words(\"spanish\"))\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "92dd54b7-abbe-6495-ccab-904d8505a33f" }, "outputs": [], "source": [ "# Now let's get rid of all the words in the product names that are in this list of \n", "# stopwords\n", "# For each row in the dataset we need to pass in all the words and only include those \n", "# that are not in stop words\n", "# The row by row is handled by just passing in the entire column but I do need to \n", "# separate out the words I believe\n", "\n", "products['short_name_processed'] = [(' ').join(word) for word in products['short_name']\n", " if word not in stopwords]\n", "print(products['short_name_processed'])\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "f3f31e15-c55a-dbaf-ab2f-0b51572be852" }, "outputs": [], "source": [ "# Hmm word in products['short_name'] left me with considering each character by itself\n", "\n", "# Let me try printing each word in one row\n", "for word in products['short_name_processed'][0]:\n", " print (word)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "81e100b4-a545-24c6-d5f9-c42887ae0e88" }, "outputs": [], "source": [ "# Once again this is by letter. Thus this isn't done by word yet! Instead I shall go\n", "# through and tokenize!\n", "words = products['short_name_processed'].split()\n", "meaningful_words = [w for w in words if not w in stopwords]\n", "return (\" \". join(meaningful_words))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "90f9e93b-5762-1fa7-afee-ecd792a597d9" }, "outputs": [], "source": [ "# Facing the problem where I can't put in the entire column at once\n", "# What is the solution doing differently?\n", "\n", "products['short_name_processed'] = (products['short_name'].map(lambda x: \" \".join([\n", " i for i in x.lower().split() if i not in stopwords])))\n", "\n", "# Series.map() will take each element of the series and perform the provided function\n", "# on it\n", "# Thus we have the first row 'No Identificado' passed in as x to the lambda function\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "da3c08de-7337-d828-e017-2e5815dbbb40" }, "outputs": [], "source": [ "# Let's look at the words we have\n", "products['short_name_processed']" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "bd7a45e9-06b0-810f-a019-63ed5715b4b5" }, "source": [ "Now we have all the words we need. Let's simplify them into the core parts of the words, the 'roots'." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "15e29dd3-d1bb-3dad-618e-b9d68c4d82ab" }, "outputs": [], "source": [ "from nltk.stem.snowball import SnowballStemmer\n", "stemmer = SnowballStemmer(\"spanish\")\n", "\n", "# Here's the example from above. It reduces a word to its stem essentially.\n", "print (stemmer.stem(\"Tortillas\"))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "5e70c606-b91c-8115-ad33-8d5d6f02d8a6" }, "outputs": [], "source": [ "# How do we replace all the processed words with their stems instead?\n", "# We will do the same process as with removing stopwords except now we will pass \n", "# through the stemmer function.\n", "\n", "products['short_name_processed'] = products['short_name_processed'].map(lambda x: stemmer.stem(a).join(' ') for a in x.lower().split())\n", " \n", "print(products['short_name_processed'])" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "7d1c7dc4-61c2-1e88-81a7-75836682ee12" }, "outputs": [], "source": [ "# How do you define x in a lambda function?\n", "# Doesn't seem like that is the problem\n", "# First of all we are joining elements of a list\n", "\n", "products['short_name_processed'] = products['short_name_processed'].map(lambda x: ' '.join(\n", " [stemmer.stem(a) for a in x.lower().split()]))\n", "\n", "print(products['short_name_processed'])\n" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "b1a76609-1088-446b-8a8d-bdca817f585e" }, "source": [ "Now we have all of the word stems for each product! Where do we go from here?\n", "\n", "We want to create a bag of words for each product. This means we create a dictionary of all the words included. Then for each entry we count the number of each word that is in that entry.\n", "We use a CountVectorizer to do this." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "9fa8480a-5eb1-5bcd-96da-0981e0117823" }, "outputs": [], "source": [ "from sklearn.feature_extraction.text import CountVectorizer\n", "\n", "# Here we initialize the CountVectorizer\n", "vectorizer = CountVectorizer(analyzer='word',\n", " tokenizer=None,\n", " preprocessor=None,\n", " stop_words=None,\n", " max_features=1000)\n", "\n", "# Now we need to use it to fit it and train it with out sample\n", "product_bag_words = vectorizer.fit_transform(products.short_name_processed).toarray()\n", "product_bag_words.shape\n", "# the shape should have the same number of rows as our product data\n", "# the number of columns is the max number of features we allowed for\n", "# " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "d2782524-2dfe-123e-4b9c-50f74db97894" }, "outputs": [], "source": [ "# What are the words in our dictionary?\n", "print(vectorizer.get_feature_names())\n", "\n", "# How can we get the number of times each word appears?\n", "# Since each column in the above array represents a single word, the sum of the column\n", "# is the number of times that word has appeared.\n", "# Let's print the most common word\n", "\n", "print('\\n\\n')\n", "print(vectorizer.get_feature_names()[np.argmax(sum(product_bag_words))], np.argmax(sum(product_bag_words)))\n", "print('\\n\\n')\n", "\n", "# It would be cool to see this in a dictionary format\n", "for word, count in zip(vectorizer.get_feature_names(), sum(product_bag_words)):\n", " print('%s: %d' %(word, count))" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "bda06b96-2e02-3e55-80ac-f3fce0244bd3" }, "source": [ "Now how do we connect these word stems with the value we have to predict - demand?\n", "First of all demand is not in the product table, it is in the train dataset.\n", "\n", "Let's first connect the word stems with the product ID so we can then join it with the train dataset demand values." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "85fe67d5-d5d9-2267-8f92-1a403a882c26" }, "outputs": [], "source": [ "train = pd.read_csv('../input/train.csv', usecols=['Producto_ID', \n", " 'Demanda_uni_equil'])\n", "train_product_agg = train.groupby('Producto_ID', as_index=False).agg('mean')\n", "print(train_product_agg)\n", "\n", "# We want to have the mean of demand for each of the product IDs\n", "# Therefore we groupby product ID and aggregate over the demand\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "0165729e-0af2-8d6c-e924-b96aa0c3e397" }, "outputs": [], "source": [ "# Thus we can now go ahead and merge the demand data with the bag of words data\n", "\n", "product_bag_words = pd.concat([products.Producto_ID, \n", " pd.DataFrame(product_bag_words, columns=vectorizer.get_feature_names(), index=products.index)], axis=1)\n", "product_bag_words.head()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "1dcd9790-eea0-26d4-a342-a3d990f2f69d" }, "source": [ "Now that we have our Bag of Words trained, let's move on to training our Random Forest.\n", "\n", "What does a Random Forest algorithm do?\n", "\n", "Real life example:\n", "You want an estimate as to whether or not you will like a new movie. \n", "\n", " - You give your friend a sample set of movies you've watched and how much you liked each one\n", " - Your friend builds a model based on this data\n", " - For a new movie, your friend asks questions akin to 20 questions 'Is it romantic?'... and compares it against the movie reviews she has from you to determine a score\n", " - For a random forest, you have multiple such people\n", " - You don't want them all to give back the same result though\n", " - Therefore you give each a different sample of movies that you liked\n", " - This leads to each person building a slightly different model with different questions\n", " - Thus to get your best estimate you average the outputs of all of the people = Random Forest\n", "\n", "Same situation. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "e11a425e-f9e2-93b5-cdca-e897d2bf2182" }, "outputs": [], "source": [ "test = pd.read_csv('../input/test.csv')" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "3bd351be-e16e-9496-cf59-35430b81a7bc" }, "outputs": [], "source": [ "# Create the model\n", "\n", "from sklearn.ensemble import RandomForestClassifier\n", "\n", "forest = RandomForestClassifier(n_estimators=100)\n", "# This means that each tree will use 100 of the estimators\n", "\n", "# Now we have to train the model on the training data\n", "# How do we create the model? Use fit with the data and the result\n", "# Where do we get the actual demand from?\n", "# Our data in product_bag_words is currently sorted alphabetically, we need to sort it\n", "# according to how it is sorted in the train data file\n", "\n", "set_train = set(train_product_agg['Producto_ID'])\n", "set_test = set(test['Producto_ID'])\n", "\n", "print(len(set_train))\n", "print(len(set_test))\n", "print(len(set_train & set_test ))\n", "\n", "forest = forest.fit_transform(product_bag_words, train[''])" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "05b4fe8d-3ff4-9c6b-87ef-a1a2ff0a8adc" }, "source": [ "Now we have to do the same thing on our test data.\n", "First we perform the same procedure to clean the data.\n", "Then we use the bag of words on it.\n", "Lastly we need to use our forest to make predictions." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "b0179fc8-a0fd-e385-a3cb-531a204f5fce" }, "outputs": [], "source": [ "# Can't actually use bag of words to do this since test doesn't have product names..." ] } ], "metadata": { "_change_revision": 0, "_is_fork": false, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.2" } }, "nbformat": 4, "nbformat_minor": 0 }