{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "9fa93300-0c8c-dea0-5883-7e668b30b658"
      },
      "source": [
        "I like to look at my data and try and get a feel for it when I start--with no domain expertise, we are limited in \"What does it mean\"  Plus with the data so sparse, there may be something interesting lurking in the details.  Let's take a look."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "dd48bdc3-0c47-95f5-b993-b8b0a390a617"
      },
      "outputs": [],
      "source": [
        "# This Python 3 environment comes with many helpful analytics libraries installed\n",
        "# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n",
        "# For example, here's several helpful packages to load in \n",
        "\n",
        "#lets take a look at that categorical data--it has been giving me fits!\n",
        "import numpy as np # linear algebra\n",
        "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
        "import collections #I had a list of lists going, the good people at Stack Overflow said DefaultDict!\n",
        "data=pd.read_csv(\"../input/train_categorical.csv\",chunksize=100000, dtype=str,usecols=list(range(1,2141)))\n",
        "uniques = collections.defaultdict(set)\n",
        "\n",
        "\n",
        "for chunk in data: \n",
        "    for col in chunk:\n",
        "        uniques[col] = uniques[col].union(chunk[col][chunk[col].notnull()].unique())\n",
        "# Input data files are available in the \"../input/\" directory.\n",
        "# Any results you write to the current directory are saved as output.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "41a25dba-9ef9-b2b6-3b89-435bfc575be5"
      },
      "outputs": [],
      "source": [
        "#Are any variables empy?\n",
        "empty=0\n",
        "for key in uniques:\n",
        "    if len(uniques[key])==0:\n",
        "        print(key)\n",
        "        empty=empty+1"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "fcbfdf15-ee85-d999-e214-561179367892"
      },
      "outputs": [],
      "source": [
        "#how bout columns with a single value?\n",
        "single=0\n",
        "for key in uniques:\n",
        "    if len(uniques[key])==1:\n",
        "        print(key,uniques[key])\n",
        "        single=single+1"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "17e80f33-c166-08ee-e3f7-1ec4e98a6164"
      },
      "outputs": [],
      "source": [
        "#how about multi-valued keys?\n",
        "multi=0\n",
        "for key in uniques:\n",
        "    if len(uniques[key])>1:\n",
        "        print(key,uniques[key])\n",
        "        multi=multi+1"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "6389e1a5-e9f0-33e2-e0b0-22ec29887673"
      },
      "outputs": [],
      "source": [
        "import matplotlib\n",
        "import matplotlib.pyplot as plt; plt.rcdefaults()\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        " \n",
        "objects = ('Empty', 'Single Value', 'Multi-Value')\n",
        "y_pos = np.arange(len(objects))\n",
        "performance = [empty,single,multi]\n",
        " \n",
        "plt.bar(y_pos, performance, align='center', alpha=0.5)\n",
        "plt.xticks(y_pos, objects)\n",
        "plt.ylabel('Usage')\n",
        "plt.title('Number of Features in Category Data')\n",
        " \n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "78269d5f-4743-bc73-79ba-ff7c068d21a3"
      },
      "source": [
        "So obviously this helps us since we can drop the empty columns (no information) and do a one-hot encode on the single values.  This makes life a lot easier.  There may be something more in the multi value columns, but now there are ~500 of them rather than 2000.\n",
        "\n",
        "Once again big shout out to the people at Stack Overflow.  Users Ami and Raymond took my code and cleaned it up quite a bit by suggesting the default dict over a list of lists.  List of lists worked, but it was difficult to follow.  I like clean clear code!"
      ]
    }
  ],
  "metadata": {
    "_change_revision": 0,
    "_is_fork": false,
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.5.2"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}