{"cells":[{"cell_type":"markdown","metadata":{"_cell_guid":"9fa93300-0c8c-dea0-5883-7e668b30b658"},"source":"I like to look at my data and try and get a feel for it when I start--with no domain expertise, we are limited in \"What does it mean\"  Plus with the data so sparse, there may be something interesting lurking in the details.  Let's take a look."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"dd48bdc3-0c47-95f5-b993-b8b0a390a617"},"outputs":[],"source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\n#lets take a look at that categorical data--it has been giving me fits!\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport collections #I had a list of lists going, the good people at Stack Overflow said DefaultDict!\ndata=pd.read_csv(\"../input/train_categorical.csv\",chunksize=100000, dtype=str,usecols=list(range(1,2141)))\nuniques = collections.defaultdict(set)\n\n\nfor chunk in data: \n    for col in chunk:\n        uniques[col] = uniques[col].union(chunk[col][chunk[col].notnull()].unique())\n# Input data files are available in the \"../input/\" directory.\n# Any results you write to the current directory are saved as output.\n"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"41a25dba-9ef9-b2b6-3b89-435bfc575be5"},"outputs":[],"source":"#Are any variables empy?\nempty=0\nfor key in uniques:\n    if len(uniques[key])==0:\n        print(key)\n        empty=empty+1"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"fcbfdf15-ee85-d999-e214-561179367892"},"outputs":[],"source":"#how bout columns with a single value?\nsingle=0\nfor key in uniques:\n    if len(uniques[key])==1:\n        print(key,uniques[key])\n        single=single+1"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"17e80f33-c166-08ee-e3f7-1ec4e98a6164"},"outputs":[],"source":"#how about multi-valued keys?\nmulti=0\nfor key in uniques:\n    if len(uniques[key])>1:\n        print(key,uniques[key])\n        multi=multi+1"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"6389e1a5-e9f0-33e2-e0b0-22ec29887673"},"outputs":[],"source":"import matplotlib\nimport matplotlib.pyplot as plt; plt.rcdefaults()\nimport numpy as np\nimport matplotlib.pyplot as plt\n \nobjects = ('Empty', 'Single Value', 'Multi-Value')\ny_pos = np.arange(len(objects))\nperformance = [empty,single,multi]\n \nplt.bar(y_pos, performance, align='center', alpha=0.5)\nplt.xticks(y_pos, objects)\nplt.ylabel('Usage')\nplt.title('Number of Features in Category Data')\n \nplt.show()"},{"cell_type":"markdown","metadata":{"_cell_guid":"78269d5f-4743-bc73-79ba-ff7c068d21a3"},"source":"So obviously this helps us since we can drop the empty columns (no information) and do a one-hot encode on the single values.  This makes life a lot easier.  There may be something more in the multi value columns, but now there are ~500 of them rather than 2000.\n\nOnce again big shout out to the people at Stack Overflow.  Users Ami and Raymond took my code and cleaned it up quite a bit by suggesting the default dict over a list of lists.  List of lists worked, but it was difficult to follow.  I like clean clear code!"}],"metadata":{"_change_revision":0,"_is_fork":false,"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.5.2"}},"nbformat":4,"nbformat_minor":0}