{"cells":[{"cell_type":"markdown","metadata":{"_cell_guid":"3b594320-8094-c257-7d7e-ed603b9bf591"},"source":"The methods used to generate the filter terms involved looking at the most frequent words/client names for clues about what types of establishments were mentioned in this data set. \n\nDeveloping the filters took a fair amount of \"human\" sleuthing while looking at TF-IDF scores, frequency counts of Client names, as well as just general knowledge of common Spanish words used to refer to certain establishment types. Especially with all the noise provided by the Clients referred to only by a proper name, the filtered data is a proportionally small figure- but, significant."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"ea1ed3af-7c78-af45-48eb-63318a2b7442"},"outputs":[],"source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nfrom subprocess import check_output\n#print(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))\n\n# Any results you write to the current directory are saved as output.\n\n# Load in the Client Name data\n# Make sure all names uppercase (there are some mixed instances)\nvf = pd.read_csv('../input/cliente_tabla.csv',header=0)\nvf['NombreCliente'] = vf['NombreCliente'].str.upper()"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"bd0d3505-f09a-a87a-cd21-2eed0ab908a9"},"outputs":[],"source":"# Begin with some analysis of the Client Name Data based on Frequency of Name\n# Notice there are a lot of Proper Names\nvf['NombreCliente'].value_counts()"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"90d7e98e-d2d2-bdb1-2ea9-cfeb284fff69"},"outputs":[],"source":"# Let's also generate a list of individual word frequency across all names\ndef tfidf_score_list(vf2, list_len):\n    from sklearn.feature_extraction.text import TfidfVectorizer\n    v = TfidfVectorizer()\n\n    vf2['New'] = 'na'\n    a = \"\".join(vf2['NombreCliente'])\n    vf2['New'][0] = a\n\n    tfidf = v.fit_transform(vf2['New'])\n\n    feature_names = v.get_feature_names()\n\n    freq = []\n    doc = 0\n    feature_index = tfidf[doc,:].nonzero()[1]\n    tfidf_scores = zip(feature_index, [tfidf[doc, x] for x in feature_index])\n    for w, s in [(feature_names[i], s) for (i, s) in tfidf_scores]:\n            freq.append((w.encode('utf-8'),s))\n\n    import numpy as np\n    names = ['word','score']\n    formats = ['S50','f8']\n    dtype = dict(names = names, formats=formats)\n    array = np.array(freq, dtype=dtype)\n\n    b = np.sort(array, order='score')\n    if list_len > len(b)+1:\n        list_len = len(b)+1\n    for i in range(1,list_len):\n        print(b[-i])\n        \ntfidf_score_list(vf, 200)"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"df442457-7ff7-9fe3-c98f-cb3364c8abbd"},"outputs":[],"source":"tfidf_score_list(vf, 200)"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"15b70b9a-5eee-60fe-30c6-734f2427e354"},"outputs":[],"source":""}],"metadata":{"_change_revision":0,"_is_fork":false,"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.5.2"}},"nbformat":4,"nbformat_minor":0}