{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Categorical Data EDA & Visualization\n\n> techniques for Categorical Data EDA & Viz\n\n![](https://media.giphy.com/media/o0vwzuFwCGAFO/giphy.gif)\n\nI participated in last competition so much, I will participate again.\n\nFirst of all, I'm going to do EDA to come up with an idea of the overall distribution or idea of the data.\n\n**Related Work**\n\n- [11 Categorical Encoders and Benchmark](https://www.kaggle.com/subinium/11-categorical-encoders-and-benchmark)"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nimport missingno as msno \n\nimport chart_studio.plotly as py\nimport cufflinks as cf\nimport plotly.express as px\nimport plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\n\nimport os \n\nprint(os.listdir('../input/cat-in-the-dat-ii'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Matplotlib Settings (for Visualization)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# matplotlib setting\nplt.rc('font', size=12) \nplt.rc('axes', titlesize=14)\nplt.rc('axes', labelsize=12) \nplt.rc('xtick', labelsize=12)\nplt.rc('ytick', labelsize=12) \nplt.rc('legend', fontsize=12) \nplt.rc('figure', titlesize=14) \nplt.rcParams['figure.dpi'] = 300\nsns.set_style(\"whitegrid\")\n\ncolors = [\"windows blue\", \"amber\", \"greyish\", \"faded green\", \"dusty purple\"]\nsns.set_palette(sns.xkcd_palette(colors))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/cat-in-the-dat-ii/train.csv')\ntest = pd.read_csv('../input/cat-in-the-dat-ii/test.csv')\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"target, train_id = train['target'], train['id']\ntest_id = test['id']\ntrain.drop(['id'], axis=1, inplace=True)\ntest.drop(['id'], axis=1, inplace=True)\nprint(train.shape)\nprint(test.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train.columns)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### feature list\n\nIt's important to know what each feature is, because you need to check how you encode or distribute based on the feature.\n\n- **bin 0~4** : Binary Feature, label encoding\n- **nom 0~9** : Nominal Feature\n- **ord 0~5** : Ordinal Feature\n- **day/month** : Date, cycle encoding \n"},{"metadata":{},"cell_type":"markdown","source":"## Total Distribution\n\nLet's first look at the overall distribution of the data."},{"metadata":{"trusted":true},"cell_type":"code","source":"msno.matrix(train)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This data seems to have a lot of missing value unlike last time.\n\nLet's look at the sorted values."},{"metadata":{"trusted":true},"cell_type":"code","source":"msno.matrix(train, sort='ascending')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"You can see that about half are empty."},{"metadata":{"trusted":true},"cell_type":"code","source":"null_rate = [train[i].isna().sum() / len(train) for i in train.columns]\nfig, ax = plt.subplots(1,1,figsize=(20, 7))\nsns.barplot(x=train.columns, y=null_rate, ax=ax,color='gray')\nax.set_title(\"Missing Value Rate (Train)\")\nax.set_xticklabels(train.columns, rotation=40)\nax.axhline(y=0.03, color='red')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"null_rate = [test[i].isna().sum() / len(train) for i in test.columns]\nfig, ax = plt.subplots(1,1,figsize=(20, 7))\nsns.barplot(x=test.columns, y=null_rate, ax=ax,color='gray')\nax.set_title(\"Missing Value Rate (Test)\")\nax.set_xticklabels(test.columns, rotation=40)\nax.axhline(y=0.02, color='red')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- The missing value (train) seems to make the data roughly 3%.\n- The missing value (test) seems to make the data roughly 2%.\n\nWhat about the target value distribution?"},{"metadata":{"trusted":true},"cell_type":"code","source":"target_dist = target.value_counts()\n\nfig, ax = plt.subplots(1, 1, figsize=(8,5))\n\nbarplot = plt.bar(target_dist.index, target_dist, color = 'lightgreen', alpha = 0.8)\nbarplot[1].set_color('darkred')\n\nax.set_title('Target Distribution')\nax.annotate(\"percentage of target 1 : {}%\".format(target.sum() / len(target)),\n              xy=(0, 0),xycoords='axes fraction', \n              xytext=(0,-50), textcoords='offset points',\n              va=\"top\", ha=\"left\", color='grey',\n              bbox=dict(boxstyle='round', fc=\"w\", ec='w'))\n\nplt.xlabel('Target', fontsize = 12, weight = 'bold')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"First of all, you can see that the target ratio is unbalanced, rather than last data."},{"metadata":{},"cell_type":"markdown","source":"## Binary Feature\n\nLet's start with the **binary feature.**"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1,5, figsize=(30, 8))\nfor i in range(5): \n    sns.countplot(f'bin_{i}', data= train, ax=ax[i])\n    ax[i].set_ylim([0, 600000])\n    ax[i].set_title(f'bin_{i}', fontsize=15)\nfig.suptitle(\"Binary Feature Distribution (Train Data)\", fontsize=20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1,5, figsize=(30, 8))\nfor i in range(5): \n    sns.countplot(f'bin_{i}', data= test, ax=ax[i], alpha=0.7,\n                 order=test[f'bin_{i}'].value_counts().index)\n    ax[i].set_ylim([0, 600000])\n    ax[i].set_title(f'bin_{i}', fontsize=15)\nfig.suptitle(\"Binary Feature Distribution (Test Data)\", fontsize=20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The overall `binary feature` distribution between `train` and `test` seems to be similar."},{"metadata":{},"cell_type":"markdown","source":"It can be seen that as $i$ of ${bin}_i$ increases, the distribution approaches 50%."},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1,5, figsize=(30, 8))\nfor i in range(5): \n    sns.countplot(f'bin_{i}', hue='target', data= train, ax=ax[i])\n    ax[i].set_ylim([0, 500000])\n    ax[i].set_title(f'bin_{i}', fontsize=15)\nfig.suptitle(\"Binary Feature Distribution (Train Data)\", fontsize=20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Nominal Feature\n\nFrom nominal data, we need to look more closely at the distribution."},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(2,3, figsize=(30, 15))\nfor i in range(5): \n    sns.countplot(f'nom_{i}', data= train, ax=ax[i//3][i%3],\n                 order=train[f'nom_{i}'].value_counts().index)\n    ax[i//3][i%3].set_ylim([0, 350000])\n    ax[i//3][i%3].set_title(f'nom_{i}', fontsize=15)\nfig.suptitle(\"Nominal Feature Distribution (Train Data)\", fontsize=20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(2,3, figsize=(30, 15))\nfor i in range(5): \n    sns.countplot(f'nom_{i}', data= test, ax=ax[i//3][i%3],\n                 order=test[f'nom_{i}'].value_counts().index,\n                 alpha=0.7)\n    ax[i//3][i%3].set_ylim([0, 250000])\n    ax[i//3][i%3].set_title(f'nom_{i}', fontsize=15)\nfig.suptitle(\"Nominal Feature Distribution (Test Data)\", fontsize=20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The overall `nominal feature` distribution between `train` and `test` seems to be similar."},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(2,3, figsize=(30, 15))\nfor i in range(5): \n    sns.countplot(f'nom_{i}', hue='target', data= train, ax=ax[i//3][i%3],\n                 order=train[f'nom_{i}'].value_counts().index)\n    ax[i//3][i%3].set_ylim([0, 300000])\n    ax[i//3][i%3].set_title(f'nom_{i}', fontsize=15)\nfig.suptitle(\"Nominal Feature Distribution (Train Data)\", fontsize=20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in range(5):\n    data = train[[f'nom_{i}', 'target']].groupby(f'nom_{i}')['target'].value_counts().unstack()\n    data['rate'] = data[1]  / (data[0] + data[1] )\n    data.sort_values(by=['rate'], inplace=True)\n    display(data.style.highlight_max(color='lightgreen').highlight_min(color='#cd4f39'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The visualization took too long and we looked at the rest of the features and found many unique elements:"},{"metadata":{"trusted":true},"cell_type":"code","source":"train[[f'nom_{i}' for i in range(5, 10)]].describe(include='O')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There seems to be something similar between `nom_7` and `nom_8`."},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(2,1, figsize=(30, 10))\nfor i in range(7,9): \n    sns.countplot(f'nom_{i}', data= train, ax=ax[i-7],\n                  order = train[f'nom_{i}'].dropna().value_counts().index)\n    ax[i-7].set_ylim([0, 5500])\n    ax[i-7].set_title(f'bin_{i}', fontsize=15)\n    ax[i-7].set_xticks([])\nfig.suptitle(\"Nominal Feature Distribution (Train Data)\", fontsize=20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(2,1, figsize=(30, 10))\nfor i in range(7,9): \n    sns.countplot(f'nom_{i}', hue='target', data= train, ax=ax[i-7],\n                  order = train[f'nom_{i}'].dropna().value_counts().index)\n    ax[i-7].set_ylim([0, 5000])\n    ax[i-7].set_title(f'bin_{i}', fontsize=15)\n    ax[i-7].set_xticks([])\nfig.suptitle(\"Nominal Feature Distribution (Train Data)\", fontsize=20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The comparison after sorting **does not seem** to have **high similarity**. (The distribution looks similar, but it's too different in detail.)\n\nHowever, given that the numbers are the same and that the bending points on the graph are at similar points in the sort order by size, we assume that there is some preprocessing to see the relationship between the two features."},{"metadata":{},"cell_type":"markdown","source":"## Ordinal Feature"},{"metadata":{"trusted":true},"cell_type":"code","source":"train[[f'ord_{i}' for i in range(6)]].describe(include='all')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1,3, figsize=(30, 8))\n\nord_order = [\n    [1.0, 2.0, 3.0],\n    ['Novice', 'Contributor', 'Expert', 'Master', 'Grandmaster'],\n    ['Freezing', 'Cold', 'Warm', 'Hot', 'Boiling Hot', 'Lava Hot']\n]\n\nfor i in range(3): \n    sns.countplot(f'ord_{i}', hue='target', data= train, ax=ax[i],\n                  order = ord_order[i]\n                 )\n    ax[i].set_ylim([0, 200000])\n    ax[i].set_title(f'ord_{i}', fontsize=15)\nfig.suptitle(\"Ordinal Feature Distribution (Train Data)\", fontsize=20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Oddly, it feels like the 3 graphs are gradually expanding, which may be useful to check again later with correlation."},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1,2, figsize=(24, 8))\n\nfor i in range(3, 5): \n    sns.countplot(f'ord_{i}', hue='target', data= train, ax=ax[i-3],\n                  order = sorted(train[f'ord_{i}'].dropna().unique())\n                 )\n    ax[i-3].set_ylim([0, 75000])\n    ax[i-3].set_title(f'ord_{i}', fontsize=15)\nfig.suptitle(\"Ordinal Feature Distribution (Train Data 3~4)\", fontsize=20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Oddly... it feels like the 2 graphs are gradually expanding...?\n\nUsing this part seems to minimize the feature."},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in range(5):\n    data = train[[f'ord_{i}', 'target']].groupby(f'ord_{i}')['target'].value_counts().unstack()\n    data['rate'] = data[1]  / (data[0] + data[1] )\n    data.sort_values(by=['rate'], inplace=True)\n    display(data.style.highlight_max(color='lightgreen').highlight_min(color='#cd4f39'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**I personally guess**\n\nBy sorting the values by the ratio of 1 in the target value, multiple ord features were ordered. This is thought to be intentional when the data is created, and data that is out of sync is considered to be an error from missing values.\n\n---\n\nord_6 has a large number of unique values, so let's sort them by size."},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(2,1, figsize=(24, 16))\n\nxlabels = train['ord_5'].dropna().value_counts().index\n\nprint(len(xlabels))\n\n# just counting\nsns.countplot('ord_5', data= train, ax=ax[0], order = xlabels )\nax[0].set_ylim([0, 12000])\nax[0].set_xticklabels(xlabels, rotation=90, rotation_mode=\"anchor\", fontsize=7)\n\n# with hue\nsns.countplot('ord_5', hue='target', data= train, ax=ax[1], order = xlabels )\nax[1].set_ylim([0, 10000])\nax[1].set_xticklabels(xlabels, rotation=90, rotation_mode=\"anchor\", fontsize=7)\n\nfig.suptitle(\"Ordinal Feature Distribution (Train Data 5)\", fontsize=20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Day & Month \n\nTODO : sin & cos transform"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(2,1, figsize=(24, 16))\n\nsns.countplot('day', hue='target', data= train, ax=ax[0])\nax[0].set_ylim([0, 100000])\n\nsns.countplot('month', hue='target', data= train, ax=ax[1])\nax[1].set_ylim([0, 80000])\n\nfig.suptitle(\"Day & Month Distribution\", fontsize=20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Surprisingly, the graph feels like something else.\n\n> Any advice would be appreciated if it was just me."},{"metadata":{"trusted":true},"cell_type":"code","source":"data = train[['day', 'target']].groupby('day')['target'].value_counts().unstack()\ndata['rate'] = data[1]  / (data[0] + data[1] )\ndata.sort_values(by=['rate'], inplace=True)\ndisplay(data.style.highlight_max(color='lightgreen').highlight_min(color='#cd4f39'))\n\ndata = train[['month', 'target']].groupby('month')['target'].value_counts().unstack()\ndata['rate'] = data[1]  / (data[0] + data[1] )\ndata.sort_values(by=['rate'], inplace=True)\ndisplay(data.style.highlight_max(color='lightgreen').highlight_min(color='#cd4f39'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"---\n\n## With Categofical Data with Encoding\n\nTo get the correlation, let's do some basic encodings and get the correlation."},{"metadata":{},"cell_type":"markdown","source":"### Binary"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nbin_encoding = {'F':0, 'T':1, 'N':0, 'Y':1}\ntrain['bin_3'] = train['bin_3'].map(bin_encoding)\ntrain['bin_4'] = train['bin_4'].map(bin_encoding)\n\ntest['bin_3'] = test['bin_3'].map(bin_encoding)\ntest['bin_4'] = test['bin_4'].map(bin_encoding)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Nominal\n\nI'll go ahead and target based encoding to believe the relationship between nom_7 and nom_8."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nfrom category_encoders.target_encoder import TargetEncoder\n\nfor i in range(10):\n    label = TargetEncoder()\n    train[f'nom_{i}'] = label.fit_transform(train[f'nom_{i}'].fillna('NULL'), target)\n    test[f'nom_{i}'] = label.transform(test[f'nom_{i}'].fillna('NULL'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Ordinal"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nord_order = [\n    [1.0, 2.0, 3.0],\n    ['Novice', 'Contributor', 'Expert', 'Master', 'Grandmaster'],\n    ['Freezing', 'Cold', 'Warm', 'Hot', 'Boiling Hot', 'Lava Hot']\n]\n\nfor i in range(1, 3):\n    ord_order_dict = {i : j for j, i in enumerate(ord_order[i])}\n    train[f'ord_{i}'] = train[f'ord_{i}'].fillna('NULL').map(ord_order_dict)\n    test[f'ord_{i}'] = test[f'ord_{i}'].fillna('NULL').map(ord_order_dict)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nfor i in range(3, 6):\n    ord_order_dict = {i : j for j, i in enumerate(sorted(list(set(list(train[f'ord_{i}'].dropna().unique()) + list(test[f'ord_{i}'].dropna().unique())))))}\n    train[f'ord_{i}'] = train[f'ord_{i}'].fillna('NULL').map(ord_order_dict)\n    test[f'ord_{i}'] = test[f'ord_{i}'].fillna('NULL').map(ord_order_dict)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nf, ax = plt.subplots(1, 3, figsize=(45, 14))\nfor idx, tp in  enumerate(['pearson', 'kendall', 'spearman']) :\n    corr = train.fillna(-1).corr(tp)\n    mask = np.zeros_like(corr, dtype=np.bool)\n    mask[np.triu_indices_from(mask)] = True\n    cmap = sns.diverging_palette(220, 10, as_cmap=True)\n    sns.heatmap(corr, mask=mask, cmap=cmap, vmax=.2, center=0,square=True, linewidths=.5, cbar_kws={\"shrink\": .5}, ax=ax[idx])\n    ax[idx].set_title(f'{tp} correlation viz')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nf, ax = plt.subplots(1, 3, figsize=(45, 14))\nfor idx, tp in  enumerate(['pearson', 'kendall', 'spearman']) :\n    corr = test.fillna(-1).corr(tp)\n    mask = np.zeros_like(corr, dtype=np.bool)\n    mask[np.triu_indices_from(mask)] = True\n    cmap = sns.diverging_palette(220, 10, as_cmap=True)\n    sns.heatmap(corr, mask=mask, cmap=cmap, vmax=.1, center=0,square=True, linewidths=.5, cbar_kws={\"shrink\": .5}, ax=ax[idx])\n    ax[idx].set_title(f'{tp} correlation viz (test)')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"First of all, it depends on the encoding, but in the case of `ord`, sorting in order has some correlation with the **target value**.\nI think ord is definitely useful as a feature.\n\nIf you look at the correlation for other features and find the encoding, I think you will get good insights. (Or target based encoding methods would be nice.)\n\nnom is a target based encoding, so I'll skip further thinking.\n\nI don't know if the features are not correlated with each other. Please let me know in the comments if I made a mistake during the process.\n\n\n## Keep Going!!"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}