{"cells":[{"metadata":{},"cell_type":"markdown","source":"In this kernel let's go through data imputation and also explore different types of data encoding techniques. As I go on in this journey and learn new topics, I will incorporate them with each new updates. Please upvote if you like this kernel and let me know your feedback."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/train.csv')\ntest = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/test.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## How many Null values ?"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Number of nulls per each column\nnulls = train.isna().sum().sort_values(ascending = False)[train.isna().sum() != 0]\nperc = round(train.isna().sum()/len(train) * 100,2)[train.isna().sum()/len(train) != 0]\npd.concat([nulls,perc], axis = 1, sort = False, keys=['Total','Percent'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"On average 3% of the values in all the columns except ID and target are nulls. Also, This dataset only contains categorical features, and includes:\n\n* binary features\n* low- and high-cardinality nominal features\n* low- and high-cardinality ordinal features\n* (potentially) cyclical features"},{"metadata":{},"cell_type":"markdown","source":"# Imputing Nulls"},{"metadata":{},"cell_type":"markdown","source":"Let's impute missing using the most frequent value along each column."},{"metadata":{"trusted":true},"cell_type":"code","source":"for column in train.columns:\n    train[column].fillna(train[column].mode()[0], inplace=True)\nfor column in test.columns:\n    test[column].fillna(test[column].mode()[0], inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# EDA"},{"metadata":{},"cell_type":"markdown","source":"## Binary Features"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Plotting binary columns\ncols = ['bin_0','bin_1','bin_2','bin_3','bin_4']\nfig, axes = plt.subplots(ncols=3, nrows=2, figsize=(16,8), dpi= 60)\naxes = axes.flatten()\nfor i, ax in zip(cols, axes):\n    sns.countplot(x = i, ax = ax, data = train, palette = 'Paired')\n    total = len(train[i])\n    for p in ax.patches:\n        percentage = '{:.1f}%'.format(100 * p.get_height()/total)\n        x = p.get_x() + p.get_width() - 0.5\n        y = p.get_y() + p.get_height() + 1000\n        ax.annotate(percentage, (x, y))\n        ax.set_ylabel('Count')\nfig.delaxes(axes[5])\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Clearly all the binary columns has No/False/0 as their mode values. Interestingly the percentage of Yes/True/1 increases as we go from bin_0 to bin_4."},{"metadata":{},"cell_type":"markdown","source":"## Nominal Features"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Plotting nominal columns\ncols = ['nom_0','nom_1','nom_2','nom_3','nom_4']\nfig, axes = plt.subplots(ncols=3, nrows=2, figsize=(18,8), dpi= 60)\naxes = axes.flatten()\nfor i, ax in zip(cols, axes):\n    sns.countplot(x = i, ax = ax, data = train, palette = 'Paired')\n    total = len(train[i])\n    for p in ax.patches:\n        percentage = '{:.1f}%'.format(100 * p.get_height()/total)\n        x = p.get_x() + p.get_width() - 0.5\n        y = p.get_y() + p.get_height() + 1000\n        ax.annotate(percentage, (x, y))\n        ax.set_ylabel('Count')\nfig.delaxes(axes[5])\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's have a look at the unique values in other nominal features."},{"metadata":{"trusted":true},"cell_type":"code","source":"cols = ['nom_5','nom_6','nom_7','nom_8','nom_9']\nfor col in cols:\n    print (train[col].nunique())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Ordinal Features"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Plotting ordinal columns\ncols = ['ord_0','ord_1','ord_2']\nfig, axes = plt.subplots(ncols=3, nrows=1, figsize=(18,4), dpi= 60)\naxes = axes.flatten()\nfor i, ax in zip(cols, axes):\n    sns.countplot(x = i, ax = ax, data = train, palette = 'Paired')\n    total = len(train[i])\n    for p in ax.patches:\n        percentage = '{:.1f}%'.format(100 * p.get_height()/total)\n        x = p.get_x() + p.get_width() - 0.5\n        y = p.get_y() + p.get_height() + 1000\n        ax.annotate(percentage, (x, y))\n        ax.set_ylabel('Count')\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The data set has more number novices than any other kaggle title. Also from the temperature stand point Freezing has occured more times. Why does kaggle titles and temperature are in the same data set ? Definitely random. May be not. Let's have a look at the unique values in other ordinal features."},{"metadata":{"trusted":true},"cell_type":"code","source":"cols = ['ord_3','ord_4','ord_5']\nfor col in cols:\n    print (train[col].nunique())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Encoding"},{"metadata":{},"cell_type":"markdown","source":"## Target Encoding\nLet's use Target encoding for high cardinal features\n\nCredits: https://www.kaggle.com/caesarlupum/2020-20-lines-target-encoding"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import StratifiedKFold\nimport category_encoders as ce\n\ncols = ['nom_5','nom_6','nom_7','nom_8','nom_9']\nteTrain = train[cols]; teTest = test[cols]\ntrain_y = train['target']; test_id = test['id']\ncol_to_encode = teTrain.columns.tolist()\noof = pd.DataFrame([])\n\nfor tr_idx, oof_idx in StratifiedKFold(n_splits=5, random_state=42, shuffle=True).split(teTrain, train_y):\n    ce_target_encoder = ce.TargetEncoder(cols = col_to_encode, smoothing=0.2)\n    ce_target_encoder.fit(teTrain.iloc[tr_idx, :], train_y.iloc[tr_idx])\n    oof = oof.append(ce_target_encoder.transform(teTrain.iloc[oof_idx, :]), ignore_index=False)\nce_target_encoder = ce.TargetEncoder(cols = col_to_encode, smoothing=0.2)\nce_target_encoder.fit(teTrain, train_y)\nteTrain = oof.sort_index()\nteTest = ce_target_encoder.transform(teTest)\n\ntrain[cols] = teTrain[cols]\ntest[cols] = teTest[cols]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Mapping\nLet's use mapping for ordinal features\n\nCredits: https://www.kaggle.com/drcapa/categorical-feature-engineering-2-xgb"},{"metadata":{"trusted":true},"cell_type":"code","source":"map_ord_1 = {'Novice':1, 'Contributor':2, 'Expert':3, 'Master':4, 'Grandmaster':5}\nmap_ord_2 = {'Freezing': 1, 'Cold':2, 'Warm':3, 'Hot':4, 'Boiling Hot': 5, 'Lava Hot':6}\nmap_ord_3 = dict(zip(train['ord_3'].value_counts().sort_index().keys(),range(1, len(train['ord_3'].value_counts())+1)))\nmap_ord_4 = dict(zip(train['ord_4'].value_counts().sort_index().keys(),range(1, len(train['ord_4'].value_counts())+1)))\n\ntemp_ord_5 = pd.DataFrame(train['ord_5'].value_counts().sort_index().keys(), columns=['ord_5'])\ntemp_ord_5['First'] = temp_ord_5['ord_5'].astype(str).str[0].str.upper()\ntemp_ord_5['Second'] = temp_ord_5['ord_5'].astype(str).str[1].str.upper()\ntemp_ord_5['First'] = temp_ord_5['First'].replace(map_ord_4)\ntemp_ord_5['Second'] = temp_ord_5['Second'].replace(map_ord_4)\ntemp_ord_5['Add'] = temp_ord_5['First']+temp_ord_5['Second']\ntemp_ord_5['Mul'] = temp_ord_5['First']*temp_ord_5['Second']\nmap_ord_5 = dict(zip(temp_ord_5['ord_5'],temp_ord_5['Mul']))\n\ntrain['ord_1'] = train['ord_1'].replace(map_ord_1)\ntrain['ord_2'] = train['ord_2'].replace(map_ord_2)\ntrain['ord_3'] = train['ord_3'].replace(map_ord_3)\ntrain['ord_4'] = train['ord_4'].replace(map_ord_4)\ntrain['ord_5'] = train['ord_5'].replace(map_ord_5)\n\ntest['ord_1'] = test['ord_1'].replace(map_ord_1)\ntest['ord_2'] = test['ord_2'].replace(map_ord_2)\ntest['ord_3'] = test['ord_3'].replace(map_ord_3)\ntest['ord_4'] = test['ord_4'].replace(map_ord_4)\ntest['ord_5'] = test['ord_5'].replace(map_ord_5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Mapping False/No to 0 and True/Yes to 1 in binary features."},{"metadata":{"trusted":true},"cell_type":"code","source":"map_bin_3 = {'F':0, 'T':1}\nmap_bin_4 = {'N': 0, 'Y':1}\n\ntrain['bin_3'] = train['bin_3'].replace(map_bin_3)\ntrain['bin_4'] = train['bin_4'].replace(map_bin_4)\n\ntest['bin_3'] = test['bin_3'].replace(map_bin_3)\ntest['bin_4'] = test['bin_4'].replace(map_bin_4)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## One Hot Encoding\nLet's encode nominal features with low cardinality."},{"metadata":{"trusted":true},"cell_type":"code","source":"cols = ['nom_0','nom_1','nom_2','nom_3','nom_4']\ntrain = pd.concat((train, pd.get_dummies(train[cols], drop_first=True)),1)\ntrain = train.drop(cols, axis = 1)\ntest = pd.concat((test, pd.get_dummies(test[cols], drop_first=True)),1)\ntest = test.drop(cols, axis = 1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Encoding Cyclical Features\nCredits: https://www.kaggle.com/drcapa/categorical-feature-engineering-2-xgb"},{"metadata":{"trusted":true},"cell_type":"code","source":"cols = ['day', 'month']\nfor feature in cols:\n    train[feature+'_sin'] = np.sin((2*np.pi*train[feature])/max(train[feature]))\n    train[feature+'_cos'] = np.cos((2*np.pi*train[feature])/max(train[feature]))\n    test[feature+'_sin'] = np.sin((2*np.pi*test[feature])/max(test[feature]))\n    test[feature+'_cos'] = np.cos((2*np.pi*test[feature])/max(test[feature]))\ntrain = train.drop(cols, axis=1)\ntest = test.drop(cols, axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}