{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Welcome to the great Categorical Feature Encoding Challenge #2\nThis notebook is a starter code for all beginners and easy to understand. I used the following notebook to improve knowledge about encoding:\nhttps://www.kaggle.com/shahules/an-overview-of-encoding-techniques\n\nAdditionally there are created new features based on the relationsship between the nominal features. <br>\nIt is used the XGB Classifier with a simple setting and great results."},{"metadata":{},"cell_type":"markdown","source":"# Load Libraries"},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport scipy.special\nimport matplotlib.pyplot as plt\nimport os","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.impute import SimpleImputer\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import LabelEncoder\nfrom sklearn import metrics\nfrom xgboost import XGBClassifier","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Input path"},{"metadata":{"trusted":true},"cell_type":"code","source":"path_in = '../input/cat-in-the-dat-ii/'\nprint(os.listdir(path_in))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Read input data"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data = pd.read_csv(path_in+'train.csv', index_col=0)\ntest_data = pd.read_csv(path_in+'test.csv', index_col=0)\nsamp_subm = pd.read_csv(path_in+'sample_submission.csv', index_col=0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Plot bar function"},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_bar(data, name):\n    data_label = data[name].value_counts()\n    dict_train = dict(zip(data_label.keys(), ((data_label.sort_index())).tolist()))\n    names = list(dict_train.keys())\n    values = list(dict_train.values())\n    plt.bar(names, values)\n    plt.grid()\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_bar_compare(train, test, name, rot=False):\n    \"\"\" Compare the distribution between train and test data \"\"\"\n    fig, axs = plt.subplots(1, 2, figsize=(9, 3), sharey=True)\n    \n    train_label = train[name].value_counts().sort_index()\n    dict_train = dict(zip(train_label.keys(), ((100*(train_label)/len(train.index)).tolist())))\n    train_names = list(dict_train.keys())\n    train_values = list(dict_train.values())\n    \n    test_label = test[name].value_counts().sort_index()\n    dict_test = dict(zip(test_label.keys(), ((100*(test_label)/len(test.index)).tolist())))\n    test_names = list(dict_test.keys())\n    test_values = list(dict_test.values())\n    \n    axs[0].bar(train_names, train_values, color='yellowgreen')\n    axs[1].bar(test_names, test_values, color = 'sandybrown')\n    axs[0].grid()\n    axs[1].grid()\n    axs[0].set_title('Train data')\n    axs[1].set_title('Test data')\n    axs[0].set_ylabel('%')\n    if(rot==True):\n        axs[0].set_xticklabels(train_names, rotation=45)\n        axs[1].set_xticklabels(test_names, rotation=45)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Overview\nWe have a look on the number of samples and check missing data. "},{"metadata":{"trusted":true},"cell_type":"code","source":"print('# samples train:', len(train_data))\nprint('# samples test:', len(test_data))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cols_with_missing_train_data = [col for col in train_data.columns if train_data[col].isnull().any()]\ncols_with_missing_test_data = [col for col in test_data.columns if test_data[col].isnull().any()]\nprint('train cols with missing data:', cols_with_missing_train_data)\nprint('test cols with missing data:', cols_with_missing_test_data)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Handle missing data\nFirst we handle missing data by a simple imputer with the most-frequent strategy."},{"metadata":{"trusted":true},"cell_type":"code","source":"imp_cat = SimpleImputer(strategy='most_frequent')\ntrain_data[cols_with_missing_train_data] = imp_cat.fit_transform(train_data[cols_with_missing_train_data])\ntest_data[cols_with_missing_test_data] = imp_cat.fit_transform(test_data[cols_with_missing_test_data])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Show features"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.columns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Show target\nWe have a look on the distribution of the target values."},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_bar(train_data, 'target')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Classify the features"},{"metadata":{"trusted":true},"cell_type":"code","source":"features_bin = ['bin_0', 'bin_1', 'bin_2', 'bin_3', 'bin_4']\nfeatures_cat = ['nom_0', 'nom_1', 'nom_2', 'nom_3', 'nom_4']\nfeatures_hex = ['nom_5', 'nom_6', 'nom_7', 'nom_8', 'nom_9']\nfeatures_ord = ['ord_0', 'ord_1', 'ord_2', 'ord_3', 'ord_4', 'ord_5']\nfeatures_cyc = ['day', 'month']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Define the scaler mappings for the ordinal features\nFor ord_0 is nothing to do."},{"metadata":{"trusted":true},"cell_type":"code","source":"map_ord_1 = {'Novice':1, 'Contributor':2, 'Expert':3, 'Master':4, 'Grandmaster':5}\nmap_ord_2 = {'Freezing': 1, 'Cold':2, 'Warm':3, 'Hot':4, 'Boiling Hot': 5, 'Lava Hot':6}\nmap_ord_3 = dict(zip(train_data['ord_3'].value_counts().sort_index().keys(),\n                     range(1, len(train_data['ord_3'].value_counts())+1)))\nmap_ord_4 = dict(zip(train_data['ord_4'].value_counts().sort_index().keys(),\n                     range(1, len(train_data['ord_4'].value_counts())+1)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"temp_ord_5 = pd.DataFrame(train_data['ord_5'].value_counts().sort_index().keys(), columns=['ord_5'])\ntemp_ord_5['First'] = temp_ord_5['ord_5'].astype(str).str[0].str.upper()\ntemp_ord_5['Second'] = temp_ord_5['ord_5'].astype(str).str[1].str.upper()\ntemp_ord_5['First'] = temp_ord_5['First'].replace(map_ord_4)\ntemp_ord_5['Second'] = temp_ord_5['Second'].replace(map_ord_4)\ntemp_ord_5['Add'] = temp_ord_5['First']+temp_ord_5['Second']\ntemp_ord_5['Mul'] = temp_ord_5['First']*temp_ord_5['Second']\nmap_ord_5 = dict(zip(temp_ord_5['ord_5'],\n                     temp_ord_5['Mul']))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Plot the nominal features and think about a relationsship\n## Feature nom_0 - the color\nBlue and Red are colors of the RGB color space, Green not."},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_bar_compare(train_data, test_data, 'nom_0')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data['rgb'] = np.where(train_data['nom_0'] == 'Green', 0, 1)\ntest_data['rgb'] = np.where(test_data['nom_0'] == 'Green', 0, 1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Feature nom_1 the geometric shape\nOnly the circle has no corners."},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_bar_compare(train_data, test_data, 'nom_1', rot=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data['round'] = np.where(train_data['nom_1'] == 'Circle', 1, 0)\ntest_data['round'] = np.where(test_data['nom_1'] == 'Circle', 1, 0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Feature nom_2 - animals\nThe snake has no feet."},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_bar_compare(train_data, test_data, 'nom_2', rot=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data['feet'] = np.where(train_data['nom_2'] == 'Snake', 0, 1)\ntest_data['feet'] = np.where(test_data['nom_2'] == 'Snake', 0, 1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Feature nom_3 - countries\nCanda is a monarchy."},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_bar_compare(train_data, test_data, 'nom_3', rot=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data['monarchy'] = np.where(train_data['nom_3'] == 'Canada', 1, 0)\ntest_data['monarchy'] = np.where(test_data['nom_3'] == 'Canada', 1, 0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Feature nom_4 - musical instruments\nThe Theremin is a electronical instrument."},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_bar_compare(train_data, test_data, 'nom_4', rot=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data['electro'] = np.where(train_data['nom_4'] == 'Theremin', 1, 0)\ntest_data['electro'] = np.where(test_data['nom_4'] == 'Theremin', 1, 0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Define y_train"},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train = train_data['target']\ndel train_data['target']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Encode the features"},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train = train_data.copy()\nX_test = test_data.copy()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Binary features\nUse the simple LabelEncoder."},{"metadata":{"trusted":true},"cell_type":"code","source":"le = LabelEncoder()\nfor col in features_bin:\n    le.fit(X_train[col])\n    X_train[col] = le.transform(X_train[col])\n    X_test[col] = le.transform(X_test[col])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Categorical features\nUse the simple LabelEncoder."},{"metadata":{"trusted":true},"cell_type":"code","source":"le = LabelEncoder()\nfor col in features_cat:\n    le.fit(X_train[col])\n    X_train[col] = le.transform(X_train[col])\n    X_test[col] = le.transform(X_test[col])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Hexadecimal features\nUse the simple LabelEncoder."},{"metadata":{"trusted":true},"cell_type":"code","source":"le = LabelEncoder()\nfor col in features_hex:\n    le.fit(X_train[col].append(X_test[col]))\n    X_train[col] = le.transform(X_train[col])\n    X_test[col] = le.transform(X_test[col])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Ordinal features\nUse the mapping for ord_1 to ord_5."},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train['ord_1'] = X_train['ord_1'].replace(map_ord_1)\nX_train['ord_2'] = X_train['ord_2'].replace(map_ord_2)\nX_train['ord_3'] = X_train['ord_3'].replace(map_ord_3)\nX_train['ord_4'] = X_train['ord_4'].replace(map_ord_4)\nX_train['ord_5'] = X_train['ord_5'].replace(map_ord_5)\nX_test['ord_1'] = X_test['ord_1'].replace(map_ord_1)\nX_test['ord_2'] = X_test['ord_2'].replace(map_ord_2)\nX_test['ord_3'] = X_test['ord_3'].replace(map_ord_3)\nX_test['ord_4'] = X_test['ord_4'].replace(map_ord_4)\nX_test['ord_5'] = X_test['ord_5'].replace(map_ord_5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Cyclic features"},{"metadata":{"trusted":true},"cell_type":"code","source":"for feature in features_cyc:\n    X_train[feature+'_sin'] = np.sin((2*np.pi*X_train[feature])/max(X_train[feature]))\n    X_train[feature+'_cos'] = np.cos((2*np.pi*X_train[feature])/max(X_train[feature]))\n    X_test[feature+'_sin'] = np.sin((2*np.pi*X_test[feature])/max(X_test[feature]))\n    X_test[feature+'_cos'] = np.cos((2*np.pi*X_test[feature])/max(X_test[feature]))\nX_train = X_train.drop(features_cyc, axis=1)\nX_test = X_test.drop(features_cyc, axis=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Scale data"},{"metadata":{"trusted":true},"cell_type":"code","source":"mean = X_train[features_hex].mean(axis=0)\nX_train[features_hex] = X_train[features_hex].astype('float32')\nX_train[features_hex] -= X_train[features_hex].mean(axis=0)\nstd = X_train[features_hex].std(axis=0)\nX_train[features_hex] /= X_train[features_hex].std(axis=0)\nX_test[features_hex] = X_test[features_hex].astype('float32')\nX_test[features_hex] -= mean\nX_test[features_hex] /= std","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mean = X_train[features_ord].mean(axis=0)\nX_train[features_ord] = X_train[features_ord].astype('float32')\nX_train[features_ord] -= X_train[features_ord].mean(axis=0)\nstd = X_train[features_ord].std(axis=0)\nX_train[features_ord] /= X_train[features_ord].std(axis=0)\nX_test[features_ord] = X_test[features_ord].astype('float32')\nX_test[features_ord] -= mean\nX_test[features_ord] /= std","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mean = X_train[features_cat].mean(axis=0)\nX_train[features_cat] = X_train[features_cat].astype('float32')\nX_train[features_cat] -= X_train[features_cat].mean(axis=0)\nstd = X_train[features_cat].std(axis=0)\nX_train[features_cat] /= X_train[features_cat].std(axis=0)\nX_test[features_cat] = X_test[features_cat].astype('float32')\nX_test[features_cat] -= mean\nX_test[features_cat] /= std","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train.columns","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Split train and val data"},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size = 0.1, random_state=2020)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Calc the class wights of the target"},{"metadata":{"trusted":true},"cell_type":"code","source":"weight = float(len(y_train[y_train == 0]))/float(len(y_train[y_train == 1]))\nw1 = np.array([1]*y_train.shape[0])\nw1[y_train==1]=weight","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train[features_cat].head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Define XGBClassifier and Predict\nDetermine the parameters of the XGB Classifier with a simple grid search.\n## Set model and fit"},{"metadata":{"trusted":true},"cell_type":"code","source":"model = XGBClassifier(objective ='binary:logistic',\n                      colsample_bytree = 0,\n                      learning_rate = 0.2,\n                      max_depth = 15,\n                      n_estimators = 400,\n                      scale_pos_weight = 2,\n                      random_state = 2020,\n                      subsample = 0.8)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.fit(X_train, y_train, eval_set=[(X_val, y_val)], verbose=False, sample_weight=w1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Evalaute the model with the val data"},{"metadata":{"trusted":true},"cell_type":"code","source":"preds_val = model.predict_proba(X_val)[:,1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"score = metrics.roc_auc_score(y_val ,preds_val)\nprint(\"score: %f\" % (score))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Confusion Matrix\nExample of confusion matrix usage to evaluate the quality of the output of a classifier data set. The diagonal elements represent the number of points for which the predicted label is equal to the true label, while off-diagonal elements are those that are mislabeled by the classifier. The higher the diagonal values of the confusion matrix the better, indicating many correct predictions."},{"metadata":{"trusted":true},"cell_type":"code","source":"# metrics.plot_confusion_matrix(model,\n#                               X_val, y_val,\n#                               cmap=plt.cm.Blues,\n#                               normalize=None,\n#                               values_format='d')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Predict test data"},{"metadata":{"trusted":true},"cell_type":"code","source":"y_test = model.predict_proba(X_test)[:,1]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Write the Export"},{"metadata":{"trusted":true},"cell_type":"code","source":"num = samp_subm.index\noutput = pd.DataFrame({'id': num,\n                       'target': y_test})\noutput.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}