{"cells":[{"metadata":{},"cell_type":"markdown","source":"### Predict Cat\n**Problem statement**:\nPredict Cat given features with a lot categorical types \n\n**Type**: \nBinary Classification\n\n**Performance metric**: \nArea Under the ROC Curve (AUC score)\n\n**What I did in this notebook:**\n\n1. **Experimenting with Imputing Methods**\n    a. Impute Constant: \"missing_value\" for string features; -1 for numeric features;\n2. **Experimenting with Categorical Encoding**\n    a. One-Hot Encoding\n    b. Label Encoding\n    c. Target Encoding -> Possibly use smoothing for correcting value from small sample.\n3. **Feature Engineering**\n    a. Power and Logaritmic transformation\n    b. Cyclical features\n    \n#### **Learning**:\n1. With plain Logistic Regression, I generate features that give public score of **0.77515**\n2. Final feature engineering are:\n\n    a. Label encoding for features with binary value. Impute -1 for the missing values in these features.\n    \n    b. Ordinal encoding for features with ordinal value. Impute -1 for the missing values in these features. Apply power and log transformation in these features.\n    \n    c. Cyclical features for day and month with several cycle period.\n    \n    d. Create interactions features.\n    \n    e. Target encoding. (This really improves perfomance on this dataset)\n    \n    f. One-hot encoding."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Library\nimport pandas as pd\nimport numpy as np\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport scipy\n\nfrom sklearn.metrics import roc_auc_score\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import StandardScaler\n\nfrom sklearn.preprocessing import LabelEncoder\nfrom sklearn.preprocessing import OrdinalEncoder\nfrom sklearn.impute import SimpleImputer\nimport category_encoders as cat_encoder\n\nimport warnings\nwarnings.filterwarnings('ignore')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Read file\ntrain = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/train.csv')\ntest = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/test.csv')\nsubmission = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/sample_submission.csv')\n\n# Drop id on train and test dataset\ntrain = train.drop('id', axis = 1)\ntest = test.drop('id', axis = 1)\n\n# Select only baseline features\nbaseline_features = ['bin_0', 'bin_1', 'bin_2', 'bin_3', 'bin_4', 'day', 'month', 'nom_0',\n                     'nom_1', 'nom_2', 'nom_3', 'nom_4', 'nom_7', 'nom_8',\n                     'ord_0', 'ord_1', 'ord_2', 'ord_3', 'ord_4', 'ord_5']\n\ntrain_target = train['target']\ntrain = train[baseline_features]\ntest = test[baseline_features]\n\n\ntrain.shape, train_target.shape, test.shape\n# We have 600k samples for training and 400k for test","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# See number of unique values of each features in train dataset\ntrain.nunique()\n\n# We could see there are features with high number of unique value such as nom_5 - nom_9\n# There are also features with low number of unique value, or we commonly call result of one-hot encoding, such as bin_0 - bin_4\n# Then let see the missing value in the dataset","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# See number of missing values of each features in train dataset\ntrain.isnull().sum()\n\n# All of features have missing values with quite high of number","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Baseline\nHaving the conditions on the training dataset. I want to create baseline model with following methods:\n1. Simple Impute with constant (integer features will be imputed with 0. string features will be imputed with \"missing_value\"). 2. Categorical Encoding using Label Encoding\n3. Standardize features\n4. Using Logistic Regression as baseline model"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Impute with constant\ncolumns = train.columns\n\nfor i in columns:\n    imputer = SimpleImputer(missing_values=np.nan, strategy= 'constant', add_indicator= True)\n    imputer.fit(train[i].to_numpy().reshape(-1,1))\n    \n    train[i] = imputer.transform(train[i].to_numpy().reshape(-1,1))\n    test[i] = imputer.transform(test[i].to_numpy().reshape(-1,1))\n\ntrain.shape, test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Split train validation dataset\nX_train, X_val, y_train, y_val = train_test_split(train,\n                                                  train_target,\n                                                  test_size = 0.2,\n                                                  stratify = train_target,\n                                                  random_state = 41)\nX_train.shape, X_val.shape, y_train.shape, y_val.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Label encoding to all columns\ncolumns = X_train.columns\n\nfor i in columns:\n    label_encoder = LabelEncoder()\n    label_encoder.fit(X_train[i])\n    \n    X_train[i] = label_encoder.transform(X_train[i])\n    X_val[i] = label_encoder.transform(X_val[i])\n\nX_train.shape, X_val.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Standardize the values\nscaler = StandardScaler()\nX_train = scaler.fit_transform(X_train)\nX_val = scaler.transform(X_val)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Measure performance on Validation dataset\nlogit = LogisticRegression()\nlogit.fit(X_train, y_train)\ny_pred =  logit.predict_proba(X_val)\n\nscore = roc_auc_score(y_val, y_pred[:,1])\nprint(\"Baseline model AUC Score: {}\".format(score))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Train on all dataset and make submission\n\n# Label encoding on Training and Test dataset\ncolumns = train.columns\n\nfor i in columns:\n    label_encoder = LabelEncoder()\n    label_encoder.fit(train[i])\n    \n    train[i] = label_encoder.transform(train[i])\n    test[i] = label_encoder.transform(test[i])\n    \n# Standardize the values\nscaler = StandardScaler()\nX_train = scaler.fit_transform(train)\nX_test = scaler.transform(test)\n\n# Training model\nlogit = LogisticRegression()\nlogit.fit(X_train, train_target)\n\n# Predict\ny_pred = logit.predict_proba(X_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Make file for submission\nbaseline_submission = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/sample_submission.csv')\nbaseline_submission['target'] = y_pred[:,1]\nbaseline_submission.to_csv('baseline_model.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Pearson correlation of each features to target\ncor_mtx = pd.concat( [train[baseline_features], train_target], axis = 1).corr()\nplt.subplots()\nsns.heatmap(cor_mtx)\n\nprint(cor_mtx['target'].sort_values(ascending=False))\n\n# Knowing that we only have a few features that have high linearly correlated to target\n# We need to look another way to do features engineering","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Experiment 1\nIn this iteration, I want to do following tasks to improve baseline pipeline:\n1. Imputation with \"constant\". Previously missing value in integer features are imputed with 0. It might cause problem if there are actually value 0 in features. The model will not able to distinguish them. Let's try to impute it with high value so the model will learn it was a missing value.\n2. Categorical Encoding. Previously all the categorical features are encoded by label integer without order even though there are ordinal degree for some features."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Read file\ntrain = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/train.csv')\ntest = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/test.csv')\n\n# Drop id on train and test dataset\ntrain = train.drop('id', axis = 1)\ntest = test.drop('id', axis = 1)\n\n# Select only baseline features\nbaseline_features = ['bin_0', 'bin_1', 'bin_2', 'bin_3', 'bin_4', 'day', 'month', 'nom_0',\n                     'nom_1', 'nom_2', 'nom_3', 'nom_4', 'nom_7', 'nom_8',\n                     'ord_0', 'ord_1', 'ord_2', 'ord_3', 'ord_4', 'ord_5']\n\ntrain_target = train['target']\ntrain = train[baseline_features]\ntest = test[baseline_features]\n\ntrain.shape, train_target.shape, test.shape\n# We have 600k samples for training and 400k for test","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Impute with constant\n\ncolumns = train.columns\n\nfor i in columns:\n    \n    if train[i].dtype == object:\n        imputer = SimpleImputer(missing_values=np.nan, strategy= 'constant', add_indicator= True)\n    else:\n        imputer = SimpleImputer(missing_values=np.nan, strategy= 'constant', fill_value= -1, add_indicator= True)\n\n    imputer.fit(train[i].to_numpy().reshape(-1,1))\n    \n    train[i] = imputer.transform(train[i].to_numpy().reshape(-1,1))\n    test[i] = imputer.transform(test[i].to_numpy().reshape(-1,1))\n\ntrain.shape, test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Split train validation dataset\nX_train, X_val, y_train, y_val = train_test_split(train,\n                                                  train_target,\n                                                  test_size = 0.2,\n                                                  stratify = train_target,\n                                                  random_state = 41)\nX_train.shape, X_val.shape, y_train.shape, y_val.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Ordinal Encoding for features ord_1, ord_2, ord_3, ord_4\n\ndef encode_ord_1(x):\n    if x == \"Novice\":\n        return 0\n    elif x == \"Contributor\":\n        return 1\n    elif x == \"Expert\":\n        return 2\n    elif x == \"Master\":\n        return 3\n    elif x == \"Grandmaster\":\n        return 4\n    else:\n        return -1\n    \n\ndef encode_ord_2(x):\n    if x == \"Freezing\":\n        return 0\n    elif x == \"Cold\":\n        return 1\n    elif x == \"Warm\":\n        return 2\n    elif x == \"Hot\":\n        return 3\n    elif x == \"Boiling Hot\":\n        return 4\n    elif x == \"Lava Hot\":\n        return 5\n    else:\n        return -1\n\ndef encode_ord_3(x):\n    if x == \"a\":\n        return 0\n    elif x == \"b\":\n        return 1\n    elif x == \"c\":\n        return 2\n    elif x == \"d\":\n        return 3\n    elif x == \"e\":\n        return 4\n    elif x == \"f\":\n        return 5\n    elif x == \"g\":\n        return 6\n    elif x == \"h\":\n        return 7\n    elif x == \"i\":\n        return 8\n    elif x == \"j\":\n        return 9\n    elif x == \"k\":\n        return 10\n    elif x == \"l\":\n        return 11\n    elif x == \"m\":\n        return 12\n    elif x == \"n\":\n        return 13\n    elif x == \"o\":\n        return 14\n    elif x == \"p\":\n        return 15\n    elif x == \"q\":\n        return 16\n    elif x == \"r\":\n        return 17\n    elif x == \"s\":\n        return 18\n    elif x == \"t\":\n        return 19\n    elif x == \"u\":\n        return 20\n    elif x == \"v\":\n        return 21\n    elif x == \"w\":\n        return 22\n    elif x == \"x\":\n        return 23\n    elif x == \"y\":\n        return 24\n    elif x == \"z\":\n        return 25\n    else:\n        return -1\n\ndef encode_bin_3(x):\n    if x == \"T\":\n        return 1\n    elif x == \"F\":\n        return 0\n    else:\n        return -1\n\ndef encode_bin_4(x):\n    if x == \"Y\":\n        return 1\n    elif x == \"N\":\n        return 0\n    else:\n        return -1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train['ord_1'] = X_train.ord_1.apply(lambda x: encode_ord_1(x))\nX_train['ord_2'] = X_train.ord_2.apply(lambda x: encode_ord_2(x))\nX_train['ord_3'] = X_train.ord_3.apply(lambda x: encode_ord_3(x))\nX_train['ord_4'] = X_train.ord_4.str.lower().apply(lambda x: encode_ord_3(x))\nX_val['ord_1'] = X_val.ord_1.apply(lambda x: encode_ord_1(x))\nX_val['ord_2'] = X_val.ord_2.apply(lambda x: encode_ord_2(x))\nX_val['ord_3'] = X_val.ord_3.apply(lambda x: encode_ord_3(x))\nX_val['ord_4'] = X_val.ord_4.str.lower().apply(lambda x: encode_ord_3(x))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Label encoding to all columns\ncolumns = X_train.columns\n\nfor i in columns:\n    if X_train[i].dtype == object:        \n        label_encoder = LabelEncoder()\n        label_encoder.fit(X_train[i])\n        X_train[i] = label_encoder.transform(X_train[i])\n        X_val[i] = label_encoder.transform(X_val[i])\n\nX_train.shape, X_val.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Standardize the values\nscaler = StandardScaler()\nX_train = scaler.fit_transform(X_train)\nX_val = scaler.transform(X_val)\n\n# Measure performance on Validation dataset\nlogit = LogisticRegression()\nlogit.fit(X_train, y_train)\ny_pred =  logit.predict_proba(X_val)\n\nscore = roc_auc_score(y_val, y_pred[:,1])\nprint(\"Experiment 1 model AUC Score: {}\".format(score))\n\n# Experiment 1 perform better than baseline model.\n# Our efforts resulting improvement","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Train on all dataset and make submission\n\ntrain = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/train.csv')\ntest = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/test.csv')\n\n# Drop id on train and test dataset\ntrain = train.drop('id', axis = 1)\ntest = test.drop('id', axis = 1)\n\n# Select only baseline features\nbaseline_features = ['bin_0', 'bin_1', 'bin_2', 'bin_3', 'bin_4', 'day', 'month', 'nom_0',\n                     'nom_1', 'nom_2', 'nom_3', 'nom_4', 'nom_7', 'nom_8',\n                     'ord_0', 'ord_1', 'ord_2', 'ord_3', 'ord_4', 'ord_5']\n\ntrain_target = train['target']\ndf_train = train[baseline_features]\ndf_test = test[baseline_features]\n\n# Impute with Simple\ncolumns = df_train.columns\n\nfor i in columns:\n    \n    if df_train[i].dtype == object:\n        imputer = SimpleImputer(missing_values=np.nan, strategy= 'constant', add_indicator= True)\n    else:\n        imputer = SimpleImputer(missing_values=np.nan, strategy= 'constant', fill_value= -1, add_indicator= True)\n\n    imputer.fit(df_train[i].to_numpy().reshape(-1,1))\n    \n    df_train[i] = imputer.transform(df_train[i].to_numpy().reshape(-1,1))\n    df_test[i] = imputer.transform(df_test[i].to_numpy().reshape(-1,1))\n    \n# Ordinal Encoding\ndf_train['ord_1'] = df_train.ord_1.apply(lambda x: encode_ord_1(x))\ndf_train['ord_2'] = df_train.ord_2.apply(lambda x: encode_ord_2(x))\ndf_train['ord_3'] = df_train.ord_3.apply(lambda x: encode_ord_3(x))\ndf_train['ord_4'] = df_train.ord_4.str.lower().apply(lambda x: encode_ord_3(x))\ndf_test['ord_1'] = df_test.ord_1.apply(lambda x: encode_ord_1(x))\ndf_test['ord_2'] = df_test.ord_2.apply(lambda x: encode_ord_2(x))\ndf_test['ord_3'] = df_test.ord_3.apply(lambda x: encode_ord_3(x))\ndf_test['ord_4'] = df_test.ord_4.str.lower().apply(lambda x: encode_ord_3(x))\n\n# Label Encoding to only object features\nfor i in columns:\n    if df_train[i].dtype == object:        \n        label_encoder = LabelEncoder()\n        label_encoder.fit(df_train[i])\n        df_train[i] = label_encoder.transform(df_train[i])\n        df_test[i] = label_encoder.transform(df_test[i])\n    \n# Standardize the values\nscaler = StandardScaler()\ndf_train = scaler.fit_transform(df_train)\ndf_test = scaler.transform(df_test)\n\n# Training model\nlogit = LogisticRegression()\nlogit.fit(df_train, train_target)\n\n# Predict\ny_pred = logit.predict_proba(df_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Make file for submission\nexp1_submission = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/sample_submission.csv')\nexp1_submission['target'] = y_pred[:,1]\nexp1_submission.to_csv('exp1_model.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Pearson correlation of each features to target\ncor_mtx = pd.concat( [pd.DataFrame(df_train, columns = columns), train_target], axis = 1).corr()\nplt.subplots()\nsns.heatmap(cor_mtx)\n\nprint(cor_mtx['target'].sort_values(ascending=False))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Experiment 2\nIn this iteration, I want to do following tasks to improve baseline pipeline:\n\n1. Imputation with \"constant\". Improve the missing_value on each string features need to label encoding with -1\n2. Feature Engineering: (a) Combine interaction between nom_0 nom_1, nom_2, nom_3, nom_4. (b) Create cyclical features from day and month.\n3. Categorical Encoding. Still do Ordinal and Label Encoding\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Read file\ntrain = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/train.csv')\ntest = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/test.csv')\n\n# Drop id on train and test dataset\ntrain = train.drop('id', axis = 1)\ntest = test.drop('id', axis = 1)\n\n# Select only baseline features\nbaseline_features = ['bin_0', 'bin_1', 'bin_2', 'bin_3', 'bin_4', 'day', 'month', 'nom_0',\n                     'nom_1', 'nom_2', 'nom_3', 'nom_4', 'nom_7', 'nom_8',\n                     'ord_0', 'ord_1', 'ord_2', 'ord_3', 'ord_4', 'ord_5']\n\ntrain_target = train['target']\ntrain = train[baseline_features]\ntest = test[baseline_features]\n\n# Impute with constant\ncolumns = train.columns\n\nfor i in columns:\n    \n    if train[i].dtype == object:\n        imputer = SimpleImputer(missing_values=np.nan, strategy= 'constant', add_indicator= True)\n    else:\n        imputer = SimpleImputer(missing_values=np.nan, strategy= 'constant', fill_value= -1, add_indicator= True)\n\n    imputer.fit(train[i].to_numpy().reshape(-1,1))\n    \n    train[i] = imputer.transform(train[i].to_numpy().reshape(-1,1))\n    test[i] = imputer.transform(test[i].to_numpy().reshape(-1,1))\n\ntrain.shape, train_target.shape, test.shape\n# We have 600k samples for training and 400k for test","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Feature Engineering\n\n# Create interactions\ntrain['nom_1_nom_2'] = train.nom_1 + \"_\" + train.nom_2\ntrain['nom_1_nom_3'] = train.nom_1 + \"_\" + train.nom_3\ntrain['nom_1_nom_4'] = train.nom_1 + \"_\" + train.nom_4\ntrain['nom_2_nom_3'] = train.nom_2 + \"_\" + train.nom_3\ntrain['nom_2_nom_4'] = train.nom_2 + \"_\" + train.nom_4\ntrain['nom_3_nom_4'] = train.nom_3 + \"_\" + train.nom_4\ntest['nom_1_nom_2'] = test.nom_1 + \"_\" + test.nom_2\ntest['nom_1_nom_3'] = test.nom_1 + \"_\" + test.nom_3\ntest['nom_1_nom_4'] = test.nom_1 + \"_\" + test.nom_4\ntest['nom_2_nom_3'] = test.nom_2 + \"_\" + test.nom_3\ntest['nom_2_nom_4'] = test.nom_2 + \"_\" + test.nom_4\ntest['nom_3_nom_4'] = test.nom_3 + \"_\" + test.nom_4\n\n# Create cyclical features from day and month\ntrain['day_sin7'] = np.sin(2*np.pi*train['day']/7)\ntrain['day_sin14'] = np.sin(2*np.pi*train['day']/14)\ntrain['day_sin14add'] = np.sin(2*np.pi*train['day']/14)*3.5\ntrain['month_sin12'] = np.sin(2*np.pi*train['month']/12)\ntrain['month_sin24'] = np.sin(2*np.pi*train['month']/24)\ntrain['month_sin24_add'] = np.sin(2*np.pi*train['month']/24)*6\ntest['day_sin7'] = np.sin(2*np.pi*test['day']/7)\ntest['day_sin14'] = np.sin(2*np.pi*test['day']/14)\ntest['day_sin14add'] = np.sin(2*np.pi*test['day']/14)*3.5\ntest['month_sin12'] = np.sin(2*np.pi*test['month']/12)\ntest['month_sin24'] = np.sin(2*np.pi*test['month']/24)\ntest['month_sin24_add'] = np.sin(2*np.pi*test['month']/24)*6\n\n# Ordinal Encoding\ntrain['ord_1'] = train.ord_1.apply(lambda x: encode_ord_1(x))\ntrain['ord_2'] = train.ord_2.apply(lambda x: encode_ord_2(x))\ntrain['ord_3'] = train.ord_3.apply(lambda x: encode_ord_3(x))\ntrain['ord_4'] = train.ord_4.str.lower().apply(lambda x: encode_ord_3(x))\ntest['ord_1'] = test.ord_1.apply(lambda x: encode_ord_1(x))\ntest['ord_2'] = test.ord_2.apply(lambda x: encode_ord_2(x))\ntest['ord_3'] = test.ord_3.apply(lambda x: encode_ord_3(x))\ntest['ord_4'] = test.ord_4.str.lower().apply(lambda x: encode_ord_3(x))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Split train validation dataset\nX_train, X_val, y_train, y_val = train_test_split(train,\n                                                  train_target,\n                                                  test_size = 0.2,\n                                                  stratify = train_target,\n                                                  random_state = 41)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Label encoding to all columns\ncolumns = X_train.columns\n\nfor i in columns:\n    if X_train[i].dtype == object:        \n        label_encoder = LabelEncoder()\n        label_encoder.fit(X_train[i])\n        X_train[i] = label_encoder.transform(X_train[i])\n        X_val[i] = label_encoder.transform(X_val[i])\n\n# Standardize the values\nscaler = StandardScaler()\nX_train = scaler.fit_transform(X_train)\nX_val = scaler.transform(X_val)\n\n# Measure performance on Validation dataset\nlogit = LogisticRegression()\nlogit.fit(X_train, y_train)\ny_pred =  logit.predict_proba(X_val)\n\nscore = roc_auc_score(y_val, y_pred[:,1])\nprint(\"Experiment 2 model AUC Score: {}\".format(score))\n\n# Experiment 2 perform better than Experiment 1 model.\n# Our efforts resulting improvement of (0.7305 - 0.7272) = 0.0033!","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Read file\ntrain = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/train.csv')\ntest = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/test.csv')\n\n# Drop id on train and test dataset\ntrain = train.drop('id', axis = 1)\ntest = test.drop('id', axis = 1)\n\n# Select only baseline features\nbaseline_features = ['bin_0', 'bin_1', 'bin_2', 'bin_3', 'bin_4', 'day', 'month', 'nom_0',\n                     'nom_1', 'nom_2', 'nom_3', 'nom_4', 'nom_7', 'nom_8',\n                     'ord_0', 'ord_1', 'ord_2', 'ord_3', 'ord_4', 'ord_5']\n\ntrain_target = train['target']\ntrain = train[baseline_features]\ntest = test[baseline_features]\n\n# Impute with constant\ncolumns = train.columns\n\nfor i in columns:\n    \n    if train[i].dtype == object:\n        imputer = SimpleImputer(missing_values=np.nan, strategy= 'constant', add_indicator= True)\n    else:\n        imputer = SimpleImputer(missing_values=np.nan, strategy= 'constant', fill_value= -1, add_indicator= True)\n\n    imputer.fit(train[i].to_numpy().reshape(-1,1))\n    \n    train[i] = imputer.transform(train[i].to_numpy().reshape(-1,1))\n    test[i] = imputer.transform(test[i].to_numpy().reshape(-1,1))\n\n# Feature Engineering\n\n# Create interactions\ntrain['nom_1_nom_2'] = train.nom_1 + \"_\" + train.nom_2\ntrain['nom_1_nom_3'] = train.nom_1 + \"_\" + train.nom_3\ntrain['nom_1_nom_4'] = train.nom_1 + \"_\" + train.nom_4\ntrain['nom_2_nom_3'] = train.nom_2 + \"_\" + train.nom_3\ntrain['nom_2_nom_4'] = train.nom_2 + \"_\" + train.nom_4\ntrain['nom_3_nom_4'] = train.nom_3 + \"_\" + train.nom_4\ntest['nom_1_nom_2'] = test.nom_1 + \"_\" + test.nom_2\ntest['nom_1_nom_3'] = test.nom_1 + \"_\" + test.nom_3\ntest['nom_1_nom_4'] = test.nom_1 + \"_\" + test.nom_4\ntest['nom_2_nom_3'] = test.nom_2 + \"_\" + test.nom_3\ntest['nom_2_nom_4'] = test.nom_2 + \"_\" + test.nom_4\ntest['nom_3_nom_4'] = test.nom_3 + \"_\" + test.nom_4\n\n# Create cyclical features from day and month\ntrain['day_sin7'] = np.sin(2*np.pi*train['day']/7)\ntrain['day_sin14'] = np.sin(2*np.pi*train['day']/14)\ntrain['day_sin14add'] = np.sin(2*np.pi*train['day']/14)*3.5\ntrain['month_sin12'] = np.sin(2*np.pi*train['month']/12)\ntrain['month_sin24'] = np.sin(2*np.pi*train['month']/24)\ntrain['month_sin24_add'] = np.sin(2*np.pi*train['month']/24)*6\ntest['day_sin7'] = np.sin(2*np.pi*test['day']/7)\ntest['day_sin14'] = np.sin(2*np.pi*test['day']/14)\ntest['day_sin14add'] = np.sin(2*np.pi*test['day']/14)*3.5\ntest['month_sin12'] = np.sin(2*np.pi*test['month']/12)\ntest['month_sin24'] = np.sin(2*np.pi*test['month']/24)\ntest['month_sin24_add'] = np.sin(2*np.pi*test['month']/24)*6\n\n# Ordinal Encoding\ntrain['ord_1'] = train.ord_1.apply(lambda x: encode_ord_1(x))\ntrain['ord_2'] = train.ord_2.apply(lambda x: encode_ord_2(x))\ntrain['ord_3'] = train.ord_3.apply(lambda x: encode_ord_3(x))\ntrain['ord_4'] = train.ord_4.str.lower().apply(lambda x: encode_ord_3(x))\ntest['ord_1'] = test.ord_1.apply(lambda x: encode_ord_1(x))\ntest['ord_2'] = test.ord_2.apply(lambda x: encode_ord_2(x))\ntest['ord_3'] = test.ord_3.apply(lambda x: encode_ord_3(x))\ntest['ord_4'] = test.ord_4.str.lower().apply(lambda x: encode_ord_3(x))\n\n# Update columns\ncolumns = train.columns\n\n# Label Encoding to only object features\nfor i in columns:\n    if train[i].dtype == object:        \n        label_encoder = LabelEncoder()\n        label_encoder.fit(train[i])\n        train[i] = label_encoder.transform(train[i])\n        test[i] = label_encoder.transform(test[i])\n    \n# Standardize the values\nscaler = StandardScaler()\ntrain = scaler.fit_transform(train)\ntest = scaler.transform(test)\n\n# Training model\nlogit = LogisticRegression()\nlogit.fit(train, train_target)\n\n# Predict\ny_pred = logit.predict_proba(test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Make file for submission\nexp2_submission = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/sample_submission.csv')\nexp2_submission['target'] = y_pred[:,1]\nexp2_submission.to_csv('exp2_model.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cor_mtx = pd.concat([pd.DataFrame(train, columns = columns), train_target], axis = 1).corr()\nsns.heatmap(cor_mtx)\n\nprint(cor_mtx['target'].sort_values(ascending = False))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"### Experiment 3\nIn this iteration, I want to do following tasks to improve baseline pipeline:\n1. Imputation with \"constant\".\n2. Feature Engineering: (a) Combine interaction between nom_0 nom_1, nom_2, nom_3, nom_4. \n    (b) Create cyclical features from day and month.\n    (c) Create square values for each ordinal features\n    (d) Create logaritmic values for each ordinal features\n\n3. Categorical Encoding. Still do Ordinal and Label Encoding"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Read file\ntrain = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/train.csv')\ntest = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/test.csv')\n\n# Drop id on train and test dataset\ntrain = train.drop('id', axis = 1)\ntest = test.drop('id', axis = 1)\n\n# Select only baseline features\nexp3_features = ['bin_0', 'bin_1', 'bin_2', 'bin_3', 'bin_4', 'day', 'month', 'nom_0',\n                     'nom_1', 'nom_2', 'nom_3', 'nom_4', 'nom_5', 'nom_6', 'nom_7', 'nom_8', 'nom_9',\n                     'ord_0', 'ord_1', 'ord_2', 'ord_3', 'ord_4', 'ord_5']\n\ntrain_target = train['target']\ntrain = train[exp3_features]\ntest = test[exp3_features]\n\n# Replacing nom_6 value 'a885aacec' in test_dataset with 'missing_value' because the value is not seen at training data\ntest.loc[test.nom_6 == \"a885aacec\", 'nom_6'] = \"missing_value\"\n\n\n# Label encode first bin_3 and bin_4\ntrain['bin_3'] = train.bin_3.apply(lambda x: encode_bin_3(x))\ntrain['bin_4'] = train.bin_4.apply(lambda x: encode_bin_4(x))\ntest['bin_3'] = test.bin_3.apply(lambda x: encode_bin_3(x))\ntest['bin_4'] = test.bin_4.apply(lambda x: encode_bin_4(x))\n\n# Impute with constant\ncolumns = train.columns\n\nfor i in columns:\n    \n    if train[i].dtype == object:\n        imputer = SimpleImputer(missing_values=np.nan, strategy= 'constant', add_indicator= True)\n    else:\n        imputer = SimpleImputer(missing_values=np.nan, strategy= 'constant', fill_value= -1, add_indicator= True)\n\n    imputer.fit(train[i].to_numpy().reshape(-1,1))\n    \n    train[i] = imputer.transform(train[i].to_numpy().reshape(-1,1))\n    test[i] = imputer.transform(test[i].to_numpy().reshape(-1,1))\n\n# Feature Engineering\n# Create interactions\ntrain['nom_1_nom_2'] = train.nom_1 + \"_\" + train.nom_2\ntrain['nom_1_nom_3'] = train.nom_1 + \"_\" + train.nom_3\ntrain['nom_1_nom_4'] = train.nom_1 + \"_\" + train.nom_4\ntrain['nom_2_nom_3'] = train.nom_2 + \"_\" + train.nom_3\ntrain['nom_2_nom_4'] = train.nom_2 + \"_\" + train.nom_4\ntrain['nom_3_nom_4'] = train.nom_3 + \"_\" + train.nom_4\ntest['nom_1_nom_2'] = test.nom_1 + \"_\" + test.nom_2\ntest['nom_1_nom_3'] = test.nom_1 + \"_\" + test.nom_3\ntest['nom_1_nom_4'] = test.nom_1 + \"_\" + test.nom_4\ntest['nom_2_nom_3'] = test.nom_2 + \"_\" + test.nom_3\ntest['nom_2_nom_4'] = test.nom_2 + \"_\" + test.nom_4\ntest['nom_3_nom_4'] = test.nom_3 + \"_\" + test.nom_4\n\n# Interaction Exp 3\ntrain['bin_all_sum'] = train.bin_0 + train.bin_1 + train.bin_2 + train.bin_3 + train.bin_4\ntrain['bin_all_mul'] = train.bin_0 * train.bin_1 * train.bin_2 * train.bin_3 * train.bin_4\ntest['bin_all_sum'] = test.bin_0 + test.bin_1 + test.bin_2 + test.bin_3 + test.bin_4\ntest['bin_all_mul'] = test.bin_0 * test.bin_1 * test.bin_2 * test.bin_3 * test.bin_4\n\n# Create cyclical features from day and month\ntrain['day_sin7'] = np.sin(2*np.pi*train['day']/7)\ntrain['day_sin14'] = np.sin(2*np.pi*train['day']/14)\ntrain['day_sin14add'] = np.sin(2*np.pi*train['day']/14)*3.5\ntrain['month_sin12'] = np.sin(2*np.pi*train['month']/12)\ntrain['month_sin24'] = np.sin(2*np.pi*train['month']/24)\ntrain['month_sin24_add'] = np.sin(2*np.pi*train['month']/24)*6\ntest['day_sin7'] = np.sin(2*np.pi*test['day']/7)\ntest['day_sin14'] = np.sin(2*np.pi*test['day']/14)\ntest['day_sin14add'] = np.sin(2*np.pi*test['day']/14)*3.5\ntest['month_sin12'] = np.sin(2*np.pi*test['month']/12)\ntest['month_sin24'] = np.sin(2*np.pi*test['month']/24)\ntest['month_sin24_add'] = np.sin(2*np.pi*test['month']/24)*6\n\n# Ordinal Encoding\ntrain['ord_1'] = train.ord_1.apply(lambda x: encode_ord_1(x))\ntrain['ord_2'] = train.ord_2.apply(lambda x: encode_ord_2(x))\ntrain['ord_3'] = train.ord_3.apply(lambda x: encode_ord_3(x))\ntrain['ord_4'] = train.ord_4.str.lower().apply(lambda x: encode_ord_3(x))\ntest['ord_1'] = test.ord_1.apply(lambda x: encode_ord_1(x))\ntest['ord_2'] = test.ord_2.apply(lambda x: encode_ord_2(x))\ntest['ord_3'] = test.ord_3.apply(lambda x: encode_ord_3(x))\ntest['ord_4'] = test.ord_4.str.lower().apply(lambda x: encode_ord_3(x))\n\n# Ordinal Encoding Square\n# First normalize with maximum label for faster convergence,\n# Subtract with 0.5 and square it\ntrain['ord_1_sqr_mid'] = ((train.ord_1 / 4) - 0.5)**2\ntrain['ord_2_sqr_mid'] = ((train.ord_2 / 5) - 0.5)**2\ntrain['ord_3_sqr_mid'] = ((train.ord_3 / 25) - 0.5)**2\ntrain['ord_4_sqr_mid'] = ((train.ord_4 / 25) - 0.5)**2\ntest['ord_1_sqr_mid'] = ((test.ord_1 / 4) - 0.5)**2\ntest['ord_2_sqr_mid'] = ((test.ord_2 / 5) - 0.5)**2\ntest['ord_3_sqr_mid'] = ((test.ord_3 / 25) - 0.5)**2\ntest['ord_4_sqr_mid'] = ((test.ord_4 / 25) - 0.5)**2\n# Square bot not centered\ntrain['ord_1_sqr'] = ((train.ord_1 / 4))**2\ntrain['ord_2_sqr'] = ((train.ord_2 / 5))**2\ntrain['ord_3_sqr'] = ((train.ord_3 / 25))**2\ntrain['ord_4_sqr'] = ((train.ord_4 / 25))**2\ntest['ord_1_sqr'] = ((test.ord_1 / 4))**2\ntest['ord_2_sqr'] = ((test.ord_2 / 5))**2\ntest['ord_3_sqr'] = ((test.ord_3 / 25))**2\ntest['ord_4_sqr'] = ((test.ord_4 / 25))**2\n# Log Transform\ntrain['ord_1_log'] = np.log1p((train.ord_1 / 4))\ntrain['ord_2_log'] = np.log1p((train.ord_2 / 5))\ntrain['ord_3_log'] = np.log1p((train.ord_3 / 25))\ntrain['ord_4_log'] = np.log1p((train.ord_4 / 25))\ntest['ord_1_log'] = np.log1p((test.ord_1 / 4))\ntest['ord_2_log'] = np.log1p((test.ord_2 / 5))\ntest['ord_3_log'] = np.log1p((test.ord_3 / 25))\ntest['ord_4_log'] = np.log1p((test.ord_4 / 25))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Splitting train data for validation\nX_train, X_val, y_train, y_val = train_test_split(train,\n                                                 train_target,\n                                                 test_size = 0.2,\n                                                 stratify = train_target,\n                                                 random_state = 41)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Label encoding to all columns\ncolumns = X_train.columns\n\nfor i in columns:\n    if X_train[i].dtype == object:        \n        label_encoder = LabelEncoder()\n        X_train[i] = label_encoder.fit_transform(X_train[i])\n        label_encoder = LabelEncoder()\n        X_val[i] = label_encoder.fit_transform(X_val[i])\n        \n# Create ordinal square after label encoding - centered\nX_train['ord_5_sqr_mid'] = ((X_train.ord_5 / X_train.ord_5.max()) - 0.5)**2\nX_val['ord_5_sqr_mid'] = ((X_val.ord_5 / X_train.ord_5.max()) - 0.5)**2\n# Create ordinal square after label encoding - not centered\nX_train['ord_5_sqr'] = ((X_train.ord_5 / X_train.ord_5.max()) )**2\nX_val['ord_5_sqr'] = ((X_val.ord_5 / X_train.ord_5.max()) )**2\n# Log transform\nX_train['ord_5_log'] = np.log1p((X_train.ord_5 / X_train.ord_5.max()))\nX_val['ord_5_log'] = np.log1p((X_val.ord_5 / X_train.ord_5.max()))\n\n# Standardize the values\nscaler = StandardScaler()\nX_train = scaler.fit_transform(X_train)\nX_val = scaler.transform(X_val)\n\n# Measure performance on Validation dataset\nlogit = LogisticRegression()\nlogit.fit(X_train, y_train)\ny_pred =  logit.predict_proba(X_val)\n\nscore = roc_auc_score(y_val, y_pred[:,1])\nprint(\"Experiment 3 model AUC Score: {}\".format(score))\n\n# Experiment 3 perform better than Experiment 2 model.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Read file\ntrain = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/train.csv')\ntest = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/test.csv')\n\n# Drop id on train and test dataset\ntrain = train.drop('id', axis = 1)\ntest = test.drop('id', axis = 1)\n\n# Select only baseline features\nexp3_features = ['bin_0', 'bin_1', 'bin_2', 'bin_3', 'bin_4', 'day', 'month', 'nom_0',\n                     'nom_1', 'nom_2', 'nom_3', 'nom_4', 'nom_5', 'nom_6', 'nom_7', 'nom_8', 'nom_9',\n                     'ord_0', 'ord_1', 'ord_2', 'ord_3', 'ord_4', 'ord_5']\n\ntrain_target = train['target']\ntrain = train[exp3_features]\ntest = test[exp3_features]\n\n# Replacing nom_6 value 'a885aacec' in test_dataset with 'missing_value' because the value is not seen at training data\ntest.loc[test.nom_6 == \"a885aacec\", 'nom_6'] = \"missing_value\"\n\n# Label encode first bin_3 and bin_4\ntrain['bin_3'] = train.bin_3.apply(lambda x: encode_bin_3(x))\ntrain['bin_4'] = train.bin_4.apply(lambda x: encode_bin_4(x))\ntest['bin_3'] = test.bin_3.apply(lambda x: encode_bin_3(x))\ntest['bin_4'] = test.bin_4.apply(lambda x: encode_bin_4(x))\n\n# Impute with constant\ncolumns = train.columns\n\nfor i in columns:\n    \n    if train[i].dtype == object:\n        imputer = SimpleImputer(missing_values=np.nan, strategy= 'constant', add_indicator= True)\n    else:\n        imputer = SimpleImputer(missing_values=np.nan, strategy= 'constant', fill_value= -1, add_indicator= True)\n\n    imputer.fit(train[i].to_numpy().reshape(-1,1))\n    \n    train[i] = imputer.transform(train[i].to_numpy().reshape(-1,1))\n    test[i] = imputer.transform(test[i].to_numpy().reshape(-1,1))\n\n# Feature Engineering\n# Create interactions\ntrain['nom_1_nom_2'] = train.nom_1 + \"_\" + train.nom_2\ntrain['nom_1_nom_3'] = train.nom_1 + \"_\" + train.nom_3\ntrain['nom_1_nom_4'] = train.nom_1 + \"_\" + train.nom_4\ntrain['nom_2_nom_3'] = train.nom_2 + \"_\" + train.nom_3\ntrain['nom_2_nom_4'] = train.nom_2 + \"_\" + train.nom_4\ntrain['nom_3_nom_4'] = train.nom_3 + \"_\" + train.nom_4\ntest['nom_1_nom_2'] = test.nom_1 + \"_\" + test.nom_2\ntest['nom_1_nom_3'] = test.nom_1 + \"_\" + test.nom_3\ntest['nom_1_nom_4'] = test.nom_1 + \"_\" + test.nom_4\ntest['nom_2_nom_3'] = test.nom_2 + \"_\" + test.nom_3\ntest['nom_2_nom_4'] = test.nom_2 + \"_\" + test.nom_4\ntest['nom_3_nom_4'] = test.nom_3 + \"_\" + test.nom_4\n\n# Interaction Exp 3\ntrain['bin_all_sum'] = train.bin_0 + train.bin_1 + train.bin_2 + train.bin_3 + train.bin_4\ntrain['bin_all_mul'] = train.bin_0 * train.bin_1 * train.bin_2 * train.bin_3 * train.bin_4\ntest['bin_all_sum'] = test.bin_0 + test.bin_1 + test.bin_2 + test.bin_3 + test.bin_4\ntest['bin_all_mul'] = test.bin_0 * test.bin_1 * test.bin_2 * test.bin_3 * test.bin_4\n\n# Create cyclical features from day and month\ntrain['day_sin7'] = np.sin(2*np.pi*train['day']/7)\ntrain['day_sin14'] = np.sin(2*np.pi*train['day']/14)\ntrain['day_sin14add'] = np.sin(2*np.pi*train['day']/14)*3.5\ntrain['month_sin12'] = np.sin(2*np.pi*train['month']/12)\ntrain['month_sin24'] = np.sin(2*np.pi*train['month']/24)\ntrain['month_sin24_add'] = np.sin(2*np.pi*train['month']/24)*6\ntest['day_sin7'] = np.sin(2*np.pi*test['day']/7)\ntest['day_sin14'] = np.sin(2*np.pi*test['day']/14)\ntest['day_sin14add'] = np.sin(2*np.pi*test['day']/14)*3.5\ntest['month_sin12'] = np.sin(2*np.pi*test['month']/12)\ntest['month_sin24'] = np.sin(2*np.pi*test['month']/24)\ntest['month_sin24_add'] = np.sin(2*np.pi*test['month']/24)*6\n\n# Ordinal Encoding\ntrain['ord_1'] = train.ord_1.apply(lambda x: encode_ord_1(x))\ntrain['ord_2'] = train.ord_2.apply(lambda x: encode_ord_2(x))\ntrain['ord_3'] = train.ord_3.apply(lambda x: encode_ord_3(x))\ntrain['ord_4'] = train.ord_4.str.lower().apply(lambda x: encode_ord_3(x))\ntest['ord_1'] = test.ord_1.apply(lambda x: encode_ord_1(x))\ntest['ord_2'] = test.ord_2.apply(lambda x: encode_ord_2(x))\ntest['ord_3'] = test.ord_3.apply(lambda x: encode_ord_3(x))\ntest['ord_4'] = test.ord_4.str.lower().apply(lambda x: encode_ord_3(x))\n\n# Ordinal Encoding Square\n# First normalize with maximum label for faster convergence,\n# Subtract with 0.5 and square it\ntrain['ord_1_sqr_mid'] = ((train.ord_1 / 4) - 0.5)**2\ntrain['ord_2_sqr_mid'] = ((train.ord_2 / 5) - 0.5)**2\ntrain['ord_3_sqr_mid'] = ((train.ord_3 / 25) - 0.5)**2\ntrain['ord_4_sqr_mid'] = ((train.ord_4 / 25) - 0.5)**2\ntest['ord_1_sqr_mid'] = ((test.ord_1 / 4) - 0.5)**2\ntest['ord_2_sqr_mid'] = ((test.ord_2 / 5) - 0.5)**2\ntest['ord_3_sqr_mid'] = ((test.ord_3 / 25) - 0.5)**2\ntest['ord_4_sqr_mid'] = ((test.ord_4 / 25) - 0.5)**2\n# Square bot not centered\ntrain['ord_1_sqr'] = ((train.ord_1 / 4))**2\ntrain['ord_2_sqr'] = ((train.ord_2 / 5))**2\ntrain['ord_3_sqr'] = ((train.ord_3 / 25))**2\ntrain['ord_4_sqr'] = ((train.ord_4 / 25))**2\ntest['ord_1_sqr'] = ((test.ord_1 / 4))**2\ntest['ord_2_sqr'] = ((test.ord_2 / 5))**2\ntest['ord_3_sqr'] = ((test.ord_3 / 25))**2\ntest['ord_4_sqr'] = ((test.ord_4 / 25))**2\n# Log Transform\ntrain['ord_1_log'] = np.log1p((train.ord_1 / 4))\ntrain['ord_2_log'] = np.log1p((train.ord_2 / 5))\ntrain['ord_3_log'] = np.log1p((train.ord_3 / 25))\ntrain['ord_4_log'] = np.log1p((train.ord_4 / 25))\ntest['ord_1_log'] = np.log1p((test.ord_1 / 4))\ntest['ord_2_log'] = np.log1p((test.ord_2 / 5))\ntest['ord_3_log'] = np.log1p((test.ord_3 / 25))\ntest['ord_4_log'] = np.log1p((test.ord_4 / 25))\n\n# Update columns\ncolumns = train.columns\n\n# Label Encoding to only object features\nfor i in columns:\n    if train[i].dtype == object:        \n        label_encoder = LabelEncoder()\n        label_encoder.fit(train[i])\n        train[i] = label_encoder.transform(train[i])\n        test[i] = label_encoder.transform(test[i])\n\n# Create ordinal square after label encoding - centered\ntrain['ord_5_sqr_mid'] = ((train.ord_5 / train.ord_5.max()) - 0.5)**2\ntest['ord_5_sqr_mid'] = ((test.ord_5 / train.ord_5.max()) - 0.5)**2\n# Create ordinal square after label encoding - not centered\ntrain['ord_5_sqr'] = ((train.ord_5 / train.ord_5.max()) )**2\ntest['ord_5_sqr'] = ((test.ord_5 / train.ord_5.max()) )**2\n# Log transform\ntrain['ord_5_log'] = np.log1p((train.ord_5 / train.ord_5.max()))\ntest['ord_5_log'] = np.log1p((test.ord_5 / train.ord_5.max()))\n\n# Update columns\ncolumns = train.columns\n\n# Standardize the values\nscaler = StandardScaler()\ntrain = scaler.fit_transform(train)\ntest = scaler.transform(test)\n\n# Training model\nlogit = LogisticRegression()\nlogit.fit(train, train_target)\n\n# Predict\ny_pred = logit.predict_proba(test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Make file for submission\nexp3_submission = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/sample_submission.csv')\nexp3_submission['target'] = y_pred[:,1]\nexp3_submission.to_csv('exp3_model.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cor_mtx = pd.concat([pd.DataFrame(train, columns = columns) ,train_target], axis = 1).corr()\nsns.heatmap(cor_mtx)\n\nprint(cor_mtx['target'].sort_values(ascending=False))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Experiment 4\nI want to experiment with Target Encoder, Sum Encoder and OneHot Encoder in this iterations. This is still the phase of collecting features"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Read file\ntrain = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/train.csv')\ntest = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/test.csv')\n\n# Drop id on train and test dataset\ntrain = train.drop('id', axis = 1)\ntest = test.drop('id', axis = 1)\n\n# Select only baseline features\nexp3_features = ['bin_0', 'bin_1', 'bin_2', 'bin_3', 'bin_4', 'day', 'month', 'nom_0',\n                     'nom_1', 'nom_2', 'nom_3', 'nom_4', 'nom_5', 'nom_6', 'nom_7', 'nom_8', 'nom_9',\n                     'ord_0', 'ord_1', 'ord_2', 'ord_3', 'ord_4', 'ord_5']\n\ntrain_target = train['target']\ntrain = train[exp3_features]\ntest = test[exp3_features]\n\n# Replacing nom_6 value 'a885aacec' in test_dataset with 'missing_value' because the value is not seen at training data\ntest.loc[test.nom_6 == \"a885aacec\", 'nom_6'] = \"missing_value\"\n\n# Label encode first bin_3 and bin_4\ntrain['bin_3'] = train.bin_3.apply(lambda x: encode_bin_3(x))\ntrain['bin_4'] = train.bin_4.apply(lambda x: encode_bin_4(x))\ntest['bin_3'] = test.bin_3.apply(lambda x: encode_bin_3(x))\ntest['bin_4'] = test.bin_4.apply(lambda x: encode_bin_4(x))\n\n# Impute with constant\ncolumns = train.columns\n\nfor i in columns:\n    \n    if train[i].dtype == object:\n        imputer = SimpleImputer(missing_values=np.nan, strategy= 'constant', add_indicator= True)\n    else:\n        imputer = SimpleImputer(missing_values=np.nan, strategy= 'constant', fill_value= -1, add_indicator= True)\n\n    imputer.fit(train[i].to_numpy().reshape(-1,1))\n    \n    train[i] = imputer.transform(train[i].to_numpy().reshape(-1,1))\n    test[i] = imputer.transform(test[i].to_numpy().reshape(-1,1))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Feature Engineering\n\n# define one-hot columns\nonehot_columns = train.dtypes[train.dtypes == object].index.to_list()\n\n# See number of samples on training data\nnum_train = len(train)\n\ndf_all = train.append(test)\n\none_hot = pd.get_dummies(\n    df_all[onehot_columns],\n    columns=onehot_columns,\n    drop_first=True,\n    dummy_na=True,\n    sparse=True,\n    dtype=\"int8\",\n # ).to_numpy()\n).sparse.to_coo().toarray()\n\none_hot_train = one_hot[:num_train]\none_hot_test = one_hot[num_train:]\n\none_hot_train = scipy.sparse.coo_matrix(one_hot_train)\none_hot_test = scipy.sparse.coo_matrix(one_hot_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create interactions\ntrain['nom_1_nom_2'] = train.nom_1 + \"_\" + train.nom_2\ntrain['nom_1_nom_3'] = train.nom_1 + \"_\" + train.nom_3\ntrain['nom_1_nom_4'] = train.nom_1 + \"_\" + train.nom_4\ntrain['nom_2_nom_3'] = train.nom_2 + \"_\" + train.nom_3\ntrain['nom_2_nom_4'] = train.nom_2 + \"_\" + train.nom_4\ntrain['nom_3_nom_4'] = train.nom_3 + \"_\" + train.nom_4\ntest['nom_1_nom_2'] = test.nom_1 + \"_\" + test.nom_2\ntest['nom_1_nom_3'] = test.nom_1 + \"_\" + test.nom_3\ntest['nom_1_nom_4'] = test.nom_1 + \"_\" + test.nom_4\ntest['nom_2_nom_3'] = test.nom_2 + \"_\" + test.nom_3\ntest['nom_2_nom_4'] = test.nom_2 + \"_\" + test.nom_4\ntest['nom_3_nom_4'] = test.nom_3 + \"_\" + test.nom_4\n\n# Interaction Exp 3\ntrain['bin_all_sum'] = train.bin_0 + train.bin_1 + train.bin_2 + train.bin_3 + train.bin_4\ntrain['bin_all_mul'] = train.bin_0 * train.bin_1 * train.bin_2 * train.bin_3 * train.bin_4\ntest['bin_all_sum'] = test.bin_0 + test.bin_1 + test.bin_2 + test.bin_3 + test.bin_4\ntest['bin_all_mul'] = test.bin_0 * test.bin_1 * test.bin_2 * test.bin_3 * test.bin_4\n\n# Create cyclical features from day and month\ntrain['day_sin7'] = np.sin(2*np.pi*train['day']/7)\ntrain['day_sin14'] = np.sin(2*np.pi*train['day']/14)\ntrain['day_sin14add'] = np.sin(2*np.pi*train['day']/14)*3.5\ntrain['month_sin12'] = np.sin(2*np.pi*train['month']/12)\ntrain['month_sin24'] = np.sin(2*np.pi*train['month']/24)\ntrain['month_sin24_add'] = np.sin(2*np.pi*train['month']/24)*6\ntest['day_sin7'] = np.sin(2*np.pi*test['day']/7)\ntest['day_sin14'] = np.sin(2*np.pi*test['day']/14)\ntest['day_sin14add'] = np.sin(2*np.pi*test['day']/14)*3.5\ntest['month_sin12'] = np.sin(2*np.pi*test['month']/12)\ntest['month_sin24'] = np.sin(2*np.pi*test['month']/24)\ntest['month_sin24_add'] = np.sin(2*np.pi*test['month']/24)*6\n\n# Ordinal Encoding\ntrain['ord_1'] = train.ord_1.apply(lambda x: encode_ord_1(x))\ntrain['ord_2'] = train.ord_2.apply(lambda x: encode_ord_2(x))\ntrain['ord_3'] = train.ord_3.apply(lambda x: encode_ord_3(x))\ntrain['ord_4'] = train.ord_4.str.lower().apply(lambda x: encode_ord_3(x))\ntest['ord_1'] = test.ord_1.apply(lambda x: encode_ord_1(x))\ntest['ord_2'] = test.ord_2.apply(lambda x: encode_ord_2(x))\ntest['ord_3'] = test.ord_3.apply(lambda x: encode_ord_3(x))\ntest['ord_4'] = test.ord_4.str.lower().apply(lambda x: encode_ord_3(x))\n\n# Ordinal Encoding Square\n# First normalize with maximum label for faster convergence,\n# Subtract with 0.5 and square it\ntrain['ord_1_sqr_mid'] = ((train.ord_1 / 4) - 0.5)**2\ntrain['ord_2_sqr_mid'] = ((train.ord_2 / 5) - 0.5)**2\ntrain['ord_3_sqr_mid'] = ((train.ord_3 / 25) - 0.5)**2\ntrain['ord_4_sqr_mid'] = ((train.ord_4 / 25) - 0.5)**2\ntest['ord_1_sqr_mid'] = ((test.ord_1 / 4) - 0.5)**2\ntest['ord_2_sqr_mid'] = ((test.ord_2 / 5) - 0.5)**2\ntest['ord_3_sqr_mid'] = ((test.ord_3 / 25) - 0.5)**2\ntest['ord_4_sqr_mid'] = ((test.ord_4 / 25) - 0.5)**2\n# Square bot not centered\ntrain['ord_1_sqr'] = ((train.ord_1 / 4))**2\ntrain['ord_2_sqr'] = ((train.ord_2 / 5))**2\ntrain['ord_3_sqr'] = ((train.ord_3 / 25))**2\ntrain['ord_4_sqr'] = ((train.ord_4 / 25))**2\ntest['ord_1_sqr'] = ((test.ord_1 / 4))**2\ntest['ord_2_sqr'] = ((test.ord_2 / 5))**2\ntest['ord_3_sqr'] = ((test.ord_3 / 25))**2\ntest['ord_4_sqr'] = ((test.ord_4 / 25))**2\n# Log Transform\ntrain['ord_1_log'] = np.log1p((train.ord_1 / 4))\ntrain['ord_2_log'] = np.log1p((train.ord_2 / 5))\ntrain['ord_3_log'] = np.log1p((train.ord_3 / 25))\ntrain['ord_4_log'] = np.log1p((train.ord_4 / 25))\ntest['ord_1_log'] = np.log1p((test.ord_1 / 4))\ntest['ord_2_log'] = np.log1p((test.ord_2 / 5))\ntest['ord_3_log'] = np.log1p((test.ord_3 / 25))\ntest['ord_4_log'] = np.log1p((test.ord_4 / 25))\n\n# Update columns\ncolumns = train.columns\n\n# Target Encoding to only object features\nfor i in columns:\n    if train[i].dtype == object:\n        target_encoder = cat_encoder.TargetEncoder(smoothing = 0.1)\n        target_encoder.fit(train[i], train_target)\n        train[i+\"_target\"] = target_encoder.transform(train[i])\n        test[i+\"_target\"] = target_encoder.transform(test[i])\n        \n# Update columns\ncolumns = train.columns\n\n# Label Encoding to only object features\nfor i in columns:\n    if train[i].dtype == object:        \n        label_encoder = LabelEncoder()\n        label_encoder.fit(train[i])\n        train[i] = label_encoder.transform(train[i])\n        test[i] = label_encoder.transform(test[i])\n\n# Create ordinal square after label encoding - centered\ntrain['ord_5_sqr_mid'] = ((train.ord_5 / train.ord_5.max()) - 0.5)**2\ntest['ord_5_sqr_mid'] = ((test.ord_5 / train.ord_5.max()) - 0.5)**2\n# Create ordinal square after label encoding - not centered\ntrain['ord_5_sqr'] = ((train.ord_5 / train.ord_5.max()) )**2\ntest['ord_5_sqr'] = ((test.ord_5 / train.ord_5.max()) )**2\n# Log transform\ntrain['ord_5_log'] = np.log1p((train.ord_5 / train.ord_5.max()))\ntest['ord_5_log'] = np.log1p((test.ord_5 / train.ord_5.max()))\n\n# Update columns\ncolumns = train.columns\n\n# Standardize the values\nscaler = StandardScaler()\ntrain = scaler.fit_transform(train)\ntest = scaler.transform(test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Horizontal stack array result from one-hot encoding\ntrain = scipy.sparse.hstack(([one_hot_train, train])).tocsr()\ntest = scipy.sparse.hstack(([one_hot_test, test])).tocsr()\n\n# Training model\nlogit = LogisticRegression()\nlogit.fit(train, train_target)\n\n# Predict\ny_pred = logit.predict_proba(test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Make file for submission\nexp4_submission = pd.read_csv('/kaggle/input/cat-in-the-dat-ii/sample_submission.csv')\nexp4_submission['target'] = y_pred[:,1]\nexp4_submission.to_csv('exp4_model.csv', index=False)\n\n# Experimentation 4 result better than experiment 3 on public leaderboard. It achieves score 0.77515 using plain LogisticRegression","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}