{"cells":[{"metadata":{},"cell_type":"markdown","source":"In this notebook I illustrate the effectiveness of Weight of Evidence Encoding (`WOEEncoder`) on a simple Logistic Regression model and compare it to other target-based encodings, such as Target Encoding and CatBoost Encoding. \n\nThe bottom line is: it performs quite well, generally better than Target Encoding and Catboost, at least for binary classification problems. Casting `WOEEncoder` on all nominal and cyclical features was enough to get an AUC score of 78.355% on the test set, and replacing `TargetEncoder` with `WOEEncoder` on \"stratified\" encodings (such as seen in [this notebook by caesarlupum](https://www.kaggle.com/caesarlupum/2020-20-lines-target-encoding) and also in [this clickbait one](https://www.kaggle.com/muhammad4hmed/easily-get-78-5-accuracy)) improved the AUC score from about 78.50% to about 78.56%.\n\nSee [this article](https://www.listendata.com/2015/03/weight-of-evidence-woe-and-information.html) for a theoretical explanaition, and [this page at Category Encoders](https://contrib.scikit-learn.org/categorical-encoding/woe.html) for documentation. In few words, what this does on a categorical feature $F$ is:\n* for each unique value $x,$ consider the corresponding rows in the training set\n* compute what percentage of positives is in these rows, compared to the whole set\n* compute what percentage of negatives is in these rows, compared to the whole set\n* take the ratio of these percentages\n* take the natural logarithm of that ratio to get the weight of evidence corresponding to $x,$ so that $WOE(x)$ is either positive or negative according to whether $x$ is more representative of positives or negatives\n* NaN's are set to have WOE=0, or according to the `handle_missing` option\n\nFor numerical features, it does the same with respect to bins.","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport os\n\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.model_selection import train_test_split, StratifiedKFold\nfrom sklearn.metrics import confusion_matrix, classification_report, roc_auc_score, roc_curve\n\nimport category_encoders as ce\n\n%matplotlib inline\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\ntest_features = pd.read_csv(\"../input/cat-in-the-dat-ii/test.csv\")\ntrain_set = pd.read_csv(\"../input/cat-in-the-dat-ii/train.csv\")\n\ntrain_targets = train_set.target\ntrain_features = train_set.drop(['target'], axis=1)\npercentage = train_targets.mean() * 100\nprint(\"The percentage of ones in the training target is {:.2f}%\".format(percentage))\ntrain_features.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Example","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"columns = [col for col in train_features.columns if col != 'id']\nwoe_encoder = ce.WOEEncoder(cols=columns)\nwoe_encoded_train = woe_encoder.fit_transform(train_features[columns], train_targets).add_suffix('_woe')\ntrain_features = train_features.join(woe_encoded_train)\n\nwoe_encoded_cols = woe_encoded_train.columns","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's see what this does on feature `nom_0`.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df = train_features.copy()\ndf['target'] = train_targets\n\noverall_number_of_ones = train_targets.sum()\noverall_number_of_zeroes = 600000 - overall_number_of_ones\nprint(\"There are {} ones and {} zeroes in the training set\".format(\n    overall_number_of_ones, overall_number_of_zeroes\n))\n\ngrouped = pd.DataFrame()\ngrouped['Total'] = df.groupby('nom_0').id.count()\ngrouped['number of ones'] = df.groupby('nom_0').target.sum()\ngrouped['number of zeroes'] = grouped['Total'] - grouped['number of ones']\n\ngrouped['percentage of ones'] = grouped['number of ones'] / overall_number_of_ones\ngrouped['percentage of zeroes'] = grouped['number of zeroes'] / overall_number_of_zeroes\ngrouped['(% ones) > (% zeroes)'] = grouped['percentage of ones'] > grouped['percentage of zeroes']\n\ngrouped['weight of evidence'] = df.groupby('nom_0').nom_0_woe.mean()\n\ngrouped","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's see what it does on another column.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"grouped = pd.DataFrame()\ngrouped['Total'] = df.groupby('month').id.count()\ngrouped['number of ones'] = df.groupby('month').target.sum()\ngrouped['number of zeroes'] = grouped['Total'] - grouped['number of ones']\n\ngrouped['percentage of ones'] = grouped['number of ones'] / overall_number_of_ones\ngrouped['percentage of zeroes'] = grouped['number of zeroes'] / overall_number_of_zeroes\ngrouped['(% ones) > (% zeroes)'] = grouped['percentage of ones'] > grouped['percentage of zeroes']\n\ngrouped['weight of evidence'] = df.groupby('month').month_woe.mean()\n\ngrouped","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Performance comparison with other encoders","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Define helper function\ndef logreg_test(cols, encoder):\n    df = train_features[cols]\n    auc_scores = []\n    acc_scores = []\n    \n    skf = StratifiedKFold(n_splits=6, shuffle=True).split(df, train_targets)\n    for train_id, valid_id in skf:\n        enc_tr = encoder.fit_transform(df.iloc[train_id,:], train_targets.iloc[train_id])\n        enc_val = encoder.transform(df.iloc[valid_id,:])\n        regressor = LogisticRegression(solver='lbfgs', max_iter=1000, C=0.6)\n        regressor.fit(enc_tr, train_targets.iloc[train_id])\n        acc_scores.append(regressor.score(enc_val, train_targets.iloc[valid_id]))\n        probabilities = [pair[1] for pair in regressor.predict_proba(enc_val)]\n        auc_scores.append(roc_auc_score(train_targets.iloc[valid_id], probabilities))\n        \n    acc_scores = pd.Series(acc_scores)\n    mean_acc = acc_scores.mean() * 100\n    print(\"Mean accuracy score: {:.3f}%\".format(mean_acc))\n    \n    auc_scores = pd.Series(auc_scores)\n    mean_auc = auc_scores.mean() * 100\n    print(\"Mean AUC score: {:.3f}%\".format(mean_auc))\n\n##########################################\nprint(\"Using Weight of Evidence Encoder\")\nwoe_encoder = ce.WOEEncoder(cols=columns)\nlogreg_test(columns, woe_encoder)\n\n##########################################\nprint(\"\\nUsing Target Encoder\")\ntarg_encoder = ce.TargetEncoder(cols=columns, smoothing=0.2)\nlogreg_test(columns, targ_encoder)\n\n##########################################\nprint(\"\\nUsing CatBoost Encoder\")\ncb_encoder = ce.CatBoostEncoder(cols=columns)\nlogreg_test(columns, cb_encoder)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Correlation with target","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Target-encoded features generally show greater correlation with target that WOE-encoded ones. This is an example where correlation shouldn't be trusted too much as a metric of feature importance.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Encode again, this time on the whole training set. WOEE was done above.\nencoder = ce.TargetEncoder(cols=columns, smoothing=0.2)\nencoded_train = encoder.fit_transform(train_features[columns], train_targets).add_suffix('_targ_enc')\ntrain_features = train_features.join(encoded_train)\n\nencoder = ce.CatBoostEncoder(cols=columns)\nencoded_train = encoder.fit_transform(train_features[columns], train_targets).add_suffix('_catboost')\ntrain_features = train_features.join(encoded_train)\n\ntraining_set = train_features.copy()\ntraining_set['target'] = train_targets\ncorrmat = training_set.corr()\nplt.subplots(figsize=(20,20))\nsns.heatmap(corrmat, vmax=0.9, square=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"corr_with_target = corrmat['target'].apply(abs).sort_values(ascending=False)\ncorr_with_target.drop(['target'], inplace=True)\ndf = pd.DataFrame(data={'features': corr_with_target.index, 'target': corr_with_target.values})\nplt.figure(figsize=(20, 20))\nsns.barplot(x=\"target\", y=\"features\", data=df)\nplt.title('Correlation with target')\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Other remarks gathered from previous notebooks\n* for `day` and `month`, both target encoding and WoE encoding show higher correlation with target than the original labeling and also than the \"trigonometrical\" encoding\n* for binary and ordinal features, WoE encoding alone showed little improvement with respect to the obvious ordinal encodings\n* Casting `WOEEncoder` on nominal and cyclical features, and using ordinal encoding for binary and ordinal features resulted in an AUC score of 0.78355 on the test set\n* Casting `TargetEncoder` on all features resulted in an AUC score of 0.78302 on the test set","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Using stratified WOE encoding for final output","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Encoding training data\ndf = train_features[columns]\ntrain_encoded = pd.DataFrame()\nskf = StratifiedKFold(n_splits=5,shuffle=True).split(df, train_targets)\nfor tr_in,fold_in in skf:\n    encoder = ce.WOEEncoder(cols=columns)\n    encoder.fit(df.iloc[tr_in,:], train_targets.iloc[tr_in])\n    train_encoded = train_encoded.append(encoder.transform(df.iloc[fold_in,:]),ignore_index=False)\n\ntrain_encoded = train_encoded.sort_index()\n\n# Encoding test data\nencoder = ce.WOEEncoder(cols=columns)\nencoder.fit(df, train_targets)\ntest_encoded = encoder.transform(test_features[columns])\n\n# Fitting\nregressor = LogisticRegression(solver='lbfgs', max_iter=1000, C=0.6)\nregressor.fit(train_encoded, train_targets)\n\n# Predicting\nprobabilities = [pair[1] for pair in regressor.predict_proba(test_encoded)]\noutput = pd.DataFrame({'id': test_features['id'],\n                       'target': probabilities})\noutput.to_csv('submission.csv', index=False)\noutput.describe()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}