{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Understanding feature importance with Shap\n\nThe goal of this notebook is to clean up the data a little bit, i.e get it in a usable format, make some predictions using a pretty vanila CatBoost algorithm and explore the model with Shap."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# Libraries\nimport os.path\n\nimport numpy as np\nimport pandas as pd\n\nfrom datetime import timedelta \nfrom datetime import datetime\n\nfrom sklearn.metrics import roc_auc_score\nfrom sklearn.model_selection import train_test_split, StratifiedKFold\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.preprocessing import LabelEncoder, OneHotEncoder, OrdinalEncoder, MinMaxScaler\nfrom category_encoders import TargetEncoder\n\nimport shap\n\nfrom catboost import CatBoostClassifier, Pool\n\nfrom hyperopt import fmin, hp, tpe\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n%matplotlib inline\n\npd.options.display.max_columns = None\npd.options.display.max_rows = None","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# import data\ntrain = pd.read_csv(\"../input/cat-in-the-dat-ii/train.csv\", index_col='id')\ntest = pd.read_csv(\"../input/cat-in-the-dat-ii/test.csv\", index_col='id')\nsample = pd.read_csv(\"../input/cat-in-the-dat-ii/sample_submission.csv\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 1. Feature Preperation"},{"metadata":{"trusted":true},"cell_type":"code","source":"## I'm not going to spend much time worrying about the preparing the data at this time\n## I'll reuse some code from other excellent notebooks to save time","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Source: https://www.kaggle.com/vikassingh1996/don-t-underestimate-the-power-of-a-logistic-reg\n\n'''Variable Description'''\ndef description(df):\n    print(f\"Dataset Shape: {df.shape}\")\n    summary = pd.DataFrame(df.dtypes,columns=['dtypes'])\n    summary = summary.reset_index()\n    summary['Name'] = summary['index']\n    summary = summary[['Name','dtypes']]\n    summary['Missing'] = df.isnull().sum().values\n    summary['PercMissing'] = df.isnull().sum().values / df.isnull().count().values\n    summary['Uniques'] = df.nunique().values\n    summary['First Value'] = df.iloc[0].values\n    summary['Second Value'] = df.iloc[1].values\n    summary['Third Value'] = df.iloc[2].values\n    return summary\nprint('**Variable Description of  train Data:**')\ndescription(train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Some intial Comments\n\nThere's lots of interesting features here...\n\n1. id, target\n    * Row identifier and target variable. Not much to be said here. \n    * Might be interesting to know what the target variable distribution is\n\n2. binary variables\n    * All of them are missing roughly 3% of their data\n    * Some are integers and some are text\n    * As CatBoost can deal with categorical text variables, I just need to figure out how best to deal with the missing data\n        * Replace null's with the mode maybe? Or make them thier own seperate feature? \n\n3. nominal variables\n    * 9 features\n        * nom_0 to nom_4 have very low cardinality - One Hot Encoding might be the best approach to dealing with these variables\n        * nom_5 to nom_9 have very high cardinality (loads of categories) - I'm not sure what the best approach with dealing with them is yet.\n\n4. ordinal features\n    * 5 features\n        * 4 of them look ok. One has pretty high cardinality so will have to be processed a little differently.\n        * Need to check if CatBoost deals with ordinal data in any specific way or if I should process these running the model.\n\n5. time features\n    * day and month"},{"metadata":{},"cell_type":"markdown","source":"# 2. Dealing with Nulls"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Source: https://www.kaggle.com/vikassingh1996/don-t-underestimate-the-power-of-a-logistic-reg\n\n## To start, let's just replace all null values with the mode of that column\ndef replace_nan(data):\n    for column in data.columns:\n        if data[column].isna().sum() > 0:\n            data[column] = data[column].fillna(data[column].mode()[0])\n\n\nreplace_nan(train)\nreplace_nan(test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"target = train.pop('target')\ntarget.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Feature Encoding\n\nWhile the CatBoost algorithm doesn't necessarily need and encoding (it has built in tools itself to deal with categories), to get the best out of the Shap package we need to make sure each feature is numerical.\n\nSo, on that note:"},{"metadata":{"trusted":true},"cell_type":"code","source":"## Source: https://www.kaggle.com/carlodnt/catboost-shap-fastai\n\n# bin_3\ntrain['bin_3'] = train['bin_3'].apply(lambda x: 0 if x == 'F' else 1)\ntest['bin_3'] = test['bin_3'].apply(lambda x: 0 if x == 'F' else 1)\n\n# bin_4\ntrain['bin_4'] = train['bin_4'].apply(lambda x: 0 if x == 'N' else 1)\ntest['bin_4'] = test['bin_4'].apply(lambda x: 0 if x == 'N' else 1)\n\n# ord_1\ntrain.ord_1.replace(to_replace = ['Novice', 'Contributor','Expert', 'Master', 'Grandmaster'],\n                         value = [0, 1, 2, 3, 4], inplace = True)\ntest.ord_1.replace(to_replace = ['Novice', 'Contributor','Expert', 'Master', 'Grandmaster'],\n                         value = [0, 1, 2, 3, 4], inplace = True)\n\n# ord_2\ntrain.ord_2.replace(to_replace = ['Freezing', 'Cold', 'Warm', 'Hot','Boiling Hot', 'Lava Hot'],\n                         value = [0, 1, 2, 3, 4, 5], inplace = True)\ntest.ord_2.replace(to_replace = ['Freezing', 'Cold', 'Warm', 'Hot','Boiling Hot', 'Lava Hot'],\n                         value = [0, 1, 2, 3, 4, 5], inplace = True)\n\n# ord_3\ntrain.ord_3.replace(to_replace = ['a', 'b', 'c', 'd', 'e', 'f', 'g','h', 'i', 'j', 'k', 'l', 'm', 'n', 'o'],\n                         value = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14], inplace = True)\ntest.ord_3.replace(to_replace = ['a', 'b', 'c', 'd', 'e', 'f', 'g','h', 'i', 'j', 'k', 'l', 'm', 'n', 'o'],\n                         value = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14], inplace = True)\n\n# ord_4\ntrain.ord_4.replace(to_replace = ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I','J', 'K', 'L', 'M', 'N', 'O', \n                                     'P', 'Q', 'R','S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z'],\n                         value = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, \n                                  22, 23, 24, 25], inplace = True)\ntest.ord_4.replace(to_replace = ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I','J', 'K', 'L', 'M', 'N', 'O', \n                                     'P', 'Q', 'R','S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z'],\n                         value = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, \n                                  22, 23, 24, 25], inplace = True)\n\nhigh_card = ['nom_0','nom_1','nom_2','nom_3','nom_4','nom_5', 'nom_6', 'nom_7', 'nom_8', 'nom_9','ord_5']\nfor col in high_card:\n    enc_nom = (train.groupby(col).size()) / len(train)\n    train[f'{col}'] = train[col].apply(lambda x: hash(str(x)) % 5000)\n    test[f'{col}'] = test[col].apply(lambda x: hash(str(x)) % 5000)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.shape, test.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. CatBoost Algorithm"},{"metadata":{"trusted":true},"cell_type":"code","source":"# create a training and validation set\nX_train, X_validation, y_train, y_validation = train_test_split(train, target, train_size=0.8, random_state=42)\n\nX_test = test.copy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train.shape, X_validation.shape, y_train.shape, y_validation.shape, X_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"categorical_features_indices = np.where(train.dtypes != np.float)[0]\ncategorical_features_indices","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## Source: https://www.kaggle.com/lucamassaron/catboost-in-action-with-dnn\n\n# Initializing a CatBoostClassifier with best parameters\nbest_params = {'bagging_temperature': 0.8,\n               'depth': 5,\n               'iterations': 500,\n               'l2_leaf_reg': 30,\n               'learning_rate': 0.05,\n               'random_strength': 0.8}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":" model = CatBoostClassifier(\n        **best_params,\n        loss_function='Logloss',\n        eval_metric='AUC',         \n#         task_type=\"GPU\",\n        nan_mode='Min',\n        verbose=False\n    )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.fit(\n        X_train, y_train,\n        verbose_eval=100, \n        early_stopping_rounds=50,\n        cat_features=categorical_features_indices,\n        eval_set=(X_validation, y_validation),\n        use_best_model=False,\n        plot=True\n);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"# 5. Model explanation with Shap"},{"metadata":{"trusted":true},"cell_type":"code","source":"shap.initjs()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"explainer = shap.TreeExplainer(model)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"shap_values = explainer.shap_values(Pool(X_train, y_train, cat_features=categorical_features_indices))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"First, let's take some rows at random and look at the shapley contribution each feature made to each prediction."},{"metadata":{"trusted":true},"cell_type":"code","source":"shap.force_plot(explainer.expected_value, shap_values[0,:], X_train.iloc[0,:])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Row 0** output is lower than the model baseline. This means the row would be predicted as being `0`. What's contributing to this? Well `nom_7` and `nom_8` appear to have the largest negative impact on the output, while `ord_3` has the largest positive impact. Outside of `ord_3` no other features are having any meaningful positive impact on the predcition. \n\nTo get a deeper insight into what's actually going on here, decoding the features back into their original states might shed some more light."},{"metadata":{"trusted":true},"cell_type":"code","source":"shap.force_plot(explainer.expected_value, shap_values[4,:], X_train.iloc[4,:])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Row 4** output is higher than the baseline. So, what's contributing to this? `ord_3` once again is having a significant positive impact on the output. Interestingly, it is very close in value to the same feature in `row 0`. Unlike `row 0`, `nom_7` and `nom_8` have a far lower impact on the model predictions. Again decoding the features back into their original states might shed some more light into whats going on here."},{"metadata":{},"cell_type":"markdown","source":"This would be a long, long process to run through this exercise with many rows. So to speed up the insights, lets stack a number of them horizontally and see what we get."},{"metadata":{"trusted":true},"cell_type":"code","source":"# visualize the training set predictions\nshap.force_plot(explainer.expected_value, shap_values[0:50,:], X_train.iloc[0:50,:])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is a really nice plot! It gives a great visual representation of how different values affect predictions. It also highlights certain rows to look into deeper using the single row view above."},{"metadata":{},"cell_type":"markdown","source":"### Feature Importance"},{"metadata":{"trusted":true},"cell_type":"code","source":"# feature importance plot\nshap.summary_plot(shap_values, X_train, plot_type=\"bar\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Ok, so the most importance features in this model are `ord_3`, `ord_5` and `ord_2` while the least important features are `bin_1` and `bin_4`. This is useful information, but let's use Shap to go deeper."},{"metadata":{"trusted":true},"cell_type":"code","source":"# summarize the effects of all the features\nshap.summary_plot(shap_values, X_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### So, whats going on here? \nFirst, a quick recap as to what's going on in this plot.\n\nThe plot shows a cumulation of many dots. The dots have three characteristics: \n* The vertical location of the dots show what feature they are depicting\n* Color shows whether that feature value was high or low for that row of the dataset\n* Horizontal location shows whether the effect of that value caused a higher or lower prediction.\n\n\n### Let's look at some features in particular.\n\n#### ord_3\n\n**Context:** This feature has 14 categories, letters `a` to `o`. For the purpose of the model, they were ordinally encoded with `a` becoming 1 up to `o` becoming 14.\n\nThis feature has the largest impact on the model and interestingly the higher the value, the more positive the contribution the feature gave to the model prediction. So any rows with letters, `m`, `n`, or `o` are more likely to associate with target variable 1 while `a`, `b`, or `c` are more likely to be associated with target variable 0.\n\n#### bin_1\n\n**Context:** This is a binary feature of `0's` and `1's`.\n\nUnlike `ord_3` this feature has very little impact on the models predictions. When this feature is 1, it has slightly negative impact on predictions, i.e. they are more likely to be 0. And conversely, when this feature is 0, is has a slighly positive impact on predictions."},{"metadata":{},"cell_type":"markdown","source":"# Conclusion\n\nI found this to be a very useful exercise in understanding the data better and I have a number of new avenues lined up to explore as a result.\n\nAny and all comments are welcome!"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}