{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Using the loan-default-dataset, we aim to build a model that can accurately predict whether or not a loan-applicant will default or not based on a set of features.","metadata":{}},{"cell_type":"code","source":"# Libraries\nimport pandas as pd # Dataframes\nimport numpy as np\nimport matplotlib.pyplot as plt # Visualization\nimport seaborn as sns # Visualization\nfrom sklearn.preprocessing import OneHotEncoder # Encoding with categorical data\nfrom sklearn.preprocessing import LabelEncoder\nfrom sklearn.model_selection import train_test_split\nimport missingno as msno # Visualizing null values\nfrom sklearn.impute import KNNImputer # Filling in null values for numeric data\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.metrics import classification_report\nfrom sklearn.metrics import confusion_matrix\nfrom xgboost import XGBClassifier\nfrom sklearn.model_selection import cross_val_score, KFold\nfrom sklearn.naive_bayes import GaussianNB, MultinomialNB, BernoulliNB\n\n# Read in data\ndf = pd.read_csv(\"/kaggle/input/loan-default-dataset/Loan_Default.csv\")","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-12-16T01:55:23.795218Z","iopub.execute_input":"2022-12-16T01:55:23.795777Z","iopub.status.idle":"2022-12-16T01:55:25.061992Z","shell.execute_reply.started":"2022-12-16T01:55:23.795714Z","shell.execute_reply":"2022-12-16T01:55:25.060654Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's get a feel for the data and polish up the column names. We can also drop the 'year' column since all entries are from 2019.","metadata":{}},{"cell_type":"code","source":"# First 100 entries\nhead100 = df.head(100)\n\n# Convert all column names to lowercase\ndf.columns=df.columns.str.lower()\n\n# Make ID string type\ndf['id'] = df['id'].astype(str)\n\n# Drop year since all entries are 2019\ndf.drop(columns=['year'], inplace=True)\n\n# Datatypes\ndf.info()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's check for null values.","metadata":{}},{"cell_type":"code","source":"# Null values\nnas = df.isna().sum()\nprint(nas)\n\n# Visualizing Null values (image)\nmsno.matrix(df)\nplt.figure(figsize = (15,9))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-16T01:58:00.720780Z","iopub.execute_input":"2022-12-16T01:58:00.721394Z","iopub.status.idle":"2022-12-16T01:58:03.290087Z","shell.execute_reply.started":"2022-12-16T01:58:00.721348Z","shell.execute_reply":"2022-12-16T01:58:03.289156Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have quite a few columns with null values. Let's go down the list and start handling them properly. But first, lets create new dataframes based on data type.","metadata":{}},{"cell_type":"code","source":"# Getting dataframes by datatype\ndtypes = pd.DataFrame(df.dtypes).reset_index()\n\ncat_vars = []\nnum_vars = []\nfor i, l in zip(dtypes['index'], dtypes[0]):\n    if l == 'object':\n        cat_vars.append(i)\n    else:\n        num_vars.append(i)","metadata":{"execution":{"iopub.status.busy":"2022-12-16T02:03:53.908564Z","iopub.execute_input":"2022-12-16T02:03:53.909503Z","iopub.status.idle":"2022-12-16T02:03:53.917844Z","shell.execute_reply.started":"2022-12-16T02:03:53.909457Z","shell.execute_reply":"2022-12-16T02:03:53.916605Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We'll start with numerical data first. An effective way of filling NaN values for numerical data is using a KNNImputer from sklearn. We could fill in the null values using the column median but this will often bias the data, especially when there are many missing values.\nKNNImputer uses the columns that have values (at that index) to estimate the null value of a certain entry.","metadata":{}},{"cell_type":"code","source":"# ====================================================================\n# Imputing: Numeric Data\n# ====================================================================\n\n# Numeric Dataframe\ndf_num = df[num_vars]\n\n# knn\nknn = KNNImputer(n_neighbors = 3)\nknn.fit(df_num)\nX = knn.fit_transform(df_num)\n\n# Check for any nas\ndf_num = pd.DataFrame(X, columns=num_vars)\nnas_num = df_num.isna().sum()\nprint(nas_num)","metadata":{"execution":{"iopub.status.busy":"2022-12-16T02:03:56.904272Z","iopub.execute_input":"2022-12-16T02:03:56.904857Z","iopub.status.idle":"2022-12-16T02:17:16.864776Z","shell.execute_reply.started":"2022-12-16T02:03:56.904823Z","shell.execute_reply":"2022-12-16T02:17:16.863399Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let's take on the missing categorical data. Here, we will take the most likely value (or the value that appears most frequently) for that column. Once all null values are imputed, we'll concatenate the numeric and categorical dataframes.","metadata":{}},{"cell_type":"code","source":"# ====================================================================\n# Imputing: Categorical Data\n# ====================================================================\n\n# Categorical Dataframe\ndf_cat = df[cat_vars]\n\nfor i in cat_vars:\n    mode = df[i].mode()\n    mode = mode[0]\n    df_cat[i].fillna(value=mode, inplace=True)\n\n# Check for any nas\nnas_cat = df_cat.isna().sum()\nprint(nas_cat)\n\n# Combining dataframes\ndf_full = pd.concat([df_num, df_cat], axis=1, join='inner')\n\n# Full dataframe visualization of null values\nmsno.matrix(df_full)\nplt.figure(figsize = (15,9))\nplt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next, lets find the outliers.","metadata":{}},{"cell_type":"code","source":"df_num = df_full[num_vars]\n\n# Function for finding outliers using 3 times the IQR\ndef find_outliers_IQR(col):\n   Q1=col.quantile(0.25)\n   Q3=col.quantile(0.75)\n   IQR=Q3-Q1\n   outliers = col[((col<(Q1-3*IQR)) | (col>(Q3+3*IQR)))]\n   return(outliers)\n\n# Function for finding proportion of column that is null\ndef outlier_prop(outliers, col):\n    outlier_size = len(outliers)\n    return outlier_size / (len(col) + outlier_size)\n\n# Termporary dataframe\ndf_temp = df_full.copy()\n\n# Both term and status are 15-20% outliers and can be treated as categorical variables\ndf_temp['term'] = df_temp['term'].astype(str)\ndf_temp['status'] = df_temp['status'].astype(str)\n\n\n# Getting num_vars and cat_vars\ndtypes = pd.DataFrame(df_temp.dtypes).reset_index()\ncat_vars = []\nnum_vars = []\nfor i, l in zip(dtypes['index'], dtypes[0]):\n    if l == 'object':\n        cat_vars.append(i)\n    else:\n        num_vars.append(i)\n\n# Getting proportion of \noutlier_props = []\ncols = []\nfor i in num_vars:\n    outliers = find_outliers_IQR(df_temp[i])\n    cols.append(i)\n    prop = outlier_prop(outliers, df_temp[i])\n    outlier_props.append(prop)\n\noutlier_props_df = pd.DataFrame([cols, outlier_props], index=['Variable', 'OutlierProp']).transpose()\n\n# Deleting outliers\nfor col in num_vars:\n    q1 = df_temp[col].quantile(0.25)\n    q3 = df_temp[col].quantile(0.75)\n    IQR = q3-q1\n    lower = q1-3*IQR\n    upper = q3+3*IQR\n    df_temp = df_temp[(df_temp[col] < upper) & (df_temp[col] > lower)]\n    \ndf_full = df_temp\n\n# Term\nterm_vals = pd.DataFrame(df_full['term'].value_counts().reset_index())\n\n# Drop terms that have less than 10 appearences in dataset\nterms_to_drop = []\nfor i, l in zip(term_vals['index'], term_vals['term']):\n    if l < 10:\n        terms_to_drop.append(i)\n\nfor i in terms_to_drop:\n    df_full = df_full[df_full['term'] != i]\n\n# Remaining Data\nproportion_remaining = round(len(df_full) / len(df), 5)\nproportion_dropped = round(1 - proportion_remaining, 2) * 100\ndropped = len(df) - len(df_full)\nprint(\"We dropped \", dropped, ' rows from the dataset')\nprint(\"That is about\", proportion_dropped, \"% of the original dataset\" )\nprint(\"The proportion of the original dataset remaining is:  \", proportion_remaining)\n\n\n\n# Make term categorical dtype\ndf_full['term'] = df_full['term'].astype('float')\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now that all our data is clean, lets do some exploration and analysis. We'll start with our numerical data.","metadata":{}},{"cell_type":"code","source":"# ====================================================================\n# Numerical Data\n# ====================================================================\n\n# Correlation\ncorr = df_num.corr()\n\n# Correlation Heatmap\nplt.figure(figsize=(16, 6))\nheatmap = sns.heatmap(corr, vmin=-1, vmax=1, annot=True)\nheatmap.set_title('Correlation Heatmap', fontdict={'fontsize':12}, pad=12);","metadata":{"execution":{"iopub.status.busy":"2022-12-16T02:17:17.896936Z","iopub.execute_input":"2022-12-16T02:17:17.897486Z","iopub.status.idle":"2022-12-16T02:17:19.003359Z","shell.execute_reply.started":"2022-12-16T02:17:17.897433Z","shell.execute_reply":"2022-12-16T02:17:19.002434Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Nothing too surprising about our correlation matrix. Let's plot each of our numerical attributes against our target ('status').","metadata":{}},{"cell_type":"code","source":"# Boxplots\nfor i in num_vars:\n    plt.figsize=(16,6)\n    sns.set_theme(style='darkgrid')\n    sns.boxplot(x=i, y='status', data=df_full)\n    plt.show()\n    \n# Histograms\nfor i in num_vars:\n    plt.figsize=(16,6)\n    sns.set_theme(style='darkgrid')\n    sns.histplot(data=df_full, x=i, hue=\"status\", multiple=\"dodge\", shrink=.8, bins=4)\n    plt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next, lets do some data visualization with our categorical data. ","metadata":{}},{"cell_type":"code","source":"# ====================================================================\n# Categorical Data\n# ====================================================================\n\ndef plot_hist(col):\n    plt.figsize=(16,6)\n    sns.set_theme(style='darkgrid')\n    sns.histplot(data=df_full, x=col, hue=\"status\", multiple=\"dodge\", shrink=.8, stat='count')\n    plt.show()\n\n\n# loan_limit\nplot_hist(df_full['loan_limit'])\n\n# Gender\nplot_hist(df_full['gender'])\n\n# approved in advance\nplot_hist(df_full['approv_in_adv'])\n\n# loan_type\nplot_hist(df_full['loan_type'])\n\n# loan_purpose\nplot_hist(df_full['loan_purpose'])\n\n# Credit worthiness\nplot_hist(df_full['credit_worthiness'])\n\n# Open credit\nplot_hist(df_full['open_credit'])\n\n# Business or commercial\nplot_hist(df_full['business_or_commercial'])\n\n# neg_ammortization\nplot_hist(df_full['neg_ammortization']) # neg_amm more likely to default\n\n# interest_only\nplot_hist(df_full['interest_only'])\n\n# Lump sum payment\nplot_hist(df_full['lump_sum_payment'])\ndf_full['lump_sum_payment'].value_counts()\nlpsm = df_full[df_full['lump_sum_payment'] == 'lpsm']\nlpsm['status'].value_counts() # Vast majority of lumpsum payments default\n\n# Construction type\nplot_hist(df_full['construction_type'])\ndf['construction_type'].value_counts() # Only 33 values for mh\nmh = df_full[df_full['construction_type'] == 'mh']\nmh['status'].value_counts() # All mh vals were defaults\n\n# occupancy_type\nplot_hist(df_full['occupancy_type'])\n\n# secured_by\nplot_hist(df_full['secured_by'])\ndf_full['secured_by'].value_counts()\nland = df_full[df_full['secured_by'] == 'land']\nland['status'].value_counts() # All land security defaulted\n\n# total_units\nplot_hist(df_full['total_units'])\ndf_full['total_units'].value_counts()\n\n# credit_type\nplot_hist(df_full['credit_type']) # All EQUI credit type defaults\n\n# co-applicant_credit_type\nplot_hist(df_full['co-applicant_credit_type'])\n\n# age\nplot_hist(df_full['age'])\n\n# submission_of_application\nplot_hist(df_full['submission_of_application'])\n\n# region\nplot_hist(df_full['region'])\n\n# security_type\nplot_hist(df_full['security_type'])\ndf_full['security_type'].value_counts()\nindirect = df_full[df_full['security_type'] == 'Indriect']\nindirect['status'].value_counts() # All indirect security type defaults","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Some important things to note:\n* There are a six columns where a particular value was not seen often in the dataset and would be considered an outlier. However, for each of these columns, the instances where the infrequent values occur are instances where the loan-borrower defaulted. Thus, this is important data to keep.\n\n* 14 of the categorical variables are binary, which will be important for us when we encode the categorical attributes.\n\n* It is clear that the high property values do NOT result in more defaults. The contrary is true.","metadata":{}},{"cell_type":"markdown","source":"Let's now preprocess our data for machine learning models. First, let's encode our categorical attributes.","metadata":{}},{"cell_type":"code","source":"# ====================================================================\n# Encoding Categorical Variables\n# ====================================================================\n\ndf_full.drop(columns=['id'], inplace=True)\n\ndtypes = pd.DataFrame(df_full.dtypes).reset_index()\ncat_vars = []\nnum_vars = []\nfor i, l in zip(dtypes['index'], dtypes[0]):\n    if l == 'object':\n        cat_vars.append(i)\n    else:\n        num_vars.append(i)\n\n\n\n# Binary variables\nbinary_vars = ['security_type', 'submission_of_application', 'co-applicant_credit_type', 'secured_by',\n               'lump_sum_payment', 'interest_only', 'neg_ammortization', 'construction_type', 'business_or_commercial',\n               'open_credit', 'credit_worthiness', 'approv_in_adv', 'loan_limit', 'status']","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The reason we are creating a list of the binary variables is to reduce the size of our dataframe after encoding. We want to use a Label Encoder for the binary variables which will result in one column of 1's and 0's. If we were to use sklearn's OneHotEncoder on binary variables, it would create two columns, one for each possible value, which is redundant considering we can use one column where the value \"1\" represents one possible value and \"0\" represents the other.","metadata":{}},{"cell_type":"code","source":"# Label Encoder\nlabel = LabelEncoder()\nfor i in binary_vars:\n    df_full[i] = label.fit_transform(df_full[i])\n\n\n\n# OneHotEncoding\ndf_cat = df_full[cat_vars]\ndf_cat.drop(columns=binary_vars, inplace=True)\n\ndf_cat.columns\n\ncat_encoder = OneHotEncoder()\ndf_cat_1hot = cat_encoder.fit_transform(df_cat)\ndf_cat_encoded = pd.DataFrame(df_cat_1hot.toarray())\n\n\n# Column names\ncat_encoder.categories_\ncat_columns = ['Female', 'Joint', 'Male', 'Sex Not Available',\n               'type1', 'type2', 'type3',\n               'p1', 'p2', 'p3', 'p4',\n               'ir', 'pr', 'sr',\n               'U1', 'U2', 'U3', 'U4',\n               'CIB', 'CRIF', 'EQUI', 'EXP',\n               'age_25-34', 'age_35-44', 'age_45-54', 'age_55-64', 'age_65-74', 'under_25', 'over_74',\n               'North', 'North-East', 'central', 'south']\n\ndf_cat_encoded.columns = cat_columns\ndf_full.drop(columns=df_cat.columns, inplace=True)\n# Concat\ndf = pd.concat([df_full, df_cat_encoded], axis=1, join='inner')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We are now ready to split our data into a training set and test set.","metadata":{}},{"cell_type":"code","source":"# ====================================================================\n# Splitting Data\n# ====================================================================\n\n# train_test_split\ntrain_set, test_set = train_test_split(df, test_size=0.2, random_state=42)\n\ny_train = train_set['status']\nX_train = train_set.drop(columns=['status'])\ny_test = test_set['status']\nX_test = test_set.drop(columns=['status'])\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Time for modeling.\nWe will use three different classifiers and choose the best one:\n1. RandomForestClassifer\n2. XGBoostClassifier\n3. NaiveBayesClassifer","metadata":{}},{"cell_type":"code","source":"# ====================================================================\n# Random Forest Classifier: 0.911 accuracy on test set\n# ====================================================================\n\n# Fitting Model\nrf = RandomForestClassifier()\nrf.fit(X_train, y_train)\n\n# Predictions\nrf_preds = rf.predict(X_test)\n\n# Performance\naccuracy_score(y_test, rf_preds)\nconfusion_matrix(y_test, rf_preds)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The RandomForestClassifier works very well with an accuracy score of 91.1%! Let's try the XGBoostClassifier and see if it can outperform the RandomForest.","metadata":{}},{"cell_type":"code","source":"# ====================================================================\n# XGBoost Classifier: 0.9733 accuracy on test set\n# ====================================================================\n\n# Fitting Model\nxgbc = XGBClassifier()\nxgbc.fit(X_train, y_train)\n\n# Scores on train set\nscores = cross_val_score(xgbc, X_train, y_train, cv=5)\nprint(\"Mean cross-validation score: %.2f\" % scores.mean())\n\n# Predictions\nxgb_preds = xgbc.predict(X_test)\n\n# Test set performance\naccuracy_score(y_test, xgb_preds)\nconfusion_matrix(y_test, xgb_preds)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"97.33% accuracy on the test set! XGBoost is an astoundingly strong ensemble-learning model that uses gradient-boosting algorithms to strengthen the model. Finally, we will try Bayesian-Classification, which, may not work well with this data considering the context. ","metadata":{}},{"cell_type":"code","source":"# ====================================================================\n# Naive Bayesian Classifer: 0.7865\n# ====================================================================\n\n# Fitting Model: Guassian\nnaive_bayes = GaussianNB()\nnaive_bayes.fit(X_train , y_train)\n\n#Predict on test data\nnb_preds = naive_bayes.predict(X_test)\n\n# Test set performance: 0.7516\naccuracy_score(y_test, nb_preds)\nconfusion_matrix(y_test, nb_preds)\n\n\n# Fitting Model: Bernouli\nnaive_bayes = BernoulliNB()\nnaive_bayes.fit(X_train , y_train)\n\n\n#Predict on test data\nnb_preds = naive_bayes.predict(X_test)\n\n# Test set performance: 0.7865\naccuracy_score(y_test, nb_preds)\nconfusion_matrix(y_test, nb_preds)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, the NaiveBayesClassifier did not perform as well as the ensemble methods. The GaussianNB performed the worst due to the lack of normality in our data's distribution.","metadata":{}}]}