{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Learn Tensorflow 2: An easy way\n\n### Objective\nThe key objective of this kernel is \n- To introduce the new programming style for creating a neural network using Keras on TF2\n- One should be capable of starting Deep Learning without going through the conventional way of ML algorithms and landing here eventually. Ref:[Learn ML - ML101, Rank 4500 to ~450 in a day](https://www.kaggle.com/gowrishankarin/learn-ml-ml101-rank-4500-to-450-in-a-day)\n- Instill knowledge on key principles includes\n    - Train, Validation and Test data splitting\n    - A simple mechanism to fill the missing values in the dataset\n    - Bias and Overfit handlers like class weights and initial bias calculation\n    - Handling categorical columns using One Hot Encoding or Embedding principles\n    - Elegant way of creating dataset of tensor slices from pandas dataframes\n    - Build a simple and flat a NN architecture using Keras\n    - Predict the targets via predict functions\n    - Analyse the results using various metrics include accuracy, precision, ROC curve etc\n\nThis is an attempt to make an engineer novice to expert on approach and process of building a neural network for classification problem."},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"from __future__ import absolute_import, division, print_function, unicode_literals\n\nimport numpy as np\nimport pandas as pd\n\nimport tensorflow as tf\nfrom sklearn.model_selection import train_test_split","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Pre-Processing\n\n### Read Data\n- Read the data from the data source using Pandas package.\n- We have 3 files - train.csv, test.csv and sample_submission.csv\n- Train set has 23 feature columns and 60,000 observations\n- Test set has 23 feature columns and 40,000 observations\n\nHint: \n- Test set volume is huge, there is a chance of imbalance target value. Ensure it is handled to avoid bias.\n- Also note, if the data is imbalanced - Right metrics is AUC(area under the curve) and not accuracy."},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"is_local = False\nINPUT_DIR = \"/kaggle/input/cat-in-the-dat-ii/\"\n\nimport tensorflow as tf; \nprint(tf.__version__)\n\nif(is_local):\n    INPUT_DIR = \"../input/\"\n\nimport os\nfor dirname, _, filenames in os.walk(INPUT_DIR):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(INPUT_DIR + \"train.csv\")\ntest_df = pd.read_csv(INPUT_DIR + \"test.csv\")\nsubmission_df = pd.read_csv(INPUT_DIR + \"sample_submission.csv\")\nprint(\"Shape of the train data is \", train_df.shape)\nprint(\"Shape of the test data is \", test_df.shape)\n\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Constants\nLet us initialize the constants \n\n- **Embedding Dimensions:** An embedding a low-dimensional space into which a high dimensional vector is represented. \n    It makes a large space of information into a sparse vectors. \n    Here we are randomly picking 9 dimensions to represent the sparse vector of certain features. More about embeddings below.\n- **Batch Size:** Batch size tells the network, the number of observations to propagate through the networks. The key aspect of the batch size is \n    how quickly a model trains/learns.\n- **Epochs:** Epoch is the number of times a learning algorithms is iterated on the entire training data set. ie Count of every observation in the training dataset\n    involved in the learning process of the model. \n- **Train and Validation Split:** A model is better if it is generalized rather than work well for the given dataset. So the train dataset is split into train and validation data. During the training process, accuracy of the model is measured through validation data metrics rather than train.\n- **Metrics:** We are observing 8 metrics to understand a model. Ref: [Precision, Recall, ROC, AUC - Validation Metrics](https://www.kaggle.com/gowrishankarin/precision-recall-roc-auc-validation-metrics)"},{"metadata":{"trusted":true},"cell_type":"code","source":"EMBEDDING_DIMENSIONS=9\nBATCH_SIZE = 1024\nEPOCHS = 25\nTRAIN_VAL_SPLIT_RATIO = 0.3\n\nMETRICS = [\n    tf.keras.metrics.TruePositives(name='tp'),\n    tf.keras.metrics.FalsePositives(name='fp'),\n    tf.keras.metrics.TrueNegatives(name='tn'),\n    tf.keras.metrics.FalseNegatives(name='fn'),\n    tf.keras.metrics.BinaryAccuracy(name='accuracy'),\n    tf.keras.metrics.Precision(name='precision'),\n    tf.keras.metrics.Recall(name='recall'),\n    tf.keras.metrics.AUC(name='auc'),\n]\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Understanding the Features\nUnderstanding the dataset by doing an **Exploratory Data Analysis and Visualization** brings significant insights for solving the problem. Ref: [Interactive EDA using Plotly](https://www.kaggle.com/gowrishankarin/interactive-eda-using-plotly)  \nIn our dataset, we have 5 types of features\n- Binary Data - Discrete (1, 0) or (True, False), (Yes, No)\n- Nominal Data - Discrete (Male, Female) or (Eye Colors) or (Hair Color)\n- Ordinal Data - Discrete Sequence (Hot, Hotter, Hottest) or (Scale of 1-10)\n- Cyclic Data - Continuous, Numeric (Weekdays) or (Month Dates)\n- Target Data - Our target data is a binary data"},{"metadata":{"trusted":true},"cell_type":"code","source":"COLUMN_TYPES = {\n    'id': 'index',\n    'bin_0': 'binary', 'bin_1': 'binary', 'bin_2': 'binary', 'bin_3': 'binary', \n    'bin_4': 'binary', 'nom_0': 'categorical', 'nom_1': 'categorical',\n    'nom_2': 'categorical', 'nom_3': 'categorical', 'nom_4': 'categorical', \n    'nom_5': 'categorical', 'nom_6': 'categorical', 'nom_7': 'categorical', \n    'nom_8': 'categorical', 'nom_9': 'categorical',\n    'ord_0': 'ordinal', 'ord_1': 'ordinal', 'ord_2': 'ordinal', \n    'ord_3': 'ordinal', 'ord_4': 'ordinal', 'ord_5': 'ordinal', \n    'day': 'cyclic', 'month': 'cyclic',\n    'target': 'target'\n}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Dealing with Missing Data\nThe most time consuming aspect of a dataset is dealing with missing data. We are presenting a simple way to address with this\n- All categorical and binary data are filled with a special value NaN\n- Cyclic data is filled with the median value for simplicity\n\nThere are sophisticated imputation mechanisms available in sklearn packages. Ref: [Fancy Impute](https://github.com/iskandr/fancyimpute), [sklearn.impute](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.impute)"},{"metadata":{"trusted":true},"cell_type":"code","source":"def fill_missing_values(dataframe, ignore_cols=['id', 'target']):\n    feature_cols = [column for column in dataframe.columns if column not in ignore_cols]\n    for a_column in feature_cols:\n        typee = COLUMN_TYPES[a_column]\n        if(typee == 'binary'):\n            dataframe.loc[:, a_column] = dataframe.loc[:, a_column].astype(str).fillna(-9999999)\n        elif(typee == 'numeric'):\n            pass\n        elif(typee == 'categorical'):\n            dataframe.loc[:, a_column] = dataframe.loc[:, a_column].astype(str).fillna(-9999999)\n        elif(typee == 'ordinal'):\n            dataframe.loc[:, a_column] = dataframe.loc[:, a_column].astype(str).fillna(-9999999)\n        elif(typee == 'cyclic'):\n            median_val = np.median(dataframe[a_column].values)\n            if(np.isnan(median_val)):\n                median_val = np.median(dataframe[~np.isnan(dataframe[a_column])][a_column].values)\n            print(a_column, median_val)\n            dataframe.loc[:, a_column] = dataframe.loc[:, a_column].astype(float).fillna(median_val)\n            \n    return dataframe.copy(deep=True)\n\ntrain_df = fill_missing_values(train_df, ignore_cols=['id', 'target'])\ntest_df = fill_missing_values(test_df, ignore_cols=['id'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Bias and Class Weights\n- Our goal is to avoid overfitting and generalization of the model we create.\n- As a first measure we split the data into train and validation set\n- Further we shall intervene by calculating initial bias and class weights\n- How to find imbalance in the target value\n    - Find the ratio of positive and negatives in the target distribution\n    - In our dataset, we have 18.72 percent are positives and rest are negatives\n    - We assume a similar behavior in the test dataset as well\n    - However there is no guarantee that it will be true\n    \n### Initial Bias\nSince the dataset is imbalanced, our output layer to reflect the bias. Bias can be calculated as follows\n\n\\begin{equation*}\np_0 = \\frac{pos}{pos + neg} = \\frac{1}{1 + b^0} \\\\\nb_0 = - log_e(\\frac{1}{p_0} - 1) \\\\\nb_0 = log_e(\\frac{pos}{neg})\n\\end{equation*}\n\nWith this inialization, Initial loss/cost function or **cross entropy**\n\n<div align=\"center\" style=\"font-size: 18px\">Loss/Cost Function or Cross Entropy</div><br>\n\n\n\\begin{equation*}\n- p_0 log(p_0) - (1 - p_0) log(1 - p_0)\n\\end{equation*}"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_initial_bias(df, col_name='target'):\n    neg, pos = np.bincount(df[col_name])\n    total = neg + pos\n    print('Examples:\\n    Total: {}\\n    Positive: {} ({:.2f}% of total)\\n'.format(\n        total, pos, 100 * pos / total))\n\n    initial_bias = np.log([pos/neg])\n    \n    return initial_bias\n\n\n\ninitial_bias = get_initial_bias(train_df)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Class Weights\n- The idea is to have the model heavily weight the few positive option available for training. \n- This is done using \"class_weight\" param of the model to pay more attention towards under represented class. \n- Incorporation of class weights is made aware through the cost function\n- The loss/cost function discussed earlier transform as follows\n\n<div align=\"center\" style=\"font-size: 18px\">Weighted Cost Entropy</div><br>\n\n\\begin{equation*}\n- w_0 p_0 log(p_0) - w1 (1 - p_0) log(1 - p_0)\n\\end{equation*}"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_class_weights(df, col_name='target'):\n    neg, pos = np.bincount(df[col_name])\n    weight_for_0 = (1 / neg) * (neg + pos) / 2.0\n    weight_for_1 = (1 / pos) * (neg + pos) / 2.0\n\n    class_weight = {\n        0: weight_for_0,\n        1: weight_for_1\n    }\n\n    print(\"Class 0: \", weight_for_0, \"Weightage\")\n    print(\"Class 1: \", weight_for_1, \"Weightage\")\n    \n    return class_weight\n\nclass_weight = get_class_weights(train_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Stratified Split of Train and Validation Data\nTo avoid Sampling bias, we shall split the data using Stratification. Stratified split ensure the imbalance ratio is maintained in train and validation dataset.\n\nWhat is Sampling Bias?  \nWhen some members of the population have lower sampling probability than others, a sampling bias occurs. \nIt results in samples favoring a particular group of the population and the model end up with bias."},{"metadata":{"trusted":true},"cell_type":"code","source":"### Stratified Split\n\nfrom sklearn.model_selection import StratifiedShuffleSplit\n\ndef split_train_validation_data(df, col_name='target', stratify=True, test_size=0.3):\n    train = None\n    val = None\n    \n    if(stratify):\n        \n\n        sss = StratifiedShuffleSplit(n_splits=2, test_size=0.2, random_state=21)\n        sss.get_n_splits(df, df.target)\n\n        splits = sss.split(df, df.target) \n        \n        indices = []\n        for train_index, test_index in splits:\n            indices.append({\n                'train': train_index,\n                'test': test_index\n            })\n\n        train = df.iloc[indices[0]['train']]\n        val = df.iloc[indices[0]['test']]\n        \n    else:\n        train, val = train_test_split(train_df, test_size=test_size)\n    return train, val\n\ntrain, val = split_train_validation_data(train_df, test_size=TRAIN_VAL_SPLIT_RATIO)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"get_initial_bias(train)\nget_initial_bias(val)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Tensorflow 2 - Feature Columns\n- Tensorflow feature columns are the most awesome capability released recently\n- Feature columns are the bridge between raw data and the neural network operated data\n- What kind of data a NN will operate on - Numbers, mostly floating point numbers\n- How to translate the categorical columns like the few we discussed earlier. eg Color of eye, Gender, All retail shops of a market etc\n- The rich nature of feature columns enable one to transform diverse range of raw format to NN operatable format\n- Naturally, the output of the feature column becomes the input to the model.\n\n\n## One Hot Encoding\n- For Categorical features, transformation of non-number data to number data is the goal\n- Categorical variables are nominal, The process of transforming into binary value is One Hot Encoding\n- The raw data in long format is converted into wide format\n- It is nothing but, binarization of a categorical values\n\n|Company   \t|Rank   \t|Price   \t|   \t|   \t|\n|:-:\t|:-:\t|:-:\t|:-:\t|:-:\t|\n|VW   \t|4   \t|   \t|100   \t|   \t|\n|Honda   \t|2   \t|   \t|10000   \t|   \t|\n|Ford   \t|3   \t|   \t|1000   \t|   \t|\n|Tesla   \t|1   \t|   \t|100000   \t|   \t|\n\n<div align=\"center\" style=\"font-size: 30px\">TO</div>\n\n|Company   \t|Rank 1  \t|Rank 2   \t|Rank 3   \t|Rank 4   \t|Price   \t|\n|:-:\t|:-:\t|:-:\t|:-:\t|:-:\t|:-:\t|\n|VW   \t|0   \t|0   \t|0   \t|1   \t|100   \t|\n|Honda   \t|0   \t|1   \t|0   \t|0   \t|10000   \t|\n|Ford   \t|0   \t|0   \t|1   \t|0   \t|1000   \t|\n|Tesla   \t|1   \t|0   \t|0   \t|0   \t|100000   \t|\n\n"},{"metadata":{},"cell_type":"markdown","source":"## Embeddings\n- Another popular scheme used for transformation of categorical variables into numbers is embeddings.\n- This scheme represents discrete values as continuous vectors\n- This process for machine translation yields significan improvement to the model performance\n- In an NN, embeddings are low-dimensional continuous vectors\n- The reduction of dimensionality of a categorical variable and meaningful representation in the transformed space is the goal\n- Dimensionality reduction addresses the high cardinality of the categorical value\n- Embeddings place the similar things closer in the embedding space\n\nExamples  \n1. Books on Data Science: Let us say there are 20000 books covering the wide gamut of all data science problems. Actual number of dimension here is 2000. By reducing the dimensonality of the dataset, From 20000 to 200 - We can represent the whole dataset.\n2. Retail Outlets of a Country: Let us say there are 1.2 million retail outlets present in a country. 1000 odd number can represent the characteristics can represent the attributes of every outlet.\n\n**Representation of Embedding with 2 vectors**\n<pre><code>\nshops = [\"Sainsbury\", \"Tesco\", \"Reliance\", \"Lulu\", \"Costco\"]  \nembeddings = [\n    [ 0.11, 0.52],  \n    [0.32, 0.56], \n    [-0.56, -0.91], \n    [-0.21, 0.21]\n]\n</code></pre>                \nHere we reduced the dimensionality to 2 from 5 to represent the property of a variable"},{"metadata":{},"cell_type":"markdown","source":"### Numeric Columns\nNumeric columns are pretty straight forword where the raw data is already with numeric value. Feature columns of TF2's numeric_column api comes handy to address the problem\n"},{"metadata":{},"cell_type":"markdown","source":"### Random Decisions\n- If the variation is less than 100 on the categorical columns, One Hot Encoding taken in this code\n- Beyond 100 variation, Embeddings are preferred"},{"metadata":{"trusted":true},"cell_type":"code","source":"def handle_feature_columns(df, columns_to_remove=['id', 'target'], all_categorical_as_ohe=True):\n    \n    def demo(feature_column):\n        feature_layer = tf.keras.layers.DenseFeatures(feature_column)\n    \n    def one_hot_encode(col_name, unique_values):\n        from_vocab = tf.feature_column.categorical_column_with_vocabulary_list(\n            col_name, unique_values\n        )\n        ohe = tf.feature_column.indicator_column(from_vocab)\n        data.append(ohe)\n        demo(ohe)\n    \n    def embedd(col_name, unique_values):\n        from_vocab = tf.feature_column.categorical_column_with_vocabulary_list(\n            col_name, unique_values\n        )\n        embeddings = tf.feature_column.embedding_column(from_vocab, dimension=EMBEDDING_DIMENSIONS)\n        data.append(embeddings)\n        demo(embeddings)\n        \n    def numeric(col_name, unique_values):\n        from_numeric = tf.feature_column.numeric_column(\n            col_name, dtype=tf.float32\n        )\n        data.append(from_numeric)\n        demo(from_numeric)\n    \n    dataframe = df.copy()\n    for pop_col in columns_to_remove:\n        dataframe.pop(pop_col)\n    data = []\n    \n    for a_column in dataframe.columns:\n        typee = COLUMN_TYPES[a_column]\n        nunique = dataframe[a_column].nunique()\n        unique_values = dataframe[a_column].unique()\n        print('Column :', a_column, nunique, unique_values[:10])                \n        if(typee == 'binary'):\n            one_hot_encode(a_column, unique_values)\n        elif(typee == 'cyclic'):\n            numeric(a_column, unique_values)\n            \n        else:\n            if(all_categorical_as_ohe):\n                one_hot_encode(a_column, unique_values)\n            else:\n                if(typee == 'categorical'):\n                    if(nunique < 100):\n                        one_hot_encode(a_column, unique_values)\n                    else:\n                        embedd(a_column, unique_values)\n                elif(typee == 'ordinal'):\n                    embedd(a_column, unique_values)\n            \n    return data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"feature_columns = handle_feature_columns(train, all_categorical_as_ohe=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Data Preparation\nTensors are the centrol data types for a Tensorflow NN framework. Crafting a tensors for the feature columns ensures the raw data translation into model acceptable one.\nSimply speaking, a tensor is a multi-dimensional numerical array. To get full picture of a tensor, we have to understand few key words\n- **Rank:** Number of dimension of a tensor is its Rank.\n- **Shape:** Shape of a tenser is its count of rows and columns.\n    - A rank zero tensor is a single number or it is a **scalar**\n    - A rank one tensor is an array of numbers or it is called as **vector**\n    - A rank two tensor is a matrix of numers or it has rows and columns\n- **Tensor Slice:** A tensor slice is a portion of data from the population based the batch size given"},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train = train.pop('target')\ny_val = val.pop('target')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def df_to_dataset(dataframe, y, shuffle=True, batch_size=32, is_test_data=False):\n    dataframe = dataframe.copy()\n    ds = None\n    if(is_test_data):\n        ds = tf.data.Dataset.from_tensor_slices(dict(dataframe))\n    else:\n        \n        ds = tf.data.Dataset.from_tensor_slices((dict(dataframe), y))\n        if(shuffle):\n            ds = ds.shuffle(buffer_size=len(dataframe))\n    ds = ds.batch(batch_size)\n    return ds\n\ntrain_ds = df_to_dataset(train, y_train, shuffle=False, batch_size=BATCH_SIZE)\nval_ds = df_to_dataset(val, y_val, shuffle=False, batch_size=BATCH_SIZE)\ntest_ds = df_to_dataset(test_df, None, shuffle=False, batch_size=BATCH_SIZE, is_test_data=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model, Training and Prediction\nOnce the dataset is ready, a model is created, trained and evaluated.\nThe current model has 6 unique items stacked one after another... {WIP}\n\n- **Sequential Model:** \n- **Feature Layer:**\n- **Dense Layer:**\n- **Batch Normalization:**\n- **Dropouts:**\n- **Activation Function - Relu:**\n- **Activation Function - Sigmoid:**\n- **Loss Function: Binary Cross Entropy:**\n- **From Logits:**\n- **Optimizer - Adam:**\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_silly_model_2(feature_layer, initial_bias=None):\n    bias = None\n    if(initial_bias):\n        bias = tf.keras.initializers.Constant(initial_bias)\n        \n    model = tf.keras.Sequential([\n        feature_layer,\n        tf.keras.layers.Dense(512, activation='relu'),\n        tf.keras.layers.Dropout(0.3),\n        tf.keras.layers.BatchNormalization(),\n        tf.keras.layers.Dense(256, activation='relu'),\n        tf.keras.layers.Dropout(0.3),\n        tf.keras.layers.BatchNormalization(),\n        tf.keras.layers.Dense(128, activation='relu'),\n        tf.keras.layers.Dropout(0.3),\n        tf.keras.layers.BatchNormalization(),\n        tf.keras.layers.Dense(1, activation='sigmoid', bias_initializer=bias)\n    ])\n    model.compile(\n        optimizer='adam',\n        loss=tf.keras.losses.BinaryCrossentropy(from_logits=False),\n        metrics=METRICS\n    )\n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Fit and Run the Model\nTo run the model, we have 14 items to ensure {WIP}\n- **Callbacks:**\n- **Early Stopping:**\n- **Reduce Learning on Plateau**\n- **Accuracy Monitoring:**\n- **Patience:**\n- **Baseline:**\n- **Restore Best Weights:**\n- **History:**"},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndef run(\n    train_data, val_data, feature_columns, \n    epochs=EPOCHS, es=False, rlr=False, \n    class_weights=None, initial_bias=None\n):\n    feature_layer = tf.keras.layers.DenseFeatures(feature_columns)\n    model = create_silly_model_2(feature_layer, initial_bias)\n\n    callbacks = []\n    if(es):\n        callbacks.append(\n            tf.keras.callbacks.EarlyStopping(\n                monitor='val_auc', min_delta=0.00001, patience=5, \n                mode='auto', verbose=1, baseline=None, restore_best_weights=True\n            )\n        )\n    if(rlr):\n        callbacks.append(\n            tf.keras.callbacks.ReduceLROnPlateau(\n                monitor='val_auc', factor=0.5, patience=3, \n                min_lr=3e-6, mode='auto', verbose=1\n            )\n        )\n\n    history = model.fit(\n        train_ds, \n        validation_data=val_ds, \n        epochs=epochs, \n        callbacks=callbacks,\n        class_weight=class_weights\n    )\n    \n    return model, history","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"model, history = run(\n    train_ds, val_ds, feature_columns, \n    epochs=EPOCHS, es=False, rlr=False, \n    class_weights=None, initial_bias=None\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions = model.predict(test_ds)\nsubmit = pd.DataFrame()\nsubmit[\"id\"] = test_df[\"id\"]\nsubmit['target'] = predictions\nsubmit.to_csv('submission_dl_stratify.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model Performance and Metrics\n- A model is rated through various metrics and the list we saw in the data preparation part. These metrics gives us an opportunity get insight over the model's performance.\n- The primary goal of building a model is to avoid overfit over the training data. Achieving a high accuracy on train data almost always result in poor performance over the real data\n- That is nothing but the neural network failed to learn the critical patterns hidden deeply inside the dataset.\n- Training for longer time will result in overfit. That is control over the number of epochs.\n\nIn this section we shall plot the following metrics across train and validation datasets\n- Loss\n- AUC, Area Under the Curve\n- Precision\n- Recall\n\nFurther we shall draw the ROC(Receiver Operating Curve) and observe the model performance from the training history."},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib as mpl\nimport matplotlib.pyplot as plt\nmpl.rcParams['figure.figsize'] = (15, 10)\ncolors = plt.rcParams['axes.prop_cycle'].by_key()['color']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_metrics(history):\n    metrics = ['loss', 'auc', 'precision', 'recall']\n    \n    for n, metric in enumerate(metrics):\n        name = metric.replace(\"_\", \" \").capitalize()\n        plt.subplot(2, 2, n+1)\n        plt.plot(\n            history.epoch, \n            history.history[metric], \n            color=colors[0], \n            label='Train'\n        )\n        plt.plot(\n            history.epoch, \n            history.history['val_' + metric], \n            color=colors[0], \n            linestyle=\"--\", \n            label='val'\n        )\n        plt.title(metric.upper())\n        plt.xlabel('Epoch')\n        plt.ylabel(name)\n        if(metric == 'loss'):\n            plt.ylim([0, plt.ylim()[1]])\n        elif(metric == 'auc'):\n            plt.ylim([0, 1])\n        else:\n            plt.ylim([0, 1])\n        plt.legend()\n        \nplot_metrics(history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Observations\n**Loss:**  \n    Loss of validation dataset to go down, inference: overfitting  \n**Area Under the Curve(AUC):**  \n    AUC of validation dataset to be more than train set, inference: overfitting  \n**Precision:**   \n    Precision of Validation and Train to be similar, inference: overfitting  \n**Recall:**   \n    Recall progression validation set to be along or above the train set: overfitting  "},{"metadata":{},"cell_type":"markdown","source":"## ROC - Receiver Operating Characteristics\nFor a binary classification problems, a ROC curve illustrates the ability of the model to predict the actual.\n- It is plotted between True Positive Rate(TPR) and False Positive Rate(FPR)\n- TPR is probability of predicting rightly\n- FPR is the probability of a false prediction\n- ROC's goal is to play as a tool to select optimal models"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_predictions = model.predict(train_ds)\nval_predictions = model.predict(val_ds)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn import metrics\n\ndef roc(name, labels, predictions, **kwargs):\n    fp, tp, _ = metrics.roc_curve(labels, predictions)\n    \n    plt.plot(100*fp, 100*tp, label=name, linewidth=2, **kwargs)\n    plt.xlabel('False Positives [%]')\n    plt.ylabel('True Positives [%]')\n    plt.xlim([-0.5, 110])\n    plt.ylim([1, 110])\n    plt.grid(True)\n    ax = plt.gca()\n    ax.set_aspect('equal')\n    plt.legend()\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"roc('Train', y_train, train_predictions, color=colors[0])\nroc('Validate', y_val, val_predictions, color=colors[0], linestyle='--')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Intervention 1: Early Stopping and Reduce Learning Rate on Plateau\nLet us try to fix the overfitting problem by\n- Early stop the training and\n- Reduce learning rate when a plateau is encountered"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"model, history = run(\n    train_ds, val_ds, feature_columns, \n    epochs=EPOCHS, es=True, rlr=True, \n    class_weights=None, initial_bias=None\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Plot Metrics: Loss vs AUC vs Precision vs Recall"},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_metrics(history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Observations\n**Loss:**  \n    Loss of validation dataset got better compared to earlier, inference: overfitting is addressed to some extent  \n**Area Under the Curve(AUC):**  \n    AUC of validation dataset is increased compared to earlier, inference: overfitting is addressed to some extent  \n**Precision:**   \n    No significant change to precision, inference: overfitting  \n**Recall:**   \n    Recall of train and validation set are overalapping: overfitting  "},{"metadata":{"trusted":true},"cell_type":"code","source":"train_predictions_es_rlr = model.predict(train_ds)\nval_predictions_es_rlr = model.predict(val_ds)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### ROC comparison\nLet us compare the baseline model results with the model having intervention of Early Stopping and RLR"},{"metadata":{"trusted":true},"cell_type":"code","source":"\nroc('Train Baseline', y_train, train_predictions, color=colors[0])\nroc('Validate Baseline', y_val, val_predictions, color=colors[0], linestyle='--')\nroc('Train [ES, RLR]', y_train, train_predictions_es_rlr, color=colors[1])\nroc('Validate [ES, RLR]', y_val, val_predictions_es_rlr, color=colors[1], linestyle='--')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Intervention 2: Class Weights and Initial Bias\nLet us try to fix further the overfitting problem by\n- Incorporating Class Weights and\n- Initial Bias"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"model, history = run(\n    train_ds, val_ds, feature_columns, \n    epochs=EPOCHS, es=True, rlr=True, \n    class_weights=class_weight, initial_bias=initial_bias\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_metrics(history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Observations\n**Loss:**  \n    Loss increased compared to earlier, inference: overfitting, model degraded  \n**Area Under the Curve(AUC):**  \n    AUC of previous and current intervention remain same, inference: no improvement  \n**Precision:**   \n    Precision decreased from earlier, inference: model degraded  \n**Recall:**   \n    Recall increased: model degraded  "},{"metadata":{"trusted":true},"cell_type":"code","source":"train_predictions_bias_cws = model.predict(train_ds)\nval_predictions_bias_cws = model.predict(val_ds)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"roc('Train Baseline', y_train, train_predictions, color=colors[0])\nroc('Validate Baseline', y_val, val_predictions, color=colors[0], linestyle='--')\nroc('Train [ES, RLR]', y_train, train_predictions_es_rlr, color=colors[1])\nroc('Validate [ES, RLR]', y_val, val_predictions_es_rlr, color=colors[1], linestyle='--')\nroc('Train [ES, RLR, BIAS, IniWts]', y_train, train_predictions_bias_cws, color=colors[2])\nroc('Validate [ES, RLR, BIAS, IniWts]', y_val, val_predictions_bias_cws, color=colors[2], linestyle='--')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Inference of Interventions\nIntervention 1 improved the model but intervention 2 did not make any significant change to the model performance. "},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions = model.predict(test_ds)\nsubmit = pd.DataFrame()\nsubmit[\"id\"] = test_df[\"id\"]\nsubmit['target'] = predictions\nsubmit.to_csv('submission_dl_final.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.5"}},"nbformat":4,"nbformat_minor":4}