{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Cat Challenge - A Workflow Guide: PCA + Logistic Regression + Wide and Deep Neural Networks\n### How can we encode different types of categorical features?\n\n#### Guilherme Vieira Dantas\n\n![GentleCat](https://encrypted-tbn0.gstatic.com/images?q=tbn%3AANd9GcTSFF8K1-XmiGDkB7i_eIqxD_6ImLH4BXMsLMhZNAbiDeqR9oO0)\n\nThe aim of this notebook is to show the classical steps in a Machine Learning project that depends on a dataset characterized by many different types of categorical features. We will follow the conventional data science approach:\n\n1. Data reading + Exploration / Transformation\n2. Dimentionality Reduction\n3. Model Selection + Validation\n4. Stacking Individual Models\n5. Conclusions\n\nI will be direct and I will also try to show different techniques that can be applied here, without losing time or lines of code. Let's begin!"},{"metadata":{},"cell_type":"markdown","source":"# 0. Importing the Libraries\n\nHere we can take a look at all the libraries and resources that will be necessary to our project. It's interesting to aways start by this step to create a code with more quality and maintenability:"},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport category_encoders as ce\nimport seaborn as sns\n\nimport tensorflow as tf\nimport tensorflow.keras.backend as K\n\nfrom sklearn.base import BaseEstimator, TransformerMixin\nfrom sklearn.preprocessing import OneHotEncoder, StandardScaler\nfrom sklearn.impute import SimpleImputer\nfrom sklearn.pipeline import Pipeline \nfrom sklearn.compose import ColumnTransformer\nfrom sklearn.decomposition import PCA\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.model_selection import GridSearchCV, StratifiedKFold\nfrom sklearn.metrics import roc_auc_score\n\nfrom keras.layers import Dense, Dropout, Concatenate, Input, BatchNormalization\nfrom keras.models import Sequential\nfrom keras.callbacks import EarlyStopping, ReduceLROnPlateau\nfrom keras.models import Model\nfrom keras.optimizers import RMSprop, Adam\nfrom keras.utils import plot_model\n\nfrom category_encoders import TargetEncoder\nfrom plotnine import *\n\nimport os\nfor dirname, _, filenames in os.walk('..'):\n    for filename in filenames:\n       print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 1. Reading and Exploring / Transforming the Data\n\nFinally, let's start by taking a look at what we have:"},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nplt.rcParams['figure.figsize'] = [15, 6]\nplt.rc('xtick', labelsize=13) \nplt.rc('ytick', labelsize=13) \n\ninput_dir = '/kaggle/input/cat-in-the-dat-ii/'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train_raw = pd.read_csv(input_dir + 'train.csv')\ndf_test_raw = pd.read_csv(input_dir + 'test.csv')\n\ndf_train_raw.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Transforming some binary features to zeros and ones:"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train_raw_transformed = df_train_raw.\\\n    assign(bin_3 = df_train_raw['bin_3'].map(lambda X: 0 if (X == 'F') else\\\n                                             (1 if (X == 'T') else np.nan))).\\\n    assign(bin_4 = df_train_raw['bin_4'].map(lambda X: 0 if (X == 'N') else\\\n                                             (1 if (X == 'Y') else np.nan)))\n\ndf_train_raw_transformed.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test_raw_transformed = df_test_raw.\\\n    assign(bin_3 = df_test_raw['bin_3'].map(lambda X: 0 if (X == 'F') else\\\n                                             (1 if (X == 'T') else np.nan))).\\\n    assign(bin_4 = df_test_raw['bin_4'].map(lambda X: 0 if (X == 'N') else\\\n                                             (1 if (X == 'Y') else np.nan)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"ORD_0:\")\nprint(df_train_raw_transformed['ord_0'].unique())\nprint(\"---\\nORD1:\")\nprint(df_train_raw_transformed['ord_1'].unique())\nprint(\"---\\nORD2:\")\nprint(df_train_raw_transformed['ord_2'].unique())\nprint(\"---\\nORD3:\")\nprint(df_train_raw_transformed['ord_3'].unique())\nprint(\"---\\nORD4:\")\nprint(df_train_raw_transformed['ord_4'].unique())\nprint(\"---\\nORD5:\")\nprint(df_train_raw_transformed['ord_5'].unique())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"OK. We have not many columns in our problem. But we do have some columns with many different values, which can be a problem. Why? Well, when we transform string features to binary features we will create a number of dimentions equal to the number of different features less $1$ (we can subtract one because the case where all the columns are equal to zero corresponds to one of the strings). \n\nThis kind of encoding is called \"Binary Encoding\" and when we get many dimentions thanks to the great number of different string values we get the problem known as the \"curse of dimentionality\" and the number of features with a given dummy column equal to one decreases exponentially:\n\n![CurseOfDimentionality](https://www.elasticfeed.com/wp-content/uploads/3e5fca2cb938bbc9f5a1cae43bac0944.jpg)\n\nThat's why it's interesting to find ordinal features in our dataset: we can just number them following some \"sorting criteria\". This is specially good when we are working with trees but to get assured that this choice is suitable to a model such like a Logistic Regression, we must have values that follow a linear relationship with the probability of getting a one in the output. It makes sense to say that since we apply a logistic function in a linear combination of the inputs to get the output of the model:\n\n![LogisticRegression](https://d2bwk5eec7cz2z.cloudfront.net/2018/09/9.png)\n\nAnyway, we will use, for now, the simplest form of encoding, we will:\n* Keep the binary features just like they are\n* Label with $1$, $2$, $3$,... all the features that seem to follow an ordering rule\n\nWe will transform the categorical features in the next section."},{"metadata":{"trusted":true},"cell_type":"code","source":"def apply_dict(X, dict_in):\n    try:\n        out = dict_in[X]\n    except:\n        out = X\n    return out\n\ndict_ord_1 = dict(Novice=1, Contributor=2, Expert=3, \n                  Master=4, Grandmaster=5)\n\ndict_ord_2 = {'Freezing': 1, 'Cold': 2, 'Warm': 3, \n              'Hot': 4, 'Boiling Hot': 5, 'Lava Hot': 6}\n\ndict_ord_3 = dict_ord_4 = dict()\nfor i in range(ord('a'), (ord('z') + 1)):\n    dict_ord_3[chr(i)] = dict_ord_4[chr(i + ord('A') - ord('a'))] = i - ord('a') + 1\n    \ndf_train = df_train_raw_transformed.\\\n    assign(ord_1 = df_train_raw_transformed['ord_1'].map(lambda X: apply_dict(X, dict_ord_1))).\\\n    assign(ord_2 = df_train_raw_transformed['ord_2'].map(lambda X: apply_dict(X, dict_ord_2))).\\\n    assign(ord_3 = df_train_raw_transformed['ord_3'].map(lambda X: apply_dict(X, dict_ord_3))).\\\n    assign(ord_4 = df_train_raw_transformed['ord_4'].map(lambda X: apply_dict(X, dict_ord_4)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test = df_test_raw_transformed.\\\n    assign(ord_1 = df_test_raw_transformed['ord_1'].map(lambda X: apply_dict(X, dict_ord_1))).\\\n    assign(ord_2 = df_test_raw_transformed['ord_2'].map(lambda X: apply_dict(X, dict_ord_2))).\\\n    assign(ord_3 = df_test_raw_transformed['ord_3'].map(lambda X: apply_dict(X, dict_ord_3))).\\\n    assign(ord_4 = df_test_raw_transformed['ord_4'].map(lambda X: apply_dict(X, dict_ord_4)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Checking the results:"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"How many samples we have to train?"},{"metadata":{"trusted":true},"cell_type":"code","source":"len(list(df_train_raw_transformed.index))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Yes, that's a huge number. We will probably face some difficulties to finetune our models...let's check graphically how many different features we have per column:"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_count_per_col = pd.DataFrame(df_train.nunique())\ndf_count_per_col.columns = ['Values']\ndf_count_per_col.index.name = 'Feature'\ndf_count_per_col.reset_index(inplace=True)\n\nggplot(df_count_per_col[df_count_per_col['Feature'] != 'id'], \n       aes(x = 'Feature', y = 'Values', fill = 'Feature')) + geom_bar(stat = 'identity', color = 'black') +\\\n    theme(axis_text_x = element_text(angle = 90, hjust = 1), legend_position = 'none') +\\\n    ggtitle('Different Features per Column') + ylab('Count')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Another important information to explore is the missing data rate: how many \"NAN's\" do we have per column?"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_missing_col = pd.DataFrame(dict(PercMissing = df_train.isnull().sum() / \n                                   len(df_train.index))).reset_index()\ndf_missing_col.loc[df_missing_col['PercMissing'] > 0, :]\n\nggplot(df_missing_col[df_missing_col['Feature'] != 'id'], \n       aes(x = 'Feature', y = 'PercMissing', fill = 'Feature')) + geom_bar(stat = 'identity', color = 'black') +\\\n    theme(axis_text_x = element_text(angle = 90, hjust = 1), legend_position = 'none') +\\\n    ggtitle('Different Features per Column') + ylab('Count')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It would be nice to follow a similar approach with the rows and check if we have a row with many missing values. If so, we can just drop them out without the risk of losing many informations.\n\nIt's not a good idea to try to do that with 600.000 different index values. So, let's just check the maximum missing data rate along all the rows:"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_missing_row = pd.DataFrame(dict(PercMissing = df_train.isnull().sum(axis=1) / \n                                   len(df_train.index))).reset_index()\n\ndf_missing_row.columns = ['Index', 'PercMissing']\nprint(str(100 * max(df_missing_row['PercMissing'])) + ' %')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It's really small, so, we will not drop out any row and we are ready to start transforming our dataset. Also, since we have no high missing NA values per columns or rows, we don't need to spend a lot of time in the missing data imputation step.\n\nLet's start out data transformation step by separing the variables in different classifications:"},{"metadata":{"trusted":true},"cell_type":"code","source":"binary_features = ['bin_' + str(i) for i in range(0, 5)]\nnominal_features_low_count = ['nom_' + str(i) for i in range(0, 5)]\nnominal_features_high_count = ['nom_' + str(i) for i in range(5, 10)]\nordinal_features_low_count = ['ord_' + str(i) for i in range(0, 5)]\nordinal_features_high_count = ['ord_5']\ndate_features = ['day', 'month']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And we will use the \"SimpleImputer\" class to fill the missing data. But I will just \"correct\" a problem in its source code: [Stack Overflow Question](https://datascience.stackexchange.com/questions/66034/sklearn-simpleimputer-too-slow-for-categorical-data-represented-as-string-values). Imputing the median of the values is slow and using the \"constant\" imputation method for each column with the most frequent value as constant can speed up the code considerably!\n\nThanks to the techniques of object orientation, we can inherit the methods of the original function and create a new \"most_frequent\" imputation method where we use, in each column, the Stack Overflow's purposed solution:"},{"metadata":{"trusted":true},"cell_type":"code","source":"class SimpleImputerCorrected(BaseEstimator, TransformerMixin):\n    \n    def __init__(self, strategy='most_frequent', verbose=False):\n        \n        self.strategy = strategy\n        self.preprocessor = None\n        self.verbose = verbose\n\n    def fit(self, X, y=None):\n        \n        if self.strategy == 'most_frequent':\n            \n            col_list = list(X.columns)\n            col_transformers = list()\n            col_mode = X.mode(axis=0).iloc[0]\n        \n            for curr_col in col_list:\n                curr_transformer_name = 'T_' + curr_col\n                curr_imputer = SimpleImputer(strategy='constant', \n                                             fill_value=col_mode[curr_col])\n                col_transformers.append((curr_transformer_name, curr_imputer, \n                                         [curr_col]))\n            \n            self.preprocessor = ColumnTransformer(transformers=col_transformers, verbose=self.verbose)\n            \n        else:\n            self.preprocessor = SimpleImputer(strategy=self.strategy)\n            \n        self.preprocessor.fit(X)\n        return self\n\n    def transform(self, X):\n        return self.preprocessor.transform(X)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"All the steps described until now will be inserted in a Scikit's Pipeline! It's a functionality that allow us to define a sequence of transformation and modeling steps as a class that can be used in different datasets when, for example, our model is running in production. Let's see some Pipeline examples in the next figure:\n\n![Pipelines](https://www.researchgate.net/publication/334565019/figure/fig1/AS:782364141690881@1563541555043/The-Auto-Sklearn-pipeline-12-contains-three-main-building-blocks-a-Data.png)\n\nOur first pipelines will be not so complicated and the code is self-explainable:"},{"metadata":{},"cell_type":"markdown","source":"What can we do with the ordinal and categorical features with many different values? I will apply the same threatment: the **Target Encoding**. It consists in transforming each category into the averages of the targets obtained for this category. \n\nI am considering that our ordinal feature with a high number of different values (ord_5) is categorical: we have $2$ random letters together, some of them in uppercase, some of them in lowercase characters and I am not really sure about how could I possibly transform it to an ordinal number that makes sense (should we sum the ASCII values? Should we sum the ASCII values after transforming the characters to their lowercase versions?)"},{"metadata":{"trusted":true},"cell_type":"code","source":"pass_features = binary_features + ordinal_features_low_count\none_hot_features = nominal_features_low_count\navg_features = nominal_features_high_count + ordinal_features_high_count\n\npass_pipeline = Pipeline(steps = [\n    ('imputation', SimpleImputerCorrected(strategy='most_frequent')),\n    ('scalling', StandardScaler())\n], verbose = True)\n\none_hot_pipeline  = Pipeline(steps = [\n    ('imputation', SimpleImputerCorrected(strategy='most_frequent')),\n    ('encoding', OneHotEncoder(sparse = False))\n], verbose = True)\n\navg_pipeline = Pipeline(steps = [\n    ('imputation', SimpleImputerCorrected(strategy='most_frequent')),\n    ('encoding', TargetEncoder()),\n    ('scalling', StandardScaler())\n], verbose = True)\n\nencoder = ColumnTransformer(\n    transformers=[\n        ('pass_pipeline', pass_pipeline, pass_features),\n        ('one_hot_pipeline', one_hot_pipeline, one_hot_features),\n        ('avg_pipeline', avg_pipeline, avg_features)\n], verbose = True, sparse_threshold=0)\n\nprint(one_hot_features)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train.iloc[1:5, 1:-2]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"array_train = encoder.fit_transform(df_train.iloc[:, 1:-2], df_train['target'])\nprint(array_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now, we will use in the next section the Principal Component Analysis technique to remove redundant dimentions of our variables.\n\n# 2. Dimentionality Reduction With Principal Component Analysis\n## 2.1. Linear Transformations\n\nThe principal component analysis is, itself, a Scikit-Learn model. It's basically a simple linear transformation that is applied to out dataset. When I say that I am applying a linear transformation over a vector, I'm saying that I'm multiplying this vector by a matrix:\n\n$$V_{Transformed} = A.V_{Before}$$\n\nWhere $V_{Before}$ is a single vector, with a number of dimentions equal to the number of columns of our original dataframe and we have as many vectors as rows in our dataset.\n\nInitially, we apply a $N \\times N$ matrix over our vectors, where $N$ is the number of columns. It means that we are using all the output dimentions of the vector.\n\n![LinearTransformation](http://duriansoftware.com/joe/media/gl3-linear-transformation-matrices-01.png)\n\nIf we apply the same linear transformation over different vectors, as we can see in the figure above, the angle between the two vectors will change and some components will be \"stretched\" ou \"shrinked\" as we can see in the figure above.\n\nIn the Principal Component Analysis we don't apply any linear transformation over our dataset. We will change the angle of each pair of vectors to $90$ degrees and obtain independent components (i.e vectors with no mutual projections over each other):\n\n![PCA](https://i2.wp.com/www.sportscidata.com/wp-content/uploads/2019/08/Principal_Component_Analysis_print.png?fit=1024%2C683&ssl=1)\n\n** This transformation is applied over the centered vectors, as we will explain in the next section **"},{"metadata":{},"cell_type":"markdown","source":"## 2.2. Variances and Features Importance\n\nSuppose we have two non orthogonal vectors in a bidimentional space and we want to calculate the sum of the variance along the $2$ dimensions. Then we have:\n\n$$Var(X_{12}) = Var(X_1) + Var(X_2) + 2.Cov(X_1, X_2)$$\n\nAfter applying a PCA over this pair, we obtain $2$ orthogonal vectors and the total variance can be calculated without the covariance term:\n\n$$Var(X^*_{12}) = Var(X^*_1) + Var(X^*_2)$$\n\nWhere:\n$$ X^*_1 = \\Lambda . (X_1 - \\mu_{1,2}) $$\n$$ X^*_2 = \\Lambda . (X_2 - \\mu_{1,2}) $$\n$$ X^*_1 \\perp X^*_2 \\rightarrow X^*_1 \\bullet X^*_2 = 0 $$\n$$ \\mu_{1,2} = \\frac{X_1 + X_2}{2} $$\n\nAnd $\\Lambda$ is our linear transformation PCA matrix. When we lose the covariance term and subtract the mean of each vector, something magic happens:\n\n$$ Var({X^*_k}) = \\mathbb{E} [{X^*_k}^2 - \\mathbb{E} [{X^*_k}]^2] $$\n\nSince we centered our data:\n\n$ \\mathbb{E} [{X^*_k}]^2 = 0 $\n\nThen:\n\n$\\frac{Var({X^*_k})}{Var_{Total}} = \\frac{\\mathbb{E} [{X^*_k}^2]}{Var_{Total}} = \\frac{|| {X^*_k}^2 ||_2}{Var_{Total}}$\n\nAnd the participation of each feature over the total variance is proportional to the squared norm of the principal component! It also applies to more dimentions (we would do exactly the same steps).\n\nComponents with neglectable importances may be ignored without any risk to spoil the final model and with the benefit of reducting the computational weight of the model.\n\nApplying our Scikit's PCA:"},{"metadata":{"trusted":true},"cell_type":"code","source":"data_pca = PCA().fit_transform(array_train) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is the transformed dataset:"},{"metadata":{"trusted":true},"cell_type":"code","source":"data_pca","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data_pca.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I usually like to plot the feature importances from the most important variable to the less important one in a Paretto's diagram:"},{"metadata":{"trusted":true},"cell_type":"code","source":"var_pca = data_pca.var(axis = 0)\nimp_pca = var_pca / sum(var_pca)\ncum_imp_pca = np.cumsum(imp_pca)\n\nlist_vars = var_pca.tolist() + imp_pca.tolist() + cum_imp_pca.tolist()\nnamed_vars = ['Variance'] * len(var_pca) +\\\n             ['Importance'] * len(imp_pca) +\\\n             ['Cumulative_Importance'] * len(cum_imp_pca)\n        \ndf_pca = pd.DataFrame(dict(\n    IndexStr=[str(i) for i in 3 * list(range(0, len(named_vars) // 3))],\n    Index=3 * list(range(0, len(named_vars) // 3)),\n    Variable=named_vars,\n    Value=list_vars\n))\n\ndf_pca.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"After processing the Paretto's plot data in a dataframe, let's check the diagram:"},{"metadata":{"trusted":true},"cell_type":"code","source":"ggplot(aes(x = 'Index', y = 'Value')) +\\\n    geom_bar(aes(fill = 'IndexStr'), color = 'black', stat = 'identity', data = df_pca[df_pca['Variable'] == 'Importance']) +\\\n    theme(legend_position = 'none') + ggtitle('Cumulative Variables Importance') +\\\n    geom_line(data = df_pca[df_pca['Variable'] == 'Cumulative_Importance'], color = 'blue') +\\\n    geom_point(data = df_pca[df_pca['Variable'] == 'Cumulative_Importance'], color = 'blue') +\\\n    geom_point(data = df_pca[(df_pca['Variable'] == 'Cumulative_Importance') &\\\n                             (df_pca['Value'] > 0.9999999999)], color = 'red', shape = 'x', size = 5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The red \"X\" is marked over the points where the cumulative importance is really, really near of $100 \\%$. It seems that the feature importances of the last $4$ terms are numerically equal to zero:"},{"metadata":{"trusted":true},"cell_type":"code","source":"imp_pca[35:-1]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It's true! And we are ready to create our final pipeline, including the dimentionality reduction. It will be created with the aid of a parametric function:"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_preprocessor(pass_features=binary_features + ordinal_features_low_count, \n                     one_hot_features=nominal_features_low_count, \n                     avg_features=nominal_features_high_count + ordinal_features_high_count, \n                     te_smoothing=1,\n                     pca_threshold=0.9999):\n\n    pass_pipeline = Pipeline(steps = [\n        ('imputation', SimpleImputerCorrected(strategy='most_frequent')),\n        ('scalling', StandardScaler())])\n\n    one_hot_pipeline  = Pipeline(steps = [\n        ('imputation', SimpleImputerCorrected(strategy='most_frequent')),\n        ('encoding', OneHotEncoder(sparse = False))])\n\n    avg_pipeline = Pipeline(steps = [\n        ('imputation', SimpleImputerCorrected(strategy='most_frequent')),\n        ('encoding', TargetEncoder(smoothing = te_smoothing)),\n        ('scalling', StandardScaler())])\n\n    encoder = ColumnTransformer(\n    transformers=[\n        ('pass_pipeline', pass_pipeline, pass_features),\n        ('one_hot_pipeline', one_hot_pipeline, one_hot_features),\n        ('avg_pipeline', avg_pipeline, avg_features)\n    ], sparse_threshold=0)\n    \n    if (pca_threshold > 0):\n        preprocessor = Pipeline(steps = [('encoder', encoder), ('pca', PCA(n_components=pca_threshold))])\n    else:\n        preprocessor = Pipeline(steps = [('encoder', encoder)])\n    return preprocessor","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Logistic Regression - Regularization + Validation"},{"metadata":{},"cell_type":"markdown","source":"The kind of transformations that we will apply to our features will be a hyperparameter of a bigger model. We will aways use the \"pass_features\" (aka binary features) without any kind of transformation, since it's not strictly necessary. Also, all the features with many different values will be target encoded since we have to avoid the curse of dimentionality and our notebook memory limitations.\n\nThen, we can make three choices:\n* Encode nominal features with few different values and ordinal features with few different values with one hot encoding (we call this \"minimum_te\")\n* Encode just nominal features with low count with one hot encoding and use target encoding in the ordinal features with few different values (\"medium_te\") or\n* Encode all nominal and ordinal features with target encoding (\"maximum_te\" configuration)\n\nWe will test the three possibilities."},{"metadata":{"trusted":true},"cell_type":"code","source":"pass_list = [pass_features, pass_features, pass_features]\n\none_hot_list = [nominal_features_low_count + ordinal_features_low_count,\n                nominal_features_low_count, []]\n\navg_list = [avg_features,\n            avg_features + ordinal_features_low_count,\n            avg_features + ordinal_features_low_count + nominal_features_low_count]\n\ntransformers_name = ['minimum_te', 'medium_te', 'maximum_te']\ntransformers_list = zip(transformers_name, pass_list, one_hot_list, avg_list)\n\nhyper_pipe_dict = {\n    'lor_model__solver': ['saga'],\n    'lor_model__penalty': ['elasticnet'],\n    'lor_model__C': [0.1, 1, 10],\n    'lor_model__l1_ratio': [0, 0.25, 0.5, 0.75, 1],\n    'preprocessor__transformers': transformers_list,\n    'preprocessor__target_smoothing': [0.1, 1, 10],\n    'preprocessor__pca_threshold': [0.9999]\n}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Preparing a list with all hyperparameters possible combinations:"},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train = df_train['target']\nk_fold_n_splits, k_fold_use = 20, 3\n\niter_list = []\nC_list = []\nl1_list = []\nt_list = []\nsmoothing_list = []\nsolver_list = []\npenalty_list = []\ntrain_score_list = []\npca_list = []\ntest_score_list = []\n\nfor t_name, t_pass, t_one_hot, t_avg in hyper_pipe_dict['preprocessor__transformers']:\n    for C in hyper_pipe_dict['lor_model__C']:\n        for l1_ratio in hyper_pipe_dict['lor_model__l1_ratio']:\n            for target_smoothing in hyper_pipe_dict['preprocessor__target_smoothing']:\n                for solver in hyper_pipe_dict['lor_model__solver']:\n                    for penalty in hyper_pipe_dict['lor_model__penalty']:\n                        for pca_threshold in hyper_pipe_dict['preprocessor__pca_threshold']:\n                            iter_list.append([C, l1_ratio, t_name, t_pass, t_one_hot, t_avg, \n                                              target_smoothing, solver, penalty, pca_threshold])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The cross validation takes time. So, I saved the results in a .csv that will be loaded if the flag \"cross_validate\" is equal to \"False\". But you can easily reproduce my gridsearch by setting the flag to \"True\" :)"},{"metadata":{"trusted":true},"cell_type":"code","source":"cross_validate = False","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here we have the cross validation loop:"},{"metadata":{"trusted":true},"cell_type":"code","source":"if cross_validate:\n    \n    for i in range(0, len(iter_list)):\n        \n        print('Progress:')\n        print('{0:.2%}'.format(i / len(iter_list)))\n        print('-----')\n        \n        C, l1_ratio, t_name, t_pass, t_one_hot, t_avg, target_smoothing, solver, penalty, pca_threshold = iter_list[i]\n        preprocessing_pipeline = get_preprocessor(t_pass, t_one_hot, t_avg, target_smoothing, pca_threshold=pca_threshold)\n        x_train = preprocessing_pipeline.fit_transform(df_train.iloc[:, 1:-2], y_train)\n                        \n        k_fold_obj = StratifiedKFold(n_splits=k_fold_n_splits, shuffle=True)\n        k_fold_count = 0\n        for train_index, test_index in k_fold_obj.split(x_train, y_train):\n            if k_fold_count >= k_fold_use:\n                break\n            else:\n                k_fold_count += 1\n                            \n                x_cv_train, y_cv_train = x_train[train_index], y_train[train_index]\n                x_cv_test, y_cv_test = x_train[test_index], y_train[test_index]\n                            \n                curr_lor = LogisticRegression(C=C, l1_ratio=l1_ratio, solver=solver, penalty=penalty, n_jobs=-1)\n                curr_lor.fit(x_cv_train, y_cv_train)\n                            \n                y_cv_train_pred = curr_lor.predict_proba(x_cv_train)[:, 1]\n                y_cv_test_pred = curr_lor.predict_proba(x_cv_test)[:, 1]\n                            \n                roc_auc_train = roc_auc_score(y_cv_train, y_cv_train_pred)\n                roc_auc_test = roc_auc_score(y_cv_test, y_cv_test_pred)\n                            \n                C_list.append(C)\n                l1_list.append(l1_ratio)\n                t_list.append(t_name)\n                smoothing_list.append(target_smoothing)\n                solver_list.append(solver)\n                penalty_list.append(penalty)\n                pca_list.append(pca_threshold)\n                train_score_list.append(roc_auc_train)\n                test_score_list.append(roc_auc_test)\n            \n    print('Progress:')\n    print('100%')\n    print('-----')\n    \n    df_cv = pd.DataFrame(\n        dict(\n            C=C_list,\n            L1=l1_list,\n            T=t_list,\n            Smooth=smoothing_list,\n            Solver=solver_list,\n            Penalty=penalty_list,\n            PCA_Threshold=pca_list,\n            Train_AUC=train_score_list,\n            Test_AUC=test_score_list\n        )\n    )\n    df_cv.to_csv('/kaggle/working/df_cv.csv', index=False)\n            \nelse:\n    df_cv = pd.read_csv('../input/cv-results-cat-challenge/df_cv.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The $10$ best hyperparameter combinations are:"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_cv.sort_values(by='Test_AUC', ascending=False).iloc[0:10, :]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Grouping the cross-validation folds and taking the average:"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_cv_grouped = df_cv.groupby(['C', 'L1', 'T', 'Smooth', 'Solver', 'Penalty']).mean().sort_values(by='Test_AUC', ascending=False)\ndf_cv_grouped.iloc[0:10, :]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can check the distribution of the scores before grouping the AUC values per folds and after taking the mean. Before taking the average:"},{"metadata":{"trusted":true},"cell_type":"code","source":"ggplot(df_cv, aes(x='Test_AUC')) + geom_histogram(bins = 50, fill = 'lightblue') +\\\n    ggtitle('CV Test AUC Distribution (Before Grouping)') + xlab('ROC-AUC') + ylab('Count - 50 Bins')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"After taking the mean:"},{"metadata":{"trusted":true},"cell_type":"code","source":"ggplot(df_cv_grouped, aes(x='Test_AUC')) + geom_histogram(bins = 50, fill = 'lightgreen') +\\\n    ggtitle('CV Test AUC Distribution (After Grouping)') + xlab('ROC-AUC') + ylab('Count - 50 Bins')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So, let's take the best $2$ models:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# C \tL1 \t    T \t        Smooth \tSolver \tPenalty\n# 0.1 \t0.25 \tmaximum_te \t0.1 \tsaga \telasticnet\n# 1.0 \t0.50 \tminimum_te \t10.0 \tsaga \telasticnet\n\nopt_pipe_A = get_preprocessor(pass_features, [], avg_features + ordinal_features_low_count + nominal_features_low_count, 0.1)\nopt_pipe_B = get_preprocessor(pass_features, ordinal_features_low_count + nominal_features_low_count, avg_features, 10.)\n\nopt_pipe_A.fit(df_train.iloc[:, 1:-2], y_train)\nopt_pipe_B.fit(df_train.iloc[:, 1:-2], y_train)\n\nopt_model_A = LogisticRegression(C=0.1, l1_ratio=0.25, solver='saga', penalty='elasticnet')\nopt_model_B = LogisticRegression(C=1.0, l1_ratio=0.50, solver='saga', penalty='elasticnet')\n\nx_pipe_A = opt_pipe_A.transform(df_train.iloc[:, 1:-2])\nx_pipe_B = opt_pipe_B.transform(df_train.iloc[:, 1:-2])\n\nopt_model_A.fit(x_pipe_A, y_train)\nopt_model_B.fit(x_pipe_B, y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"target_A = opt_model_A.predict_proba(opt_pipe_A.transform(df_test.iloc[:, 1:-1]))[:, 1]\ntarget_B = opt_model_B.predict_proba(opt_pipe_B.transform(df_test.iloc[:,1:-1]))[:, 1]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It's interesting to notice that the model $B$ (the second best model) let us get better results in the submission ($76.601 \\%$)"},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.DataFrame({\n    'id': df_test['id'],\n    'target': target_A\n}).to_csv('/kaggle/working/df_out_logistic_model_A.csv', index=False)\n\npd.DataFrame({\n    'id': df_test['id'],\n    'target': target_B\n}).to_csv('/kaggle/working/df_out_logistic_model_B.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In out next step, we will try to make a Neural Network and combine its results with the predictions of the Logistic Regression.\n\n# 4. Neural Network - Finding a Good Architecture"},{"metadata":{},"cell_type":"markdown","source":"In our Neural Network, we will use One Hot Encoding in every possible feature (except the binary ones, which are already encoded and the ordinal / nominal features that have many different values).\n\nWe can create a model with the necessary complexity to \"decode\" all the dummified features."},{"metadata":{"trusted":true},"cell_type":"code","source":"default_pipeline = get_preprocessor(pass_features=binary_features, \n                                    one_hot_features=nominal_features_low_count + ordinal_features_low_count, \n                                    avg_features=nominal_features_high_count + ordinal_features_high_count)\n\nx_train = default_pipeline.fit_transform(df_train.iloc[:, 1:-2], y_train)\nx_test = default_pipeline.transform(df_test.iloc[:, 1:-1])\nx_train","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Using the function Keras API we will create a \"Wide and Deep\" neural network to solve our problem.\n\n![Wide And Deep NN](https://www.researchgate.net/profile/Kaveh_Bastani/publication/328161216/figure/fig3/AS:679665219928064@1539056224036/Illustration-of-the-wide-and-deep-model-which-is-an-integration-of-wide-component-and.ppm)\n\nWe will not explicitly use embedding layers in our model but the principle is the same. Wide and Deep Neural Networks were originally proposed by Google in $2016$, to use in the APP Store recommendations system:"},{"metadata":{"trusted":true},"cell_type":"code","source":"def auc(y_true, y_pred):\n    def fallback_auc(y_true, y_pred):\n        try:\n            return roc_auc_score(y_true, y_pred)\n        except:\n            return 0.5\n    return tf.py_function(fallback_auc, (y_true, y_pred), tf.double)\n\ndef get_wide_and_deep(use_montecarlo=False):\n\n    inputs = Input(shape=(81,))\n\n    deep = Dense(81, activation='elu')(inputs)\n    deep = Dropout(0.5)(deep) if not use_montecarlo else Dropout(0.5)(deep, training=True)\n    deep = Dense(40, activation='elu')(deep)\n    deep = Dropout(0.5)(deep) if not use_montecarlo else Dropout(0.5)(deep, training=True)\n    deep = Dense(20, activation='elu')(deep)\n    deep = Dropout(0.5)(deep) if not use_montecarlo else Dropout(0.5)(deep, training=True)\n    deep = Dense(10, activation='elu')(deep)\n\n    deep_and_wide = Concatenate()([deep, inputs])\n    deep_and_wide = BatchNormalization()(deep_and_wide)\n    deep_and_wide = Dense(1, activation='sigmoid')(deep_and_wide)\n\n    model_nn = Model(inputs=inputs, outputs=deep_and_wide)\n    model_nn.compile(optimizer=Adam(lr=0.001),loss='binary_crossentropy', metrics=['accuracy', auc])\n    \n    return model_nn","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Printing the layers of our model:"},{"metadata":{"trusted":true},"cell_type":"code","source":"model_nn = get_wide_and_deep()\nmodel_nn.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can even draw the layers of our Wide and Deep NN:"},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_model(model_nn, to_file='model.png', show_shapes=True, show_layer_names=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The Dropout layers represent a regularization mechanism that trains our neural network after removing $X \\%$ random weights of the training algorithm. The $X \\%$ value in our model is $50 \\%$. It's a smart mechanism that is able to create a robust neural network, that is capable to take reasonable predictions evn without some of its synaptic weights! \n\nThe BatchNormalization layer standardizes the values by learning the variance and the means of previous batch values. It can help us to don't have explosive (positive or negative) values in out Neural Network.\n\nFinally, the EarlyStopping callback will train our Neural Network while the validation score increases. We give $30$ chances to our model to get a higher score, otherwise, we just stop training. This parameter is called the Early Stopping \"patience\":"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train['target'].mean()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_idx.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"es = EarlyStopping(monitor='val_auc', min_delta=0.001, patience=5,verbose=1, mode='max', baseline=None, restore_best_weights=True)\nrlr = ReduceLROnPlateau(monitor='val_auc', factor=0.5,patience=3, min_lr=1e-6, mode='max', verbose=1)\n\nn_folds = 10\nsfk = StratifiedKFold(n_splits=n_folds, random_state=42, shuffle=True)\npred_list, hist_list = [], []\n\nfold_index = 1\nfor train_idx, val_idx in sfk.split(x_train, y_train):\n    \n    print('\\n---\\nFold Index\\n---\\n: ' + str(fold_index))\n    fold_index += 1\n    \n    model_nn = get_wide_and_deep()\n    history_nn = model_nn.fit(x_train[train_idx], y_train[train_idx],\n                              validation_data = (x_train[val_idx], y_train[val_idx]),\n                              callbacks=[es, rlr], \n                              epochs=100, batch_size=1024, \n                              class_weight={0: 0.2, 1: 0.8},\n                              verbose=0)\n    \n    pred_list.append(model_nn.predict(x_test))\n    hist_list.append(history_nn)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Also, notice that the validation set is composed of $10 \\%$ of the training dataset. Finally, predicting the Neural Network final result:"},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_list = [list(X) for X in pred_list]\npred_list_formatted = [np.array([Y[0] for Y in X]) for X in pred_list]\ntarget_nn = list(np.mean(pred_list_formatted, axis=0))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"out_nn = model_nn.predict(x_test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's plot the training and validations curves of our fitting process:"},{"metadata":{"trusted":true},"cell_type":"code","source":"history_nn.history.keys()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history_nn.history.keys()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"list_val = history_nn.history['val_loss'] +\\\n           history_nn.history['val_accuracy'] +\\\n           history_nn.history['val_auc'] +\\\n           history_nn.history['loss'] +\\\n           history_nn.history['accuracy'] +\\\n           history_nn.history['auc']\n\nn_epochs = len(history_nn.history['val_loss'])\n\nlist_steps = 6 * (list(range(1, n_epochs + 1)))\n\nlist_metrics = (n_epochs * ['Loss']) + (n_epochs * ['Accuracy']) +\\\n               (n_epochs * ['AUC']) + (n_epochs * ['Loss']) +\\\n               (n_epochs * ['Accuracy']) + (n_epochs * ['AUC'])\n\nlist_kind = (n_epochs * ['Validation']) + (n_epochs * ['Validation']) +\\\n            (n_epochs * ['Validation']) + (n_epochs * ['Training']) +\\\n            (n_epochs * ['Training']) + (n_epochs * ['Training'])\n\ndf_nn_history = pd.DataFrame(dict(Step=list_steps, Value=list_val, Metric=list_metrics, Kind=list_kind))\ndf_nn_history.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ggplot(df_nn_history[df_nn_history['Step'] > 5], aes(x='Step', y='Value', colour='Kind')) +\\\n    geom_line(aes(group='Kind')) +\\\n    facet_grid('Metric ~ .', scales='free') +\\\n    ggtitle('Training / Validation Metrics') + geom_point(aes(shape = 'Kind'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"target_nn = [X[0] for X in out_nn.tolist()]\ntarget_nn[1:5]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.DataFrame({\n    'id': df_test['id'],\n    'target': target_nn\n}).to_csv('/kaggle/working/df_out_wide_and_deep.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can also blend the models, and consider the average of the neural network output with the best logistic regression we found (model B):"},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.DataFrame({\n    'id': df_test['id'],\n    'target': (np.array(target_nn) + np.array(target_B)) / 2\n}).to_csv('/kaggle/working/df_out_models_avg.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The best model is the last one: when we take the average of the best models and they have different predictions, then we create a better estimator, thanks to the central limit theorem:\n\n![Central Limit Theorem](https://i.stack.imgur.com/wPGzI.png)\n\nWe can imagine that each model is a \"coin\" with a good probability to take a $1$ (imagine that we we have a $0$ in one side and a $1$ on the other side). So, if the coins generate reasonably independent results and we take the average value that we get after flipping them, we will tend to get a normal distribution, with a low standard deviation and centered around a value that is near to one (closer than the expected value of the Bernouilli distribution of each individual coin!)\n\nThe theory justifies the better results :)\n\nSo, maybe, we can apply a **MonteCarlo Dropout** instead of removing the Dropout effect during the testing phase (Keras does it by default). With this approach, we convert our deterministic neural network into a probabilistic one and then we can create many different estimators and get the average predictions. Many different configurations with few weights will be considered:\n\n![Dropout Figure](https://encrypted-tbn0.gstatic.com/images?q=tbn%3AANd9GcSz-CH9Jv9p3gJjn8pSF3EHebiRHUof84YOzExRI3QuCzqubwkn)"},{"metadata":{},"cell_type":"markdown","source":"Creating a model with the same weights with the MonteCarlo parameter equal to \"True\":"},{"metadata":{"trusted":true},"cell_type":"code","source":"model_nn_mc = get_wide_and_deep(use_montecarlo=True)\nmodel_nn_mc.set_weights(model_nn.get_weights())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Running $100$ simulations:"},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions_list = []\nn_montecarlo_sims = 100\nfor i in range(n_montecarlo_sims):\n    if (i % 10) == 0:\n        print('Monte Carlo Simulation - Iteration: ' + str(i + 1) + '/' + str(n_montecarlo_sims))\n    predictions_list.append([X[0] for X in model_nn_mc.predict(x_test)])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Taking the average:"},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions_list = [np.array(X) for X in predictions_list]\ntarget_nn_mc = sum(predictions_list) / n_montecarlo_sims","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Saving the obtained model and the average of the MonteCarlo NN with the logistic regression:"},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.DataFrame({\n    'id': df_test['id'],\n    'target': target_nn_mc\n}).to_csv('/kaggle/working/df_out_models_mc.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.DataFrame({\n    'id': df_test['id'],\n    'target': (target_nn_mc + np.array(target_B)) / 2\n}).to_csv('/kaggle/working/df_out_models_avg_mc.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We also have a small improvement after applying the Monte Carlo Dropout model and blending it with the Logistic Regression."},{"metadata":{},"cell_type":"markdown","source":"# 5. Final Conclusions\n\nSo, here I can list what I learned with this challenge:\n\n1. We can use different encoding schemes to tranform our categorical features in numbers that can be processed by machine learning models. The best encoding scheme depends not only on the number of different features of the model, but also on the kind of estimator that we will use.\n2. Sometimes, simple models as Logistic Regressions may already give us great results and additional efforts are, sometimes, useless. I mean: would it by really important to improve the score in $0.5 \\%$ after applying a huge effort in a complex neural network? Well, it depends on the kind of business or application.\n3. Wide and Deep neural networks may be seen as an optimal linear combination between a simple logistic regression and a deep neural network. So, it's a mix between a complex model and an optimal one. It was interesting to use it here.\n4. When working with neural networks, we should consider alternative regularization mechanisms such as Dropout layers or BatchNormalization layers to improve the consistency and prediction power of the estimator.\n5. The EarlyStopping is an elegant way to avoid overfitting. But I got better results when I increased the patience from $10$ to $30$. Why? If the convergence of the weights is slow, we can wait more in order to obtain a better estimator.\n6. Blending independent models may help us to reach better results thanks to the Central Limit Theorem :)\n\nIf you read until this point, thanks a lot! Take this ASCII lucky cat for you!\n\n         ,_         _,\n         |\\\\.-\"\"\"-.//|\n         \\`         `/\n        /    _   _    \\\n        |    a _ a    |\n        '.=    Y    =.'\n          >._  ^  _.<\n         /   `````   \\\n         )           (\n        ,(           ),\n       / )   /   \\   ( \\\n       ) (   )   (   ) (\n       ( )   (   )   ( )\n       )_(   )   (   )_(-.._\n      (  )_  (._.)  _(  )_, `\\\n       ``(   )   (   )`` .' .'\n    jgs   ```     ```   ( (`\n                         '-'\n"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.9"}},"nbformat":4,"nbformat_minor":4}