{"cells":[{"metadata":{},"cell_type":"markdown","source":"# My First Kaggle Notebook!\n---\nThis notebook doesn't have any EDA, nor any feature engineering. It was initially made just for submission to [this competition.](https://www.kaggle.com/c/cat-in-the-dat-ii)\n## This notebook contains two approaches: A standard tree-based(Random Forest) approach  and a neural-net based approach (Keras)"},{"metadata":{},"cell_type":"markdown","source":"So Let's start!\n## 1. Importing Libraries\n_(You know, the standard) :p_"},{"metadata":{"trusted":true},"cell_type":"code","source":"# For the random forest model\nimport numpy as np\nimport pandas as pd\nfrom sklearn import preprocessing\nfrom sklearn.ensemble import RandomForestClassifier\nimport joblib\n\n# Continued for keras model\nimport tensorflow as tf\nfrom keras import layers, optimizers, callbacks, metrics, utils, regularizers\nfrom keras.models import Model\nfrom sklearn import metrics\nfrom sklearn import model_selection\nfrom keras import backend as K\nimport gc\nimport warnings\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2. Reading the CSV files"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv(\"../input/cat-in-the-dat-ii/train.csv\")\ntest = pd.read_csv(\"../input/cat-in-the-dat-ii/test.csv\")\ntrain.columns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Adding a column in Test data, to concatenate it with Training data [more on this later]\ntest[\"target\"] = -1\ntest.columns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Full dataset\nfull_data = pd.concat([train, test]).reset_index(drop=True)\nfull_data.shape ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3. Label Encoding\nNow comes the interesting part!\n#### NOTE: I used Label Encoding for all the types of categorical data, which results in the model failing to capture some in-built features in the different categories!"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Categorical Features don't include the ID and the Target (obviously)\nCATEGORICAL_FEATURES = [c for c in full_data.columns if c not in [\"id\", \"target\"]]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# I combined the datasets, just so that I could handle the 'previously unseen values' while inferencing\nlabels = dict()\nfor feature in CATEGORICAL_FEATURES:\n    full_data.loc[:, feature] = full_data.loc[:, feature].fillna(\"-1\").astype(str)\n    label = preprocessing.LabelEncoder()\n    label.fit(full_data.loc[:, feature].values.tolist())\n    full_data.loc[:, feature] = label.transform(full_data.loc[:, feature].values.tolist())\n    labels[feature] = label\n# joblib.dump(labels, f\"../input/label_dict.pkl\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Wait, wasn't that a data leakage issue?\nProbably, but I'm a newbie and still learning how to handle the new labels. Any help on this would be appreciated! ;)"},{"metadata":{},"cell_type":"markdown","source":"## 4. Converting the combined data back to Train and Test "},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data = full_data[full_data.target != -1].reset_index(drop=True)\ntest_data = full_data[full_data.target == -1].drop([\"target\"], axis=1).reset_index(drop=True)\nprint(f\"Train shape: {train_data.shape} ; Test shape: {test_data.shape}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### As we can see, All the labels are now encoded in integers.<br>\nOne interesting thing I noticed was if I flipped the _fillna_ and _astype_ commands in Label Encoding, All the NaN values would have the last integer alloted to it, and not the first (i.e, 0).<br>\n### Do we want that? I'd love to hear your thoughts!"},{"metadata":{},"cell_type":"markdown","source":"## 5. Saving processed inputs"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.to_csv(\"train_categorical.csv\", index=False)\ntest_data.to_csv(\"test_categorical.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Here my notebook splits in two parts:\n#### 1. Keras model which trains on entity embeddings\n#### 2. Random Forest model which trains on the label encodings"},{"metadata":{},"cell_type":"markdown","source":"# 6.1 Random Forest model which trains on label encoded data"},{"metadata":{},"cell_type":"markdown","source":"## 6.1.1 Creating Folds"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Number of folds defined defined for StratifiedKFold, for the Random Forest model\nNUMBER_OF_FOLDS = 5\n\n# Just for sanity check\ntrain_data = pd.read_csv(\"train_categorical.csv\")\ntest_data = pd.read_csv(\"test_categorical.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Creating a column to incorporate the folds\ntrain_data.loc[:, \"kfold\"] = -1\nkfold = model_selection.StratifiedKFold(n_splits=NUMBER_OF_FOLDS, shuffle=True, random_state=7) # 7 is my lucky number! (I know about 42 don't worry :p)\n\nfor fold, (train_idx, valid_idx) in enumerate(kfold.split(X = train_data, y = train_data.target.values)):\n    train_data.loc[valid_idx, \"kfold\"] = fold\n    print(f\"Shape of {fold} fold: ({len(train_data.loc[train_idx])},{len(train_data.loc[valid_idx])})\")\n# train_data.to_csv(\"../input/train_folds.csv\", index = False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6.1.2 Training!"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Creating and saving a classifier for each fold\nclf_dict = dict()\nfor fold in range(NUMBER_OF_FOLDS):\n    train_df = train_data[train_data.kfold != fold].reset_index(drop=True)\n    valid_df = train_data[train_data.kfold == fold].reset_index(drop=True)\n    training_y = train_df.target.values\n    training_x = train_df.drop([\"id\", \"target\", \"kfold\"], axis=1)\n    validation_y = valid_df.target.values\n    validation_x = valid_df.drop([\"id\", \"target\", \"kfold\"], axis=1)\n    \n    clf = RandomForestClassifier(n_estimators=200, n_jobs = 12, verbose=0)\n    clf.fit(training_x, training_y)\n    pred_y = clf.predict_proba(validation_x)[:, 1]\n    print(metrics.roc_auc_score(validation_y, pred_y))\n    clf_dict[fold] = clf\n#     joblib.dump(clf, f\"./{MODEL}_{fold}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### AUC Score: ~0.72\nNot that good, but not that bad either, considering no feature engineering done, no handling of the imbalanced dataset."},{"metadata":{},"cell_type":"markdown","source":"## 6.1.3 Inferencing\nBut first let us look at the sample submission.<br>\n_Confession: I made a mistake in my first submission because I neglected this. :P_"},{"metadata":{"trusted":true},"cell_type":"code","source":"# test_data = pd.read_csv(\"../input/test_categorical.csv\")\nsample = pd.read_csv(\"../input/cat-in-the-dat-ii/sample_submission.csv\")\nsample.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Averaging out the 5 classifiers' output\npredictions = None\ntest_idx = test_data[\"id\"]\ntest_data = test_data.drop([\"id\"], axis=1)\n\nfor fold in range(NUMBER_OF_FOLDS):\n#     clf = joblib.load(f\"./{MODEL}_{fold}\")\n    clf = clf_dict[fold]\n    predict = clf.predict_proba(test_data)[:,1]\n    if fold == 0:\n        predictions = predict\n    else:\n        predictions += predict\nrf_prediction = predictions / float(NUMBER_OF_FOLDS)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6.1.4 Submitting!\n#### Fun Fact: When I first submitted, I got an AUC score of 0.52!"},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.DataFrame(np.column_stack((test_idx, rf_prediction)), columns=[\"id\", \"target\"])\nsubmission.id = submission.id.astype(int)\nsubmission.to_csv(\"randomforest_submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Before moving on, let me refresh the kernel, as it ran into memory issues earlier! :'("},{"metadata":{"trusted":true},"cell_type":"code","source":"%reset -f","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6.1.5 Importing Again!\nAs the kernel is refreshed, we need to import everything all over again!"},{"metadata":{"trusted":true},"cell_type":"code","source":"# For the random forest model\nimport numpy as np\nimport pandas as pd\nfrom sklearn import preprocessing\nfrom sklearn.ensemble import RandomForestClassifier\nimport joblib\n\n# Continued for keras model\nimport tensorflow as tf\nfrom keras import layers, optimizers, callbacks, metrics, utils, regularizers\nfrom keras.models import Model\nfrom sklearn import metrics\nfrom sklearn import model_selection\nfrom keras import backend as K\nimport gc\nimport warnings\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 6.2 Keras model which trains on Entity Embeddings"},{"metadata":{},"cell_type":"markdown","source":"#### Note: Many of the below ideas are inspired from the world's first 4x GM.\nCheck out [his profile!](https://www.kaggle.com/abhishek)"},{"metadata":{},"cell_type":"markdown","source":"## 6.2.1 Defining the metric"},{"metadata":{"trusted":true},"cell_type":"code","source":"# There's a keras metric AUC with which I was initially training, however I realised it wasn't a reliable metric\n# Therefore after searching for reliable metrics, I came across this function which will act as our metric.\ndef auc(y_true, y_pred):\n    def fallback_auc(y_true, y_pred):\n        try:\n            return metrics.roc_auc_score(y_true, y_pred)\n        except:\n            return 0.5\n    return tf.py_function(fallback_auc, (y_true, y_pred), tf.double)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Number of folds defined defined for StratifiedKFold, for the keras model\nNUMBER_OF_FOLDS = 3\n\n# Just for sanity check\ntrain_data = pd.read_csv(\"train_categorical.csv\")\ntest_data = pd.read_csv(\"test_categorical.csv\")\n\n# Since we cleared the memory\nCATEGORICAL_FEATURES = [c for c in train_data.columns if c not in [\"id\", \"target\"]]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6.2.2 Defining the model"},{"metadata":{"trusted":true},"cell_type":"code","source":"# This function will return the created keras model\ndef make_model(data, features):\n    input_cols = []\n    output_emb = []\n    for column in features:\n        embedding_dim = min(int(len(data[column].unique())/2)+1, 65)\n        input_layer = layers.Input(shape=(1,)) #input will be batches of 1 dimension\n        mid_layer = layers.Embedding(len(data[column].values.tolist())+1, embedding_dim)(input_layer) \n        out = layers.SpatialDropout1D(0.1)(mid_layer)\n        emb = layers.Reshape(target_shape=(embedding_dim,))(out)\n        input_cols.append(input_layer)\n        output_emb.append(emb)\n    x = layers.Concatenate()(output_emb)\n    x = layers.BatchNormalization()(x)\n    x = layers.Dense(4096, activation=\"relu\", kernel_regularizer=regularizers.l2(0.001))(x)\n    x = layers.Dropout(0.1)(x)\n    x = layers.BatchNormalization()(x)\n    x = layers.Dense(512, activation=\"relu\", kernel_regularizer=regularizers.l2(0.001))(x)\n    x = layers.Dropout(0.1)(x)\n    x = layers.BatchNormalization()(x)\n    y = layers.Dense(2, activation=\"softmax\", kernel_regularizer=regularizers.l2(0.001))(x)\n    model = Model(inputs = input_cols, outputs = y)\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"final_valid_preds = np.zeros((len(train_data)))\nfinal_test_preds = np.zeros((len(test_data)))\n\nkfold = model_selection.StratifiedKFold(n_splits=NUMBER_OF_FOLDS, shuffle=True)\n\n# Defining callbacks\nearlystop = callbacks.EarlyStopping(monitor='val_auc', min_delta=0.001, patience=5, verbose=1, mode='max', baseline=None, restore_best_weights=True)\nreducelr = callbacks.ReduceLROnPlateau(monitor='val_auc', factor=0.5, patience=3, min_lr=1e-6, mode='max', verbose=1)\n\ntest_idx = test_data[\"id\"]\ntest_df = test_data.drop([\"id\"], axis=1)\ntest = [test_df.values[:, k] for k in range(test_df.values.shape[1])]\n\nfor (train_idx, valid_idx) in kfold.split(X = train_data, y = train_data.target.values):\n    print(f\"Shape of fold: ({len(train_data.loc[train_idx])},{len(train_data.loc[valid_idx])})\")\n    train_df = train_data.loc[train_idx]\n    valid_df = train_data.loc[valid_idx]\n    ytrain = train_df.target.values\n    valid_y = valid_df.target.values\n    \n    # The input to the model will be list of lists such that each list will represent encoded data of a single feature\n    X = [train_df.loc[:, CATEGORICAL_FEATURES].values[:, k] for k in range(train_df.loc[:, CATEGORICAL_FEATURES].values.shape[1])]\n    Xvalid = [valid_df.loc[:, CATEGORICAL_FEATURES].values[:, k] for k in range(valid_df.loc[:, CATEGORICAL_FEATURES].values.shape[1])]\n\n    # Defining the model\n    model = make_model(train_df, CATEGORICAL_FEATURES)\n    \n    # Metric is the metric function defined above\n    model.compile(loss=\"binary_crossentropy\", optimizer=\"adam\", metrics=[auc])\n \n    history = model.fit(X, utils.to_categorical(ytrain), validation_data=(Xvalid, utils.to_categorical(valid_y)),\n                        batch_size = 1024, callbacks=[earlystop, reducelr], epochs=100, verbose=1)\n    # Predict on test data\n    test_preds = model.predict(test)[:,1]\n    # Predict on validation data per fold\n    valid_preds = model.predict(Xvalid)[:, 1]\n    \n    final_valid_preds[valid_idx] = valid_preds.ravel()\n    final_test_preds += test_preds.ravel()\n    \n    print(metrics.roc_auc_score(valid_y, valid_preds))\n    \n    # To clear out the GPU memory held by the model\n    K.clear_session()    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# The final prediction is taken as the average of the predictions per fold\nkeras_prediction = final_test_preds / float(NUMBER_OF_FOLDS)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6.2.3 Creating a submission file for the keras model"},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.DataFrame(np.column_stack((test_idx, keras_prediction)), columns=[\"id\", \"target\"])\nsubmission.id = submission.id.astype(int)\nsubmission.to_csv(\"keras_submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### AUC Score: ~0.783\nThat was quite an improvement on our previous Random Forest model!"},{"metadata":{},"cell_type":"markdown","source":"# With this, we come to an end to my first notebook on Kaggle!\n### Any suggestions, any mistakes pointed out would be highly appreciated. ;D"},{"metadata":{},"cell_type":"markdown","source":"![  <- I attached an image, but it's broken; probably because you didn't upvote ;(](nan)"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.6"}},"nbformat":4,"nbformat_minor":4}