{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":53666,"databundleVersionId":6589269,"sourceType":"competition"}],"dockerImageVersionId":30615,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 11th Place Gold in Sleep Comp\nKaggle's Sleep Comp was a fun competition! My 11th place gold solution discussion is [here][1]. This notebook shows the code for my 2 main models that I used in my solution. Each model is a GPU Deep Learning model built from the following 3 DL building blocks:\n\n![](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Dec-2023/blocks.png)\n\n[1]: https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states/discussion/459596","metadata":{}},{"cell_type":"markdown","source":"# Data Loader\nThe dataloader randomly produces 24 hour crops from the train data where **every crop contains onset and wakeup**. It **standardizes** all the features with `feature = (feature-mean)/(standard deviation)`\n\nMy `CNN-Transformer-GRU` model uses the original `anglez` and `enmo` features standardized. And it uses `hour`, `minute` and 76 engineered features. My `Deep-GRU` **uses only** the original `anglez` and `enmo` features standardized. Wow!\n\n![](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Dec-2023/dataloader.png)","metadata":{}},{"cell_type":"markdown","source":"# Scheduler\nWe train each model for 15 epochs using the following `cosine scheduler` without warmup.","metadata":{}},{"cell_type":"code","source":"import math, numpy as np\nimport matplotlib.pyplot as plt\nimport tensorflow as tf\n\nLR_START = 1e-6\nLR_MAX = 1e-3\nLR_MIN = 1e-6\nLR_RAMPUP_EPOCHS = 0\nLR_SUSTAIN_EPOCHS = 0\nEPOCHS = 15\n\ndef lrfn(epoch):\n    if epoch < LR_RAMPUP_EPOCHS:\n        lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n    elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n        lr = LR_MAX\n    else:\n        decay_total_epochs = EPOCHS - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS - 1\n        decay_epoch_index = epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS\n        phase = math.pi * decay_epoch_index / decay_total_epochs\n        cosine_decay = 0.5 * (1 + math.cos(phase))\n        lr = (LR_MAX - LR_MIN) * cosine_decay + LR_MIN\n    return lr\n\nrng = [i for i in range(EPOCHS)]\nlr_y = [lrfn(x) for x in rng]\nplt.figure(figsize=(10, 4))\nplt.plot(rng, lr_y, '-o')\nplt.xlabel('Epoch'); plt.ylabel('LR')\nprint(\"Learning rate schedule: {:.3g} to {:.3g} to {:.3g}\". \\\n      format(lr_y[0], max(lr_y), lr_y[-1]))\nLR = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose = True)","metadata":{"execution":{"iopub.status.busy":"2023-12-09T16:59:47.178388Z","iopub.execute_input":"2023-12-09T16:59:47.179028Z","iopub.status.idle":"2023-12-09T17:00:01.670409Z","shell.execute_reply.started":"2023-12-09T16:59:47.178981Z","shell.execute_reply":"2023-12-09T17:00:01.668955Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# CNN-Transformer-GRU\nMy CNN-Transformer-GRU model uses the original `anglez` and `enmo` features standardized. And it uses `hour`, `minute` and 76 engineered features. In total it uses 80 features. By itself, it achieves approximately `CV 0.800 and Public LB 0.750`.\n\n![](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Dec-2023/model2b.png)","metadata":{}},{"cell_type":"code","source":"import tensorflow.keras.backend as K\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\ntf.__version__","metadata":{"execution":{"iopub.status.busy":"2023-12-09T17:00:01.672752Z","iopub.execute_input":"2023-12-09T17:00:01.673577Z","iopub.status.idle":"2023-12-09T17:00:01.682462Z","shell.execute_reply.started":"2023-12-09T17:00:01.673533Z","shell.execute_reply":"2023-12-09T17:00:01.681472Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class TransformerBlock(layers.Layer):\n    def __init__(self, embed_dim, feat_dim, num_heads, ff_dim, rate=0.1):\n        super(TransformerBlock, self).__init__()\n        self.att = layers.MultiHeadAttention(num_heads=num_heads, key_dim=embed_dim)\n        self.ffn = keras.Sequential(\n            [layers.Dense(ff_dim, activation=\"gelu\"), layers.Dense(feat_dim),]\n        )\n        self.layernorm1 = layers.LayerNormalization(epsilon=1e-6)\n        self.layernorm2 = layers.LayerNormalization(epsilon=1e-6)\n        self.dropout1 = layers.Dropout(rate)\n        self.dropout2 = layers.Dropout(rate)\n\n    def call(self, inputs, training):\n        attn_output = self.att(inputs, inputs)\n        attn_output = self.dropout1(attn_output, training=training)\n        out1 = self.layernorm1(inputs + attn_output)\n        ffn_output = self.ffn(out1)\n        ffn_output = self.dropout2(ffn_output, training=training)\n        return self.layernorm2(out1 + ffn_output)","metadata":{"execution":{"iopub.status.busy":"2023-12-09T17:00:01.683751Z","iopub.execute_input":"2023-12-09T17:00:01.684387Z","iopub.status.idle":"2023-12-09T17:00:01.727554Z","shell.execute_reply.started":"2023-12-09T17:00:01.684348Z","shell.execute_reply":"2023-12-09T17:00:01.726608Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def positional_encoding(maxlen, num_hid):\n        depth = num_hid/2\n        positions = tf.range(maxlen, dtype = tf.float32)[..., tf.newaxis]\n        depths = tf.range(depth, dtype = tf.float32)[np.newaxis, :]/depth\n        angle_rates = tf.math.divide(1, tf.math.pow(tf.cast(10000, tf.float32), depths))\n        angle_rads = tf.linalg.matmul(positions, angle_rates)\n        pos_encoding = tf.concat(\n          [tf.math.sin(angle_rads), tf.math.cos(angle_rads)],\n          axis=-1)\n        return pos_encoding","metadata":{"execution":{"iopub.status.busy":"2023-12-09T17:00:01.729553Z","iopub.execute_input":"2023-12-09T17:00:01.730521Z","iopub.status.idle":"2023-12-09T17:00:01.747143Z","shell.execute_reply.started":"2023-12-09T17:00:01.730478Z","shell.execute_reply":"2023-12-09T17:00:01.745968Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from tensorflow.keras.layers import Input, Dense, Multiply, Add, Conv1D, Concatenate\n\ndef wave_block(x, filters, kernel_size, n):\n    dilation_rates = [2**i for i in range(n)]\n    x = Conv1D(filters = filters,\n               kernel_size = 1,\n               padding = 'same')(x)\n    res_x = x\n    for dilation_rate in dilation_rates:\n        tanh_out = Conv1D(filters = filters,\n                          kernel_size = kernel_size,\n                          padding = 'same', \n                          activation = 'tanh', \n                          dilation_rate = dilation_rate)(x)\n        sigm_out = Conv1D(filters = filters,\n                          kernel_size = kernel_size,\n                          padding = 'same',\n                          activation = 'sigmoid', \n                          dilation_rate = dilation_rate)(x)\n        x = Multiply()([tanh_out, sigm_out])\n        x = Conv1D(filters = filters,\n                   kernel_size = 1,\n                   padding = 'same')(x)\n        res_x = Add()([res_x, x])\n    return res_x","metadata":{"execution":{"iopub.status.busy":"2023-12-09T17:00:01.748789Z","iopub.execute_input":"2023-12-09T17:00:01.749234Z","iopub.status.idle":"2023-12-09T17:00:01.761521Z","shell.execute_reply.started":"2023-12-09T17:00:01.749200Z","shell.execute_reply":"2023-12-09T17:00:01.760350Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"feat_dim = 16 \nembed_dim = 32  \nnum_heads = 4  \nff_dim = 128  \ndropout_rate = 0.0\nnum_blocks = 4\n\ndef build_model_1():\n    \n    # INPUT \n    inp = tf.keras.Input(shape=(1440,80))\n    \n    # POSITIONAL ENCODING\n    x = layers.Dense(feat_dim)(inp)\n    p = positional_encoding(17280//12,feat_dim)\n    x = x + p\n    \n    # FOUR of (2xCCN + 1xTRANSFORMER BLOCKS)\n    for k in range(num_blocks):\n        skip = x\n        x = wave_block(x, feat_dim, 3, 12)\n        x = wave_block(x, feat_dim, 3, 12)\n        x = TransformerBlock(embed_dim, feat_dim, num_heads, ff_dim, dropout_rate)(x)\n        x = 0.9*x + 0.1*skip \n        \n    # RNN BLOCK\n    x = tf.keras.layers.Bidirectional(\n            tf.keras.layers.GRU(units=128, return_sequences=True))(x)\n    \n    # OUTPUT\n    x1 = tf.keras.layers.Dense(1,activation='relu')(x)\n    x1 = tf.keras.layers.Flatten()(x1)\n    x1 = tf.keras.layers.Activation('softmax', dtype='float32')(x1)\n    \n    x2 = tf.keras.layers.Dense(1,activation='relu')(x)\n    x2 = tf.keras.layers.Flatten()(x2)\n    x2 = tf.keras.layers.Activation('softmax', dtype='float32')(x2)\n    \n    # COMPILE MODEL\n    model = tf.keras.Model(inputs=inp, outputs=[x1,x2])\n    opt = tf.keras.optimizers.Adam(learning_rate = 1e-3)\n    loss = tf.keras.losses.CategoricalCrossentropy()\n    model.compile(loss=loss, optimizer = opt)\n        \n    return model","metadata":{"execution":{"iopub.status.busy":"2023-12-09T17:00:01.763620Z","iopub.execute_input":"2023-12-09T17:00:01.764410Z","iopub.status.idle":"2023-12-09T17:00:01.785588Z","shell.execute_reply.started":"2023-12-09T17:00:01.764361Z","shell.execute_reply":"2023-12-09T17:00:01.784699Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model1 = build_model_1()\nmodel1.summary()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-12-09T17:00:01.787145Z","iopub.execute_input":"2023-12-09T17:00:01.787913Z","iopub.status.idle":"2023-12-09T17:00:10.881298Z","shell.execute_reply.started":"2023-12-09T17:00:01.787851Z","shell.execute_reply":"2023-12-09T17:00:10.880257Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Deep-GRU\nMy Deep-GRU uses only the original `anglez` and `enmo` features standardized. In total it **uses only 2 features**, Wow! By itself, it achieves approximately `CV 0.805 and Public LB 0.755`.\n\n![](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Dec-2023/model1c.png)","metadata":{}},{"cell_type":"code","source":"num_blocks = 6\n\ndef build_model_2():\n    \n    # INPUT \n    inp = tf.keras.Input(shape=(17280,2))\n    \n    # CNN LAYERS\n    x = tf.keras.layers.Conv1D(64,5,strides=3,padding='same',activation='relu')(inp)\n    x = tf.keras.layers.Conv1D(64,3,strides=2,padding='same',activation='relu')(x)  \n    x = tf.keras.layers.Conv1D(64,3,strides=2,padding='same',activation='relu')(x)\n    \n    # SIX RNN BLOCKS\n    for k in range(num_blocks):\n        x = tf.keras.layers.Bidirectional(\n                tf.keras.layers.GRU(units=128, return_sequences=True))(x)\n    \n    # OUTPUT\n    x1 = tf.keras.layers.Dense(1,activation='relu')(x)\n    x1 = tf.keras.layers.Flatten()(x1)\n    x1 = tf.keras.layers.Activation('softmax')(x1)\n    \n    x2 = tf.keras.layers.Dense(1,activation='relu')(x)\n    x2 = tf.keras.layers.Flatten()(x2)\n    x2 = tf.keras.layers.Activation('softmax')(x2)\n    \n    # COMPILE MODEL\n    model = tf.keras.Model(inputs=inp, outputs=[x1,x2])\n    opt = tf.keras.optimizers.Adam(learning_rate = 1e-3)\n    loss = tf.keras.losses.CategoricalCrossentropy()\n    model.compile(loss=loss, optimizer = opt)\n    \n    return model","metadata":{"execution":{"iopub.status.busy":"2023-12-09T17:00:10.882910Z","iopub.execute_input":"2023-12-09T17:00:10.883212Z","iopub.status.idle":"2023-12-09T17:00:10.893646Z","shell.execute_reply.started":"2023-12-09T17:00:10.883187Z","shell.execute_reply":"2023-12-09T17:00:10.892382Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model2 = build_model_2()\nmodel2.summary()","metadata":{"_kg_hide-output":false,"execution":{"iopub.status.busy":"2023-12-09T17:00:10.895184Z","iopub.execute_input":"2023-12-09T17:00:10.895514Z","iopub.status.idle":"2023-12-09T17:00:14.054597Z","shell.execute_reply.started":"2023-12-09T17:00:10.895487Z","shell.execute_reply":"2023-12-09T17:00:14.053474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Sample OOF Predictions\nBelow are four sample OOF predictions. We see how using a `NLP QA head` which uses softmax produces **distinct sharp peaks**. This is different than using `NLP NER head` which uses binary cross entropy and targets converted into **Gaussian peaks**. With distinct sharp peaks, it is easy for us to find the top1, top2, top3, etc predictions per night. \n\nAlso softmax calibrates the predictions to be actual probabilities. Each night, all our model predictions add up to probability equal one. The competition metric `Average Precision` requires us to include `score` for each prediction. Using calibrated probabilties for `score` improves our CV and LB score!\n\n![](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Dec-2023/pred1.png)\n![](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Dec-2023/pred2.png)","metadata":{}}]}