{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# ResNet-34 | Art of Hyperparam Tuning! 🎨 \n\nGain a 🍭 **competitive advantage** in your notebooks with *letstune*!\n\nExplore quickly 🔎 better ML configurations!\n\n<br/>\n\nThis notebook trains ResNet Keras on MNIST with **hyperparameter tuning**.\n\nIt's a fork of [another Kaggle notebook](https://www.kaggle.com/code/pranjalsriv/resnet-34-data-augmentation-99-2-accuracy) (from Pranjal SRIVASTAVA)\nwith **hyperparameter tuning added**.","metadata":{}},{"cell_type":"markdown","source":"# letstune\n\n![letstune logo](https://github.com/mslapek/letstune/raw/main/img/logo.svg)\n\n*letstune* is an MIT-licensed Python hyperparameter tuning package.\n\nIt's better than grid-search, because it *operates on epochs*:\n\n1. Runs some parameters for a few epochs\n2. **Automatically discards low performers**\n3. Continues training for the best ones\n4. And so on... we have many *rounds* of such elimination.\n\n**Cool plots** demonstrating this ahead! 📊\n\n*Disclaimer*: I'm the author of the package. 😉\n\nMore info: https://github.com/mslapek/letstune","metadata":{}},{"cell_type":"code","source":"# based on https://www.kaggle.com/code/masterscrat/detect-if-notebook-is-running-interactively/notebook\nINTERACTIVE = 'runtime' in get_ipython().config.IPKernelApp.connection_file\nprint(f\"INTERACTIVE={INTERACTIVE}\")","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:20:05.908790Z","iopub.execute_input":"2023-11-25T17:20:05.909624Z","iopub.status.idle":"2023-11-25T17:20:05.921306Z","shell.execute_reply.started":"2023-11-25T17:20:05.909569Z","shell.execute_reply":"2023-11-25T17:20:05.920354Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# set to True to limit training time\nDEBUG = INTERACTIVE\nprint(f'DEBUG={DEBUG}')","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:20:05.922862Z","iopub.execute_input":"2023-11-25T17:20:05.923157Z","iopub.status.idle":"2023-11-25T17:20:05.932352Z","shell.execute_reply.started":"2023-11-25T17:20:05.923132Z","shell.execute_reply":"2023-11-25T17:20:05.931290Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import sys\nprint(sys.version)","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:20:05.933963Z","iopub.execute_input":"2023-11-25T17:20:05.934301Z","iopub.status.idle":"2023-11-25T17:20:05.943517Z","shell.execute_reply.started":"2023-11-25T17:20:05.934273Z","shell.execute_reply":"2023-11-25T17:20:05.942573Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import tensorflow as tf\nprint(\"Num GPUs Available: \", len(tf.config.list_physical_devices('GPU')))","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:20:05.944733Z","iopub.execute_input":"2023-11-25T17:20:05.945009Z","iopub.status.idle":"2023-11-25T17:20:18.905051Z","shell.execute_reply.started":"2023-11-25T17:20:05.944985Z","shell.execute_reply":"2023-11-25T17:20:18.903984Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-11-25T17:20:18.907809Z","iopub.execute_input":"2023-11-25T17:20:18.908797Z","iopub.status.idle":"2023-11-25T17:20:18.916689Z","shell.execute_reply.started":"2023-11-25T17:20:18.908760Z","shell.execute_reply":"2023-11-25T17:20:18.915728Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# install letstune\n!pip install letstune==0.3.0 --no-deps","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:20:18.917918Z","iopub.execute_input":"2023-11-25T17:20:18.918206Z","iopub.status.idle":"2023-11-25T17:20:21.657638Z","shell.execute_reply.started":"2023-11-25T17:20:18.918182Z","shell.execute_reply":"2023-11-25T17:20:21.656658Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import letstune\nfrom letstune import Params, rand\n\nletstune.__version__","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:20:21.659537Z","iopub.execute_input":"2023-11-25T17:20:21.659955Z","iopub.status.idle":"2023-11-25T17:20:21.701445Z","shell.execute_reply.started":"2023-11-25T17:20:21.659918Z","shell.execute_reply":"2023-11-25T17:20:21.700424Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load the Dataset","metadata":{}},{"cell_type":"code","source":"X = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\n\nif DEBUG:\n    X = X.iloc[:100]\n\nX.head()","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:20:21.702750Z","iopub.execute_input":"2023-11-25T17:20:21.703037Z","iopub.status.idle":"2023-11-25T17:20:25.399805Z","shell.execute_reply.started":"2023-11-25T17:20:21.703013Z","shell.execute_reply":"2023-11-25T17:20:25.398813Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X.shape","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:20:25.401228Z","iopub.execute_input":"2023-11-25T17:20:25.401646Z","iopub.status.idle":"2023-11-25T17:20:25.408606Z","shell.execute_reply.started":"2023-11-25T17:20:25.401590Z","shell.execute_reply":"2023-11-25T17:20:25.407665Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_test = pd.read_csv('/kaggle/input/digit-recognizer/test.csv')\n\nif DEBUG:\n    X_test = X_test.iloc[:20]\n\nX_test.head()","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:20:25.409923Z","iopub.execute_input":"2023-11-25T17:20:25.410354Z","iopub.status.idle":"2023-11-25T17:20:27.514577Z","shell.execute_reply.started":"2023-11-25T17:20:25.410278Z","shell.execute_reply":"2023-11-25T17:20:27.513701Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_test.shape","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:20:27.519262Z","iopub.execute_input":"2023-11-25T17:20:27.519770Z","iopub.status.idle":"2023-11-25T17:20:27.525517Z","shell.execute_reply.started":"2023-11-25T17:20:27.519742Z","shell.execute_reply":"2023-11-25T17:20:27.524572Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y = X['label']\ny.head()","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:20:27.526729Z","iopub.execute_input":"2023-11-25T17:20:27.527074Z","iopub.status.idle":"2023-11-25T17:20:27.538188Z","shell.execute_reply.started":"2023-11-25T17:20:27.527047Z","shell.execute_reply":"2023-11-25T17:20:27.537303Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X.drop(['label'], axis=1, inplace=True)\nX.head()","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:20:27.539475Z","iopub.execute_input":"2023-11-25T17:20:27.539835Z","iopub.status.idle":"2023-11-25T17:20:27.575297Z","shell.execute_reply.started":"2023-11-25T17:20:27.539804Z","shell.execute_reply":"2023-11-25T17:20:27.574337Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib as mpl\nimport matplotlib.pyplot as plt","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:20:27.576412Z","iopub.execute_input":"2023-11-25T17:20:27.576704Z","iopub.status.idle":"2023-11-25T17:20:27.581224Z","shell.execute_reply.started":"2023-11-25T17:20:27.576679Z","shell.execute_reply":"2023-11-25T17:20:27.580241Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Check a sample image","metadata":{}},{"cell_type":"code","source":"some_digit = np.array(X.iloc[[0]])\nsome_digit_image = some_digit.reshape(28, 28)\n\nplt.imshow(some_digit_image, cmap = mpl.cm.binary, interpolation=\"nearest\")\nplt.axis(\"off\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:20:27.582544Z","iopub.execute_input":"2023-11-25T17:20:27.583004Z","iopub.status.idle":"2023-11-25T17:20:27.693434Z","shell.execute_reply.started":"2023-11-25T17:20:27.582963Z","shell.execute_reply":"2023-11-25T17:20:27.692194Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Augmentation","metadata":{}},{"cell_type":"markdown","source":"*Generate more training data by shifting the given images 1 pixel in up/down/left/right directions to generate 4 more training samples for each input image.*","metadata":{}},{"cell_type":"code","source":"from scipy.ndimage import interpolation","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:20:27.695256Z","iopub.execute_input":"2023-11-25T17:20:27.695819Z","iopub.status.idle":"2023-11-25T17:20:27.701212Z","shell.execute_reply.started":"2023-11-25T17:20:27.695792Z","shell.execute_reply":"2023-11-25T17:20:27.699976Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"*Shift 1 pixel down*","metadata":{}},{"cell_type":"code","source":"X_aug_down = interpolation.shift(np.array(X).reshape(-1,28,28), [0,1,0], cval=0)\nX_aug_down.shape","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:20:27.703444Z","iopub.execute_input":"2023-11-25T17:20:27.704290Z","iopub.status.idle":"2023-11-25T17:20:27.793512Z","shell.execute_reply.started":"2023-11-25T17:20:27.704246Z","shell.execute_reply":"2023-11-25T17:20:27.792797Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"*Shift 1 pixel up.*","metadata":{}},{"cell_type":"code","source":"X_aug_up = interpolation.shift(np.array(X).reshape(-1,28,28), [0,-1,0], cval=0)\nX_aug_up.shape","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:20:27.794803Z","iopub.execute_input":"2023-11-25T17:20:27.795300Z","iopub.status.idle":"2023-11-25T17:20:27.839800Z","shell.execute_reply.started":"2023-11-25T17:20:27.795274Z","shell.execute_reply":"2023-11-25T17:20:27.838823Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"*Shift 1 pixel right.*","metadata":{}},{"cell_type":"code","source":"X_aug_right = interpolation.shift(np.array(X).reshape(-1,28,28), [0,0,1], cval=0)\nX_aug_right.shape","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:20:27.841024Z","iopub.execute_input":"2023-11-25T17:20:27.841288Z","iopub.status.idle":"2023-11-25T17:20:27.888694Z","shell.execute_reply.started":"2023-11-25T17:20:27.841265Z","shell.execute_reply":"2023-11-25T17:20:27.887580Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"*Shift 1 pixel left.*","metadata":{}},{"cell_type":"code","source":"X_aug_left = interpolation.shift(np.array(X).reshape(-1,28,28), [0,0,-1], cval=0)\nX_aug_left.shape","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:20:27.889941Z","iopub.execute_input":"2023-11-25T17:20:27.890251Z","iopub.status.idle":"2023-11-25T17:20:27.936486Z","shell.execute_reply.started":"2023-11-25T17:20:27.890226Z","shell.execute_reply":"2023-11-25T17:20:27.935517Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_temp = np.concatenate((X_aug_down, X_aug_up, X_aug_right, X_aug_left))\nX_temp.shape","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:20:27.937830Z","iopub.execute_input":"2023-11-25T17:20:27.938135Z","iopub.status.idle":"2023-11-25T17:20:27.945863Z","shell.execute_reply.started":"2023-11-25T17:20:27.938109Z","shell.execute_reply":"2023-11-25T17:20:27.944831Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"*Augmented training dataset generated and stored in X_aug and y_aug.*","metadata":{}},{"cell_type":"code","source":"X_aug = np.concatenate((X, X_temp.reshape(-1, 784)))\nX_aug.shape","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:20:27.947072Z","iopub.execute_input":"2023-11-25T17:20:27.947358Z","iopub.status.idle":"2023-11-25T17:20:27.956425Z","shell.execute_reply.started":"2023-11-25T17:20:27.947335Z","shell.execute_reply":"2023-11-25T17:20:27.955439Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_aug = np.concatenate((y, y, y, y, y))\ny_aug.shape","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:20:27.957560Z","iopub.execute_input":"2023-11-25T17:20:27.957885Z","iopub.status.idle":"2023-11-25T17:20:27.967645Z","shell.execute_reply.started":"2023-11-25T17:20:27.957860Z","shell.execute_reply":"2023-11-25T17:20:27.966637Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Split into Training and Validation Sets","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\nX_train, X_val, y_train, y_val = train_test_split(X_aug, y_aug, test_size=0.2, random_state=42)","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:20:27.968770Z","iopub.execute_input":"2023-11-25T17:20:27.969055Z","iopub.status.idle":"2023-11-25T17:20:28.359429Z","shell.execute_reply.started":"2023-11-25T17:20:27.969031Z","shell.execute_reply":"2023-11-25T17:20:28.358618Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Keras model and Params\n\n**Params** is an object with hyper-parameters.\n\nWe define a params with a new Python class, which inherits from\n[letstune.Params](https://letstune.readthedocs.io/en/latest/params.html).\n\nIt's similar to Python [dataclasses](https://docs.python.org/3/library/dataclasses.html).\n","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow import keras\n\ntf.__version__","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:20:28.360565Z","iopub.execute_input":"2023-11-25T17:20:28.360903Z","iopub.status.idle":"2023-11-25T17:20:28.368123Z","shell.execute_reply.started":"2023-11-25T17:20:28.360877Z","shell.execute_reply":"2023-11-25T17:20:28.367053Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from functools import partial","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:20:28.369470Z","iopub.execute_input":"2023-11-25T17:20:28.369873Z","iopub.status.idle":"2023-11-25T17:20:28.376778Z","shell.execute_reply.started":"2023-11-25T17:20:28.369829Z","shell.execute_reply":"2023-11-25T17:20:28.375808Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The **ImageNet Large Scale Visual Recognition Challenge (ILSVRC)** 2015 challenge was won using a ***Residual Network (or ResNet)***. The key to being able to train such a deep network is to use ***skip connections (also called shortcut connections)***: the signal feeding into a layer is also added to the output of a layer located a bit higher up the stack. If we add the input x to the output of the network (i.e., we add a skip connection), then the network will be forced to model ***f(x) = h(x) - x rather than h(x)***. This is called residual learning.","metadata":{}},{"cell_type":"markdown","source":"When we initialize a regular neural network, its weights are close to zero, so the network just outputs values close to zero. If we add a skip connection, the resulting network just outputs a copy of its inputs. If the target function is fairly close, this will speed up the training considerably. Moreoever, if we add many skip connections, the network can start making progress even if several layers have not started learning yet. Thanks to skip connections, the signal can easily make its way across the whole network.","metadata":{}},{"cell_type":"markdown","source":"**ResNet-34** is the ResNet with 34 layers (only counting convolutional layers and the fully connected layer) containing 3 residual units that output 64 feature maps, 4 residual units with 128 maps, 6 residual units with 256 maps, and 3 residual units with 512 maps.","metadata":{}},{"cell_type":"code","source":"DefaultConv2D = partial(keras.layers.Conv2D, kernel_size=3, strides=1, padding=\"SAME\", use_bias=False)","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:20:28.377983Z","iopub.execute_input":"2023-11-25T17:20:28.378369Z","iopub.status.idle":"2023-11-25T17:20:28.495941Z","shell.execute_reply.started":"2023-11-25T17:20:28.378335Z","shell.execute_reply":"2023-11-25T17:20:28.495108Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class ResidualUnit(keras.layers.Layer):\n    def __init__(self, filters, strides=1, activation=\"relu\", kernel_regularizer=None, **kwargs):\n        super().__init__(**kwargs)\n        self.activation = keras.activations.get(activation)\n        self.main_layers = [\n            DefaultConv2D(filters, strides=strides, kernel_regularizer=kernel_regularizer),\n            keras.layers.BatchNormalization(),\n            self.activation,\n            DefaultConv2D(filters, kernel_regularizer=kernel_regularizer),\n            keras.layers.BatchNormalization()]\n        self.skip_layers = []\n        if strides > 1:\n            self.skip_layers = [\n                DefaultConv2D(filters, kernel_size=1, strides=strides, kernel_regularizer=kernel_regularizer),\n                keras.layers.BatchNormalization()\n            ]\n            \n    def call(self, inputs):\n        Z = inputs\n        for layer in self.main_layers:\n            Z = layer(Z)\n        skip_Z = inputs\n        for layer in self.skip_layers:\n            skip_Z = layer(skip_Z)\n        return self.activation(Z + skip_Z)","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:20:28.497075Z","iopub.execute_input":"2023-11-25T17:20:28.497401Z","iopub.status.idle":"2023-11-25T17:20:28.507465Z","shell.execute_reply.started":"2023-11-25T17:20:28.497363Z","shell.execute_reply":"2023-11-25T17:20:28.506343Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Below are defined letstune **Params**. The final ``ResNet34Params`` are composed from other params.","metadata":{}},{"cell_type":"code","source":"class AdamOptimizerParams(Params):\n    # model_cls ask letstune to provide create_model() method,\n    # returning an Adam optimizer\n    #\n    # cls = class!\n    model_cls = keras.optimizers.Adam\n    \n    learning_rate: float = rand.uniform(0.001, 0.01, log=True)\n\n        \nclass L2Regularizer(Params):\n    model_cls = keras.regularizers.L2\n    \n    l2: float = rand.uniform(0.001, 0.01, log=True)\n    \n        \nclass ResNet34Params(Params):\n    optimizer_params: AdamOptimizerParams\n    regularizer_params: L2Regularizer\n    \n    \n    # Params are just a Python class, so we can define a method\n    def create_model(self):        \n        # uses Adam optimizer from model_cls\n        kernel_regularizer=self.regularizer_params.create_model()\n        \n        model = keras.models.Sequential()\n        model.add(DefaultConv2D(\n            64, kernel_size=7, strides=2,\n            input_shape=[28, 28, 1], \n            kernel_regularizer=kernel_regularizer,\n        ))\n        model.add(keras.layers.BatchNormalization())\n        model.add(keras.layers.Activation(\"relu\"))\n        model.add(keras.layers.MaxPool2D(pool_size=3, strides=2, padding=\"SAME\"))\n        prev_filters = 64\n        for filters in [64] * 3 + [128] * 4 + [256] * 6 + [512] * 3:\n            strides = 1 if filters == prev_filters else 2\n            model.add(ResidualUnit(filters, strides=strides, kernel_regularizer=kernel_regularizer))\n            prev_filters = filters\n        model.add(keras.layers.GlobalAvgPool2D())\n        model.add(keras.layers.Flatten())\n        model.add(keras.layers.Dense(10, activation=\"softmax\"))\n        \n        optimizer = self.optimizer_params.create_model()\n        \n        model.compile(\n            loss=\"sparse_categorical_crossentropy\",\n            optimizer=optimizer,\n            metrics=[\"accuracy\"],\n        )\n        \n        return model","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:20:28.514171Z","iopub.execute_input":"2023-11-25T17:20:28.514499Z","iopub.status.idle":"2023-11-25T17:20:28.763452Z","shell.execute_reply.started":"2023-11-25T17:20:28.514471Z","shell.execute_reply":"2023-11-25T17:20:28.762069Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Trainer\n\n**Trainer** is a class, which knows how to train a model!\n\nSee [keras mnist example](https://github.com/mslapek/letstune/blob/main/examples/keras/simple_mnist_epochtrainer.py)\nin *letstune* GIT repo.\n\n[Trainer docs](https://letstune.readthedocs.io/en/latest/trainer.html) describe the trainer API.","metadata":{}},{"cell_type":"code","source":"class MNISTTrainer(letstune.EpochTrainer):\n    # params_cls has our params class, which we defined in a previous cell\n    params_cls = ResNet34Params\n    metric = \"val_accuracy\"\n\n    \n    def load_dataset(self, dataset):\n        (self.X_train, self.y_train, self.X_val, self.y_val) = dataset\n\n    def train_epoch(self, epoch):\n        # default implementation of create_model\n        # sets self.model to params.create_model()\n\n        history = self.model.fit(\n            self.X_train,\n            self.y_train,\n            batch_size=256,\n            validation_data=(self.X_val, self.y_val),\n            epochs=epoch + 1,\n            initial_epoch=epoch,\n            verbose=0,\n        )\n\n        return {m: v[0] for m, v in history.history.items()}\n\n    def save(self, checkpoint):\n        # by default checkpoint saves in the local directory\n        # checkpoint can store models in other places, like AWS S3 bucket\n        checkpoint.save_keras(self.model)\n\n    def load(self, checkpoint, params):\n        self.model = checkpoint.load_keras()","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:20:28.764826Z","iopub.execute_input":"2023-11-25T17:20:28.765179Z","iopub.status.idle":"2023-11-25T17:20:28.779587Z","shell.execute_reply.started":"2023-11-25T17:20:28.765139Z","shell.execute_reply":"2023-11-25T17:20:28.778632Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"trainer = MNISTTrainer()","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:20:28.780811Z","iopub.execute_input":"2023-11-25T17:20:28.781117Z","iopub.status.idle":"2023-11-25T17:20:28.793549Z","shell.execute_reply.started":"2023-11-25T17:20:28.781090Z","shell.execute_reply":"2023-11-25T17:20:28.792520Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Tuning\n\nTuning is started with function [letstune.tune](https://letstune.readthedocs.io/en/latest/tune.html).\n\nIt returns [TuningResults](https://letstune.readthedocs.io/en/latest/results/epoch.html)\nobject with useful stats.\n\nThe training will take a few rounds - the schedule is calculated below.\nTo the next round get only 25% of the best performers.","metadata":{}},{"cell_type":"code","source":"from datetime import timedelta\n\nif DEBUG:\n    rounds=[\n        timedelta(seconds=10),\n        timedelta(seconds=20),\n    ]\n    \n    params_number = 3\nelse:\n    rounds=[\n        timedelta(minutes=1),\n        timedelta(minutes=2),\n        timedelta(minutes=2),\n    ]\n    \n    params_number = 16\n    \nprint(\n    f\"Will train {params_number} params with rounds {rounds}\"\n)","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:20:28.794919Z","iopub.execute_input":"2023-11-25T17:20:28.795384Z","iopub.status.idle":"2023-11-25T17:20:28.805674Z","shell.execute_reply.started":"2023-11-25T17:20:28.795341Z","shell.execute_reply":"2023-11-25T17:20:28.804790Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import math\n\nprint(\"== Training schedule ==\")\n\nsim_params_number = params_number\n\nfor i, round_duration in enumerate(rounds):\n    print(f\"Round {i} will run with {sim_params_number:2} parameters for {round_duration}\")\n    sim_params_number = math.ceil(sim_params_number / 4)\n\ndel sim_params_number","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:20:28.806882Z","iopub.execute_input":"2023-11-25T17:20:28.807182Z","iopub.status.idle":"2023-11-25T17:20:28.817218Z","shell.execute_reply.started":"2023-11-25T17:20:28.807140Z","shell.execute_reply":"2023-11-25T17:20:28.816311Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from pathlib import Path\n\n\nresults_dir = Path('/kaggle/temp/tuning')\nresults_dir.mkdir(parents=True, exist_ok=True)\n    \ntuning = letstune.tune(\n    trainer,\n    params_number=params_number,\n    results_dir=results_dir,\n    dataset=(\n        X_train.reshape(-1, 28, 28, 1),\n        y_train,\n        X_val.reshape(-1, 28, 28, 1),\n        y_val,\n    ),\n    passthrough_errors=DEBUG,\n    rounds=rounds,\n)","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:20:28.818260Z","iopub.execute_input":"2023-11-25T17:20:28.818542Z","iopub.status.idle":"2023-11-25T17:23:11.815566Z","shell.execute_reply.started":"2023-11-25T17:20:28.818517Z","shell.execute_reply":"2023-11-25T17:23:11.814612Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tuning","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:23:11.817020Z","iopub.execute_input":"2023-11-25T17:23:11.817763Z","iopub.status.idle":"2023-11-25T17:23:11.824361Z","shell.execute_reply.started":"2023-11-25T17:23:11.817725Z","shell.execute_reply":"2023-11-25T17:23:11.823412Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"All experiments are stored persistently on the disk, so we can easily resume the experiment or load the best model.\n\n* *letstune* saves experiment metadata in `letstune.db` SQLite database.\n* `checkpoints` have saved models.\n\n*letstune* architecture supports the cloud, ☁️ metadata could be saved in PostgreSQL and checkpoints in blob storage, like AWS S3.","metadata":{}},{"cell_type":"code","source":"!ls -lh /kaggle/temp/tuning","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:23:11.826080Z","iopub.execute_input":"2023-11-25T17:23:11.826368Z","iopub.status.idle":"2023-11-25T17:23:12.869093Z","shell.execute_reply.started":"2023-11-25T17:23:11.826337Z","shell.execute_reply":"2023-11-25T17:23:12.867995Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!ls -lh /kaggle/temp/tuning/checkpoints","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:23:12.870864Z","iopub.execute_input":"2023-11-25T17:23:12.871824Z","iopub.status.idle":"2023-11-25T17:23:13.896480Z","shell.execute_reply.started":"2023-11-25T17:23:12.871785Z","shell.execute_reply":"2023-11-25T17:23:13.895454Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!tree /kaggle/temp/tuning","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:28:38.029914Z","iopub.execute_input":"2023-11-25T17:28:38.030746Z","iopub.status.idle":"2023-11-25T17:28:39.059397Z","shell.execute_reply.started":"2023-11-25T17:28:38.030707Z","shell.execute_reply":"2023-11-25T17:28:39.058291Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# copying the database, so we can download it\n!cp /kaggle/temp/tuning/letstune.db .","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:23:13.898234Z","iopub.execute_input":"2023-11-25T17:23:13.899071Z","iopub.status.idle":"2023-11-25T17:23:14.952923Z","shell.execute_reply.started":"2023-11-25T17:23:13.899034Z","shell.execute_reply":"2023-11-25T17:23:14.951768Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's have a peek into **the database**! 👀\n\nIn the next section, we will see how to access the DB with *letstune* training results objects!","metadata":{}},{"cell_type":"code","source":"import sqlite3\n\ncon = sqlite3.connect(\"letstune.db\")\ncur = con.cursor()\n\nprint(\"== all tables ==\")\nfor t in con.execute(\"SELECT * FROM sqlite_master where type='table'\").fetchall():\n    print(t)\n\nprint(\"\\n\\n== trainings ==\")\nfor t in cur.execute(\"SELECT * FROM trainings\").fetchall():\n    print(t)\n    \ncon.close()","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:27:25.417382Z","iopub.execute_input":"2023-11-25T17:27:25.418187Z","iopub.status.idle":"2023-11-25T17:27:25.425722Z","shell.execute_reply.started":"2023-11-25T17:27:25.418150Z","shell.execute_reply":"2023-11-25T17:27:25.424645Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Don't worry! 🤯 We do not have to make SQL SELECT statements manually - *letstune* has build-in methods to query such data.","metadata":{}},{"cell_type":"markdown","source":"# Tuning results overview\n\n*letstune* summarizes tuning results in an Object Oriented way -\nwe get `tuning` with objects describing the experiment.\n\nThis can be used for:\n\n- visualization\n- analysis of parameter impact on model quality\n- best model loading\n\nWe can get a quick summary by just printing tuning object (watch out for error count!):","metadata":{}},{"cell_type":"code","source":"tuning","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:30:59.731837Z","iopub.execute_input":"2023-11-25T17:30:59.732753Z","iopub.status.idle":"2023-11-25T17:30:59.739795Z","shell.execute_reply.started":"2023-11-25T17:30:59.732714Z","shell.execute_reply":"2023-11-25T17:30:59.738816Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"With `to_df()` we get a quick Pandas summary:","metadata":{}},{"cell_type":"code","source":"pd.set_option(\"display.max_columns\", 200)\n\ntuning_df = tuning.to_df()\ntuning_df['mean_epoch_duration'] = tuning_df['duration'] / tuning_df['epochs']\n\ntuning_df.to_csv(f'tuning.csv', index=False)\ntuning_df","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:31:01.058984Z","iopub.execute_input":"2023-11-25T17:31:01.059403Z","iopub.status.idle":"2023-11-25T17:31:01.095161Z","shell.execute_reply.started":"2023-11-25T17:31:01.059360Z","shell.execute_reply":"2023-11-25T17:31:01.094187Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# *letstune* plot\n\nThe next plot shows how *letstune* sees our experiment.\n\nWe will see, how each sampled parameter did during the training. Only the strongest one will survive to the end.\n\nTo the highest round were assigned only the best performers.\nTrainings from lower rounds have less time assigned to them.","metadata":{}},{"cell_type":"code","source":"import plotly.express as px\n\nMINUTE = timedelta(minutes=1)","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:32:10.768563Z","iopub.execute_input":"2023-11-25T17:32:10.769320Z","iopub.status.idle":"2023-11-25T17:32:12.629962Z","shell.execute_reply.started":"2023-11-25T17:32:10.769282Z","shell.execute_reply":"2023-11-25T17:32:12.629107Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dfs = []\n\ny_label = \"total \" + trainer.metric\n\nfor training in tuning:\n    training_df = pd.DataFrame([\n        {\n            \"training_id\": training.training_id,\n            y_label: epoch.total_metric_value,\n            \"time (minute)\": epoch.total_duration / MINUTE,\n            \"survived round\": training.round,\n        }\n        for epoch in training\n    ])\n    dfs.append(training_df)\n    \ndf = pd.concat(dfs)\ndf\n\nfig = px.line(\n    df,\n    x=\"time (minute)\",\n    y=y_label,\n    line_group='training_id',\n    color='survived round',\n    markers=True,\n    title=f\"Total {trainer.metric} colored by <em>letstune</em> rounds\"\n)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:32:12.631576Z","iopub.execute_input":"2023-11-25T17:32:12.632197Z","iopub.status.idle":"2023-11-25T17:32:14.255092Z","shell.execute_reply.started":"2023-11-25T17:32:12.632167Z","shell.execute_reply":"2023-11-25T17:32:14.254003Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's see overfitting in action:","metadata":{}},{"cell_type":"code","source":"dfs = []\n\nfor training in tuning:\n    for fold in ['valid', 'train']:\n        training_df = pd.DataFrame([\n            {\n                \"training_id\": training.training_id,\n                \"loss\": epoch.metric_values['val_accuracy' if fold == 'valid' else 'accuracy'],\n                'fold': fold,\n                \"time (minute)\": epoch.total_duration / MINUTE,\n                \"survived round\": training.round,\n            }\n            for epoch in training\n        ])\n        dfs.append(training_df)\n    \ndf = pd.concat(dfs)\ndf\n\nfig = px.line(\n    df,\n    x=\"time (minute)\",\n    y='loss',\n    line_group='training_id',\n    color='survived round',\n    line_dash='fold',\n    markers=True,\n    title=f\"Accuracy colored by <em>letstune</em> rounds\"\n)\nfig.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:32:14.256793Z","iopub.execute_input":"2023-11-25T17:32:14.257082Z","iopub.status.idle":"2023-11-25T17:32:14.356787Z","shell.execute_reply.started":"2023-11-25T17:32:14.257058Z","shell.execute_reply":"2023-11-25T17:32:14.355649Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Parameter impact\n\nWe will compare\nhow differs parameter distribution\nbetween \n\n1. trainings quickly dropped (round 0),\n2. other trainings, which survived round 0.\n\nThis will help to choose parameter values for the next experiment.\n\nIt gives an insight into how the neural network behaves.","metadata":{}},{"cell_type":"code","source":"import seaborn as sns\n\ntuning_df.index = tuning_df.index.rename('rank')\ntuning_df = tuning_df.reset_index()\n\ntuning_df['survived round 0'] = tuning_df['round'] > 0\n\ntuning_df['log10(lr)'] = np.log10(tuning_df['params.optimizer_params.learning_rate'])\ntuning_df['log10(l2)'] = np.log10(tuning_df['params.regularizer_params.l2'])\n","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:32:14.360565Z","iopub.execute_input":"2023-11-25T17:32:14.360956Z","iopub.status.idle":"2023-11-25T17:32:14.559471Z","shell.execute_reply.started":"2023-11-25T17:32:14.360929Z","shell.execute_reply":"2023-11-25T17:32:14.558638Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Learning rate","metadata":{}},{"cell_type":"code","source":"sns.displot(\n    data=tuning_df,\n    x='log10(lr)',\n    hue='survived round 0',\n    stat='percent',\n    common_norm=False,\n    kde=True,\n    row='survived round 0',\n).set(title='Impact of learning rate on survivorship')\n","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:32:14.560815Z","iopub.execute_input":"2023-11-25T17:32:14.561198Z","iopub.status.idle":"2023-11-25T17:32:15.807270Z","shell.execute_reply.started":"2023-11-25T17:32:14.561163Z","shell.execute_reply":"2023-11-25T17:32:15.806284Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.relplot(\n    data=tuning_df,\n    y='log10(lr)',\n    x='rank',\n).set(title='Learning rate vs rank')\n\nprint(\"Lower rank is better\")","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:32:15.808713Z","iopub.execute_input":"2023-11-25T17:32:15.809082Z","iopub.status.idle":"2023-11-25T17:32:16.312634Z","shell.execute_reply.started":"2023-11-25T17:32:15.809049Z","shell.execute_reply":"2023-11-25T17:32:16.311646Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Regularization","metadata":{}},{"cell_type":"code","source":"sns.displot(\n    data=tuning_df,\n    x='log10(l2)',\n    hue='survived round 0',\n    stat='percent',\n    common_norm=False,\n    kde=True,\n    row='survived round 0',\n).set(title='Impact of regularization on survivorship')\n","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:32:16.313879Z","iopub.execute_input":"2023-11-25T17:32:16.314182Z","iopub.status.idle":"2023-11-25T17:32:17.523333Z","shell.execute_reply.started":"2023-11-25T17:32:16.314156Z","shell.execute_reply":"2023-11-25T17:32:17.522183Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.relplot(\n    data=tuning_df,\n    y='log10(l2)',\n    x='rank',\n).set(title='Regularization vs rank')\n\nprint(\"Lower rank is better\")","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:32:17.526018Z","iopub.execute_input":"2023-11-25T17:32:17.526344Z","iopub.status.idle":"2023-11-25T17:32:18.007986Z","shell.execute_reply.started":"2023-11-25T17:32:17.526313Z","shell.execute_reply":"2023-11-25T17:32:18.007047Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Final prediction and CSV\n\nWith *letstune* you can load the best model **in one line of code!** 🤩","metadata":{}},{"cell_type":"code","source":"model = tuning[0].best_epoch.checkpoint.load_keras()\nmodel","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:32:18.009695Z","iopub.execute_input":"2023-11-25T17:32:18.010115Z","iopub.status.idle":"2023-11-25T17:32:24.606134Z","shell.execute_reply.started":"2023-11-25T17:32:18.010081Z","shell.execute_reply":"2023-11-25T17:32:24.605144Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predictions = np.argmax(model.predict(X_test.values.reshape(-1, 28, 28, 1)), axis=-1)\n\noutput = pd.DataFrame({'ImageId': list(range(1, len(X_test)+1)), 'Label': predictions})\noutput.to_csv('submission.csv', index=False)\nprint(\"Your submission was successfully saved!\")","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:32:24.609940Z","iopub.execute_input":"2023-11-25T17:32:24.610312Z","iopub.status.idle":"2023-11-25T17:32:25.460492Z","shell.execute_reply.started":"2023-11-25T17:32:24.610284Z","shell.execute_reply":"2023-11-25T17:32:25.459641Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!head submission.csv","metadata":{"execution":{"iopub.status.busy":"2023-11-25T17:32:25.461576Z","iopub.execute_input":"2023-11-25T17:32:25.461916Z","iopub.status.idle":"2023-11-25T17:32:26.479321Z","shell.execute_reply.started":"2023-11-25T17:32:25.461889Z","shell.execute_reply":"2023-11-25T17:32:26.478157Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Conclusions\n\nGain a 🍭 **competitive advantage** in your notebooks with *letstune*!\n\nExplore quickly 🔎 better ML configurations!\n\nIt's MIT licensed!\n\n![letstune logo](https://github.com/mslapek/letstune/raw/main/img/logo.svg)\n\n## Please help me spread the word about **letstune**!\n\n*letstune* was created to help other people - to create better products!\n\n<br/>\n\n♻️ In my first workplace, we have been working on technology, which\nwould **reduce the use** of *artificial fertilizers* and *pesticides*,\nwhich 🧪 **contaminate our rivers**. 🧪\n\n🌎 We should build an *easy-to-use* technology, that would\nempower such people! 🌍\n\n<br/>\n\nIf you want to support my mission, **please**:\n\n- give \"like\" to this notebook\n- use it in your notebooks 🍭 to give you a competitive advantage!\n    - fork this notebook\n    - create a new ones - you can use *keras*, *tensorflow*, *pytorch*, *lightgbm*, *xgboost*, *sklearn*, ...\n- if you win with *letstune*, please mention it\n- give a star ⭐️ on GitHub: https://github.com/mslapek/letstune\n\n<br/>\n\nAsk questions! I'm happy to answer them! 😉","metadata":{}}]}