{"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.11.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"colab":{"provenance":[{"file_id":"https://github.com/simon-mellergaard/ML2/blob/main/DLP/chapter09_part02_modern_convnet_architecture_patterns_i.ipynb","timestamp":1755600034999}],"toc_visible":true},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":3362,"databundleVersionId":31148,"sourceType":"competition"}],"dockerImageVersionId":31089,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Disclaimer\n\nThis notebook is based from the book [Deep Learning with Python](https://www.amazon.com/Learning-Python-Second-Fran%C3%A7ois-Chollet/dp/1617296864) chapter 9. Some of the code has been modified.\n\n![book picture](https://m.media-amazon.com/images/I/71sQM8kzchL._SY385_.jpg)","metadata":{"id":"qdG6l02FdZaP"}},{"cell_type":"markdown","source":"# Getting the data ready\nThe below block will unzip the training data. ","metadata":{}},{"cell_type":"code","source":"!unzip -qq /kaggle/input/dogs-vs-cats/train.zip","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-19T16:06:28.127661Z","iopub.execute_input":"2025-08-19T16:06:28.127938Z","iopub.status.idle":"2025-08-19T16:06:37.699988Z","shell.execute_reply.started":"2025-08-19T16:06:28.127914Z","shell.execute_reply":"2025-08-19T16:06:37.699212Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Making the validation, training and test set\n\nWe will not be using the actual test set from the competition, but rather 2000 random images from the training set. This gives smaller datasize to train on, which will give less input for the model to train on, however it increases the speed of the model to train. For a more precise model, a larger dataset is always preferable. ","metadata":{}},{"cell_type":"code","source":"import os, shutil, pathlib\nfrom tensorflow.keras.utils import image_dataset_from_directory\n\noriginal_dir = pathlib.Path(\"/kaggle/working/train\")\nnew_base_dir = pathlib.Path(\"cats_vs_dogs_small\")\n\ndef make_subset(subset_name, start_index, end_index):\n    for category in (\"cat\", \"dog\"):\n        dir = new_base_dir / subset_name / category\n        if os.path.exists(dir) and os.path.isdir(dir):\n            shutil.rmtree(dir)\n        os.makedirs(dir)\n        fnames = [f\"{category}.{i}.jpg\" for i in range(start_index, end_index)]\n        for fname in fnames:\n            shutil.copyfile(src=original_dir / fname,\n                            dst=dir / fname)\n\nmake_subset(\"train\", start_index=0, end_index=1000)\nmake_subset(\"validation\", start_index=1000, end_index=1500)\nmake_subset(\"test\", start_index=1500, end_index=2500)\n\ntrain_dataset = image_dataset_from_directory(\n    new_base_dir / \"train\",\n    image_size=(180, 180),\n    batch_size=32)\nvalidation_dataset = image_dataset_from_directory(\n    new_base_dir / \"validation\",\n    image_size=(180, 180),\n    batch_size=32)\ntest_dataset = image_dataset_from_directory(\n    new_base_dir / \"test\",\n    image_size=(180, 180),\n    batch_size=32)","metadata":{"id":"JID7fdPydZaw","outputId":"97db9648-14b1-408f-8078-f9ebf61e80a6","trusted":true,"execution":{"iopub.status.busy":"2025-08-19T16:06:42.014857Z","iopub.execute_input":"2025-08-19T16:06:42.015591Z","iopub.status.idle":"2025-08-19T16:07:00.177023Z","shell.execute_reply.started":"2025-08-19T16:06:42.015549Z","shell.execute_reply":"2025-08-19T16:07:00.176266Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Specifying a simple convolutional neural network\n\nA simple dataugmentation process is specified:","metadata":{}},{"cell_type":"code","source":"from tensorflow import keras\nfrom tensorflow.keras import layers\ndata_augmentation = keras.Sequential(\n    [\n        layers.RandomFlip(\"horizontal\"),\n        layers.RandomRotation(0.1),\n        layers.RandomZoom(0.2),\n    ]\n)","metadata":{"id":"NryzBNmddZax","trusted":true,"execution":{"iopub.status.busy":"2025-08-19T16:07:46.436175Z","iopub.execute_input":"2025-08-19T16:07:46.436949Z","iopub.status.idle":"2025-08-19T16:07:46.456332Z","shell.execute_reply.started":"2025-08-19T16:07:46.436917Z","shell.execute_reply":"2025-08-19T16:07:46.455798Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Below is the architechture of the model built. It consists of repeated layers of residual blocks, which in the end is being pooled with the global average pooling operator. It also includes dropout to fight the overfitting.","metadata":{}},{"cell_type":"code","source":"inputs = keras.Input(shape=(180, 180, 3))\nx = data_augmentation(inputs)\n\nx = layers.Rescaling(1./255)(x)\nx = layers.Conv2D(filters=32, kernel_size=5, use_bias=False)(x)\n\nfor size in [32, 64, 128, 256, 512]:\n    residual = x\n\n    x = layers.BatchNormalization()(x)\n    x = layers.Activation(\"relu\")(x)\n    x = layers.SeparableConv2D(size, 3, padding=\"same\", use_bias=False)(x)\n\n    x = layers.BatchNormalization()(x)\n    x = layers.Activation(\"relu\")(x)\n    x = layers.SeparableConv2D(size, 3, padding=\"same\", use_bias=False)(x)\n\n    x = layers.MaxPooling2D(3, strides=2, padding=\"same\")(x)\n\n    residual = layers.Conv2D(\n        size, 1, strides=2, padding=\"same\", use_bias=False)(residual)\n    x = layers.add([x, residual])\n\nx = layers.GlobalAveragePooling2D()(x)\nx = layers.Dropout(0.5)(x)\noutputs = layers.Dense(1, activation=\"sigmoid\")(x)\nmodel = keras.Model(inputs=inputs, outputs=outputs)\nmodel.summary()","metadata":{"id":"WI0hSCnFdZax","trusted":true,"execution":{"iopub.status.busy":"2025-08-19T16:07:49.029225Z","iopub.execute_input":"2025-08-19T16:07:49.029485Z","iopub.status.idle":"2025-08-19T16:07:50.005490Z","shell.execute_reply.started":"2025-08-19T16:07:49.029464Z","shell.execute_reply":"2025-08-19T16:07:50.004895Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The final model consists of 718.000 parameters, which is quite large, but many other deep NN have a lot more parameters. ","metadata":{}},{"cell_type":"markdown","source":"## Fitting the model\n\nThe model is now ready to be fitted. It will run for 100 epochs, which should be enough to see a decent result. Callbacks is used to save the best model.","metadata":{}},{"cell_type":"code","source":"model.compile(loss=\"binary_crossentropy\",\n              optimizer=\"rmsprop\",\n              metrics=[\"accuracy\"])\ncallbacks = [\n    keras.callbacks.ModelCheckpoint(\n        filepath=\"model.keras\",\n        save_best_only=True,\n        monitor=\"val_loss\")\n]\nhistory = model.fit(\n    train_dataset,\n    epochs=100,\n    validation_data=validation_dataset,\n    callbacks=callbacks)","metadata":{"id":"cBuaj6pKdZa1","outputId":"126eae68-4767-468e-8261-58ecddabe4a2","trusted":true,"execution":{"iopub.status.busy":"2025-08-19T16:07:55.071113Z","iopub.execute_input":"2025-08-19T16:07:55.071792Z","iopub.status.idle":"2025-08-19T16:28:01.841368Z","shell.execute_reply.started":"2025-08-19T16:07:55.071765Z","shell.execute_reply":"2025-08-19T16:28:01.840741Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The below function is used to plot the results from the model.","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport numpy as np\n\ndef plot_loss(history, title='', exclude_indexes=None):\n    # Extract loss and accuracy\n    history_dict = history.history\n    train_loss = np.array(history_dict['loss'])\n    val_loss = np.array(history_dict['val_loss'])\n    train_accuracy = np.array(history_dict['accuracy'])\n    val_accuracy = np.array(history_dict['val_accuracy'])\n\n    # Convert exclude_indexes to a set of valid indices\n    if exclude_indexes is None:\n        exclude_indexes = set()\n    else:\n        exclude_indexes = set(exclude_indexes)\n\n    # Build a mask to keep only desired indexes\n    all_indexes = np.arange(len(train_loss))\n    mask = np.array([i not in exclude_indexes for i in all_indexes])\n\n    # Apply mask\n    train_loss = train_loss[mask]\n    val_loss = val_loss[mask]\n    train_accuracy = train_accuracy[mask]\n    val_accuracy = val_accuracy[mask]\n    epochs = np.arange(1, len(history_dict['loss']) + 1)[mask]\n\n    # Plotting\n    plt.figure(figsize=(14, 5))\n    plt.suptitle(f'{title}', fontsize=16)\n\n    # Loss plot\n    plt.subplot(1, 2, 1)\n    plt.plot(epochs, train_loss, 'o-', label='Training Loss', color = 'purple')\n    plt.plot(epochs, val_loss, 'o-', label='Validation Loss', color = 'navy')\n    plt.title('Training and Validation Loss')\n    plt.xlabel('Epochs')\n    plt.ylabel('Loss')\n    plt.legend()\n\n    # Accuracy plot\n    plt.subplot(1, 2, 2)\n    plt.plot(epochs, train_accuracy, 'o-', label='Training Accuracy', color = 'purple')\n    plt.plot(epochs, val_accuracy, 'o-', label='Validation Accuracy', color = 'navy')\n    plt.title('Training and Validation Accuracy')\n    plt.xlabel('Epochs')\n    plt.ylabel('Accuracy')\n    plt.legend()\n\n    plt.tight_layout()\n    plt.show()\n\ndef print_best_val_loss_and_accuracy(history):\n    # Find the best model index\n    history_dict = history.history\n    best_val_loss_index = np.argmin(history_dict['val_loss'])\n    best_val_loss = history_dict['val_loss'][best_val_loss_index]\n    best_val_accuracy = history_dict['val_accuracy'][best_val_loss_index]\n\n    # Print results\n    print(f\"Best Validation Loss: {'{0:.5g}'.format(best_val_loss)} at epoch {best_val_loss_index + 1}\")\n    print(f\"Validation Accuracy at Best Loss: {'{0:.5g}'.format(100 * best_val_accuracy)}%\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-19T16:28:03.192515Z","iopub.execute_input":"2025-08-19T16:28:03.192795Z","iopub.status.idle":"2025-08-19T16:28:03.202008Z","shell.execute_reply.started":"2025-08-19T16:28:03.192764Z","shell.execute_reply":"2025-08-19T16:28:03.201257Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plot_loss(history=history, title='Classification of cats and dogs with an Xception-like model')\nprint_best_val_loss_and_accuracy(history)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-19T16:28:06.589632Z","iopub.execute_input":"2025-08-19T16:28:06.590321Z","iopub.status.idle":"2025-08-19T16:28:07.074934Z","shell.execute_reply.started":"2025-08-19T16:28:06.590294Z","shell.execute_reply":"2025-08-19T16:28:07.074236Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The best model is found at epoch 76, and it has a validation accuracy of 89.8%, which is quite decent! Let us see how it does on the testing set. The mdoel seems to be improving consistently, however the validation loss seems to stall at the last ~30 epochs, meaning that the bet model have converged. ","metadata":{}},{"cell_type":"markdown","source":"# Evaluating on the test set\n\nThe home-made test set is being tested in the following code:","metadata":{}},{"cell_type":"code","source":"test_model = keras.models.load_model(\"model.keras\")\ntest_loss, test_acc = test_model.evaluate(test_dataset)\nprint(f\"Test accuracy: {test_acc:.3f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-19T16:28:25.478010Z","iopub.execute_input":"2025-08-19T16:28:25.478828Z","iopub.status.idle":"2025-08-19T16:28:29.501333Z","shell.execute_reply.started":"2025-08-19T16:28:25.478791Z","shell.execute_reply":"2025-08-19T16:28:29.500759Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The testing accuracy is **87.7%** which is a little lower than the validation accuracy, but still quite high. ","metadata":{}},{"cell_type":"markdown","source":"# Submitting to the competition\n\nBelow will the true test set be used, which can be used to submit to the competition. The test set is being unpacked and stored in a variable. **Note that is no longer possible to upload a submission, so it is only done for demonstration purposes!**","metadata":{}},{"cell_type":"code","source":"!unzip -qq /kaggle/input/dogs-vs-cats/test1.zip","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-19T16:28:53.482605Z","iopub.execute_input":"2025-08-19T16:28:53.483237Z","iopub.status.idle":"2025-08-19T16:28:57.979375Z","shell.execute_reply.started":"2025-08-19T16:28:53.483215Z","shell.execute_reply":"2025-08-19T16:28:57.978370Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"original_dir = pathlib.Path(\"/kaggle/working/test1\")\nnew_base_dir = pathlib.Path(\"/kaggle/working/test_comp\")\ndef make_subset2(subset_name, start_index, end_index):\n    dir = new_base_dir / subset_name\n    if os.path.exists(dir) and os.path.isdir(dir):\n        shutil.rmtree(dir)\n    os.makedirs(dir)\n    fnames = [f\"{i}.jpg\" for i in range(start_index, end_index)]\n    for fname in fnames:\n        shutil.copyfile(src=original_dir / fname,\n                        dst=dir / fname)\nmake_subset2(\"test_comp\", start_index=1, end_index=12501)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-19T16:54:45.322785Z","iopub.execute_input":"2025-08-19T16:54:45.323489Z","iopub.status.idle":"2025-08-19T16:54:46.936469Z","shell.execute_reply.started":"2025-08-19T16:54:45.323464Z","shell.execute_reply":"2025-08-19T16:54:46.935854Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"dire = pathlib.Path(\"/kaggle/working/test_comp/test_comp\")\ntest_set = image_dataset_from_directory(\n    dire,\n    labels=None,\n    image_size=(180, 180),\n    batch_size=32)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-19T16:55:11.738838Z","iopub.execute_input":"2025-08-19T16:55:11.739132Z","iopub.status.idle":"2025-08-19T16:55:12.025155Z","shell.execute_reply.started":"2025-08-19T16:55:11.739109Z","shell.execute_reply":"2025-08-19T16:55:12.024307Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The test set is now ready to be predicted. The best model is loaded, and the labels are predicted for all the insered in the submission:","metadata":{}},{"cell_type":"code","source":"preds = test_model.predict(test_set)\npredicted_labels = (preds > 0.5).astype(int)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-19T17:04:30.066362Z","iopub.execute_input":"2025-08-19T17:04:30.066620Z","iopub.status.idle":"2025-08-19T17:04:44.588297Z","shell.execute_reply.started":"2025-08-19T17:04:30.066600Z","shell.execute_reply":"2025-08-19T17:04:44.587722Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import pandas as pd\npredicted_labels = predicted_labels.tolist()\nindex = [i for i in range(1, 12501, 1)]\noutput = pd.DataFrame({'id': index,\n                       'label': predicted_labels})\noutput.to_csv('submission.csv', index=False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-19T17:08:13.574548Z","iopub.execute_input":"2025-08-19T17:08:13.574847Z","iopub.status.idle":"2025-08-19T17:08:13.598231Z","shell.execute_reply.started":"2025-08-19T17:08:13.574823Z","shell.execute_reply":"2025-08-19T17:08:13.597426Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"And that's it! We have now made a submission. It is not good enough to earn the first place in the competition, but nevertheless it is still a good starting point for making a robust neural network. ","metadata":{}}]}