{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import os\nos.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'\n\n#import important libraries\nimport numpy as np \nimport pandas as pd \nimport random\nfrom keras.datasets import cifar10\n\n# import libraries for EDA and preprocessing\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.utils import np_utils\n\n# Modeling\nfrom keras.models import *\nfrom keras.layers import *\nfrom keras.optimizers import Adam\nfrom keras.callbacks import EarlyStopping, ModelCheckpoint, ReduceLROnPlateau\nfrom tensorflow.keras.utils import plot_model\n\n# Evaluating\nfrom sklearn.metrics import roc_curve, auc, roc_auc_score\nfrom sklearn.metrics import classification_report,confusion_matrix","metadata":{"execution":{"iopub.execute_input":"2023-02-25T13:25:13.924677Z","iopub.status.busy":"2023-02-25T13:25:13.924256Z","iopub.status.idle":"2023-02-25T13:25:20.687647Z","shell.execute_reply":"2023-02-25T13:25:20.686718Z","shell.execute_reply.started":"2023-02-25T13:25:13.924579Z"},"id":"Yy2kngqr5qyL"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Part 1: Brief description of the problem and data ","metadata":{"id":"ldGU6ZiW5qyV"}},{"cell_type":"markdown","source":"### 1.1 Problem","metadata":{"id":"IaM5NOx35qyb"}},{"cell_type":"markdown","source":"Deep learning is a type of machine learning based on artificial neural networks that teaches computers to do what comes naturally to humans: learn by example. In deep learning, a computer model learns to perform classification tasks directly from images, text, or sound. Image classification is one of the most important applications of deep learning, refers to assigning labels to images based on certain characteristics or features present in them. The algorithm identifies these features and uses them to differentiate different images and assign labels to them. In this project, my goal is to classify images from CIFAR-10 dataset by training Convolutional Neural Networks (CNNs). To achieve this goal, I will:   \n+ Load in the data.     \n+ EDA - Inspect, Visualize, and Clean the data.    \n+ Building and training models:   \n    + Model 1: CNN with two layers\n    + Model 2: CNN with two layers adding dropout  \n    + Model 3: CNN with three layers adding dropout\n    + Model 4: CNN with three layers adding Earlystopping and ReduceLROnPlateau\n    + Model 5: Resnet50 Model with \"Imagenet\" pretrained weights\n    + Model 6: Resnet101 Model with \"Imagenet\" pretrained weights\n+ Results and Analysis, compare 6 models.\n+ Build the best model with tuning hyperparameters.\n+ Conclusion.   \n\nHere, I will compare these six deep learning models by roc_auc_score because it is generally seen as a more important measure of how good an algorithm is. This metric considers the trade-offs between precision and recall, while Accuracy only looks at how many predictions are correct. It tells us what is the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance. Since we care about true negatives as much as we care about true positives then it totally makes sense to use roc_auc_score.\n\nReference Sources:   \n(1) https://www.tensorflow.org/api_docs/python/tf/keras/datasets/cifar10/load_data   \n(2) https://neptune.ai/blog/f1-score-accuracy-roc-auc-pr-auc","metadata":{"id":"Dn0hvCD45qyd"}},{"cell_type":"markdown","source":"### 1.2 Data","metadata":{"id":"3q88gQlJ5qyg"}},{"cell_type":"markdown","source":"In this project, I use CIFAR-10 dataset from keras datasets module.   \n\nCIFAR-10 is a collection of images used to train Machine Learning and Computer Vision algorithms, used for object recognition. It is a subset of the 80 million tiny images dataset and consists of 60,000 32x32 color images containing one of 10 object classes, with 6000 images per class. It was collected by Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. \n\nThere are 50,000 training images and 10,000 test images, labeled over 10 categories in the official data. \n\nThe label classes in the dataset are:\n\n![image.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAVkAAAGaCAYAAACljjDBAAAKs2lDQ1BJQ0MgUHJvZmlsZQAASImVlwdUk9kSgOf/UwkJARIiICX0JkgngJTQAwhIB1EJSYBQQgwEFbuyuIJrQUQEy4quiii4FkDWigUrig0LuiCLiLouFmxY3g8cgrvvvPfOm3PumS+TuTNz77lzzvwANCZfKs1CVQGyJXmyyEAfdnxCIpvUByhoAAlsQZ0vyJVyIyJCAZMx/Xd5dweQYX3TejjWv///X0VNKMoVACARGKcIcwXZGB/G1kuBVJYHgNuJ2Y3m5EmH+QLGTBlWIMadw5w2ygPDnDLCePyIT3SkL8aaAGQqny9LA6AaY3Z2viANi0P1w9hWIhRLMMZ+g2d2do4QYywvmGM+UoyH43NSvouT9reYKYqYfH6agkfPMiJkP3GuNIs/7/+8jv8t2VnysRym2KKmy4IiMc3C7uxuZk6IgiUpYeFjLBaO+I9wujwoZowFub6JYyzk+4Uo9maFhY5xqjiAp4iTx4seY1Guf9QYy3IiFblSZb7cMebLxvPKM2MU9nQRTxG/ID06bozzxbFhY5ybGRUy7uOrsMvkkYr6RZJAn/G8AYqzZ+d+d14xT7E3Lz06SHF2/nj9Igl3PGZuvKI2ocjPf9wnRuEvzfNR5JJmRSj8RVmBCntufpRibx72IMf3RijuMIMfHDHGEAr+EAZsiAB7cIF0sAY7rP8gTzR3+I2Cb450nkyclp7H5mJdJmLzJAKbSWx7W3sHgOGeHX0SbyJHehFhnRi35WA9xHmH9cm6cVtKGUBjEYDm/XGb8VYAeiFAQ4tALssftQ23ExCAAnRgghbogRGYY5XZgzO4gzdWcTCEQzQkwEwQYDVngwzmwAJYCkVQAmthA1TCNtgBe2A/HIRGOAan4TxchutwGx5AF/TCcxiAdzCEIAgJoSEMRAvRR0wQK8Qe4SCeiD8SikQiCUgykoZIEDmyAFmOlCClSCWyHalBfkWOIqeRi0g7cg/pRvqR18gnFIdSUSaqi5qik1EOykVD0Gh0BpqGzkYL0EJ0NVqBVqP70Ab0NHoZvY12oc/RQRzglHEsnAHOGsfB+eLCcYm4VJwMtwhXjCvHVePqcM24VtxNXBfuBe4jnohn4Nl4a7w7PggfgxfgZ+MX4VfhK/F78A34s/ib+G78AP4rgUbQIVgR3Ag8QjwhjTCHUEQoJ+wiHCGcI9wm9BLeEYlEFtGM6EIMIiYQM4jziauIW4j1xFPEdmIPcZBEImmRrEgepHASn5RHKiJtIu0jnSTdIPWSPpCVyfpke3IAOZEsIS8jl5P3kk+Qb5D7yENKqkomSm5K4UpCpXlKa5R2KjUrXVPqVRqiqFHMKB6UaEoGZSmlglJHOUfppLxRVlY2VHZVnqYsVl6iXKF8QPmCcrfyR6o61ZLqS02iyqmrqbupp6j3qG9oNJopzZuWSMujrabV0M7QHtE+qDBUbFR4KkKVxSpVKg0qN1Re0pXoJnQufSa9gF5OP0S/Rn+hqqRqquqrylddpFqlelS1Q3VQjaFmpxaulq22Sm2v2kW1p+okdVN1f3WheqH6DvUz6j0MHMOI4csQMJYzdjLOMXqZRKYZk8fMYJYw9zPbmAMa6hqOGrEaczWqNI5rdLFwLFMWj5XFWsM6yLrD+jRBdwJ3gmjCygl1E25MeK85UdNbU6RZrFmveVvzkxZby18rU2udVqPWQ228tqX2NO052lu1z2m/mMic6D5RMLF44sGJ93VQHUudSJ35Ojt0rugM6urpBupKdTfpntF9ocfS89bL0CvTO6HXr8/Q99QX65fpn9R/xtZgc9lZ7Ar2WfaAgY5BkIHcYLtBm8GQoZlhjOEyw3rDh0YUI45RqlGZUYvRgLG+8VTjBca1xvdNlEw4JukmG01aTd6bmpnGma4wbTR9aqZpxjMrMKs16zSnmXuZzzavNr9lQbTgWGRabLG4bolaOlmmW1ZZXrNCrZytxFZbrNonESa5TpJMqp7UYU215lrnW9dad9uwbEJtltk02rycbDw5cfK6ya2Tv9o62WbZ7rR9YKduF2y3zK7Z7rW9pb3Avsr+lgPNIcBhsUOTwytHK0eR41bHu04Mp6lOK5xanL44uzjLnOuc+12MXZJdNrt0cJicCM4qzgVXgquP62LXY64f3Zzd8twOuv3lbu2e6b7X/ekUsymiKTun9HgYevA9tnt0ebI9kz1/9uzyMvDie1V7PfY28hZ67/Lu41pwM7j7uC99bH1kPkd83vu6+S70PeWH8wv0K/Zr81f3j/Gv9H8UYBiQFlAbMBDoFDg/8FQQISgkaF1QB0+XJ+DV8AaCXYIXBp8NoYZEhVSGPA61DJWFNk9FpwZPXT+1M8wkTBLWGA7hvPD14Q8jzCJmR/w2jTgtYlrVtCeRdpELIlujGFGzovZGvYv2iV4T/SDGPEYe0xJLj02KrYl9H+cXVxrXFT85fmH85QTtBHFCUyIpMTZxV+LgdP/pG6b3JjklFSXdmWE2Y+6MizO1Z2bNPD6LPos/61AyITkueW/yZ344v5o/mMJL2ZwyIPAVbBQ8F3oLy4T9Ig9Rqagv1SO1NPVpmkfa+rT+dK/08vQXYl9xpfhVRlDGtoz3meGZuzO/ZcVl1WeTs5Ozj0rUJZmSszl6OXNz2qVW0iJp12y32RtmD8hCZLtykdwZuU15TGw4uiI3l/8g7873zK/K/zAnds6huWpzJXOvzLOct3JeX0FAwS/z8fMF81sWGCxYuqB7IXfh9kXIopRFLYuNFhcu7l0SuGTPUsrSzKVXl9kuK132dnnc8uZC3cIlhT0/BP5QW6RSJCvqWOG+YtuP+B/FP7atdFi5aeXXYmHxpRLbkvKSz6sEqy79ZPdTxU/fVqeublvjvGbrWuJaydo767zW7SlVKy0o7Vk/dX1DGbusuOzthlkbLpY7lm/bSNko39hVEVrRtMl409pNnyvTK29X+VTVb9bZvHLz+y3CLTe2em+t26a7rWTbp5/FP9/dHri9odq0unwHcUf+jic7Y3e2/sL5pWaX9q6SXV92S3Z37Yncc7bGpaZmr87eNbVorby2f1/Svuv7/fY31VnXba9n1ZccgAPyA89+Tf71zsGQgy2HOIfqDpsc3nyEcaS4AWmY1zDQmN7Y1ZTQ1H40+GhLs3vzkd9sftt9zOBY1XGN42tOUE4Unvh2suDk4CnpqRen0073tMxqeXAm/syts9POtp0LOXfhfMD5M63c1pMXPC4cu+h28eglzqXGy86XG644XTly1enqkTbntoZrLtearrteb26f0n7ihteN0zf9bp6/xbt1+XbY7fY7MXfudiR1dN0V3n16L+veq/v594ceLOkkdBY/VH1Y/kjnUfXvFr/Xdzl3He/2677yOOrxgx5Bz/M/cv/43Fv4hPakvE+/r+ap/dNj/QH9159Nf9b7XPp86EXRn2p/bn5p/vLwX95/XRmIH+h9JXv17fWqN1pvdr91fNsyGDH46F32u6H3xR+0Puz5yPnY+inuU9/QnM+kzxVfLL40fw352vkt+9s3KV/GHxkFcNhCU1MBXu8GoCUAMK4DUKaPztQjgox+B4wQ/CcenbtHxBlgRwdA9HyA0KsAmyqxkRaLT08CiKBjdndAHRwUa2z+HZnVh8W2DoBfH28f5tBJdoV/yugc/13d/9QwHNUR/qn/BamcB6+0KOQuAAAAVmVYSWZNTQAqAAAACAABh2kABAAAAAEAAAAaAAAAAAADkoYABwAAABIAAABEoAIABAAAAAEAAAFZoAMABAAAAAEAAAGaAAAAAEFTQ0lJAAAAU2NyZWVuc2hvdCSk4lwAAAHWaVRYdFhNTDpjb20uYWRvYmUueG1wAAAAAAA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4OnhtcHRrPSJYTVAgQ29yZSA2LjAuMCI+CiAgIDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+CiAgICAgIDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiCiAgICAgICAgICAgIHhtbG5zOmV4aWY9Imh0dHA6Ly9ucy5hZG9iZS5jb20vZXhpZi8xLjAvIj4KICAgICAgICAgPGV4aWY6UGl4ZWxZRGltZW5zaW9uPjQxMDwvZXhpZjpQaXhlbFlEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlBpeGVsWERpbWVuc2lvbj4zNDU8L2V4aWY6UGl4ZWxYRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpVc2VyQ29tbWVudD5TY3JlZW5zaG90PC9leGlmOlVzZXJDb21tZW50PgogICAgICA8L3JkZjpEZXNjcmlwdGlvbj4KICAgPC9yZGY6UkRGPgo8L3g6eG1wbWV0YT4KDKa0+AAAQABJREFUeAHtnQd4VEUXhr8QAoQWqlFKIICCQCihNynSi/QmRQSVJgiiqEixKyq9hgAK0nsV6b2HAEloCQFCb5qekMb+M+O/S0ISsrvZza673zxPsvfeOXNm5p2bb+eemd04aER68OgfMJEACZAACZieQDbTu6RHEiABEiABLQGKrJYEX0mABEjADAQosmaASpckQAIkoCVAkdWS4CsJkAAJmIEARdYMUOmSBEiABLQEKLJaEnwlARIgATMQoMiaASpdkgAJkICWAEVWS4KvJEACJGAGAhRZM0ClSxIgARLQEqDIaknwlQRIgATMQIAiawaodEkCJEACWgIUWS0JvpIACZCAGQhQZM0AlS5JgARIQEuAIqslwVcSIAESMAMBiqwZoNIlCZAACWgJUGS1JPhKAiRAAmYgQJE1A1S6JAESIAEtAYqsloSer+Hh4Vjo7Y0zZ87oWSKl2alTp1T5qKiolBk8IwEbIvD06VM8fvzY6B5ltrzRFZuhoF2KrL+fH5o2aYI/li41GKkS2YULjRfZkyexUJSPpsgazJ4F0icw5uOP1T0t7+vOnTrh22++wZEjR9IvYOacefPmoX27drh3755eNd2+fRtXr17V2RpaXlfQCg+yW2GbzN6kpxoNYmNjkZCYaPa6WAEJZAWB+Ph4PHnyBB+NGgX5lHTyxAl8+sknmDBxItq2bZsVTUhRR5s2bVCwQAG4urqmuJ7eyc+TJ+PSpUvYvWePMjG0fHp+reG6XYqsPuD/+ecfnDh+HGFhYajfoAFKly6dqtjBgwdx/do1vF6xIurUqaPLF/+bEpcvX1azXRcXFzRt2hR58+bV5fOABMxBwMHBAb169VKuBw4ciI9GjsR3336LypUrw83NTV1PSIiHr+9ZJWilS5XCG40bI1u2Zw+0cjZ5Sjxtvfraa6hevTqyZ38mEXJWeuzoUWRzdESzZs0g722ZfHx88PDBA1SpWhX79u5FWzGDlZOYPHnyIC4uDklJSdgrxLNqtWq4cf06bty4gYqVKqF27dqq/J/bt+Phw4dISEjA5k2b0LBRoxTlnZ2dlZ38Wzx9+jTu3r2LiuJvrlatWup6ZGSkqrea8P/w0SNcFmJdp25dvCb6YA3pGUFraI2VtOHKlSvqBpU3R6FChTBz5kz069cPwz/8UNfCtWvWqJsnUcyG5Sxi2LBh6P/OOyp/zuzZWLZsmSobGhqKxYsWYfFvv6FgwYK68jwgAXMSkMIp39ylKJ319VUiKwV2xIcjcO7cORQuXBh///03GogJxJSpUyFjoKM++kgIsK8SJy8vLyWSC8W9W6xYMci1hM8++wzxQjSleE759VeMGzdOCeqWzZuxR4ho/vz51aSkuqcnDh44oP4G6gqxixN/Hz/++KO6/+Xfi5xxS0H9cMQI9O3bF+vWr8cDIdLy72jVqlWo8Prr2L9vn668FNmQkBB88P77kIIqJywRERFo36EDxo8fj0dCoKX/EiVKqPplHXPmzMHHIoTSo2dPc2LWy/eztzC9zO3D6OzZs+qdfLUQUvnTvEULbNy4Ub27agnId8ldu3djk7jBKlSogK1bt6oseVNLge3Xvz+2iXdobxF/lTOAFcuXa4vylQSyhICczckUEBCgXhcv/k0J7NRp07D9zz8xfPhwHBUz0+PHjqnZpRTSvmIyIScEs4VIyUmBvCafzOTjfP58+bBhwwZVtkaNGlgi1jRknkxSpPsIwdy6bZv6e1AXn/slZ587d+3C2nXrUKZMGd3fzOLFi+Hh4aHEe6UQ2fLlyz9XEpg4YQLkTH2Z+DuSdXQScedt4m9Otl+bpMhK/zv++ku1ff/+/dosi75SZNPALx+5fhXv1MHBwfhN3AAydCDjXDI4r03y0Uc+SsmZrnzkunXrlhJTOROQSb7bbhM3wzURTpDv8DLexEQCWUkgu3isl0kKoEy+YkdMPiGUctV/y5YtOoG8JEJbxYsXRwERQ5XCJXfPyNnq8hUrlJhJe3nvy8d715dfhpOTE8Z9+SXGjh2LxMQE5Vv+6tOnD4oWLarydReTHZQXkxE5w35Z+GgiZtkhImwgZ7AZpejoaAQFBcFTzJClOOfMmRNdu3VTxU6LNwFtqirCFY6iz3LmK4U6MDBQm2XRV4YL0sAv40dfffUV3N3dIeM82nfrNEzVpVdeeUW93rx5E3f+L8R/7dih3nllRpEiRRiTVYT4KysJHBUzVJkqiZisTHfu3Pn3kXzlSnUuf0nRksIkheu3339Xk4r14vFd7oCRsdyfxAxWPprL9PL/73N5LBe09F3UkvbPp5deekldkn8zGfmRQizfKLRxZVlQCrVMkens0pGzXmtJFNk0RmK5eCSR7/iLxCxWvmv/Lm4+GddKnp6KeK02yccxOautUqUKZKhBplGjR+sWw+QNknxxQVuOryRgLgJyYiDjmlJs5AKWTMXEbDVQrDd4LVig7m95TXtvypmqnPmNEItln3/xBeQk4RuxDWzd2rV4Z8AAaaoLO8hjf39/BIifTp07y1ODk/ybyZEjhwoTZFRYzrLl34+fqE+b/MQ2TJlkqM7ak12L7BXxmCQf6ZOn9u3bo7TYSSAXv/aJm1QOrlzkkumJWDGV7/gyyXf70mKmK1c8d+3cCfmoIh9TZHkZf50qFhPeEQthMmywSMwKunXvjg4iUM9EAuYiIIVV3s8R4gMzMlYpPzAjH+nlE5lMXbt0waRJkzBBLBbJ+1GGwWQ47BcRGpO7AMYJcW3VqpWKy8rdATLlypVL3ddvNm+Ow4cOQcZPywh/cjFYTkB6v/22stPn11YRopCiKENr8g1Air/0L1NJsftBrmfsFbsTZLw3eZJ/c+3EjoU/RRx5xowZeF0sjMnFZBmGa9iwIWJjYpKbW92xXYus3IIlf5InuadQvnPLLSVfi5CBjE3Vq1cPu0RAXcZX5cqpTHJb10wx4PJGlY8ucqVUJhl8nzJlCiaLx6xvvv5avVvLWFZjEbdlIgFzEpAiK7dsyXUCKWDf//AD3nzzTV2VrVq3RqiYFMg3/RNiH618Wuvw1lsqZCAnEzLOOl98iGCnmDTIx/kuXbvqRFSKdZJYtZfxWlmPXMT6RtRlSJKhidViYUuKv9yxkHy3jtwXe1R8eOJLsWNhuvi7ej6NFjsF5K4B7a4eucVM/p3JUN01sXZizclBANM8ePSPNbfRYm2T20WkyKb3qC9vtnBx0xZIZ2uWjGXJ2a18x2ciAWsiILcWyoWutGKX8lON2j2wz7dZbrOSP4bs+5Zx1x5i5jxw0CB88MEHahuWFPjnU0Z/T9JeCq2cZadV/nl/1nJu1zPZjAYho4GUN2h6Ait9y8cZJhKwRgIv2rOdnsDKfsg4qvzJTErv7yqjvydZp1z7SK98ZtpkzrLcwmVOuvRNAiSgnuRKiXUO+TFbe0wMF9jjqLPPJEACWUaAM9ksQ82KSIAE7JEARdYeR519JgESyDICFNksQ82KSIAE7JEARdYeR519JgESyDICFNksQ82KSIAE7JEARdYeR519JgESyDICFNksQ82KSIAE7JEARdYeR519JgESyDICFNksQ82KSIAE7JEARdYeR519JgESyDICFNksQ82KSIAE7JEARdYeR519JgESyDICFNksQ82KSIAE7JEARdYeR519JgESyDIC6ku7IyIis6xCVkQCJEAC9kRAfZ+sPXWYfSUBEiCBrCTAcEFW0mZdJEACdkeAImt3Q84OkwAJZCUBimxW0mZdJEACdkeAImt3Q84OkwAJZCUBimxW0mZdJEACdkeAImt3Q84OkwAJZCUBimxW0mZdJEACdkeAImt3Q84OkwAJZCUBimxW0mZdJEACdkeAImt3Q84OkwAJZCUBiqwRtOPj45GUlGRESRYhAcsQSExMzFTF8+cvQFDQ1Uz5sNfCFFkDRv7Ro0fo2/cdVPaohkqVq2LEyFGIjo42wANNScAyBOrUbYCtW7cZXfl8rwW4epUiawxAiqwB1CZO/ApP4p5g65aNWLrkN1y+fBmzZs8xwANNScAyBLzmz0GTJo0tU7md10qR1fMGiI2Nxe49ezHgnf4oX748ateuhcEffCBmB9v19EAzEjAvgZCQEPyxbDlWrlyFx48fp6js5s1bCAsLV9d27dqNK1euYNu27Vizdp0Kfa1atRqhoaHYsGEj5s6bj7Nnz6Uo//xJenXJMJr0JevaseMveC9chFu3bj9fHAEBF+Dl5a3qi4uLS5VvSxcosnqO5sOHD9XN6FHFQ1eiVCk3PHjwgPFZHREeWIrAHjEB6NipK3x9z+LEyZN4o3EznDhxUtecb779XghbgDqfM3c+Brw7CL9OmYrAK4GQawxfjBuPbt174cDBQ9i+/U9x3FOJsM5BsoMX1aX11bdff6xfvwEnRRuavdkCR44c1XmYNWsO3u7TD8HXgjF9xkx07dYDCQkJunxbO1Bf2m1rnTJHf7SzgCKFC+vcOzs7K4GNioqCi4uL7joPSCCrCVwJDMQXn49Fz549VNUTJkzCkiVLUbdunTSbUqlSJSz09lJ58ilNpuHDhqBLl87qWJZfL2a17du3U+fJf+lTV+fOnTBo4Luq2NChw7Fz5y40bNgA/v7+mDZ9Bnbt3IFy5coqga9Vux42bNyEnj26J6/GZo4psiYYSgcHBxN4oQsSMJ7A8GFD8eTJExw9egx+Qsju3r2Lm7dupeuwRg3PVHnu7u66a23atsagQR8oEcyRI4fuujzQpy7P6tV1ZSpWrIiDhw6p85OnTsPV1RXnzp1XP/Kim1tJXLxwUWdvawcMF+g5oi4u+ZVlTEyMrsSTJ7FwdHRE3rx5ddd4QAKWILBs+Qo0bNQEmzZvgRRF59y5odFojG5K8eLFlWjfu3cvlQ9j6tK2JeRGiHr623/gALQ/bm5uKFWqVKp6bOUCZ7J6juQrr7yCbNmyQQb8ixYtqkqFhNxU78ryOhMJWJLAjBmzMGzoEAwcOEA1Q4YK5OKWsen0KR/Iez4t8ctMXXLWmpAQj+nTpsDJycnY5v2nylEd9ByunDlzomWL5vjl16m4fv06LojHG++Fi9OMWenpkmYkYDIC7u6lEXDhAuT6gIx7Llr8G2Ji/o216lvJmjVr1ew1UMR3ly1fjsaN30izaGbqkjHe+PgE/PLLFLWbISIiAl+On4hTIoxgq4kia8DITvpqIp4mPUXLVm3RqXNXlCxZAiM+HGaAB5qSgHkIjB37CYKvBqNKVU+8/8EQvPFGI8hHfe2ilj61RotQWO069dG6TXu12i9jr2mlzNQlZ8fe3vOxa/duyA9I1G/QSLWxWrWqaVVlE9f432qNGEY5W5CxWLm7gIkErIlAeHi4wTtdpBBXrFQFG9avRRWxRVHOLgsWLJhht4ypK7nT0NAw5M+fT/0tJb9ua8eMyRoxolzoMgIai2QJgcxuJZSTB30EVnYms3UVLFggS5hYuhKGCyw9AqyfBCxMQG5BLFu2DHLlymXhlthm9QwX2Oa4slckQAJWQoAzWSsZCDaDBEjANglQZG1zXNkrEiABKyFAkbWSgWAzSIAEbJMARdY2x5W9IgESsBICFFkrGQg2gwRIwDYJUGRtc1zZKxIgASshQJG1koFgM0iABGyTAEXWNseVvSIBErASAhRZKxkINoMESMA2CajvLggKDrHN3rFXJEACJGBhAvxYrYUHgNWTAAnYNgGGC2x7fNk7EiABCxOgyFp4AFg9CZCAbROgyNr2+LJ3JEACFiZAkbXwALB6EiAB2yZAkbXt8WXvSIAELEyAImvhAWD1JEACtk2AImvb48vekQAJWJgARdbCA8DqSYAEbJsARda2x5e9IwESsDABiqyFB4DVkwAJ2DYBiqxtjy97RwL/CQKrVq2Gj49Pum1NSkqCRqNR+efOnccfy5ana2ttGRRZA0fkxo0bmDJlGt5o3NTAkjQnAeslEBx8DaGhoRZr4MpVa3Da50y69Y8YOQrjJ0xS+VJkl/2HRFZ9C1e6PWNGCgLDho/AgQMH8dprr+Levfsp8nhCAv9lAoOHDEX/fn3Rv38/q+zGx6NHIUeOHFbZtowaRZHNiFCy/G7duuKXn3/C6dM+eP+DIclyeEgClicQEhKCQ4ePILujI1q0aI4iRYqoRsXHx2PDho1o1aoVChYsoK75BwTg0cNHqF69Onbu3ImIiEic8fWFq6ursGupbG7fvoNDhw5Dln/jjUYoU8ZdXU9MTMS6devRsmULVd/du3dRu1ZN1KxZE4GBgTh85ChKlyqlyjg5Oaky8ld4eDiOHj2GkJs34SnqrVOnti5Pe3BYtP+8nx/Kli2L1qIdDg4OKuv+/ftwzu0MN7eSWtMUr3fu3MHeffuVfcsWLUQ/XkqRb8kThgsMoN+saRPkyZPHgBI0JYGsIbBnz1507NQVvr5nceLkSRHOaoYTJ06qymNjY/HFuPGQYqhNu3fvwQLvhYiLe4JLly4rIZVPZ9ev31AmUlybNmuO7X/+KX52oE3b9ti+/U+VJ0VX+uvVuw8O7D+AY0ePi+O+4nF+opp8SH+TvvoG3br11FYnnvzuoXWbdvjl16kICLiAPn3749vvvtfly4OlS5dh5qw5Kv/DD0fqwgMyb9nyFdiyeas8TJWOHjuG5i1a48TxE9i8eYtoazvI8IfVJBFMZjKQwP79BzTlXq1gYCmak4D5CMyeM1cjFo90FYwfP1EzZMgwdR4WFqYp7V5OI8RNlz9l6jRNz15v687fbN5Ss2TJUnUuRFRTqXJVzYIF3rr8OXPnaap71tRERUVpoqOjlb+NGzfp8t8ZMFBT4fXKmsePH6trISE3lY1Yw1Dn/foP0Lz3/mCNmAWr8zNnfFW+fJXprY5dNEOHDlfH8pcIy2kqVqqikW2RabDoy8SJX6nj335bomnZqo06TkhI0FSp6qlZtmyFOpe/Brw7SDNq9BjduaUPOJO1mrc7NoQEjCcwfNhQdOz4lnocnzffS81arwYHG+XwwoWLEEKK1q1b68q/1aGDWBgLE7PeS7pr7u7/hg/khapVq6BypUooXLiwypeP9fKp7/LlK5A7A4SYqvCCowhlyOTpWR0lSpTAsWPH1bn85VHFQ3fcoEF9tZvA5wWLYdJY+o+MjBR1yBDGBvXjkj8/Ll68qPNl6QPGZC09AqyfBExAQD5OT58+E01FSKtChfIifplbt+XJUPcyvuns7IzixYvpisoYp1x4CgsL111LfqCNnaZ1TcykERMTg1fLlUuejZIlSwh/YSmuaU+yZ8+uBFuGL+rVq6u9nOpVxqGzZcuGk6dOp8jz9PRMcW7JE4qsJemzbhIwEYEZM2Zh2NAhGDhwgPIoHv1x5coVdZwzZ05I0ZKzU31SKbFoJeO4fn7+qFatqioiY70yFlux4uv6uEhhI2e3clYrY8Raf3FxcSp+3K1rF52teKzXHT8Ui3K3b98WAltHdy2tAzljljPlIUM+gEflymmZWPwawwUWHwI2gAQyT8DdvTQCLlyAiJnC398fixb/JmaPscpxrly5UKd2bWwSi0JPnjxBYFAQNm7cnGKmK3cV3Lp1W9lXqlRR7DqohukzZkLuC5eLSLNnz0XdunVQrNiz2a0y1vNX375vY/XqNTguFqfkbHjyz78gv3isb9Sooc6DiKuqtv3999+YOnUaSpVyQ/KQhM4w2UHFihUh2/urWFCT7RQxWsg3mIWLFiezsuwhRday/Fk7CZiEwNixnyD4ajDEIpBa4ZdbruSKvpyRyjR06GCcP38elT2qib2wA/Daqykf3Xv27IFVQgRbt2mvtkHNmT1TzVxbtGyDtzp2FqECJ3jNn2t0W0eP+ggNGtaHWCBDnbr1ccbHF4sXeaNo0aI6n40aNhAz8fdFfgPI3Q/ffvO1Li+9AxnjnTd3jtglEQexGIZq1Wti67btaNe2TXpFsvw6/1ttliNnhSRgPgJyL6qLi0u6FcjFK+1e2eeNpCDLR/bcIp6rTTKWKoVMhhxMkeQeW1lPvnz50nX3zz//iDYW1O2RTdfwuQzZVsBBtN/5uRzLnlJkLcuftZMACdg4AYYLbHyA2T0SIAHLEqDIWpY/aycBErBxAhRZGx9gdo8ESMCyBCiyluXP2kmABGycAEXWxgeY3SMBErAsAYqsZfmzdhIgARsnQJG18QFm90iABCxLgCJrWf6snQRIwMYJUGRtfIDZPRIgAcsSoMhalj9rJwESsHECFFkbH2B2jwRIwLIE1PfJBgWHWLYVrJ0ESIAEbJQAvyDGRgeW3SIBErAOAgwXWMc4sBUkQAI2SoAia6MDy26RAAlYBwGKrHWMA1tBAiRgowQosjY6sOwWCZCAdRCgyFrHOLAVJEACNkqAImujA8tukQAJWAcBiqx1jANbQQIkYKMEKLI2OrDsFgmQgHUQoMhaxziwFSRAAjZKgCJrowPLbpEACVgHAYqsEeMQFxcHjUZjREkWIQHzEFi1ajV8fHzSdZ6UlGTQPRsZGYnpM2YiLCw8XZ/M0I8ARVY/TsrK398fb73VGZUqV0WDho3x3Xc/IDQ01AAPNCUB8xBYuWoNTvucSdf5iJGjMH7CpHTzn8+IiIjAjBmzEB5BkX2ejaHn6lu4DC1kj/bynX3goA/QsmULeHt74e+//8bw4SMQFh6OX3+ZbI9I2Of/EIGPR49Cjhw5/kMttp2mUmT1HMvz5/0ghXbMx6NQqFAhuLq+hK5du2DL1q16eqAZCZifwOHDR3Dezw9ly5ZF61Yt4eDgoCq9f/8+nHM7w82tJM6dO6/uZXl+4sRJDB0yGI6OjggOvoZ9+/YhT5688PSsbv7G2kkNFFk9B7phwwa4fCkghfWDBw9QtWrVFNd4QgKWIrB06TKUKFEChQsXwrRpM9CrV098/903qjnLlq+A60svoYanJ3b8tRMbN25S1ytUKI/BH7yPM2d8MeDdQShW7BV4eHhggbe3pbphc/VSZI0c0osXL2Hjps1YueIPIz2wGAmYlkD1alUxd+5s5fTgwUMYJsJZX02aACcnp1QVyWt/bt8CFxcXlffT5J/RpPEbmD17JrJlywZZXoquWC1LVZYXDCPAhS/DeCnrf/75R9zAH2LUqJHqXd8IFyxCAiYn4FHFQ+ezQYP6ajeBTzqLYeXKldUJrNx54Ofnj6bNmiqBlU5kvkr/Dzf8e8LfxhCgyBpILSYmBu+++x5q1aqF9wYNNLA0zUkgawhkz55dhA0K4/r1GxlWGBUVBSm07u6lM7SlgeEEKLIGMEtMTMTQYR/C9WVX/PTj9waUpCkJmJ9A8r3bDx8+wu3bt1GvXp0MK5Yhg7x588L3zNkMbWlgOAGKrJ7M5A089rMvEB0drbZsScGVH0qQP0+fPtXTC81IwHwEli1bgcCgILW9cOrUaShVyk3MTt31qrBzp45YvWYtTp06re7xefO9/i3HmKxe/F5kxIWvF9FJlhd09apuRbZqtRrJciA2bU/DWx3ap7jGExLIagKNxA6YgQPfh9yu5ZI/P6ZPn6p3Ez79dAxuig8s9OrdR+2nlTsTmExDgP+t1jQc6YUErIaAXJgtWLCgbo+sIQ2Tn/RydnZOc0eCIX5o+4wARfYZCx6RAAmQgMkJMCZrcqR0SAIkQALPCFBkn7HgEQmQAAmYnABF1uRI6ZAESIAEnhGgyD5jwSMSIAESMDkBiqzJkdIhCZAACTwjQJF9xoJHJEACJGByAhRZkyOlQxIgARJ4RoAi+4wFj0iABEjA5AQosiZHSockQAIk8IwARfYZCx6RAAmQgMkJUGRNjpQOSYAESOAZAfUtXEHBIc+u8IgESIAESMBkBPgFMSZDSUckQAIkkJoAwwWpmfAKCZAACZiMAEXWZCjpiARIgARSE6DIpmbCKyRAAiRgMgIUWZOhpCMSIAESSE2AIpuaCa+QAAmQgMkIUGRNhpKOSIAESCA1AYpsaia8QgIkQAImI0CRNRlKOiIBEiCB1AQosqmZ8AoJkAAJmIwARdZkKOmIBEiABFIToMimZpLhlcTExAxtaEACJEACkgBF1oD74OLFS/jk08/gWaM2mr3ZElOmTkdCQoIBHmhKAtZHICIiAoGBgdbXMBtpEUVWz4GMjY3FOwMGIrezMw4e2Ic5s2di06bN2LNnr54eaEYC1kng0KHD6NW7j3U2zgZaRZHVcxAfPXqMunXrYNSokShYsABef70CWrR4E5s2b9HTA81IwLwE/v77b3U/rlixCqGhYSkqS0pKwu7dezBvvhdOn/bB06dPVf6RI0dx9NhxxMXFY9Wq1bh7926KcjzJPAGKrJ4M3dxKYtbM6ShUqJAqodFocMbHF6VLl9bTA81IwHwErl27juYtWmHBAm+cOnUKjZs0w/btf6oKpcD26dsfU6ZMw6NHjzDuy/Ho13+Ayrt16xbu3LkDaXPp0mVERkaar5F26ll9abed9t2obgcEXMAffyzDYTEDqFjxdYwcMdwoPyxEAqYk8PkX49CsWTNM+fVn5XbtuvWYPWce2rVri5CQm3jZ1RXz581FgQIuePjwEeo3aITLl6+gd+9eyJcvHwICAvD115NM2ST6+j8BiqyBt0LevHlQpkwZPHj4EMFXg8WCQRCqV69moBeak4DpCMhZqJ+fP94d8I7OaZfOnVCrZg0VFihTxh3Tp0/FzZu3sHPnTjGbfQwnJycEBwejQoXyujI8MA8BhgsM5CrDA4MHv4/ff1uE2rVrYarYYcBEApYkEBoaKmKqcShVqpSuGY6OjiqUlS1bNoSHh6Ntuw4Y/fEYJbQFxJqCvC5DXkzmJ0CR1ZOxjFd99tkXePLkia5EpUoVcVXMBphIwJIEihQpgty5c8PHx0fXjKioKOzY8ZeKtW7b9iceP/4bK5b/gU8/HYO+fd5GjhxOcHBw0NnzwHwEKLJ6spULXlu2boOXlzfkdi756LVu3QbUq1dXTw80IwHzEejWtQtWrV4Df39/xMTE4IcfJ2PO3HmQM1r3MqURFhaGK1euQO6JnTptujgPR4y4j2VyFfHaqKjoVDsSzNda+/JMkdVzvF1dX4L3gvki9uWH2nXqq32FckvXxAnj9fRAMxIwH4GxYz/Ba6++ii5de6gPy8gPF3jNn6sqrF+vHnr26I6+/QagRs06uHXrtggtuKmYrDTw9KwOD4/KqFW7rm5Hgvlaan+e+d9qjRhzOVNwFh9K4OOWEfBYxKwE4uPjVXxW7hh4Psm8xMQkEVpwfj5LncvYrouLi4rXpmnAi0YRoMgahY2FSIAESEA/AgwX6MeJViRAAiRgFAGKrFHYWIgESIAE9CNAkdWPE61IgARIwCgCFFmjsLEQCZAACehHgCKrHydakQAJkIBRBCiyRmFjIRIgARLQjwBFVj9OtCIBEiABowhQZI3CxkIkQAIkoB8Biqx+nGhFAiRAAkYRoMgahY2FSIAESEA/AhRZ/TjRigRIgASMIqD+M0JQcIhRhVmIBEiABEjgxQT4BTEv5sNcEiABEsgUAYYLMoWPhUmABEjgxQQosi/mw1wSIAESyBQBimym8LEwCZAACbyYAEX2xXyYSwIkQAKZIkCRzRQ+FiYBEiCBFxOgyL6YD3NJgARIIFMEKLKZwsfCJEACJPBiAhTZF/NhLgmQAAlkigBFNlP4WJgESIAEXkyAIvtiPswlARIggUwRoMhmCh8Lk4B1EoiMjMT0GTMRFhZunQ20o1ZRZI0c7PnzF6BS5aoIDr5mpAcWIwHzEYiIiMCMGbMQHkGRNR9l/TxTZPXjlMLq2rXrmDV7DmJiYvD06dMUeTwhARIggeQE1FcdJr/A4xcT0Gg0+PyLcejVqwcWL/79xcbMJYEsJCCfqvbt24c8efLC07N6qprl7PbAwUO4f+8+GjSoj0qVKqawCQi4gKNHj6Fo0SJo164tcubMiaSkJKxduw4NGzbE/gMHUK5sWdSrVzdFOZ68mABnsi/mkyp3+YqViI2NxaCB76bK4wUSsBSBU6dOo8NbnbB6zVqc9vHBkKHDUjTlwYOHaNW6HZYsWYpz586ha7ce2Lp1m85m1qw5eLtPPwRfC1axXJmfkJCA+Ph4fDFuPHr2ehvr1q7H7Tt3dGV4oCcBMTNj0pPAvXv3NFWr1dCId3zN/fsPNKXdy2kCA4P0LE0zEjAfgc5dummGDh2uETNPVcmBAwfV/Xnjxg11PnDge5oPR3yka4BYU9DUq99Infv5+Wncy7yqCQq6qs7j4uI0Vap6alatXqMRITHlZ4H3Ql1ZHhhGgDNZPd+MpNmXX05QYYLnH7MMcEFTEjA5AflI7+fnj6bNmiJbtn//pMuVK/tvPQ4OEJKAU6d9ULhQIaxbt0H9yBmqmDQgNDQUJ8Us2NXVVcxwz6u8LVu2wc2tJC5euKhra80aNXTHPDCMAGOyevLaIh6tjh47jtZtWmPbtu0IC/931fbAwYNITEzE669X0NMTzUjAtASioqJU7NTdvXSajv/55x9Im6tXg/Ho8WOdTdu2bRAnxDbkRogqL2Ou2uTm5oZSpUppT/maCQIUWT3hPX70GG+80Qi7d+9RJcQjlXoVj2UoWKAgRVZPjjQzPQEXFxfkzZsXvmfOIq0ZZyExg82TJzcaN3kD7783KFUD5Kw1ISEe06dNgZOTU4p8uf7AlDkCDBfoyW/gwAFY4DVP9/Pz5J9Uya8mTUS3bl309EIzEjAPgc6dOqpFL7kAFh0djXnzvf6tSIQKHETIoHv37li2bDmOHT+uth0ePXYMYz4Zq0IJ7du3EwtcCfjllykqfCB3IXw5fiKkL6bME6DIZp4hPZCAxQl8+ukYlCxZAr1690GNmnWQPXvKGelYkV+/Xj0MGDBIfYhmnNgx0LVrZyXAr7zyCry952PX7t2oU7cB6jdopHbQVKtW1eL9soUG8L/V2sIosg8k8H8Cchbq7Oyc6rFfC0iuH0RFRaNAARftpRSvoaFhyJ8/HxwdHVNc54nxBCiyxrNjSRIgARLIkADDBRkiogEJkAAJGE+AIms8O5YkARIggQwJUGQzREQDEiABEjCeAEXWeHYsSQIkQAIZEqDIZoiIBiRAAiRgPAGKrPHsWJIESIAEMiRAkc0QEQ1IgARIwHgCFFnj2bEkCZAACWRIgCKbISIakAAJkIDxBCiyxrNjSRIgARLIkABFNkNENCABEiAB4wmo75MNCg4x3gNLkgAJkAAJpEuAXxCTLhpmkAAJkEDmCTBckHmG9EACJEAC6RKgyKaLhhkkQAIkkHkCFNnMM6QHEiABEkiXAEU2XTTMIAESIIHME6DIZp4hPZAACZBAugQosumiYQYJkAAJZJ4ARTbzDOmBBEiABNIlQJFNFw0zSIAESCDzBCiymWdIDyRAAiSQLgGKbLpomEECJEACmSdAkc08Q3ogAasjEBkZiekzZiIsLNzq2mZvDeJ3Fxgw4rdv30Gr1m1TlfA77wtHR8dU13mBBCxF4M6dO2jYqAkOHNiLUm5ulmoG6xUE1LdwkYR+BO4/uK8Mly9bmqIABTYFDp6QAAkkI0CRTQYjo8MH9x/g5ZdfRrVqVTMyZT4JZDmB4OBr2LdvH/LkyQtPz+qp6g8PD8fRo8cQcvMmPKtXR506tVPYXL0aLMrvR758edG2bRvs2PEX2rVrK87zpbDjiWEEKLIG8Hrw4CEKFSyIQ4cOIzAoCFWrVEGtWjUN8EBTEjAPgVOnTmPAu4NQrNgr8PDwwAJv7xQV3bt3D126dkeuXM6oWPF1TJkyDe+80w8Txn+p7Hx9z6JP3/6ifDFU8aiM35csRWBgEGrXrk2RTUHSiBMNk94EfvhxsqZM2dc0HTt10bRq3U5T2r2cxsvLW+/yNCQBcxHo3KWbZujQ4ZqkpCRVxYEDB9X9eePGDXXer/8AzXvvD9YkJiaq8zNnfFW+fJVp/PiJmg8GD9Xla8uL2bHK5y/jCXB3gQFvTB+N/BD79+3Bpo3r8deObRg79hMsW77cAA80JQHTExDCCj8/fzRt1hTZsv37J12uXNl/K3JwgMwXYoqWLVvoFmhlOKFEiRI4duy4stt/4CCaNW2iy3/ttVdN31A79UiRNWDgc+fODTe3kroSPbp3g9xxcOHCRd01HpBAVhOIiopSQuruXjrNqsPCwhATE4NXy5VLkV+yZAmxxevfPLkbwd3dPUU+T0xDgCJrAMeZM2dDxr60Se5BFA8RePo0SXuJrySQ5QRcXFyQN29e+J45m2bdhQsXFotheXDixEldflxcHGQcVsZn5eRB+jjj66vLl/c1k2kIUGQN4FiggAu+//5H3L17F6GhoZg330stFLz22msGeKEpCZieQOdOHbF6zVo1CYiOjlb3pqrl/2LZt+/bWL16DY4fP6E+oDD551+QP39+NGrUUJl1fKsD1qxZh5MnT0GWnztvvukbaaceubvAgIHv0aM7rl2/AbHIIEQ2DCVLlsTcObOQM2dOA7zQlARMT+DTT8fg5shR6NW7D3LkyIFevXqmqGT0qI8gwwrvDBgIBxGnrVC+PBYv8kbRokWV3Zgxo9XWrt5v94WTk5PauiUzhClTJgnwE19GAHz69CkiIiIhZ7ZMJGBNBCIiIuDs7KyEMq12id0FiI2NTbUtSy6OyQ/VyPK5cuXC3r37MPzDkQjwP6fCCWn54jX9CHAmqx+nFFZyBZcCmwIJT6yEgAwBvChlz549lcBK+0WLfoOfvz8GDXwX9x88wAyx/iB3I8h4LVPmCHAmmzl+LE0CNkEgISEBM2bMUh+0eRL3BA0bNsDHo0epBTWb6KAFO0GRtSB8Vk0CJGD7BLi7wPbHmD0kARKwIAGKrAXhs2oSIAHbJ0CRtf0xZg9JgAQsSIAia0H4rJoESMD2CVBkbX+M2UMSIAELEqDIWhA+qyYBErB9AhRZ2x9j9pAESMCCBCiyFoTPqkmABGyfAEXW9seYPSQBErAgAfXdBUHBIRZsAqsmARIgAdslwI/V2u7YsmckQAJWQIDhAisYBDaBBEjAdglQZG13bNkzEiABKyBAkbWCQWATSIAEbJcARdZ2x5Y9IwESsAICFFkrGAQ2gQRIwHYJUGRtd2zZMxIgASsgQJG1gkFgE0iABGyXAEXWdseWPSMBErACAhRZKxgENoEESMB2CVBkbXds2TMSIAErIECRtYJBYBNIwBQEYmJisGHDRnw5fiIiIyNN4ZI+TEBAfUGMCfzYjYukpCTMm+eFw0eOIEeOHPhj6e9203d21LoJjBjxEe7dv4+2bdsge3b+aVvLaHEkDBiJhIQE9H67LxISEvHh8KGoXLmSAaVpSgLmIxAeHo5Dh49gw/o18PDwMF9F9GwwAYqsAcjWrFmHwMAgnDh+BLlz5zagJE1JwHwEbt68hY0bNyExMREnTp5CREQkGjSoj127dqNUKTcEBV1FTGwsenTvphrhHxAg7uETcClQAC2aN0fBggV0jZNPatv/3IFbt26hdq1ayJUrJyKjolC/Xj2dDQ8MI0CRNYDXipUr0aFDe/z1107cf/BA3IQ1UbNmTQM80JQETE8gIiICwdeuKcdBQUHIny+fOp4zdz4ePnyAnDlzovmbb6prU6dNV+GuNm1a49y5c5g6dRqW/bEU5cqVVfnDho/Avn370bp1K+zauRuPHj8W93gNiqyiY9wvfp+sAdwqe1QTcVgnVKpUCU7ZnXDg4EFMnvwjunfraoAXmpKA6QkECnFt1aotTp86jiJFiqgKOnbqiqJFi2Cht5c69/U9i67demDL5g0qpCBnre+/PwSJSYlYuuQ3PHr0CHXqNhD5G1UoTOb36tUHrxR7BTNnTDN9o+3EI3cX6DnQcrYQHR2NgQPfVYtdixd745NPPsbKlav09EAzEsh6AjVqeOoqPXXqNIoVK6aL2To6OqJduzY4ceIkpKAePHgYrq4v6dYaZH6ZMu668jwwjgBFVk9u8oaTqW7dOroSLVu0EI9c58Uj2SPdNR6QgLUSuHPnDl59tVyK5pUsWVIs5CZAbv8KDAyEuztFNQUgE5xQZPWEmCdPHvUYduzocV2JByLepdFo4OTE0LYOCg+slkCp0qVw5oyvWiDTNvK4WAArWbIE8ok4rpzl+vn5qVmtNl8DjfaQr0YSoMgaAK5f3z5Yu249Tp/2wb1797Bw4WLUrFFDrM4WNMALTUnAMgQ6dXxLVfzzz79Cbvk6evQY1m/YgK5duqjrrdu0UtsTf/l1itihEIHz5/2we/ceyzTWhmrlFMyAwRw6dLDaVdCnb381g/XwqIzp06Ya4IGmJGA5AnJBbPEib4wd+zl++32JmL3mRZfOnTFy5IeqUS+7umLqlF/wxbjxagJRWsx8y5YtCwcHy7XZFmrm7gIjRjE+Ph5PnjxB/vz5jSjNIiRgeQKhoWFwccmPbNlSPszKBTAHoapypiuf0Fq1bosmTZrgi8/HWr7R/9EWpCT8H+1EVjdbfpyWApvV1FmfKQnIDyA8L7AxMbFKVDdt2ozbYpFMhhWCg6+hQ/t2pqza7nxxJmt3Q84Ok0D6BHzOnMGCBQtx5coVFC9eHMOGDkHDhg3SL8CcDAlQZDNERAMSIAESMJ4AwwXGs2NJEiABEsiQAEU2Q0Q0IAESIAHjCVBkjWfHkiRAAiSQIQGKbIaIaEACJEACxhOgyBrPjiVJgARIIEMCFNkMEdGABEiABIwnQJE1nh1LkgAJkECGBCiyGSKiAQmQAAkYT4Aiazw7liQBEiCBDAmob+EKCg7J0JAGJEACJEAChhPgx2oNZ8YSJEACJKA3AYYL9EZFQxIgARIwnABF1nBmLEECJEACehOgyOqNioYkQAIkYDgBiqzhzFiCBEiABPQmQJHVGxUNSYAESMBwAhRZw5mxBAmQAAnoTYAiqzcqGpIACZCA4QQosoYzYwkSIAES0JsARVZvVDQkARIgAcMJUGQNZ8YSJEACJKA3AYqs3qhoSALWS2DVqtXw8fGx3gbaccsosnoOvvfCRahUuWqaP2d8ffX0QjMSMA+BlavW4LTPGfM4p9dMEVDfwpUpD3ZSuG+ft9G1S+cUvT10+Ah++mkyqlapkuI6T0iABEhAS4AiqyWRwauzszPkT/K0du069O/fD9mzE2NyLjy2HAEZMvDx8UXdunVQrVrVFA25ffsODh06jPj4eLzxRiOUKeOu8pOSkiDv5YYNG2L/gQMoV7Ys6tWrC3ndR8yO/QMC0KhhA5QvXz6Fvzt37mDvvv1wcHBAyxYt4Or6Uop8nvxLgF91aOSdIG+8Xr364OiRQyhQwMVILyxGAqYh0LFTV0RGRiJ//nxwL10ae/buw4B3+mPMmNGqAimug977ALVr1xIimwA/Pz9MnfIL2rVri9jYWFSsVAXFihVDkcKF0bdfH3Tv1hXdu/dCfEICPDwqYc+evWje/E189+03yt/RY8fw3nuD0bRJYzx89AjXrl3D2jWrUbZsGdN0yJa8aJiMIjD8w5GaCRMmGVWWhUjA1ATe6thF8/4HQzRPnz5Vrn//fYmmeYtW6ljMXDViPUGzYIG3rto5c+dpqnvW1ERFRWliYmI0pd3LaRZ4L9Tl+5w5oylbrrwmIiJCXfP3D9D07PW2Jjo6WpOQkKCpUtVTs2zZCp39gHcHaUaNHqM758EzAlz4MuId8+bNW9i5cxcGDhxgRGkWIQHzEKhevZp6dJfeK1asiKtXg5GYmIgLFy5CiCNat26tq/itDh0QGhqGS5cu6a7VrFFDd+xWsiRc8ufHJ59+hr1iVvzaa69i1crlyJ07Ny5fvqJmzUlJiVi3boP6kbYXL17UlefBMwIU2Wcs9D5aKHYaNBGPSaXFYxkTCVgzATGfgoydyvWE4sWL6Zoq46c5cuRAWFi47lryg6JFi2Lbts14/fUKmPzzL6hRsw68vRcqk5CQEGTLlg0nT51WMVwZx00QYu7p6ZncBY//T4ArNgbeCqGhoVi7bj1+/22RgSVpTgKWIVCqVCkVd/Xz89cthvn6nlULYBUrvp5mo65du47omGiM+mik+tm1azcGDxmm4rJubiXVotiQIR/Ao3LlNMvz4jMCnMk+Y6HX0ZKlf+DVcuVQp05tvexpRAKWJlCpUkXIUML0GTNx48YNBAdfw+zZc9UOBLnYlVaSC7s9evRGQMAFyNmwo6Mj8uXLh8JiYUyGIqTPX3+dqnyJGC2WLFmKhYsWp+XK7q9xJmvALSBXYZcKkZ00aaIBpWhKApYlILdYzZk9E2M+GYsWLduoMEFdMUmYM21mug3r0L4dAq8EovfbfdWsVcZov5o0QexeyK/KzJs7B5+O/QwtW7VBrly5VFhh1szp6fqz5wxu4bLn0Wff7Y6A2EmgZqU5c+bUq+9yFhseHpHuNkXpD3AQC2Ip95Dr5dxOjCiydjLQ7CYJkIBlCDAmaxnurJUESMBOCFBk7WSg2U0SIAHLEKDIWoY7ayUBErATAhRZOxlodpMESMAyBCiyluHOWkmABOyEAEXWTgaa3SQBErAMAYqsZbizVhIgATshQJG1k4FmN0mABCxDgCJrGe6slQRIwE4IUGTtZKDZTRIgAcsQoMhahjtrJQESsBMC6lu4goJD7KS77CYJkAAJZC0BfkFM1vJmbSRAAnZGgOECOxtwdpcESCBrCVBks5Y3ayMBErAzAhRZOxtwdpcESCBrCVBks5Y3ayMBErAzAhRZOxtwdpcESCBrCVBks5Y3ayMBErAzAhRZOxtwdpcESCBrCVBks5Y3ayMBErAzAhRZOxtwdpcESCBrCVBks5Y3ayMBErAzAhRZIwY8KSkJGo3GiJIsQgJZT+DIkaPYtGlzuhXzfk4XjUkyKLIGYLx79y7GT5iI2nXqo0HDxvjhx8mIiIgwwANNSSDrCRwWIrvxBSI7YuQocV9PyvqG2UmN6lu47KSvme7mZ5+PQ1xcHLZs3oCoqCgMHjIMsTEx+PbbrzPtmw5IwFIEPh49Cjly5LBU9TZfL0XWgCE+e/Ycvv3mKxQvXlyV6tjxLezdu88ADzQlAfMRePToEXbv3gsXl/xo2rQJcufOnaKykJAQ7BH3q1vJkio/e/Z///zv378P59zOcHMriXv37uG0zxnUqV0bO3ftQkJCAlq2aIGSJUuk8MUT/QkwXKA/K1SuXAlSaLXp3Lnz8KhcWXvKVxKwGIHDh4/gzeatcO78OaxbvwE1atbB7dt3dO0JDAzE4MHDcOXKFXz9zXfo3buvLm/Z8hXi6WyrOr8s8j/77AsMeHcQ/P0D4OXljfYdOuLGjRs6ex4YRoAzWQN4/fTj9+jbbwB8xU0XHRUNlwIumD5tigEeaEoC5iGwwHshevXsgXHjPlcVTJz4FQ4ePIQ+fXqrc0fH7Fi9eoWY5boooW3TtgPkzLdo0aKpGiRnrwu85qnZqwyPde3WAzt37hYi/X4qW17ImABnshkz0ln4+p5FUlIi6tSpjVq1a4qb9DEuX76iy+cBCViKQLWqVdXi1rz5Xgi5eRPfiLCWVmBlm8qWLaMEVh6XL18eDg4OQmwD5Wmq5OjoqAsP5MyZE82aNcWhw4dT2fGCfgQosvpxUgteE8TsYOLECZgw/kv8PPkndRNPmMhVWT0R0syMBMaMGa0WYC9cuIg2bdqjW/eeuHXr9gtr1HcbolyDuHbt2gt9MTN9AhTZ9NmkyJGLBjFiJ0HF1yvorleqWBHBwdcQGxuru8YDEshqAnKfq4zJ1q9XF7NnzcDpU8eRLVs2/DrFNKGs06d9ULdu3azuls3UR5HVcyhLly6NIkWKwGvBQoSHh+PBg4dYsuQPtfDl7OyspxeakYDpCcjH+4mTvsYPP/ykJgKyBnnN3d3dqMri4+Oxes1aERpLwv4DB7B//wE0adLYKF8sBFBk9bwL5D7CRQu9EBkZicZN3kTbtu3x0ktFMWvWdD090IwEzEfAy2su/AMC4FmjtvqgTAGxwDXw3QFGVSi3dh0/dhweVarjvfcGo56YIbdu1dIoXywE8L/VGnEXPHnyBE5OTmq2YERxFiEBsxGQH5LJlSsXtHtgDa1IzlyHDBmOK5cvqDDY06dPkSdPHkPd0D4ZAW7hSgZD30N5EzORgDUSyJs3r8maxTCYaVAyXGAajvRCAjZBILdzbpQpU8Ym+mItnWC4wFpGgu0gARKwSQKcydrksLJTJEAC1kKAImstI8F2kAAJ2CQBiqxNDis7RQIkYC0EKLLWMhJsBwmQgE0SoMja5LCyUyRAAtZCgCJrLSPBdpAACdgkAYqsTQ4rO0UCJGAtBCiy1jISbAcJkIBNEqDI2uSwslMkQALWQkB9d0FQcIi1tIftIAESIAGbIsCP1drUcLIzJEAC1kaA4QJrGxG2hwRIwKYIUGRtajjZGRIgAWsjQJG1thFhe0iABGyKAEXWpoaTnSEBErA2AhRZaxsRtocESMCmCFBkbWo42RkSIAFrI0CRtbYRYXtIgARsigBF1qaGk50hARKwNgIUWWsbEbaHBEjApghQZG1qONkZEiABayNAkTViROLj46HRaIwoySIk8N8kEBISglmz5vw3G2/hVlNkDRiA+/fvo1fvPqjsUQ3VPWth3nwvA0rTlATMRyAiIgKBgYFmq+D69RuYOWu22fzbsmOKrAGj++7A91GoYEHs37cb478chzlz5mLHjr8M8EBTEjAPgUOHDqsJgHm802tmCFBk9aR38+YtXL58GR9/PBrFixdHt25d0LRpUyxfvlJPDzQjAfMQOHLkKI4eO464uHisWrUad+/eRVJSkjq+ffsO/li2HMePn4Cv71kcPnxE1wgZ8pL2Dx8+0l37+++/sWnzFqxYsQqhoWG6688fPH78WJW9c+fO81k8f44ARfY5IOmd3r5zG9mzZ0fJkiV0Jg3q18P1Gzd05zwgAUsQuHXrFqTYSWG9dOkyIiMjIdcNvhg3Hj17vY11a9fjtsjfum07lv6xTNdEKbLS5tq1a+ratWvX0bxFKyxY4I1Tp06hcZNm2L79T5299iA8PBx9+w3AMSHcr7zyivYyX9MhQJFNB8zzlyuUL4/ExEScOHFSlxVy8yZiY2N05zwgAUsQ6N27F3p074bcuZ3x9deTUF7cq9o0YEB/bN68Ad27ddVeSvf18y/GoVmzZvhz+1ZMnz4VEyZ8idlz5qWwj4mJwTsDBqKYENepU35BtmyUkBSA0jhR/xkhjeu89ByBQoUKQd7MH436GH37vI074pHs0MFDcHV9+TlLnpKA9RCoWaOGXo2Rs2A/P3+8O+AdnX2Xzp1Qq2YNPH36VHdt2PARiIqKwqqVy9WTnS6DB+kSoMimiyZ1xjdillC9ejUV2/L0rI5SpdwQ4B+Q2pBXSOA/RiA0NFTEdOPEPV1K13JHR0eULl1ady6f5OTahAxPBAQEoGbNmro8HqRPgHP99NmkypExWfnY9eMP36F3r574a8dOtG/fLpUdL5CANRLImycPoqOj02xakSJFRLghN3x8fHT5csYqd8/IWa5MUnQ3bliL994biBEjR79wYUznhAegyBp4E8jVWu+Fi9C5SzfkzJULHTq0N9ADzUnA9ARcXV3FY3z0C4WvceNGOHv2HAKDgiBjq9oPF2g/WNOtaxesWr0G/v7+Kv+HHydjztx5Slxlix0cHODi4oJPxnyMEiVKYMyYT/mhHD2GkiKrB6TkJnK1duuWbWjSpIlYtV2luwGT2/CYBLKagAxfeXhURq3addPcESDb4+npibfEpKCLmCBUrVYDN8SnuJycnHRNHTv2E7z26qvo0rUHPGvUVh9u8Jo/V5evPZAz2lmzpuP8+fPw9l6kvczXdAjwv9WmA4aXSeC/SEDGVuVs80Wr/rGxsWpWmks8iaWV5PYvGZ/Nly9fWtm8ZiABiqyBwGhOAiRAAoYQYLjAEFq0JQESIAEDCVBkDQRGcxIgARIwhABF1hBatCUBEiABAwlQZA0ERnMSIAESMIQARdYQWrQlARIgAQMJUBGpJjsAAAB9SURBVGQNBEZzEiABEjCEAEXWEFq0JQESIAEDCVBkDQRGcxIgARIwhABF1hBatCUBEiABAwlQZA0ERnMSIAESMIQARdYQWrQlARIgAQMJqC/tDgoOMbAYzUmABEiABPQhwC+I0YcSbUiABEjASAIMFxgJjsVIgARIQB8C/wPf0TjSc891BgAAAABJRU5ErkJggg==)\n","metadata":{"id":"XFmJkLWB5qyg"}},{"cell_type":"markdown","source":"#### Loading the data","metadata":{"id":"0FGBle4f5qyi"}},{"cell_type":"code","source":"(x_train, y_train), (x_test, y_test) = cifar10.load_data()","metadata":{"execution":{"iopub.execute_input":"2023-02-25T13:25:20.690015Z","iopub.status.busy":"2023-02-25T13:25:20.689396Z","iopub.status.idle":"2023-02-25T13:25:27.811359Z","shell.execute_reply":"2023-02-25T13:25:27.810319Z","shell.execute_reply.started":"2023-02-25T13:25:20.689983Z"},"id":"XFT2vJq75qyn"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Part 2: Exploratory Data Analysis (EDA)","metadata":{"id":"9eOTdbXR5qyr"}},{"cell_type":"markdown","source":"### 2.1 Inspect the data","metadata":{"id":"Ox1es-P15qys"}},{"cell_type":"code","source":"# get the shape of the dataset\nprint(\"The shape of x_train: \", x_train.shape)\nprint(\"The shape of x_test: \", x_test.shape)\nprint(\"The shape of y_train: \", y_train.shape)\nprint(\"The shape of y_test: \", y_test.shape)","metadata":{"execution":{"iopub.execute_input":"2023-02-25T13:25:27.813233Z","iopub.status.busy":"2023-02-25T13:25:27.812881Z","iopub.status.idle":"2023-02-25T13:25:27.823732Z","shell.execute_reply":"2023-02-25T13:25:27.819444Z","shell.execute_reply.started":"2023-02-25T13:25:27.813178Z"},"id":"XcvHR90k5qyt","outputId":"7ff5d3d5-6c40-4a54-903b-517c15a28a7e"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# view some first rows of y_train\ntrain_df = pd.DataFrame(y_train, columns=[\"label\"])\ntrain_df.head()","metadata":{"execution":{"iopub.execute_input":"2023-02-25T13:25:27.827386Z","iopub.status.busy":"2023-02-25T13:25:27.826649Z","iopub.status.idle":"2023-02-25T13:25:28.096689Z","shell.execute_reply":"2023-02-25T13:25:28.095714Z","shell.execute_reply.started":"2023-02-25T13:25:27.827358Z"},"id":"ln9z4blU5qyu","outputId":"dd9459a8-12e2-43bc-9ac7-fda905eac4f3"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# the structure of data also tells us the number of rows, columns and type of data\ntrain_df.info()","metadata":{"execution":{"iopub.execute_input":"2023-02-25T13:25:28.099215Z","iopub.status.busy":"2023-02-25T13:25:28.098858Z","iopub.status.idle":"2023-02-25T13:25:28.120557Z","shell.execute_reply":"2023-02-25T13:25:28.119462Z","shell.execute_reply.started":"2023-02-25T13:25:28.099180Z"},"id":"YmEJ23Zl5qyv","outputId":"3aa86e40-d8bd-4470-ae2c-81646a4518da"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# get the unique labels in training data\ntrain_df[\"label\"].unique()","metadata":{"execution":{"iopub.execute_input":"2023-02-25T13:25:28.122817Z","iopub.status.busy":"2023-02-25T13:25:28.122423Z","iopub.status.idle":"2023-02-25T13:25:28.132648Z","shell.execute_reply":"2023-02-25T13:25:28.131542Z","shell.execute_reply.started":"2023-02-25T13:25:28.122782Z"},"id":"fwrqKGeN5qyw","outputId":"796f6a26-c4f8-4e76-a680-398deb48c0f6"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# view some first rows of y_test\ntest_df = pd.DataFrame(y_test, columns=[\"label\"])\ntest_df.head()","metadata":{"execution":{"iopub.execute_input":"2023-02-25T13:25:28.135280Z","iopub.status.busy":"2023-02-25T13:25:28.134171Z","iopub.status.idle":"2023-02-25T13:25:28.145795Z","shell.execute_reply":"2023-02-25T13:25:28.144710Z","shell.execute_reply.started":"2023-02-25T13:25:28.135203Z"},"id":"fQ1i2vEm5qyx","outputId":"11d63cc7-7ffd-4fdf-c439-adb39e303562"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# the structure of data also tells us the number of rows, columns and type of data\ntest_df.info()","metadata":{"execution":{"iopub.execute_input":"2023-02-25T13:25:28.148378Z","iopub.status.busy":"2023-02-25T13:25:28.147284Z","iopub.status.idle":"2023-02-25T13:25:28.161907Z","shell.execute_reply":"2023-02-25T13:25:28.160861Z","shell.execute_reply.started":"2023-02-25T13:25:28.148279Z"},"id":"sfXkNBib5qyx","outputId":"eb801f97-6ec9-464f-cc3d-9d491c182041"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# get the unique labels in test data\ntest_df[\"label\"].unique()","metadata":{"execution":{"iopub.execute_input":"2023-02-25T13:25:28.164409Z","iopub.status.busy":"2023-02-25T13:25:28.163175Z","iopub.status.idle":"2023-02-25T13:25:28.173737Z","shell.execute_reply":"2023-02-25T13:25:28.172624Z","shell.execute_reply.started":"2023-02-25T13:25:28.164371Z"},"id":"MMDADUaF5qyy","outputId":"f972ed62-d34a-4bd2-d8cf-11b403418481"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2.2 Visualize the data","metadata":{"id":"iLbS4Fcd5qyy"}},{"cell_type":"markdown","source":"#### Training data","metadata":{"id":"Y44Qg7SW5qyz"}},{"cell_type":"code","source":"# plot the count of each label\nfig, ax = plt.subplots(figsize=(6,6))\nsns.countplot(data=train_df, y='label', ax=ax).set(title='\\nFigure 1. The Count of Each Label in Train Data\\n')\n\n# plot the proportion of each label\nlabels = train_df['label'].unique().tolist()\ncounts = train_df['label'].value_counts()\nsizes = [counts[v] for v in labels]\nfig1, ax1 = plt.subplots()\nax1.pie(sizes, labels=labels, autopct='%0.0f%%')\nax1.axis('equal')\nplt.title(\"\\nFigure 2. The Proportion of Each Label in Train Data\\n\")\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2023-02-25T13:25:28.180240Z","iopub.status.busy":"2023-02-25T13:25:28.179303Z","iopub.status.idle":"2023-02-25T13:25:28.650816Z","shell.execute_reply":"2023-02-25T13:25:28.649690Z","shell.execute_reply.started":"2023-02-25T13:25:28.180205Z"},"id":"NylMCcvx5qy0","outputId":"8c1bcd87-ab8e-4082-8d2d-081e0e7455e3"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From Figure 1 and 2, we can see that there are 5000 images per class and the proportion of each class is 10% in the training data.","metadata":{"id":"uNNFLbWp5qy1"}},{"cell_type":"markdown","source":"#### Test data","metadata":{"id":"-7UyqjY15qy2"}},{"cell_type":"code","source":"# plot the count of each label\nfig, ax = plt.subplots(figsize=(6,6))\nsns.countplot(data=test_df, y='label', ax=ax).set(title='\\nFigure 3. The Count of Each Label in Test Data\\n')\n\n# plot the proportion of each label\nlabels = test_df['label'].unique().tolist()\ncounts = test_df['label'].value_counts()\nsizes = [counts[v] for v in labels]\nfig1, ax1 = plt.subplots()\nax1.pie(sizes, labels=labels, autopct='%0.0f%%')\nax1.axis('equal')\nplt.title(\"\\nFigure 4. The Proportion of Each Label in Test Data\\n\")\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2023-02-25T13:25:28.653440Z","iopub.status.busy":"2023-02-25T13:25:28.652727Z","iopub.status.idle":"2023-02-25T13:25:29.093582Z","shell.execute_reply":"2023-02-25T13:25:29.092609Z","shell.execute_reply.started":"2023-02-25T13:25:28.653401Z"},"id":"X_zYeBQh5qy2","outputId":"e437e2e0-8bf8-4afd-f740-98d034e6244a"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From Figure 3 and 4, we can see that there are 1000 images per class and the proportion of each class is 10% in the test data.","metadata":{"id":"du0zWdi-5qy3"}},{"cell_type":"code","source":"# create a list of class\nclasses = ['Airplane','Automobile','Bird','Cat','Deer','Dog','Frog','Horse','Ship','Truck']","metadata":{"execution":{"iopub.execute_input":"2023-02-25T13:25:29.095927Z","iopub.status.busy":"2023-02-25T13:25:29.094922Z","iopub.status.idle":"2023-02-25T13:25:29.100996Z","shell.execute_reply":"2023-02-25T13:25:29.099748Z","shell.execute_reply.started":"2023-02-25T13:25:29.095888Z"},"id":"SUjdGhsv5qy3"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# visualize some images in train data\nfig, axes = plt.subplots(5, 5, figsize=(10, 10))\nfor i, ax in enumerate(axes.flat):\n    ax.imshow(x_train[i])\n    ax.set(xticks=[], yticks=[], xlabel = classes[y_train[i][0]])\nfig.suptitle('\\nFigure 5. Train image Visualizations') \nplt.show() ","metadata":{"execution":{"iopub.execute_input":"2023-02-25T13:25:29.103833Z","iopub.status.busy":"2023-02-25T13:25:29.102681Z","iopub.status.idle":"2023-02-25T13:25:30.184560Z","shell.execute_reply":"2023-02-25T13:25:30.183556Z","shell.execute_reply.started":"2023-02-25T13:25:29.103780Z"},"id":"Lc-Gpu5G5qy4","outputId":"9601e241-5d9a-4a74-913d-b99cb378e46f"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# visualize some images in test data\nfig, axes = plt.subplots(5, 5, figsize=(10, 10))\nfor i, ax in enumerate(axes.flat):\n    ax.imshow(x_test[i])\n    ax.set(xticks=[], yticks=[], xlabel = classes[y_test[i][0]])\nfig.suptitle('\\nFigure 6. Test image Visualizations') \nplt.show() ","metadata":{"execution":{"iopub.execute_input":"2023-02-25T13:25:30.185971Z","iopub.status.busy":"2023-02-25T13:25:30.185645Z","iopub.status.idle":"2023-02-25T13:25:31.524057Z","shell.execute_reply":"2023-02-25T13:25:31.523187Z","shell.execute_reply.started":"2023-02-25T13:25:30.185940Z"},"id":"LGNsQOJJ5qy4","outputId":"ec77d392-eefa-41b5-810e-0522ac5d1fdf"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the output above, we can summarize that:\n\n+ There are 50,000 images labeled over 10 categories in training data.\n+ There are 10,000 images labeled over 10 categories in test data.\n+ There is no missing values.     \n+ The shape of each image is (32,32,3).\n+ All categories are in balance, with each category distribution at 10%.","metadata":{"id":"xN4TdMIq5qy4"}},{"cell_type":"markdown","source":"### 2.3 Data Preprocessing","metadata":{"id":"ML8iwiFZ5qy5"}},{"cell_type":"markdown","source":"Before we can proceed with building the model, I will:  \n\n(1) perform the one-hot encoding for labels   \n(2) set random_state, batch_size, number of categories, input_shape, epochs, step_size_train, step_size_test   \n(3) To work with neural networks, we need to normalize the dataset, otherwise, it could take a lot longer for the network to converge on a solution. As we are working with 32 x 32 NumPy arrays representing each image and each pixel in the array has an intensity somewhere between 1 — 255, a simpler way of getting all of these images on a scale between 0–1 is to divide each array by 255. To make our model much more accurate we can add data augmentation on our data.","metadata":{"id":"3s5KN3gI5qy5"}},{"cell_type":"code","source":"# perform the one-hot encoding for labels\ny_train = np_utils.to_categorical(y_train)\ny_test = np_utils.to_categorical(y_test)\nprint(\"There are {} training images, labeled over {} categories.\".format(y_train.shape[0], y_train.shape[1]))\nprint(\"There are {} test images, labeled over {} categories.\".format(y_test.shape[0], y_test.shape[1]))","metadata":{"execution":{"iopub.execute_input":"2023-02-25T13:25:31.530962Z","iopub.status.busy":"2023-02-25T13:25:31.528263Z","iopub.status.idle":"2023-02-25T13:25:31.543619Z","shell.execute_reply":"2023-02-25T13:25:31.542457Z","shell.execute_reply.started":"2023-02-25T13:25:31.530923Z"},"id":"9pbOE3zD5qy6","outputId":"8289ead3-d863-4344-c8d2-c791f48e0cc5"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# view some first rows of y_train after encoding\ny_train[:5]","metadata":{"execution":{"iopub.execute_input":"2023-02-25T13:25:31.551380Z","iopub.status.busy":"2023-02-25T13:25:31.548533Z","iopub.status.idle":"2023-02-25T13:25:31.561670Z","shell.execute_reply":"2023-02-25T13:25:31.560759Z","shell.execute_reply.started":"2023-02-25T13:25:31.551339Z"},"id":"q-1XNRxZ5qy7","outputId":"4ee596f0-2722-4075-cb91-b4c630935905"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# set random state\nRANDOM_STATE = 42\n\n# set batch size\nBATCH_SIZE = 64\n\n# set input shape\nINPUT_SHAPE = x_train.shape[1:]\n\n# set the number of categories\nNUM_LABELS = y_train.shape[1]\n\n# set the number of epochs\nEPOCHS = 20\n\n# set step_size_train and step_size_test\nSTEP_SIZE_TRAIN = x_train.shape[0]//BATCH_SIZE\nSTEP_SIZE_TEST = x_test.shape[0]//BATCH_SIZE","metadata":{"execution":{"iopub.execute_input":"2023-02-25T13:25:31.567134Z","iopub.status.busy":"2023-02-25T13:25:31.566331Z","iopub.status.idle":"2023-02-25T13:25:31.573434Z","shell.execute_reply":"2023-02-25T13:25:31.571994Z","shell.execute_reply.started":"2023-02-25T13:25:31.567098Z"},"id":"80laYpVv5qy8"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train data augmentation\ntrain_datagen = ImageDataGenerator(featurewise_center=False,  # set input mean to 0 over the dataset  \n                              featurewise_std_normalization=False,# divide inputs by std of the dataset\n                              zoom_range = 0.2, # Randomly zoom image \n                              rotation_range = 20,  # randomly rotate images in the range (degrees, 0 to 180)\n                              width_shift_range=3/32,  # randomly shift images horizontally (fraction of total width)\n                              height_shift_range=3/32,  # randomly shift images vertically (fraction of total height)\n                              horizontal_flip = True,  # randomly flip image\n                              rescale = 1./255)    # multiply the data by the value provided\ntrain_generator = train_datagen.flow(x_train, y_train, batch_size=BATCH_SIZE)\n\n# test data augmentation\ntest_datagen = ImageDataGenerator(rescale = 1./255)\ntest_generator = test_datagen.flow(x_test, y_test, batch_size=BATCH_SIZE)\n","metadata":{"execution":{"iopub.execute_input":"2023-02-25T13:25:31.613749Z","iopub.status.busy":"2023-02-25T13:25:31.613324Z","iopub.status.idle":"2023-02-25T13:25:31.623943Z","shell.execute_reply":"2023-02-25T13:25:31.622626Z","shell.execute_reply.started":"2023-02-25T13:25:31.613716Z"},"id":"Ooa5yh8-5qy-"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Part 3: Building and Training models","metadata":{"id":"6fxaoZ6n5qy_"}},{"cell_type":"markdown","source":"Convolutional Neural Networks are used for classifying and recognizing images because of its high accuracy and its built-in convolutional layer reduces the high dimensionality of images without losing its information. A CNN follows a hierarchical model which works on building a network, like a funnel, and finally gives out a fully connected layer where all the neurons are connected to each other and the output is processed. That is why CNNs are especially suited for image classification.  \n\nIn this project, I will build and train 4 different CNNs and 2 different ResNets model:","metadata":{}},{"cell_type":"markdown","source":"### Model 1: Convolutional Neural Network (CNN)","metadata":{"id":"b5veF8gV5qy_"}},{"cell_type":"markdown","source":"Model 1 is comprised of:  \n\nA simple CNN model with 2 Convolutional layers followed by max-pooling layers. BatchNormalization normalize the activation of the previous layer at each batch. Softmax is used as the activation function for the final layer of the category classifier. Use categorical_crossentropy loss function for our category-class classification problem because we have more than two labels. For simplicity, use accuracy as our evaluation metrics to evaluate the model during training and testing.  \n\nArchitecture: \n  + two convolutional layers\n  + hidden layer activations: relu\n  + BatchNormalization\n  + pool_size: (2,2)\n  + final layer activation: softmax","metadata":{"id":"gpPmzDX85qy_"}},{"cell_type":"code","source":"# build CNN model\nmodel1 = Sequential()\n# first convolutional layer\nmodel1.add(Conv2D(64, (3, 3), input_shape=INPUT_SHAPE))\nmodel1.add(BatchNormalization())\nmodel1.add(Activation('relu'))\nmodel1.add(MaxPooling2D(pool_size=(2, 2)))\n\n# second convolutional layer\nmodel1.add(Conv2D(128, (3, 3)))\nmodel1.add(BatchNormalization())\nmodel1.add(Activation('relu'))\nmodel1.add(MaxPooling2D(pool_size=(2, 2)))\n\nmodel1.add(Flatten())\nmodel1.add(Dense(256))\nmodel1.add(BatchNormalization())\nmodel1.add(Activation('relu'))\n\n# Out layer\nmodel1.add(Dense(NUM_LABELS))\nmodel1.add(Activation('softmax'))\n\n# summary model 1\nmodel1.summary()","metadata":{"execution":{"iopub.execute_input":"2023-02-25T13:25:31.840711Z","iopub.status.busy":"2023-02-25T13:25:31.840290Z","iopub.status.idle":"2023-02-25T13:25:34.526131Z","shell.execute_reply":"2023-02-25T13:25:34.525386Z","shell.execute_reply.started":"2023-02-25T13:25:31.840676Z"},"id":"MZ0dz2H05qy_","outputId":"23c85964-a71b-4813-e279-3105855a15ca"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plot model 1\nplot_model(model1, show_shapes=True, show_layer_names=True)","metadata":{"execution":{"iopub.execute_input":"2023-02-25T13:25:34.527647Z","iopub.status.busy":"2023-02-25T13:25:34.527170Z","iopub.status.idle":"2023-02-25T13:25:35.396752Z","shell.execute_reply":"2023-02-25T13:25:35.395603Z","shell.execute_reply.started":"2023-02-25T13:25:34.527608Z"},"id":"N8vAL1BE5qzA","outputId":"f6deda7a-c2c2-4fb4-c426-b89347747606"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's compile the model now using: \n+ optimizer: Adam, we are using a lower learning rate of 0.0001 for a smoother curve.  \n+ loss function: categorical_crossentropy because we have more than two labels.\n+ metrics: accuracy     ","metadata":{"id":"hE8Qkc2m5qzA"}},{"cell_type":"code","source":"# compile the model\nmodel1.compile(loss='categorical_crossentropy',\n              optimizer='adam',\n              metrics=['accuracy'])","metadata":{"execution":{"iopub.execute_input":"2023-02-25T13:25:35.399745Z","iopub.status.busy":"2023-02-25T13:25:35.399027Z","iopub.status.idle":"2023-02-25T13:25:35.418966Z","shell.execute_reply":"2023-02-25T13:25:35.418009Z","shell.execute_reply.started":"2023-02-25T13:25:35.399697Z"},"id":"MWNr-E4p5qzA"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# let’s train our model for 20 epochs\nres1 = model1.fit(train_generator,\n                epochs = EPOCHS,\n                verbose = 1,\n                steps_per_epoch=STEP_SIZE_TRAIN,\n                validation_steps=STEP_SIZE_TEST,\n                validation_data = test_generator)          ","metadata":{"execution":{"iopub.execute_input":"2023-02-25T13:25:35.421104Z","iopub.status.busy":"2023-02-25T13:25:35.420734Z","iopub.status.idle":"2023-02-25T13:40:17.347605Z","shell.execute_reply":"2023-02-25T13:40:17.346504Z","shell.execute_reply.started":"2023-02-25T13:25:35.421067Z"},"id":"RmltMCm85qzA","outputId":"608e1f66-4b59-4930-902c-8303b82cede9"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plot the graph of accuracy\nepochs_range = range(EPOCHS)\n\nacc1 = res1.history['accuracy']\ntest_acc1 = res1.history['val_accuracy']\nplt.figure(figsize=(15, 15))\nplt.subplot(2, 2, 1)\nplt.plot(epochs_range, acc1, label='Training Accuracy', color = \"blue\", marker = \"o\")\nplt.plot(epochs_range, test_acc1, label='Test Accuracy', color = \"red\", marker = \"o\")\nplt.legend(loc='lower right')\nplt.title('Figure 7. CNN Base Model: Training and Test Accuracy')\n\n# plot the graph of loss\nloss1 = res1.history['loss']\ntest_loss1 = res1.history['val_loss']\nplt.subplot(2, 2, 2)\nplt.plot(epochs_range, loss1, label='Training Loss', color = \"blue\", marker = \"o\")\nplt.plot(epochs_range, test_loss1, label='Test Loss', color = \"red\", marker = \"o\")\nplt.legend(loc='upper right')\nplt.title('Figure 8. CNN Base Model: Training and Test Loss')\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2023-02-25T13:40:17.355818Z","iopub.status.busy":"2023-02-25T13:40:17.355492Z","iopub.status.idle":"2023-02-25T13:40:17.770642Z","shell.execute_reply":"2023-02-25T13:40:17.767484Z","shell.execute_reply.started":"2023-02-25T13:40:17.355788Z"},"id":"Fyuz64Q75qzA","outputId":"e9117f8e-d12c-4f33-dc9f-8119a9f657dc"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Model 2: CNN with two layers and adding dropout","metadata":{"id":"O_yrOmux5qzB"}},{"cell_type":"markdown","source":"Model 2 is comprised of:  \n\nA CNN with two layers adding dropout at each convolutional layer to avoid overfitting. Dropout is a technique where randomly selected neurons are ignored during training. This means that their contribution to the activation of downstream neurons is temporally removed on the forward pass, and any weight updates are not applied to the neuron on the backward pass. When we have training data, if we try to train our model too much, it might overfit, and dropout regularization is one technique used to tackle overfitting problems in deep learning.  \n\nArchitecture:\n  + two convolutional layers\n  + hidden layer activations: relu\n  + BatchNormalization\n  + pool_size: (2,2)\n  + dropout: 0.2\n  + final layer activation: softmax","metadata":{"id":"6CAanOdf5qzB"}},{"cell_type":"code","source":"# build CNN model adding dropout\n\nmodel2 = Sequential()\n# first convolutional layer\nmodel2.add(Conv2D(64, (3, 3), input_shape=INPUT_SHAPE))\nmodel2.add(BatchNormalization())\nmodel2.add(Activation('relu'))\nmodel2.add(MaxPooling2D(pool_size=(2, 2)))\nmodel2.add(Dropout(0.2))\n\n# second convolutional layer\nmodel2.add(Conv2D(128, (3, 3)))\nmodel2.add(BatchNormalization())\nmodel2.add(Activation('relu'))\nmodel2.add(MaxPooling2D(pool_size=(2, 2)))\nmodel2.add(Dropout(0.2))\n\nmodel2.add(Flatten())\nmodel2.add(Dense(256))\nmodel2.add(BatchNormalization())\nmodel2.add(Activation('relu'))\nmodel2.add(Dropout(0.2))\n\n# Out layer\nmodel2.add(Dense(NUM_LABELS))\nmodel2.add(Activation('softmax'))\n\n# summary model 2\nmodel2.summary()","metadata":{"execution":{"iopub.execute_input":"2023-02-25T13:40:17.772571Z","iopub.status.busy":"2023-02-25T13:40:17.772190Z","iopub.status.idle":"2023-02-25T13:40:18.201887Z","shell.execute_reply":"2023-02-25T13:40:18.201128Z","shell.execute_reply.started":"2023-02-25T13:40:17.772534Z"},"id":"V2pagB1Z5qzB","outputId":"1e4b3c53-a168-43a1-de54-39dde4cd3502"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# compile the model 2\nmodel2.compile(loss='categorical_crossentropy',\n              optimizer='adam',\n              metrics=['accuracy'])","metadata":{"execution":{"iopub.execute_input":"2023-02-25T13:40:18.203322Z","iopub.status.busy":"2023-02-25T13:40:18.202978Z","iopub.status.idle":"2023-02-25T13:40:18.227541Z","shell.execute_reply":"2023-02-25T13:40:18.226742Z","shell.execute_reply.started":"2023-02-25T13:40:18.203287Z"},"id":"fItoC_kR5qzC"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# let’s train our model for 20 epochs\nres2 = model2.fit(train_generator,\n                epochs = EPOCHS,\n                steps_per_epoch=STEP_SIZE_TRAIN,\n                validation_steps=STEP_SIZE_TEST, \n                verbose = 1,\n                validation_data = test_generator)              ","metadata":{"execution":{"iopub.execute_input":"2023-02-25T13:40:29.370800Z","iopub.status.busy":"2023-02-25T13:40:29.370394Z","iopub.status.idle":"2023-02-25T13:56:45.457027Z","shell.execute_reply":"2023-02-25T13:56:45.456000Z","shell.execute_reply.started":"2023-02-25T13:40:29.370763Z"},"id":"UDRDYZ3z5qzC","outputId":"a77b42a2-231b-4adb-c364-18e095f3645f"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plot the graph of accuracy\nepochs_range = range(EPOCHS)\n\nacc2 = res2.history['accuracy']\ntest_acc2 = res2.history['val_accuracy']\nplt.figure(figsize=(15, 15))\nplt.subplot(2, 2, 1)\nplt.plot(epochs_range, acc2, label='Training Accuracy', color = \"blue\", marker = \"o\")\nplt.plot(epochs_range, test_acc2, label='Test Accuracy', color = \"red\", marker = \"o\")\nplt.legend(loc='lower right')\nplt.title('Figure 9. CNN Adding Dropout Model: Training and Test Accuracy')\n\n# plot the graph of loss\nloss2 = res2.history['loss']\ntest_loss2 = res2.history['val_loss']\nplt.subplot(2, 2, 2)\nplt.plot(epochs_range, loss2, label='Training Loss', color = \"blue\", marker = \"o\")\nplt.plot(epochs_range, test_loss2, label='Test Loss', color = \"red\", marker = \"o\")\nplt.legend(loc='upper right')\nplt.title('Figure 10. CNN Adding Dropout Model: Training and Test Loss')\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2023-02-25T13:56:45.459597Z","iopub.status.busy":"2023-02-25T13:56:45.459132Z","iopub.status.idle":"2023-02-25T13:56:45.853552Z","shell.execute_reply":"2023-02-25T13:56:45.852613Z","shell.execute_reply.started":"2023-02-25T13:56:45.459560Z"},"id":"Sh1YfwYK5qzC","outputId":"de41e88f-7933-4bd5-a43c-b7c01dcf07e2"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Model 3: CNN with three layers and adding dropout","metadata":{"id":"VjS5IQc05qzD"}},{"cell_type":"markdown","source":"Model 3 is comprised of:  \n\nA CNN from model 2 but adding one more convolutional layer to see which result will we get. \n\nArchitecture:\n  + three convolutional layers\n  + hidden layer activations: relu\n  + BatchNormalization\n  + pool_size: (2,2)\n  + dropout: 0.2\n  + final layer activation: softmax","metadata":{"id":"i_2uScJr5qzD"}},{"cell_type":"code","source":"# build CNN model adding more layers with more filters\nmodel3 = Sequential()\n# first convolutional layer\nmodel3.add(Conv2D(64, (3, 3), input_shape=INPUT_SHAPE))\nmodel3.add(BatchNormalization())\nmodel3.add(Activation('relu'))\nmodel3.add(MaxPooling2D(pool_size=(2, 2)))\nmodel3.add(Dropout(0.2))\n\n# second convolutional layer\nmodel3.add(Conv2D(128, (3, 3)))\nmodel3.add(BatchNormalization())\nmodel3.add(Activation('relu'))\nmodel3.add(MaxPooling2D(pool_size=(2, 2)))\nmodel3.add(Dropout(0.2))\n  \n# third convolutional layer\nmodel3.add(Conv2D(64, (3, 3)))\nmodel3.add(BatchNormalization())\nmodel3.add(Activation('relu'))\nmodel3.add(MaxPooling2D(pool_size=(2, 2)))\nmodel3.add(Dropout(0.2))\n\nmodel3.add(Flatten())\nmodel3.add(Dense(256))\nmodel3.add(BatchNormalization())\nmodel3.add(Activation('relu'))\nmodel3.add(Dropout(0.2))\n\n# Out layer\nmodel3.add(Dense(NUM_LABELS))\nmodel3.add(Activation('softmax'))\n\n# summary model 3\nmodel3.summary()","metadata":{"execution":{"iopub.execute_input":"2023-02-25T13:56:45.855929Z","iopub.status.busy":"2023-02-25T13:56:45.855236Z","iopub.status.idle":"2023-02-25T13:56:46.042381Z","shell.execute_reply":"2023-02-25T13:56:46.041572Z","shell.execute_reply.started":"2023-02-25T13:56:45.855887Z"},"id":"zFhvPA9e5qzD","outputId":"dea379c2-02ca-4a31-a053-a7d1bb46573c"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# compile the model 3\nmodel3.compile(loss='categorical_crossentropy',\n              optimizer='adam',\n              metrics=['accuracy'])","metadata":{"execution":{"iopub.execute_input":"2023-02-25T13:56:46.044997Z","iopub.status.busy":"2023-02-25T13:56:46.044495Z","iopub.status.idle":"2023-02-25T13:56:46.070953Z","shell.execute_reply":"2023-02-25T13:56:46.069977Z","shell.execute_reply.started":"2023-02-25T13:56:46.044957Z"},"id":"z8RUHDqM5qzF"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# let’s train our model for 20 epochs\nres3 = model3.fit(train_generator,\n                epochs = EPOCHS,\n                steps_per_epoch=STEP_SIZE_TRAIN,\n                validation_steps=STEP_SIZE_TEST,\n                verbose = 1, \n                validation_data = test_generator)","metadata":{"execution":{"iopub.execute_input":"2023-02-25T13:56:46.072872Z","iopub.status.busy":"2023-02-25T13:56:46.072128Z","iopub.status.idle":"2023-02-25T14:15:27.611735Z","shell.execute_reply":"2023-02-25T14:15:27.610732Z","shell.execute_reply.started":"2023-02-25T13:56:46.072829Z"},"id":"LmBYsFXW5qzG","outputId":"56feb23c-0735-421f-f4fd-0186206a2ada"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plot the graph of accuracy\nepochs_range = range(EPOCHS)\n\nacc3 = res3.history['accuracy']\ntest_acc3 = res3.history['val_accuracy']\nplt.figure(figsize=(15, 15))\nplt.subplot(2, 2, 1)\nplt.plot(epochs_range, acc3, label='Training Accuracy', color = \"blue\", marker = \"o\")\nplt.plot(epochs_range, test_acc3, label='Test Accuracy', color = \"red\", marker = \"o\")\nplt.legend(loc='lower right')\nplt.title('Figure 11. CNN With Three Layers Model: Training and Test Accuracy')\n\n# plot the graph of loss\nloss3 = res3.history['loss']\ntest_loss3 = res3.history['val_loss']\nplt.subplot(2, 2, 2)\nplt.plot(epochs_range, loss3, label='Training Loss', color = \"blue\", marker = \"o\")\nplt.plot(epochs_range, test_loss3, label='Test Loss', color = \"red\", marker = \"o\")\nplt.legend(loc='upper right')\nplt.title('Figure 12. CNN With Three Layers Model: Training and Test Loss')\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2023-02-25T14:15:27.614089Z","iopub.status.busy":"2023-02-25T14:15:27.613463Z","iopub.status.idle":"2023-02-25T14:15:28.021473Z","shell.execute_reply":"2023-02-25T14:15:28.020603Z","shell.execute_reply.started":"2023-02-25T14:15:27.614047Z"},"id":"ON9a0GVv5qzG","outputId":"6efcb3bb-16ac-478c-a95c-b22809f901c7"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Model 4: CNN with three layers using Earlystopping and ReduceLROnPlateau","metadata":{"id":"xP-q2kgz5qzG"}},{"cell_type":"markdown","source":"Next, let's use Earlystopping to avoid overfitting by terminating the process early. Since the goal of a training is to maximize the validation accuracy, so we can set up the metrics as:  \n    + monitor = \"val_acc\": value being monitored.  \n    + mode = \"max\": training will stop when the quantity monitored has stopped inreasing.   \n    + patience = 5: number of epochs with no improvement after which training will be stopped.  \n    \nMoreover, let's use Reduce learning rate when a metric has stopped improving. Models often benefit from reducing the learning rate by a factor of 2-10 once learning stagnates. This callback monitors a quantity and if no improvement is seen for a 'patience' number of epochs, the learning rate is reduced.  \n    + factor = 0.5: factor by which the learning rate will be reduced (new_learning_rate = learning_rate * factor).  \n    + min_lr = 0.00001: lower bound on the learning rate.  \n\nA model.fit() training loop will check at end of every epoch whether the accuracy is no longer increasing, considering the monitor and patience if applicable. Once it's found no longer increasing, model.stop_training is marked True and the training terminates.","metadata":{"id":"9aqpyhQI5qzH"}},{"cell_type":"code","source":"# set up callback\nes= tf.keras.callbacks.EarlyStopping(monitor='val_accuracy', mode='max', patience=5, verbose=1)\nreduce_lr = ReduceLROnPlateau(monitor='val_accuracy', factor=0.5, patience=5, \n                                   verbose=1, mode='max', min_lr=0.00001)\ncallback=[es, reduce_lr]","metadata":{"execution":{"iopub.execute_input":"2023-02-25T14:23:26.897902Z","iopub.status.busy":"2023-02-25T14:23:26.897496Z","iopub.status.idle":"2023-02-25T14:23:26.904201Z","shell.execute_reply":"2023-02-25T14:23:26.903090Z","shell.execute_reply.started":"2023-02-25T14:23:26.897865Z"},"id":"85fRmJfk5qzH"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# build CNN model adding more layers with more filters\nmodel4 = Sequential()\n# first convolutional layer\nmodel4.add(Conv2D(32, (3, 3), input_shape=INPUT_SHAPE))\nmodel4.add(BatchNormalization())\nmodel4.add(Activation('relu'))\nmodel4.add(MaxPooling2D(pool_size=(2, 2)))\nmodel4.add(Dropout(0.2))\n\n# second convolutional layer\nmodel4.add(Conv2D(64, (3, 3)))\nmodel4.add(BatchNormalization())\nmodel4.add(Activation('relu'))\nmodel4.add(MaxPooling2D(pool_size=(2, 2)))\nmodel4.add(Dropout(0.2))\n  \n# third convolutional layer\nmodel4.add(Conv2D(128, (3, 3)))\nmodel4.add(BatchNormalization())\nmodel4.add(Activation('relu'))\nmodel4.add(MaxPooling2D(pool_size=(2, 2)))\nmodel4.add(Dropout(0.2))\n\nmodel4.add(Flatten())\nmodel4.add(Dense(256))\nmodel4.add(BatchNormalization())\nmodel4.add(Activation('relu'))\nmodel4.add(Dropout(0.2))\n\n# Out layer\nmodel4.add(Dense(NUM_LABELS))\nmodel4.add(Activation('softmax'))\n\n# summary model 4\nmodel4.summary()","metadata":{"execution":{"iopub.execute_input":"2023-02-25T14:23:29.375708Z","iopub.status.busy":"2023-02-25T14:23:29.375337Z","iopub.status.idle":"2023-02-25T14:23:29.574152Z","shell.execute_reply":"2023-02-25T14:23:29.573315Z","shell.execute_reply.started":"2023-02-25T14:23:29.375652Z"},"id":"uswkNkgy5qzH","outputId":"5b3ca1fd-ab4e-4539-b313-f09b81485d88"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# compile the model 4\nmodel4.compile(loss='categorical_crossentropy',\n              optimizer='adam',\n              metrics=['accuracy'])","metadata":{"execution":{"iopub.execute_input":"2023-02-25T14:23:35.511163Z","iopub.status.busy":"2023-02-25T14:23:35.510805Z","iopub.status.idle":"2023-02-25T14:23:35.522268Z","shell.execute_reply":"2023-02-25T14:23:35.521204Z","shell.execute_reply.started":"2023-02-25T14:23:35.511131Z"},"id":"05cCXtkZ5qzL"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# let’s train our model for 20 epochs\nres4 = model4.fit(train_generator,\n                epochs = EPOCHS,\n                steps_per_epoch=STEP_SIZE_TRAIN,\n                validation_steps=STEP_SIZE_TEST,\n                verbose = 1, \n                validation_data = test_generator,\n                callbacks=callback)","metadata":{"execution":{"iopub.execute_input":"2023-02-25T14:23:37.074142Z","iopub.status.busy":"2023-02-25T14:23:37.073390Z","iopub.status.idle":"2023-02-25T14:31:47.726101Z","shell.execute_reply":"2023-02-25T14:31:47.725179Z","shell.execute_reply.started":"2023-02-25T14:23:37.074092Z"},"id":"08r21U1K5qzL","outputId":"e72ae817-0838-4121-b785-10ce050b75e0"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plot the graph of accuracy\nepochs_range = range(20)\n\nacc4 = res4.history['accuracy']\ntest_acc4 = res4.history['val_accuracy']\nplt.figure(figsize=(15, 15))\nplt.subplot(2, 2, 1)\nplt.plot(epochs_range, acc4, label='Training Accuracy', color = \"blue\", marker = \"o\")\nplt.plot(epochs_range, test_acc4, label='Test Accuracy', color = \"red\", marker = \"o\")\nplt.legend(loc='lower right')\nplt.title('Figure 13. CNN Using Early Stopping Model: Training and Test Accuracy')\n\n# plot the graph of loss\nloss4 = res4.history['loss']\ntest_loss4 = res4.history['val_loss']\nplt.subplot(2, 2, 2)\nplt.plot(epochs_range, loss4, label='Training Loss', color = \"blue\", marker = \"o\")\nplt.plot(epochs_range, test_loss4, label='Test Loss', color = \"red\", marker = \"o\")\nplt.legend(loc='upper right')\nplt.title('Figure 14. CNN Using Early Stopping Model: Training and Test Loss')\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2023-02-25T14:36:21.764053Z","iopub.status.busy":"2023-02-25T14:36:21.763648Z","iopub.status.idle":"2023-02-25T14:36:22.129534Z","shell.execute_reply":"2023-02-25T14:36:22.128613Z","shell.execute_reply.started":"2023-02-25T14:36:21.764020Z"},"id":"b7iGIT1W5qzM","outputId":"61d16cb3-358a-4974-85cc-18a444be6a00"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Model 5: ResNet50 Model with \"Imagenet\" pretrained weights","metadata":{"id":"GGl-J6IkeMBg"}},{"cell_type":"markdown","source":"ResNet50 is a 50 layers convolutional neural network included 48 convolutional layers, one MaxPool layer, and one average pool layer. I choose to build a ResNet50 with \"Imagenet\" weights because the Imagenet database contains a pretrained version of the network that has been trained on more than a million photos of 1000 different item categories from the Imagenet dataset, including several animals, a keyboard, a mouse, and a pencil. Moreover, this model has over 23 million trainable parameters which indicates a deep architecture that makes it better for image classification.\n\nSo that's why here, we will train a ResNet50 model with taking the \"Imagenet\" pretrained weights file loaded and repurposing it for cifar10 image classification. It helps achieve higher performance even if the model is trained on a smaller datatset. Since the training time took so long, in this model we just trained with only 10 epochs.","metadata":{"id":"ZZSNu5vQeYBj"}},{"cell_type":"code","source":"# build model 5\nmodel5 = None\nmodel5 = Sequential()\nmodel5.add(UpSampling2D((2,2)))\nmodel5.add(UpSampling2D((2,2)))\nmodel5.add(UpSampling2D((2,2)))\n\nresnet = tf.keras.applications.resnet50.ResNet50(include_top=False, \n                                                 weights=\"imagenet\", \n                                                 pooling = \"max\", \n                                                 classes=NUM_LABELS)\nmodel5.add(resnet)\nmodel5.add(Dropout(0.2))\n\n# Out layer\nmodel5.add(Dense(NUM_LABELS))\nmodel5.add(Activation('softmax'))\n\nmodel5.build(input_shape = (None, 32,32,3))\n\n# summary model 5\nmodel5.summary()","metadata":{"id":"k5yJhpWKpt6S","outputId":"c280fdfe-6a0f-404f-dc69-75df49c1179f"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# compile the model 5\nmodel5.compile(loss='categorical_crossentropy',\n              optimizer='adam',\n              metrics=['accuracy'])","metadata":{"id":"Uvpv5t-beas2"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# let’s train our model for 10 epochs\nres5 = model5.fit(train_generator,\n                steps_per_epoch=STEP_SIZE_TRAIN,\n                validation_steps=STEP_SIZE_TEST, \n                epochs = 10,\n                verbose = 1,\n                validation_data = test_generator)","metadata":{"id":"siaWioO-ea01","outputId":"2a4fb32a-665d-4a53-be8a-b7fefff1c56f"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plot the graph of accuracy\nepochs_range = range(10)\n\nacc5 = res5.history['accuracy']\ntest_acc5 = res5.history['val_accuracy']\nplt.figure(figsize=(15, 15))\nplt.subplot(2, 2, 1)\nplt.plot(epochs_range, acc5, label='Training Accuracy', color = \"blue\", marker = \"o\")\nplt.plot(epochs_range, test_acc5, label='Test Accuracy', color = \"red\", marker = \"o\")\nplt.legend(loc='lower right')\nplt.title('Figure 15. Resnet50 With \"Imagenet\" Pretrained Weights Model: Training and Test Accuracy')\n\n# plot the graph of loss\nloss5 = res5.history['loss']\ntest_loss5 = res5.history['val_loss']\nplt.subplot(2, 2, 2)\nplt.plot(epochs_range, loss5, label='Training Loss', color = \"blue\", marker = \"o\")\nplt.plot(epochs_range, test_loss5, label='Test Loss', color = \"red\", marker = \"o\")\nplt.legend(loc='upper right')\nplt.title('Figure 16. Resnet50 With \"Imagenet\" Pretrained Weights Model: Training and Test Loss')\nplt.show()","metadata":{"id":"SGrqP1V0ea9t","outputId":"85bbc4c5-bde2-44a8-e9fc-ac81dd1fbc4c"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Model 6: ResNet101 With \"Imagenet\" pretrained weights\n","metadata":{"id":"3QRqQyM-qfKd"}},{"cell_type":"markdown","source":"ResNet101 is a 101 layers convolutional neural network and here our model will also be trained with \"Imagenet\" pretrained weights file loaded.","metadata":{"id":"MzJ9Y4p3tKpX"}},{"cell_type":"code","source":"# build model 6\nmodel6 = None\nmodel6 = Sequential()\nmodel6.add(UpSampling2D((2,2)))\nmodel6.add(UpSampling2D((2,2)))\nmodel6.add(UpSampling2D((2,2)))\n\nresnet101 = tf.keras.applications.resnet.ResNet101(include_top=False, \n                                                   weights=\"imagenet\", \n                                                   pooling = \"max\", \n                                                   classes=NUM_LABELS)\nmodel6.add(resnet101)\nmodel6.add(Dropout(0.2))\n\n# Out layer\nmodel6.add(Dense(NUM_LABELS))\nmodel6.add(Activation('softmax'))\n\nmodel6.build(input_shape = (None, 32, 32, 3))\n\n# summary model 6\nmodel6.summary()","metadata":{"id":"Mbs55g2UqB2s","outputId":"1610ac72-3f3f-4e71-e977-da609c0512b1"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# compile the model 6\nmodel6.compile(loss='categorical_crossentropy',\n              optimizer='adam',\n              metrics=['accuracy'])","metadata":{"id":"tMEM8oQrs_-3"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# let’s train our model for 10 epochs\nres6 = model6.fit(train_generator,\n                steps_per_epoch=STEP_SIZE_TRAIN,\n                validation_steps=STEP_SIZE_TEST,\n                epochs = 10,\n                verbose = 1,\n                validation_data = test_generator)                                ","metadata":{"id":"zHb6ZJ_7tAEa","outputId":"d09b3d54-1366-4425-cf5c-65b5a7182372"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plot the graph of accuracy\nepochs_range = range(10)\n\nacc6 = res6.history['accuracy']\ntest_acc6 = res6.history['val_accuracy']\nplt.figure(figsize=(15, 15))\nplt.subplot(2, 2, 1)\nplt.plot(epochs_range, acc6, label='Training Accuracy', color = \"blue\", marker = \"o\")\nplt.plot(epochs_range, test_acc6, label='Test Accuracy', color = \"red\", marker = \"o\")\nplt.legend(loc='lower right')\nplt.title('Figure 17. Resnet101 With \"Imagenet\" Pretrained Weights Model: Training and Test Accuracy')\n\n# plot the graph of loss\nloss6 = res6.history['loss']\ntest_loss6 = res6.history['val_loss']\nplt.subplot(2, 2, 2)\nplt.plot(epochs_range, loss6, label='Training Loss', color = \"blue\", marker = \"o\")\nplt.plot(epochs_range, test_loss6, label='Test Loss', color = \"red\", marker = \"o\")\nplt.legend(loc='upper right')\nplt.title('Figure 18. Resnet101 With \"Imagenet\" Pretrained Weights Model: Training and Test Loss')\nplt.show()","metadata":{"id":"LiNwja0etALc","outputId":"d5315009-2b00-4f66-c6b9-1cbee54e383d"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Part 4: Results and Analysis","metadata":{"id":"200D9ve85qzM"}},{"cell_type":"code","source":"# Get the true labels\ny_true = np.argmax(y_test, axis = 1)","metadata":{"execution":{"iopub.execute_input":"2023-02-25T14:36:28.228551Z","iopub.status.busy":"2023-02-25T14:36:28.228165Z","iopub.status.idle":"2023-02-25T14:36:28.233308Z","shell.execute_reply":"2023-02-25T14:36:28.232351Z","shell.execute_reply.started":"2023-02-25T14:36:28.228518Z"},"id":"MgajomHY5qzM"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# x_test augmentation\nx_test_datagen = test_datagen.flow(x_test, y=None, batch_size=BATCH_SIZE, shuffle=False)","metadata":{"id":"WG8UrlRRTfVR"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Model 1","metadata":{"id":"PRcN8U5R5qzM"}},{"cell_type":"code","source":"# get loss and accuracy score\ntrain_loss1, train_acc1 = model1.evaluate(train_generator, verbose=0)\ntest_loss1, test_acc1 = model1.evaluate(test_generator, verbose=0)\nprint('train_loss_model1:', train_loss1)\nprint('train_acc_model1:', train_acc1)\nprint('test_loss_model1:', test_loss1)\nprint('test_acc_model1:', test_acc1)","metadata":{"execution":{"iopub.execute_input":"2023-02-25T14:36:31.395342Z","iopub.status.busy":"2023-02-25T14:36:31.394638Z","iopub.status.idle":"2023-02-25T14:37:15.009390Z","shell.execute_reply":"2023-02-25T14:37:15.008348Z","shell.execute_reply.started":"2023-02-25T14:36:31.395304Z"},"id":"hXegWem55qzM","outputId":"c6139109-ab1e-4af3-bc46-64e4ffe1a788"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# make predictions on the test dataset\ny_predict1 = model1.predict(x_test_datagen)","metadata":{"execution":{"iopub.execute_input":"2023-02-25T14:41:05.477348Z","iopub.status.busy":"2023-02-25T14:41:05.476632Z","iopub.status.idle":"2023-02-25T14:41:06.871657Z","shell.execute_reply":"2023-02-25T14:41:06.870567Z","shell.execute_reply.started":"2023-02-25T14:41:05.477309Z"},"id":"PQ3Mb25Z5qzN","outputId":"4b64d526-c825-47ee-cd7f-9729947b52e9"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# calculate roc_auc_score\nauc_score1 = roc_auc_score(y_test, y_predict1)\nprint('Model 1 roc_auc_score:', auc_score1)","metadata":{"execution":{"iopub.execute_input":"2023-02-25T14:41:32.668679Z","iopub.status.busy":"2023-02-25T14:41:32.668278Z","iopub.status.idle":"2023-02-25T14:41:32.709034Z","shell.execute_reply":"2023-02-25T14:41:32.707931Z","shell.execute_reply.started":"2023-02-25T14:41:32.668630Z"},"id":"Q1TOiw7W5qzN","outputId":"22befeff-613c-4db9-8423-6346851e60e8"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print out classification report\ny_pred1 = np.argmax(y_predict1, axis=1)\nprint(classification_report(y_true, y_pred1))","metadata":{"execution":{"iopub.execute_input":"2023-02-25T14:42:28.626204Z","iopub.status.busy":"2023-02-25T14:42:28.625844Z","iopub.status.idle":"2023-02-25T14:42:28.653348Z","shell.execute_reply":"2023-02-25T14:42:28.652213Z","shell.execute_reply.started":"2023-02-25T14:42:28.626173Z"},"id":"io4lzoUu5qzN","outputId":"a8140e80-3495-4250-f2c9-c8a3051a8379"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print out the confusion matrix\ncm1 = confusion_matrix(y_true, y_pred1)\nsns.heatmap(cm1, annot=True, fmt=\".0f\")","metadata":{"execution":{"iopub.execute_input":"2023-02-25T14:42:37.042923Z","iopub.status.busy":"2023-02-25T14:42:37.042547Z","iopub.status.idle":"2023-02-25T14:42:37.705040Z","shell.execute_reply":"2023-02-25T14:42:37.704152Z","shell.execute_reply.started":"2023-02-25T14:42:37.042890Z"},"id":"K4xJSFNV5qzN","outputId":"c42b3387-817b-4f7c-a67e-b84bff05a664"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Model 2","metadata":{"id":"UVPRVz1h5qzN"}},{"cell_type":"code","source":"# get loss and accuracy score\ntrain_loss2, train_acc2 = model2.evaluate(train_generator, verbose=0)\ntest_loss2, test_acc2 = model2.evaluate(test_generator, verbose=0)\nprint('train_loss_model2:', train_loss2)\nprint('train_acc_model2:', train_acc2)\nprint('test_loss_model2:', test_loss2)\nprint('test_acc_model2:', test_acc2)","metadata":{"execution":{"iopub.execute_input":"2023-02-25T14:44:52.532468Z","iopub.status.busy":"2023-02-25T14:44:52.532027Z","iopub.status.idle":"2023-02-25T14:45:35.276298Z","shell.execute_reply":"2023-02-25T14:45:35.275348Z","shell.execute_reply.started":"2023-02-25T14:44:52.532428Z"},"id":"-FIjQjPi5qzN","outputId":"050ff0a2-1681-4fc2-e388-fa230f147680"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# make predictions on the test dataset\ny_predict2 = model2.predict(x_test_datagen)","metadata":{"execution":{"iopub.execute_input":"2023-02-25T14:45:35.278788Z","iopub.status.busy":"2023-02-25T14:45:35.278393Z","iopub.status.idle":"2023-02-25T14:45:37.950707Z","shell.execute_reply":"2023-02-25T14:45:37.949695Z","shell.execute_reply.started":"2023-02-25T14:45:35.278750Z"},"id":"LCQ5G2Np5qzN","outputId":"08e7f531-5e92-462b-c372-7d913e35f8d8"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# calculate roc_auc_score\nauc_score2 = roc_auc_score(y_test, y_predict2)\nprint('Model 2 roc_auc_score:', auc_score2)","metadata":{"execution":{"iopub.execute_input":"2023-02-25T14:45:37.952857Z","iopub.status.busy":"2023-02-25T14:45:37.952258Z","iopub.status.idle":"2023-02-25T14:45:37.990039Z","shell.execute_reply":"2023-02-25T14:45:37.988882Z","shell.execute_reply.started":"2023-02-25T14:45:37.952817Z"},"id":"ji8MRNX05qzO","outputId":"90e5438c-99e5-460d-a1cc-dba310f68baa"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print out classification report\ny_pred2 = np.argmax(y_predict2, axis= 1)\nprint(classification_report(y_true, y_pred2))","metadata":{"execution":{"iopub.execute_input":"2023-02-25T14:45:37.993027Z","iopub.status.busy":"2023-02-25T14:45:37.992602Z","iopub.status.idle":"2023-02-25T14:45:38.020798Z","shell.execute_reply":"2023-02-25T14:45:38.019689Z","shell.execute_reply.started":"2023-02-25T14:45:37.992987Z"},"id":"L46iuEVk5qzO","outputId":"b43cb33f-725f-483e-d6bb-3171f8c425de"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print out the confusion matrix\ncm2 = confusion_matrix(y_true, y_pred2)\nsns.heatmap(cm2, annot=True, fmt=\".0f\")","metadata":{"execution":{"iopub.execute_input":"2023-02-25T14:45:38.022830Z","iopub.status.busy":"2023-02-25T14:45:38.022147Z","iopub.status.idle":"2023-02-25T14:45:38.723586Z","shell.execute_reply":"2023-02-25T14:45:38.722681Z","shell.execute_reply.started":"2023-02-25T14:45:38.022792Z"},"id":"BO1skasZ5qzO","outputId":"dfc69fff-9b66-42bf-9e85-3dfd37c7e12f"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Model 3","metadata":{"id":"4IgyWonI5qzP"}},{"cell_type":"code","source":"# get loss and accuracy score\ntrain_loss3, train_acc3 = model3.evaluate(train_generator, verbose=0)\ntest_loss3, test_acc3 = model3.evaluate(test_generator, verbose=0)\nprint('train_loss_model3:', train_loss3)\nprint('train_acc_model3:', train_acc3)\nprint('test_loss_model3:', test_loss3)\nprint('test_acc_model3:', test_acc3)","metadata":{"execution":{"iopub.execute_input":"2023-02-25T14:45:49.044530Z","iopub.status.busy":"2023-02-25T14:45:49.044175Z","iopub.status.idle":"2023-02-25T14:46:31.817994Z","shell.execute_reply":"2023-02-25T14:46:31.816789Z","shell.execute_reply.started":"2023-02-25T14:45:49.044498Z"},"id":"PbYtVShf5qzQ","outputId":"ffd2ab52-10ee-49e8-9632-a88098210298"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# make predictions on the test dataset\ny_predict3 = model3.predict(x_test_datagen)","metadata":{"execution":{"iopub.execute_input":"2023-02-25T14:46:31.820911Z","iopub.status.busy":"2023-02-25T14:46:31.820037Z","iopub.status.idle":"2023-02-25T14:46:34.505488Z","shell.execute_reply":"2023-02-25T14:46:34.504504Z","shell.execute_reply.started":"2023-02-25T14:46:31.820870Z"},"id":"2CVr1Dhj5qzQ","outputId":"27e37350-1f1c-435e-f351-9261af9d042b"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# calculate test roc_auc_score\nauc_score3 = roc_auc_score(y_test, y_predict3)\nprint('Model 3 roc_auc_score:', auc_score3)","metadata":{"execution":{"iopub.execute_input":"2023-02-25T14:46:34.507564Z","iopub.status.busy":"2023-02-25T14:46:34.507205Z","iopub.status.idle":"2023-02-25T14:46:34.545964Z","shell.execute_reply":"2023-02-25T14:46:34.544893Z","shell.execute_reply.started":"2023-02-25T14:46:34.507527Z"},"id":"ySkDU_3M5qzR","outputId":"d02c51fc-3350-4db5-8e6c-fd4b6f1ba320"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print out classification report\ny_pred3 = np.argmax(y_predict3, axis= 1)\nprint(classification_report(y_true, y_pred3))","metadata":{"execution":{"iopub.execute_input":"2023-02-25T14:46:34.548918Z","iopub.status.busy":"2023-02-25T14:46:34.548289Z","iopub.status.idle":"2023-02-25T14:46:34.576939Z","shell.execute_reply":"2023-02-25T14:46:34.575927Z","shell.execute_reply.started":"2023-02-25T14:46:34.548878Z"},"id":"8k-AZsCQ5qzR","outputId":"410c9048-c668-41fd-f73f-f4d6052fac92"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print out the confusion matrix\ncm3 = confusion_matrix(y_true, y_pred3)\nsns.heatmap(cm3, annot=True, fmt=\".0f\")","metadata":{"execution":{"iopub.execute_input":"2023-02-25T14:46:34.579224Z","iopub.status.busy":"2023-02-25T14:46:34.578322Z","iopub.status.idle":"2023-02-25T14:46:35.289504Z","shell.execute_reply":"2023-02-25T14:46:35.288596Z","shell.execute_reply.started":"2023-02-25T14:46:34.579184Z"},"id":"tkPHZocg5qzR","outputId":"a29e336e-8157-4921-cacc-1c8e5416d6dc"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Model 4","metadata":{"id":"RZCVGq0p5qzR"}},{"cell_type":"code","source":"# get loss and accuracy score\ntrain_loss4, train_acc4 = model4.evaluate(train_generator, verbose=0)\ntest_loss4, test_acc4 = model4.evaluate(test_generator, verbose=0)\nprint('train_loss_model4:', train_loss4)\nprint('train_acc_model4:', train_acc4)\nprint('test_loss_model4:', test_loss4)\nprint('test_acc_model4:', test_acc4)","metadata":{"execution":{"iopub.execute_input":"2023-02-25T14:46:41.571229Z","iopub.status.busy":"2023-02-25T14:46:41.570828Z","iopub.status.idle":"2023-02-25T14:47:22.433272Z","shell.execute_reply":"2023-02-25T14:47:22.432203Z","shell.execute_reply.started":"2023-02-25T14:46:41.571193Z"},"id":"i6CqU0qy5qzR","outputId":"eb840fa1-bb39-44c4-d049-e79a03956faa"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# make predictions on the test dataset\ny_predict4 = model4.predict(x_test_datagen)","metadata":{"execution":{"iopub.execute_input":"2023-02-25T14:47:22.440629Z","iopub.status.busy":"2023-02-25T14:47:22.437933Z","iopub.status.idle":"2023-02-25T14:47:24.447062Z","shell.execute_reply":"2023-02-25T14:47:24.446081Z","shell.execute_reply.started":"2023-02-25T14:47:22.440588Z"},"id":"OP79A-KQ5qzS","outputId":"1338291d-2f2e-4736-aad9-bd44e9be4566"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# calculate roc_auc_score\nauc_score4 = roc_auc_score(y_test, y_predict4)\nprint('Model 4 roc_auc_score:', auc_score4)","metadata":{"execution":{"iopub.execute_input":"2023-02-25T14:47:24.449205Z","iopub.status.busy":"2023-02-25T14:47:24.448759Z","iopub.status.idle":"2023-02-25T14:47:24.487128Z","shell.execute_reply":"2023-02-25T14:47:24.485855Z","shell.execute_reply.started":"2023-02-25T14:47:24.449163Z"},"id":"03VjW_Oi5qzS","outputId":"f507a9f0-cc67-484a-e591-5545bb7e9b2c"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print out classification report\ny_pred4 = np.argmax(y_predict4, axis= 1)\nprint(classification_report(y_true, y_pred4))","metadata":{"execution":{"iopub.execute_input":"2023-02-25T14:47:24.490738Z","iopub.status.busy":"2023-02-25T14:47:24.490419Z","iopub.status.idle":"2023-02-25T14:47:24.516639Z","shell.execute_reply":"2023-02-25T14:47:24.515574Z","shell.execute_reply.started":"2023-02-25T14:47:24.490710Z"},"id":"FSLk_YsU5qzS","outputId":"95e941d9-4a73-4c36-af1c-51e181ef1b1a"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print out the confusion matrix\ncm4 = confusion_matrix(y_true, y_pred4)\nsns.heatmap(cm4, annot=True, fmt=\".0f\")","metadata":{"execution":{"iopub.execute_input":"2023-02-25T14:47:24.518587Z","iopub.status.busy":"2023-02-25T14:47:24.518034Z","iopub.status.idle":"2023-02-25T14:47:25.345657Z","shell.execute_reply":"2023-02-25T14:47:25.344545Z","shell.execute_reply.started":"2023-02-25T14:47:24.518549Z"},"id":"XcO5tIu85qzS","outputId":"85b30f60-a97b-4771-e99c-10fdf61e1683"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Model 5","metadata":{"id":"CT56Qr1QtzR2"}},{"cell_type":"code","source":"# get loss and accuracy score\ntrain_loss5, train_acc5 = model5.evaluate(train_generator, verbose=0)\ntest_loss5, test_acc5 = model5.evaluate(test_generator, verbose=0)\nprint('train_loss_model5:', train_loss5)\nprint('train_acc_model5:', train_acc5)\nprint('test_loss_model5:', test_loss5)\nprint('test_acc_model5:', test_acc5)","metadata":{"id":"2XqADOcxt1wt","outputId":"3cd16ff7-8f88-4391-9800-861d78f314bc"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# make predictions on the test dataset\ny_predict5 = model5.predict(x_test_datagen)","metadata":{"id":"B1l-vBWJt16E","outputId":"ef94b053-ce36-4bf0-ddcb-838ceca4beb4"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# calculate roc_auc_score\nauc_score5 = roc_auc_score(y_test, y_predict5)\nprint('Model 5 roc_auc_score:', auc_score5)","metadata":{"id":"JtNyGeuRt2D7","outputId":"9e2aad48-a3ef-44f7-c873-49387f206317"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print out classification report\ny_pred5 = np.argmax(y_predict5,  axis = 1)\nprint(classification_report(y_true, y_pred5))","metadata":{"id":"k4UXpZ7Kt2K7","outputId":"dfbb9cda-9c05-4b70-de4c-5a28d0a516bb"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print out the confusion matrix\ncm5 = confusion_matrix(y_true, y_pred5)\nsns.heatmap(cm5, annot=True, fmt=\".0f\")","metadata":{"id":"0lSEAUG1t2Sw","outputId":"ac2a21be-858b-4c2a-8c10-8839baad09d3"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Model 6","metadata":{"id":"t-VhJBMduU9c"}},{"cell_type":"code","source":"# get loss and accuracy score\ntrain_loss6, train_acc6 = model6.evaluate(train_generator, verbose=0)\ntest_loss6, test_acc6 = model6.evaluate(test_generator, verbose=0)\nprint('train_loss_model6:', train_loss6)\nprint('train_acc_model6:', train_acc6)\nprint('test_loss_model6:', test_loss6)\nprint('test_acc_model6:', test_acc6)","metadata":{"id":"54xg1zAQt2aN","outputId":"77bf3389-bb1d-40b6-c16d-e5a4e9aec3d2"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# make predictions on the test dataset\ny_predict6 = model6.predict(x_test_datagen)","metadata":{"id":"1c93bL3wt2pM","outputId":"25da9267-5039-4b45-a547-c7f5c0d033da"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# calculate roc_auc_score\nauc_score6 = roc_auc_score(y_test, y_predict6)\nprint('Model 6 roc_auc_score:', auc_score6)","metadata":{"id":"2Wa8_4wUt2_S","outputId":"618325c1-4cb2-4cf6-c4d1-f07f8603a623"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print out classification report\ny_pred6 = np.argmax(y_predict6, axis= 1)\nprint(classification_report(y_true, y_pred6))","metadata":{"id":"B9HEZIbKt3Ip","outputId":"c8b4cebc-e2d6-4e08-a672-2219925fcb3b"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print out the confusion matrix\ncm6 = confusion_matrix(y_true, y_pred6)\nsns.heatmap(cm6, annot=True, fmt=\".0f\")","metadata":{"id":"C1IqWET8t3RQ","outputId":"4db17a20-6b8b-4d67-9f49-4106b6eb1e1a"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Comparing 6 models","metadata":{"id":"SLmAu8Yq5qzS"}},{"cell_type":"code","source":"# create compare dataframe to compare five models\nmodels = {'Model': [\"Model 1\", \"Model 2\", \"Model 3\", \"Model 4\", \"Model5\", \"Model6\"],\n        'Training Accuracy': [train_acc1, train_acc2, train_acc3, train_acc4, train_acc5, train_acc6],\n        'Training Loss': [train_loss1, train_loss2, train_loss3, train_loss4, train_loss5, train_loss6],      \n        'Test Accuracy': [test_acc1, test_acc2, test_acc3, test_acc4, test_acc5, test_acc6],\n        'Test Loss': [test_loss1, test_loss2, test_loss3, test_loss4, test_loss5, test_loss6],\n        'roc_auc_score': [auc_score1, auc_score2, auc_score3, auc_score4, auc_score5, auc_score6]}\ncompare_models = pd.DataFrame(models)\ncompare_models = compare_models.sort_values(by='roc_auc_score', ascending = False, ignore_index=True)\nprint(\"\\nCompare six deep learning models: \\n\")\ndisplay(compare_models)","metadata":{"execution":{"iopub.execute_input":"2023-02-25T14:56:59.416286Z","iopub.status.busy":"2023-02-25T14:56:59.415531Z","iopub.status.idle":"2023-02-25T14:56:59.432746Z","shell.execute_reply":"2023-02-25T14:56:59.431738Z","shell.execute_reply.started":"2023-02-25T14:56:59.416244Z"},"id":"Vet71_975qzS","outputId":"237041e5-0088-4c08-bdf1-00a3b057a825"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Looking at the compare table above, we can see that model 5 - Resnet50 trained with \"Imagenet\" pretrained weights file loaded is the best model with the highest roc_auc_score and we also see that this model also has the best training accuracy and test accuracy. Thus, I will use model 5 and tune hyperparameters included dropout and learning rate to find the best model. Moreover, I will tune hyperparameter for model 2 (CNN with two layers adding dropout) as well. Let's see which best model we can define and then I will use this model to predict on test data.","metadata":{"id":"uKaOR2LX5qzS"}},{"cell_type":"markdown","source":"## Part 5: Hyperparameters tuning and Train Best Model","metadata":{"id":"WH8FqMy45g7b"}},{"cell_type":"markdown","source":"### 5.1 Hyperparameters tuning for ResNet50 Model ","metadata":{"id":"IKVOEkwn5qzT"}},{"cell_type":"markdown","source":"Because our above result shows that Resnet50 trained with \"Imagenet\" pretrained weights file loaded is the best model, so now I would like to build this model with trying 3 different learning rate: [0.01, 0.001, 0.0001]. However, since it took too much time for training with more epochs, I just train with 5 epochs to find the best model which has the highest val_auc score.\n\nWhile we are creating many models in a loop, this global state will consume an increasing amount of memory over time, so we should clear it. Calling clear_session() releases the global state: this helps avoid clutter from old models and layers, especially when memory is limited.  ","metadata":{"id":"jCFZk52x5qzT"}},{"cell_type":"code","source":"for lr in [0.01, 0.001, 0.0001]:\n    #  clear session:\n      tf.keras.backend.clear_session()\n\n      # define new model\n      model7 = None\n      model7 = Sequential()\n      model7.add(UpSampling2D((2,2)))\n      model7.add(UpSampling2D((2,2)))\n      model7.add(UpSampling2D((2,2)))\n\n      resnet = tf.keras.applications.resnet50.ResNet50(include_top=False, \n                                                      weights=\"imagenet\", \n                                                      pooling = \"max\", \n                                                      classes=NUM_LABELS)\n      model7.add(resnet)\n      model7.add(Dropout(0.2))\n\n      # Out layer\n      model7.add(Dense(NUM_LABELS))\n      model7.add(Activation('softmax'))\n\n      model7.build(input_shape = (None, 32,32,3))\n\n      # compile new model\n      model7.compile(loss = 'categorical_crossentropy',\n                      optimizer = Adam(learning_rate = lr),\n                      metrics=['accuracy'])\n\n      # train model 7\n      res7 = model7.fit(train_generator,\n                      epochs = 5 , \n                      steps_per_epoch=STEP_SIZE_TRAIN,\n                      validation_data = test_generator,\n                      validation_steps=STEP_SIZE_TEST,\n                      verbose=0)\n      \n      # get result\n      y_predict7 = model7.predict(x_test_datagen)\n      auc_score7 = roc_auc_score(y_test, y_predict7)\n      \n      # best result\n      print(\"For Dropout = \",dropout,\"and learning rate = \", lr, \", roc_auc_score: \", auc_score7)","metadata":{"id":"AG4pefyHTd68","outputId":"18bd6d86-2822-479f-8d10-9b565e1c0fba"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 5.2 Hyperparameters tuning for Model 2(CNN with 2 layers adding dropout)","metadata":{"id":"fihqI8px5tJT"}},{"cell_type":"code","source":"# tuning dropout and learning rate for GRU model\nfor dropout in [0.2, 0.3, 0.5]:\n    for lr in [0.01, 0.001, 0.0001]:\n    #  clear session:\\\n        tf.keras.backend.clear_session()\n\n        # define new model\n        model8 = Sequential()\n        # first convolutional layer\n        model8.add(Conv2D(64, (3, 3), input_shape=INPUT_SHAPE))\n        model8.add(BatchNormalization())\n        model8.add(Activation('relu'))\n        model8.add(MaxPooling2D(pool_size=(2, 2)))\n        model8.add(Dropout(dropout))\n\n        # second convolutional layer\n        model8.add(Conv2D(128, (3, 3)))\n        model8.add(BatchNormalization())\n        model8.add(Activation('relu'))\n        model8.add(MaxPooling2D(pool_size=(2, 2)))\n        model8.add(Dropout(dropout))\n\n        model8.add(Flatten())\n        model8.add(Dense(256))\n        model8.add(BatchNormalization())\n        model8.add(Activation('relu'))\n        model8.add(Dropout(dropout))\n\n        # Out layer\n        model8.add(Dense(NUM_LABELS))\n        model8.add(Activation('softmax'))\n\n        # compile new model\n        model8.compile(loss = 'binary_crossentropy',\n                            optimizer = Adam(learning_rate = lr),\n                            metrics=['accuracy'])\n\n        # train new model\n        res8 = model8.fit(train_generator,\n                                epochs = 10,\n                                steps_per_epoch=STEP_SIZE_TRAIN,\n                                validation_steps=STEP_SIZE_TEST, \n                                verbose = 0,\n                                validation_data = test_generator)  \n        \n        # get result\n        y_predict8 = model8.predict(x_test_datagen)\n        auc_score8 = roc_auc_score(y_test, y_predict8)\n      \n        # best result\n        print(\"For Dropout = \",dropout,\"and learning rate = \", lr, \", roc_auc_score: \", auc_score8)","metadata":{"id":"CCXbe0OO6ApN","outputId":"f0d731e8-5951-48be-a963-87a1a0a60f37"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Compare the tuned CNN model and the tuned ResNet50 model, we can see that ResNet50 model with dropout = 0.2 and learning rate = 0.0001 has better roc_auc_score. So now I will apply these hyperparameters to train the best model and predict on test data.","metadata":{"id":"Vml9Dk_u5qzU"}},{"cell_type":"markdown","source":"### 5.3 Train the best model","metadata":{"id":"jAS-L_mx5qzU"}},{"cell_type":"markdown","source":"Let's train the best model with tuned hyperparameters from result above and visualize image predictions on test data, here, I will set green label for correct predicted images and red label for wrong predicted images.","metadata":{"id":"w2AnK7um_LpO"}},{"cell_type":"code","source":"# define best model\nbest_model = None\nbest_model = Sequential()\nbest_model.add(UpSampling2D((2,2)))\nbest_model.add(UpSampling2D((2,2)))\nbest_model.add(UpSampling2D((2,2)))\n\nresnet = tf.keras.applications.resnet50.ResNet50(include_top=False, \n                                                weights=\"imagenet\", \n                                                pooling = \"max\", \n                                                classes=NUM_LABELS)\nbest_model.add(resnet)\nbest_model.add(Dropout(0.2))\n\n# Out layer\nbest_model.add(Dense(NUM_LABELS))\nbest_model.add(Activation('softmax'))\n\nbest_model.build(input_shape = (None, 32,32,3))\n\n# set up callback\nes= tf.keras.callbacks.EarlyStopping(monitor='val_accuracy', mode='max', patience=3, verbose=0)\nreduce_lr = ReduceLROnPlateau(monitor='val_accuracy', factor=0.5, patience=3, \n                                   verbose=0, mode='max', min_lr=0.00001)\ncallback=[es, reduce_lr]\n\n# compile best model\nbest_model.compile(loss = 'categorical_crossentropy',\n                optimizer = Adam(learning_rate = 0.0001),\n                metrics=['accuracy'])\n\n# train best model \nbest_res = best_model.fit(train_generator,\n                epochs = 10 , \n                steps_per_epoch=STEP_SIZE_TRAIN,\n                validation_data = test_generator,\n                validation_steps=STEP_SIZE_TEST,\n                callbacks = callback,\n                verbose = 0)\n             ","metadata":{"execution":{"iopub.execute_input":"2023-02-25T18:22:22.031200Z","iopub.status.busy":"2023-02-25T18:22:22.030828Z","iopub.status.idle":"2023-02-25T19:04:12.126043Z","shell.execute_reply":"2023-02-25T19:04:12.125053Z","shell.execute_reply.started":"2023-02-25T18:22:22.031167Z"},"id":"fM1fzW9p5qzV"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plot the graph of accuracy\nepochs_range = range(len(best_res.history['loss']))\n\ntrain_acc = best_res.history['accuracy']\ntest_acc = best_res.history['val_accuracy']\nplt.figure(figsize=(15, 15))\nplt.subplot(2, 2, 1)\nplt.plot(epochs_range, train_acc, label='Training Accuracy', color = \"blue\", marker = \"o\")\nplt.plot(epochs_range, test_acc, label='Test Accuracy', color = \"red\", marker = \"o\")\nplt.legend(loc='lower right')\nplt.title('Figure 19. Best Model: Training and Test Accuracy')\n\n# plot the graph of loss\ntrain_loss = best_res.history['loss']\ntest_loss = best_res.history['val_loss']\nplt.subplot(2, 2, 2)\nplt.plot(epochs_range, train_loss, label='Training Loss', color = \"blue\", marker = \"o\")\nplt.plot(epochs_range, test_loss, label='Test Loss', color = \"red\", marker = \"o\")\nplt.legend(loc='upper right')\nplt.title('Figure 20. Best Model: Training and Test Loss')\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2023-02-25T19:05:59.557325Z","iopub.status.busy":"2023-02-25T19:05:59.556950Z","iopub.status.idle":"2023-02-25T19:05:59.936611Z","shell.execute_reply":"2023-02-25T19:05:59.935710Z","shell.execute_reply.started":"2023-02-25T19:05:59.557292Z"},"id":"CM2hlcBy5qzW","outputId":"643b841d-3d01-44c6-8d99-ded58a0f7f9f"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# get loss and accuracy score\nbest_train_loss, best_train_acc = best_model.evaluate(train_generator)\nbest_test_loss, best_test_acc = best_model.evaluate(test_generator)\nprint('best_train_loss:', best_train_loss)\nprint('best_train_acc:', best_train_acc)\nprint('best_test_loss:', best_test_loss)\nprint('best_test_acc:', best_test_acc)","metadata":{"execution":{"iopub.execute_input":"2023-02-25T19:06:00.896355Z","iopub.status.busy":"2023-02-25T19:06:00.895993Z","iopub.status.idle":"2023-02-25T19:06:40.564224Z","shell.execute_reply":"2023-02-25T19:06:40.563134Z","shell.execute_reply.started":"2023-02-25T19:06:00.896323Z"},"id":"pAEIk2DK5qzX","outputId":"18971cd2-e7fc-48fa-c060-ec05c35ba6b4"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# make predictions on the test dataset\nbest_y_predict = best_model.predict(x_test_datagen)","metadata":{"execution":{"iopub.execute_input":"2023-02-25T19:06:40.567339Z","iopub.status.busy":"2023-02-25T19:06:40.566255Z","iopub.status.idle":"2023-02-25T19:06:42.167886Z","shell.execute_reply":"2023-02-25T19:06:42.166910Z","shell.execute_reply.started":"2023-02-25T19:06:40.567299Z"},"id":"DT5mkGoP5qzZ","outputId":"27248700-e470-4fb6-8aeb-c29c78fba0a3"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# calculate roc_auc_score\nbest_auc_score = roc_auc_score(y_test, best_y_predict)\nprint('Best model roc_auc_score:', best_auc_score)","metadata":{"execution":{"iopub.execute_input":"2023-02-25T19:06:42.170203Z","iopub.status.busy":"2023-02-25T19:06:42.169314Z","iopub.status.idle":"2023-02-25T19:06:42.231157Z","shell.execute_reply":"2023-02-25T19:06:42.230066Z","shell.execute_reply.started":"2023-02-25T19:06:42.170165Z"},"id":"mP2ARpEb5qzZ","outputId":"b2f0dfe9-2e26-4eab-c186-7d3e4ebefbf7"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print out classification report\nbest_y_pred = np.argmax(best_y_predict, axis= 1)\nprint(classification_report(y_true, best_y_pred))","metadata":{"execution":{"iopub.execute_input":"2023-02-25T19:06:42.235267Z","iopub.status.busy":"2023-02-25T19:06:42.233504Z","iopub.status.idle":"2023-02-25T19:06:42.267694Z","shell.execute_reply":"2023-02-25T19:06:42.266054Z","shell.execute_reply.started":"2023-02-25T19:06:42.234922Z"},"id":"C9L1-vFL5qza","outputId":"af6aadff-277a-4ae4-d5fe-99916d0339c0"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print out the confusion matrix\nbest_cm = confusion_matrix(y_true, best_y_pred)\nsns.heatmap(best_cm, annot=True, fmt=\".0f\")","metadata":{"execution":{"iopub.execute_input":"2023-02-25T19:06:42.270040Z","iopub.status.busy":"2023-02-25T19:06:42.269605Z","iopub.status.idle":"2023-02-25T19:06:43.155217Z","shell.execute_reply":"2023-02-25T19:06:43.154287Z","shell.execute_reply.started":"2023-02-25T19:06:42.269998Z"},"id":"y7iGMGTj5qza","outputId":"ac260018-53ff-4e8f-b119-330d73e8ffbe"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, axes = plt.subplots(4, 6, figsize=(15, 15))\nfor i, ax in enumerate(axes.flat):\n  ax.imshow(x_test[i])\n  ax.set(xticks=[], yticks=[], xlabel=classes[y_pred1[i]])\n  if classes[y_true[i]] == classes[y_pred1[i]]:  \n    ax.xaxis.label.set_color('green')\n  else:\n    ax.xaxis.label.set_color('red')\nfig.suptitle('\\nFigure 21. Test Image Predictions Visualization')\nplt.show()","metadata":{"id":"kcuZDLJ0RYKH","outputId":"9dad31a5-aee8-465c-d84c-7b4880ffaa41"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Part 6: Conclusion and Takeaways","metadata":{"id":"KIX2mPz85qza"}},{"cell_type":"markdown","source":"In this project, there are 6 parts:  \n\n(1) Brief description of the problem and data.     \n(2) EDA - Inspect, Visualize, and Clean the data.    \n(3) Building and training models:   \n    + Model 1: CNN with two layers   \n    + Model 2: CNN with two layers adding dropout    \n    + Model 3: CNN with three layers adding dropout   \n    + Model 4: CNN with three layers adding Earlystopping and ReduceLROnPlateau   \n    + Model 5: ResNet50 with \"Imagenet\" pretrained weights   \n    + Model 6: ResNet101 with \"Imagenet\" pretrained weights    \n(4) Results and Analysis:   \n    + Print out results   \n    + Compare six deep learning models.  \n(5) Tuning hyperparameter to find the best model.   \n(6) Conclusion.      \n\nThe goal of this project is to recognize objects in photos with cifa-10 dataset which contains 60,000 32x32 color images in 10 different categories. These categories represent airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks. By comparing six different deep learning models including: CNN with two layers, CNN with two layers adding dropout, CNN with three layers adding dropout, CNN with three layers adding Earlystopping and ReduceLROnPlateau, ResNet50 with \"Imagenet\" pretrained weights and ResNet101 with \"Imagenet\" pretrained weights, we can conclude that in this case, ResNet50 with \"Imagenet\" pretrained weights is the model that has the best performance with the highest roc_auc_score of 0.98, train accuracy of 0.8 and test accuracy of 0.8. \n\nAfter tuning learning rate for ResNet50 model and tuning dropout and learning rate for model 2, we got the roc_auc_score increased from 0.98 to 0.99 at dropout = 0.2 and learning rate = 0.0001 for ResNet50. Next we used these best hyperparameters for building the best model to predict on test datatset, print out the result and visualize image with label predictions.\n\nI tried my best and with the result of roc_auc_score of 0.99 and test accuracy 0.92, I think it is good enough, however, because of the limitation of data and the running time was too costly, the models just were trained on limited approach. There are many other ways can improve this kind of project such as:     \n    + building more deep learning models by tuning other hyperparameters to get optimal results such as: try different dropout at each convolutional layer, try with more different learning rate, ...    \n    + or we can run models by increasing the number of epochs   \n    + or adding more layers and filters.    \nI believe these ways might get better results.   \n","metadata":{"id":"nx-YFSRd5qzb"}},{"cell_type":"code","source":"","metadata":{"id":"51o59QW_5qzb"},"execution_count":null,"outputs":[]}]}