{"cells":[{"metadata":{"id":"AmFyQOgK1_Y5"},"cell_type":"markdown","source":"# Introduction\nOne way to learn deep leaning is putting in practice all concepts, and trying many times with different configurations.\n\nIn this case, I will build some models using Convolutional Neural Networks (CNN) in order to get the best model.\n\nI do not want to spend much time in training, so I chose CIFAR-10. There are few images and these images are small, that model will not spend a lot of time in training.\n\n#### Data\nhttps://www.kaggle.com/c/cifar-10/overview\n\n#### References\nTo build these model I read the following readings:\n\nhttps://medium.com/analytics-vidhya/cnns-architectures-lenet-alexnet-vgg-googlenet-resnet-and-more-666091488df5\n\nhttps://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf\n\nhttps://towardsdatascience.com/a-guide-to-an-efficient-way-to-build-neural-network-architectures-part-ii-hyper-parameter-42efca01e5d7\n\n**Version: 1.0**","execution_count":null},{"metadata":{"id":"vZAm9PR5zyjO"},"cell_type":"markdown","source":"# 1. Load data","execution_count":null},{"metadata":{"id":"33pxUyksz05N","trusted":true},"cell_type":"code","source":"import tensorflow as tf\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport pandas as pd","execution_count":null,"outputs":[]},{"metadata":{"id":"p6785v5x2YqS","outputId":"21ba2a66-cabb-45e4-a9af-910fd81d944f","trusted":true},"cell_type":"code","source":"print(tf.__version__)","execution_count":null,"outputs":[]},{"metadata":{"id":"chv7yZ6t22gt","outputId":"95e732c6-b5fa-45a5-b363-49db29397406","trusted":true},"cell_type":"code","source":"(train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.cifar10.load_data()","execution_count":null,"outputs":[]},{"metadata":{"id":"gybnV7Nc2U-Y"},"cell_type":"markdown","source":"# 2. Data understanding","execution_count":null},{"metadata":{"id":"6pg0R45g2ZC_","outputId":"c948adfe-b989-41cc-a32b-f09c8d872e97","trusted":true},"cell_type":"code","source":"train_images.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_images.shape","execution_count":null,"outputs":[]},{"metadata":{"id":"MbGR4_OP2ZGH","outputId":"25763391-31fe-4972-f2bf-454d43b0be88","trusted":true},"cell_type":"code","source":"class_names = ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']\n\nplt.figure(figsize=(10,10))\nfor i in range(25):\n    plt.subplot(5,5,i+1)\n    plt.xticks([])\n    plt.yticks([])\n    plt.grid(False)\n    plt.imshow(train_images[i], cmap=plt.cm.binary)\n    # The CIFAR labels happen to be arrays, \n    # which is why you need the extra index\n    plt.xlabel(class_names[train_labels[i][0]])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"id":"_zdriROxyszW"},"cell_type":"markdown","source":"Frequency by label","execution_count":null},{"metadata":{"id":"bpM6kESlyXRs","outputId":"8c8a60b3-2fef-4947-b57d-45e0a9e03bc1","trusted":true},"cell_type":"code","source":"np.unique(train_labels, return_counts=True)","execution_count":null,"outputs":[]},{"metadata":{"id":"kU2IpZCs3aN2"},"cell_type":"markdown","source":"# 3. Modeling","execution_count":null},{"metadata":{"id":"wCzoip9A5CwO"},"cell_type":"markdown","source":"## 3.1. Transform data","execution_count":null},{"metadata":{"id":"a1ZCsXDQ5HZe","trusted":true},"cell_type":"code","source":"# Normalizing data\ntrain_images = train_images / 255.0\ntest_images = test_images / 255.0","execution_count":null,"outputs":[]},{"metadata":{"id":"DyTxlr1CKccP"},"cell_type":"markdown","source":"## 3.2. Metrics\nFor this project I will use accuracy and loss metrics, bacuase they are the most common and I want to keep it easy.\n\nAdditionally, I will use avoidable bias and variance metrics to evaluate if it is necessary to focus in reduce bias (training) or variance (testing).","execution_count":null},{"metadata":{"id":"R_WlIra5_7iS","trusted":true},"cell_type":"code","source":"# How well a human can classify an image\nhumanLevelPerformance = 0.9\n\n# How bad a human can classify an image\nhumanLevelError = 0.1","execution_count":null,"outputs":[]},{"metadata":{"id":"GT3RTEvnRMXu"},"cell_type":"markdown","source":"## 3.3. Create model","execution_count":null},{"metadata":{"id":"MVOPDpQuMA9p","trusted":true},"cell_type":"code","source":"# Input configuration\ninputHeight = train_images.shape[1]\ninputWeight = train_images.shape[2]\nnumberChannels = train_images.shape[3]","execution_count":null,"outputs":[]},{"metadata":{"id":"nwU19-Ub1Qdg","trusted":true},"cell_type":"code","source":"# Hyperparameters\n\n# 1028 because it is fast and the data is small\nbatchSize = 1028\n\n# 500 epochs because it is necessary a large number of iteration to get the best results\nepochs = 500\nAUTOTUNE = tf.data.experimental.AUTOTUNE","execution_count":null,"outputs":[]},{"metadata":{"id":"B0b0I0Uz-IJN"},"cell_type":"markdown","source":"### 3.3.1. Create a start model\nThis model is based on LeNet-5 model, because the dimension of the image is too small as LeNet-5 uses.","execution_count":null},{"metadata":{"id":"GvLV70hz2ZJc","outputId":"8f4c093f-5f3f-4416-8880-4fe46a5b079c","trusted":true},"cell_type":"code","source":"# Large size kernel use to large amount of pixels (big images)\n# For small images or many elements, small kernels\n\nmodel = tf.keras.models.Sequential()\n\n# 3x3 filter because the input image is small and I want to capture as many details as posible.\n# 32 filters because I follow LeNet-5 recommendation as start point in order to get.\n# padding='same' because I want to capture image's borders.\n# Activation function='relu' because it is the most recommended.\n# MaxPooling2D(2, 2) to shrink convolution layer size and speed training and reduce risk of overfitting.\nmodel.add(tf.keras.layers.Conv2D(32, (3, 3), activation='relu', padding='same', input_shape=(inputHeight, inputWeight, numberChannels)))\nmodel.add(tf.keras.layers.MaxPooling2D((2, 2)))\n\n# 3x3 filter because to keep capturing as many details as posible\n# 64 filters because more neurons process more information (feature maps).\n# MaxPooling2D(2, 2) to shrink convolution layer size and speed training and reduce risk of overfitting\nmodel.add(tf.keras.layers.Conv2D(64, (3, 3), activation='relu'))\nmodel.add(tf.keras.layers.MaxPooling2D((2, 2)))\n\n# 3x3 filter because to keep capturing as many details as posible\n# 64 filters because more neurons process more information (feature maps).\n# MaxPooling2D(2, 2) to shrink convolution layer size and speed training and reduce risk of overfitting\nmodel.add(tf.keras.layers.Conv2D(64, (3, 3), activation='relu'))\nmodel.add(tf.keras.layers.MaxPooling2D((2, 2)))\n\n# LeNet-5 recommendations\nmodel.add(tf.keras.layers.Flatten())\nmodel.add(tf.keras.layers.Dense(128, activation='relu'))\nmodel.add(tf.keras.layers.Dense(64, activation='relu'))\nmodel.add(tf.keras.layers.Dense(10))\n\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"id":"MR_7Tfh82ZQO","trusted":true},"cell_type":"code","source":"# Adam optimizer because it is most recommended\nmodel.compile(optimizer='adam',\n              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n              metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"id":"YL_Xf9Ap38_6","outputId":"d515c2c3-b1ca-4d25-86de-d8e09316909e","trusted":true},"cell_type":"code","source":"# EarlyStopping to capture the best loss\nhistory = model.fit(train_images, train_labels, epochs=epochs, batch_size=batchSize, validation_data=(test_images, test_labels),\n                    callbacks = tf.keras.callbacks.EarlyStopping(monitor='loss', patience=5))","execution_count":null,"outputs":[]},{"metadata":{"id":"mDBwwZAqzkbh","outputId":"517726fd-2b76-424c-e8a0-2f34e4ea1118","trusted":true},"cell_type":"code","source":"# Best results\nprint('Loss:', history.history['loss'][-1])\nprint('Accuracy:', history.history['accuracy'][-1])\nprint('Val Loss:', history.history['val_loss'][-1])\nprint('Val Accuracy:', history.history['val_accuracy'][-1])","execution_count":null,"outputs":[]},{"metadata":{"id":"q-YwTHTcByij","outputId":"b1f0a81a-4f89-4b36-b4d1-6d443fd674e9","trusted":true},"cell_type":"code","source":"avoidableBias = history.history['loss'][-1] - humanLevelError\nvariance = history.history['val_loss'][-1] - history.history['loss'][-1]\n\nprint('Avoidable bias:', avoidableBias)\nprint('Variance:', variance)\n\nif avoidableBias < variance:\n  print('It is necessary to reduce variance')\nelse:\n  print('It is necessary to reduce bias')","execution_count":null,"outputs":[]},{"metadata":{"id":"XJNKCJ09Mrfh"},"cell_type":"markdown","source":"Observations\n* Both accuracy and loss are close to human perfomance vision, but unfortunately there is overfitting. The difference between training metrics and validation metrics is big.\n* Variance is more than avoidable bias, that means that it is necessary to reduce variance.","execution_count":null},{"metadata":{"id":"o5L4FNY8QcPf"},"cell_type":"markdown","source":"### 3.3.2. Modify model to reduce variance\nThis model is to reduce the variance.","execution_count":null},{"metadata":{"id":"a4OUVL-ISNbA","outputId":"0739cf49-3ae6-4f0a-d0cf-78afd9bd6716","trusted":true},"cell_type":"code","source":"model = tf.keras.models.Sequential()\n\n# Add Dropout performed better in CNN layers\nmodel.add(tf.keras.layers.Conv2D(32, (3, 3), activation='relu', padding='same', input_shape=(inputHeight, inputWeight, numberChannels)))\nmodel.add(tf.keras.layers.MaxPooling2D((2, 2)))\nmodel.add(tf.keras.layers.Dropout(0.2))\n\nmodel.add(tf.keras.layers.Conv2D(64, (3, 3), activation='relu'))\nmodel.add(tf.keras.layers.MaxPooling2D((2, 2)))\nmodel.add(tf.keras.layers.Dropout(0.2))\n\nmodel.add(tf.keras.layers.Conv2D(64, (3, 3), activation='relu'))\nmodel.add(tf.keras.layers.MaxPooling2D((2, 2)))\nmodel.add(tf.keras.layers.Dropout(0.2))\n\n# Add regulation L2 performed better in Fully connected layers\nmodel.add(tf.keras.layers.Flatten())\nmodel.add(tf.keras.layers.Dense(128, activation='relu', kernel_regularizer = tf.keras.regularizers.L2(0.01)))\nmodel.add(tf.keras.layers.Dense(64, activation='relu', kernel_regularizer = tf.keras.regularizers.L2(0.01)))\nmodel.add(tf.keras.layers.Dense(10))\n\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"id":"Lyt_9ky5SNej","trusted":true},"cell_type":"code","source":"model.compile(optimizer='adam',\n              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n              metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"id":"ozl-gD_oSNiq","outputId":"29bdd4df-bc68-4782-ea33-79678e3ddb19","trusted":true},"cell_type":"code","source":"history = model.fit(train_images, train_labels, epochs=epochs, batch_size=batchSize, validation_data=(test_images, test_labels),\n                    callbacks = tf.keras.callbacks.EarlyStopping(monitor='loss', patience=5))","execution_count":null,"outputs":[]},{"metadata":{"id":"u-2FP9hHYCtj","outputId":"6689a408-093d-4c1c-fba0-8c41ca21c24f","trusted":true},"cell_type":"code","source":"# Best results\nprint('Loss:', history.history['loss'][-1])\nprint('Accuracy:', history.history['accuracy'][-1])\nprint('Val Loss:', history.history['val_loss'][-1])\nprint('Val Accuracy:', history.history['val_accuracy'][-1])","execution_count":null,"outputs":[]},{"metadata":{"id":"5CZ9HtYxYCxS","outputId":"b7fdb7f9-5df2-47a6-e1be-0e6a62122bb8","trusted":true},"cell_type":"code","source":"avoidableBias = history.history['loss'][-1] - humanLevelError\nvariance = history.history['val_loss'][-1] - history.history['loss'][-1]\n\nprint('Avoidable bias:', avoidableBias)\nprint('Variance:', variance)\n\nif avoidableBias < variance:\n  print('It is necessary to reduce variance')\nelse:\n  print('It is necessary to reduce bias')","execution_count":null,"outputs":[]},{"metadata":{"id":"YgHGdmyGjCmy"},"cell_type":"markdown","source":"Observations:\n* The overfitting was eliminated. In fact, validation metrics are less than training metrics (bias < variance).\n* Avoidable bias is greater than variance, so a new model will focus on reducing the bias (training).","execution_count":null},{"metadata":{"id":"uI86Myc6tKPo"},"cell_type":"markdown","source":"# 4. Conclusions\nThe first model was focused on reducing bias, because my first step looked for a model that performs well on training. Once I get a good start model, I need to eliminate overfitting.","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}