{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":3362,"databundleVersionId":31148,"sourceType":"competition"},{"sourceId":5441,"databundleVersionId":38425,"sourceType":"competition"},{"sourceId":30378,"sourceType":"datasetVersion","datasetId":23777}],"dockerImageVersionId":29869,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction\nIn this notebook, i will be bringing you through the use of **Transfer Learning** in pretrained models like VGG16, Resnet etc to solve similar tasks; image classification of Dogs and Cats. We will see how we can use these pretrained models as feature extractors to generate features to be fed into another classifier, as well as finetuning these models to solve our problem. \nThe notebook will be using Keras, a high-level API that allow us to build prototypes quickly with minimum coding, without loss in performance. \n* Building a custom CNN\n* Using a pre-trained model as feature extractor (VGG16)\n* Fine tuning an existing pre-trained model (ResNet50)\n","metadata":{}},{"cell_type":"markdown","source":"I hope that after reading the notebook, you guys can not only apply these transfer learning techniques for future problems, but also appreciate the power of using pre-trained models that the awesome researchers have came up with :)","metadata":{}},{"cell_type":"markdown","source":"# Table of Content\n1. [Dog vs Cat Dataset](#sec1) \n * [1.1 Extracting Zip file](#sec1.1) \n * [1.2 Data Formatting](#sec1.2)\n * [1.3 Generating mini batches for training](#sec1.3) \n2. [CNN Architecture](#sec2) \n * [2.1 CNN from scratch](#sec2.1) \n * [2.2 Transfer Learning](#sec2.2) \n * [2.2.1 Feature Extraction](#sec2.2.1) \n * [2.2.2 Fine Tuning](#sec2.2.2) \n3. [Results](#sec3) \n","metadata":{}},{"cell_type":"code","source":"#Download Dependencies\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport zipfile\nimport glob\nimport os\nimport torch as th\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom sklearn.model_selection import train_test_split\nimport tensorflow as tf\nfrom kaggle_datasets import KaggleDatasets\n\nfrom keras.preprocessing.image import ImageDataGenerator, array_to_img, img_to_array, load_img\nfrom keras.applications.resnet50 import preprocess_input","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_kg_hide-output":false,"execution":{"iopub.status.busy":"2023-11-22T21:16:47.792885Z","iopub.execute_input":"2023-11-22T21:16:47.793196Z","iopub.status.idle":"2023-11-22T21:16:56.175268Z","shell.execute_reply.started":"2023-11-22T21:16:47.793156Z","shell.execute_reply":"2023-11-22T21:16:56.174541Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# [1. Dogs vs Cats Dataset](#sec1)","metadata":{}},{"cell_type":"markdown","source":"\n## [1.1 Extracting Zip file](#sec1.1)\nThe data files provided are in .zip extension, which needs to be unzipped before we can access the images for our training later on. We will be using the [zipfile](https://docs.python.org/3/library/zipfile.html) library that provides a simple way for us to unzip these .zip files","metadata":{}},{"cell_type":"code","source":"import zipfile\nimport glob\n\nzip_file = glob.glob('/kaggle/input/dogs-vs-cats/*.zip') #return any files with .zip extension\nprint(zip_file)\n\n#extract file into a temp folder\ndef extract_zip(file):\n with zipfile.ZipFile(file,\"r\") as zip_ref:\n zip_ref.extractall(\"temp\")\n \n#extract both train and test1 zip\nfor files in zip_file:\n extract_zip(files)","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","execution":{"iopub.status.busy":"2023-11-22T21:16:56.177259Z","iopub.execute_input":"2023-11-22T21:16:56.177473Z","iopub.status.idle":"2023-11-22T21:17:12.030265Z","shell.execute_reply.started":"2023-11-22T21:16:56.177448Z","shell.execute_reply":"2023-11-22T21:17:12.029642Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#instantiate the constants\nbatch_size = 16\nimg_size = 224\nepochs = 5","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-11-22T21:17:12.031461Z","iopub.execute_input":"2023-11-22T21:17:12.031702Z","iopub.status.idle":"2023-11-22T21:17:12.035530Z","shell.execute_reply.started":"2023-11-22T21:17:12.031670Z","shell.execute_reply":"2023-11-22T21:17:12.034681Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(len(os.listdir('/kaggle/working/temp/train')), \"training data\")\nprint(len(os.listdir('/kaggle/working/temp/test1')), \"test data\")\nos.listdir(\"temp\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-11-22T21:17:12.036690Z","iopub.execute_input":"2023-11-22T21:17:12.036923Z","iopub.status.idle":"2023-11-22T21:17:12.073685Z","shell.execute_reply.started":"2023-11-22T21:17:12.036891Z","shell.execute_reply":"2023-11-22T21:17:12.072953Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n## [1.2 Data Formatting](#sec1.2)","metadata":{}},{"cell_type":"markdown","source":"In our data formatting, we will generate the respective labels for **dogs (1)** and **cats (0)** for our training data. File path will also be collected as a column for our dataframe so that it can be used to load and train our images","metadata":{}},{"cell_type":"code","source":"def gen_label(directory):\n label = []\n for file in os.listdir(directory):\n if (file.split('.')[0] == 'dog'):\n label.append(str(1))\n elif (file.split('.')[0] == 'cat'):\n label.append(str(0))\n return label\n #print(len(label),\"files in\", directory)\n \ndef get_path(directory):\n path = []\n for files in os.listdir(directory):\n path.append(files)\n return path\n\ntrain_y = gen_label('temp/train')\ntrain_x = get_path('temp/train')\ntest_x = get_path('temp/test1')","metadata":{"execution":{"iopub.status.busy":"2023-11-22T21:17:12.076350Z","iopub.execute_input":"2023-11-22T21:17:12.076566Z","iopub.status.idle":"2023-11-22T21:17:12.152283Z","shell.execute_reply.started":"2023-11-22T21:17:12.076540Z","shell.execute_reply":"2023-11-22T21:17:12.151517Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.DataFrame({'filename': train_x,\n 'category': train_y})\nprint(df.head())\n\nsns.countplot(x='category',data=df).set_title(\"Data Distribution\")","metadata":{"execution":{"iopub.status.busy":"2023-11-22T21:17:12.154770Z","iopub.execute_input":"2023-11-22T21:17:12.154976Z","iopub.status.idle":"2023-11-22T21:17:12.329067Z","shell.execute_reply.started":"2023-11-22T21:17:12.154952Z","shell.execute_reply":"2023-11-22T21:17:12.328260Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Lets visualise one of the training image","metadata":{}},{"cell_type":"code","source":"# Change working directory\nos.chdir('temp/train')\n\nimg = load_img(df['filename'].iloc[0]) \n \n# Displaying the image \nplt.figure(figsize=(8,8))\nplt.imshow(img)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-11-22T21:17:12.330393Z","iopub.execute_input":"2023-11-22T21:17:12.330633Z","iopub.status.idle":"2023-11-22T21:17:12.616356Z","shell.execute_reply.started":"2023-11-22T21:17:12.330597Z","shell.execute_reply":"2023-11-22T21:17:12.615694Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n## [1.3 Generating mini batches for training](#sec1.3)","metadata":{}},{"cell_type":"markdown","source":"Before we move on from here, i would like discuss the following terms that you might have often come across; **Batch Gradient Descent**, **Stochastic Gradient Descent**, **Mini-Batch Gradient Descent**. \n\nIn machine learning terms, the word **gradient descent** refers to a specific optimization algorithm whose purpose it to minimise the loss function of any machine learning algorithm. Gradient Descent does so by calculating the gradient of the weights parameters with respect to the loss function, and using it to find a global minimum of the loss function.\n\n**Batch Gradient Descent** uses the entire training data to make one gradient computation; which results in slow convergence since the gradient descent only makes a single step with every epoch. \n**Stochastic Gradient Descent** makes a gradient calculation with a single training sample, which will result in loss of speedup from the vectorization. \n**Mini-Batch Gradient Descent** uses a mini-batches of training example for each gradient calculation; which makes convergence faster, and at the same time, utilise the speed up from vectorization.","metadata":{}},{"cell_type":"markdown","source":"We will first split our training data into training and validation set\n","metadata":{}},{"cell_type":"code","source":"train_df, valid_df = train_test_split(df, test_size=0.25)\nprint(train_df.shape)\nprint(valid_df.shape)","metadata":{"execution":{"iopub.status.busy":"2023-11-22T21:17:12.617828Z","iopub.execute_input":"2023-11-22T21:17:12.618050Z","iopub.status.idle":"2023-11-22T21:17:12.627368Z","shell.execute_reply.started":"2023-11-22T21:17:12.618025Z","shell.execute_reply":"2023-11-22T21:17:12.626703Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Keras library provides module for generating mini batches of augmented data that we can directly feed into our CNN model, which provides convenience as we just have to feed in the dataframe with relevant file path and labels. Not only that, the dataset generated can be wrapped with transformation for data augmentation so that our mini-batches will consist of data with transformation, adding more data to prevent model overfitting. \n*ImageDataGenerator* is a class that provides a variety of ways we can futher augment our data, find more [here](https://keras.io/preprocessing/image/). Data augmentation is only added to our training data. \n*flow_from_dataframe()* method will accept dataframe with filenames as x_column and labels as y_column to generate mini-batches","metadata":{}},{"cell_type":"code","source":"def generate_train_batch(model):\n \n if model == 'resnet': #use of resnet requires its specific preprocessing_function for better accuracy for augmentation\n print('resnet data')\n train_datagen = ImageDataGenerator(\n rotation_range=10,\n zoom_range=0.1,\n horizontal_flip=True,\n fill_mode='nearest',\n width_shift_range=0.1,\n height_shift_range=0.1,\n preprocessing_function = preprocess_input)\n\n else:\n train_datagen = ImageDataGenerator( #standard augmentation\n rotation_range=10,\n rescale=1./255,\n zoom_range=0.1,\n horizontal_flip=True,\n fill_mode='nearest',\n width_shift_range=0.1,\n height_shift_range=0.1)\n\n if model == 'vgg': #VGG16 will only generate mini-batches of x_features; y_col=None as feature extractor\n print('vgg data')\n train_gen = train_datagen.flow_from_dataframe(\n train_df[['filename']],\n x_col='filename',\n y_col=None,\n target_size=(img_size, img_size),\n batch_size = batch_size,\n class_mode=None,\n shuffle=False)\n \n else:\n train_gen = train_datagen.flow_from_dataframe(\n train_df,\n x_col='filename',\n y_col='category',\n target_size=(img_size, img_size),\n batch_size = batch_size,\n class_mode='binary')\n\n return train_gen\n\n\ndef generate_valid_batch(model):\n if model == 'resnet':\n print('resnet validation set')\n valid_datagen = ImageDataGenerator(preprocessing_function = preprocess_input)\n else:\n valid_datagen = ImageDataGenerator(rescale=1./255)\n \n valid_gen = valid_datagen.flow_from_dataframe(\n valid_df,\n x_col='filename',\n y_col='category',\n target_size=(img_size, img_size),\n batch_size = batch_size,\n class_mode='binary')\n \n return valid_gen\n\ntrain_gen = generate_train_batch('others')\nvalid_gen = generate_valid_batch('others')","metadata":{"execution":{"iopub.status.busy":"2023-11-22T21:17:12.628730Z","iopub.execute_input":"2023-11-22T21:17:12.628962Z","iopub.status.idle":"2023-11-22T21:17:12.946984Z","shell.execute_reply.started":"2023-11-22T21:17:12.628936Z","shell.execute_reply":"2023-11-22T21:17:12.946079Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Data Augmentation\nWe can visualize the effect of a random image after data augmentation","metadata":{}},{"cell_type":"code","source":"visual_datagen = ImageDataGenerator( #standard augmentation\n rotation_range=10,\n rescale=1./255,\n zoom_range=0.1,\n horizontal_flip=True,\n fill_mode='nearest',\n width_shift_range=0.1,\n height_shift_range=0.1)\n\nvisualise_df = train_df.sample(n=1).reset_index(drop=True)\nvisualisation_generator = visual_datagen.flow_from_dataframe(\n visualise_df, \n x_col='filename',\n y_col='category'\n)\nplt.figure(figsize=(8, 8))\nfor i in range(0, 9):\n plt.subplot(3, 3, i+1)\n for X_batch, Y_batch in visualisation_generator:\n image = X_batch[0]\n plt.imshow(image)\n break\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-11-22T21:17:12.948381Z","iopub.execute_input":"2023-11-22T21:17:12.948710Z","iopub.status.idle":"2023-11-22T21:17:14.556545Z","shell.execute_reply.started":"2023-11-22T21:17:12.948663Z","shell.execute_reply":"2023-11-22T21:17:14.555867Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# [2. CNN Architecture](#sec2)","metadata":{}},{"cell_type":"markdown","source":"Convolutional Neural Network (CNN) has been the to-go neural network architecture when dealing with images. From Yann LeCunn's LeNet in 1990s predicting digits on the popular MNIST data, to Alexnet impressive performance on ImageNet, the area of CNN has evolved tremendously. \nA simple architecture of CNN is shown below: \n\n \n \n**Convolution Layer** uses filters to extract meaningful edges, features in an image, through a combination of linear and non-linear process. Linear operation invloves the correlation between filters and image pixels.Non-linear operations includes ReLU, Tanh. \n \n\n\n**Pooling Layer** aims at reducing the spatial size of the feature map in order to reduce the number of parameters and computation complexity. Some commonly used pooling methods include average pooling, max pooling. \n\n\nImages are retreived from [google](https://developers.google.com/machine-learning/practica/image-classification/convolutional-neural-networks)","metadata":{}},{"cell_type":"markdown","source":"\n## [2.1 CNN from scratch](#sec2.1)","metadata":{}},{"cell_type":"markdown","source":"In this section, we will try to implement a CNN architecture from scratch, using the Keras library. We will be using just 5 epochs for all our training","metadata":{}},{"cell_type":"code","source":"from keras.applications import VGG16, resnet\nfrom keras.layers import *\nfrom keras.models import Model,Sequential\nfrom keras import optimizers\nfrom keras import regularizers","metadata":{"execution":{"iopub.status.busy":"2023-11-22T21:17:14.557590Z","iopub.execute_input":"2023-11-22T21:17:14.557806Z","iopub.status.idle":"2023-11-22T21:17:14.562452Z","shell.execute_reply.started":"2023-11-22T21:17:14.557780Z","shell.execute_reply":"2023-11-22T21:17:14.561473Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from keras import backend as K\nK.clear_session()","metadata":{"execution":{"iopub.status.busy":"2023-11-22T21:17:14.563608Z","iopub.execute_input":"2023-11-22T21:17:14.563885Z","iopub.status.idle":"2023-11-22T21:17:14.581925Z","shell.execute_reply.started":"2023-11-22T21:17:14.563857Z","shell.execute_reply":"2023-11-22T21:17:14.581372Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For our simple CNN, we will use 3 convolutional layers with all having the same filter size of (3x3). Each convolutional layers will be followed by a Max pooling layer that will reduce the dimensions of the convolutional features maps. The output of the final convolutional layer will be passed into a Fully connected layer with 500 neurons, and connecting to the last layer, which acts as sigmoid activation to predict 1/0.","metadata":{}},{"cell_type":"code","source":"#model instantiation\nmodelcnn=Sequential()\nmodelcnn.add(Conv2D(16, (3,3), activation=\"relu\", input_shape=(img_size, img_size, 3)))\nmodelcnn.add(Conv2D(16, (3,3), activation=\"relu\",))\nmodelcnn.add(MaxPooling2D((3,3)))\n\nmodelcnn.add(Conv2D(32, (3,3), activation=\"relu\"))\nmodelcnn.add(Conv2D(32, (3,3), activation=\"relu\"))\nmodelcnn.add(MaxPooling2D(2,2))\n\nmodelcnn.add(Conv2D(64, (3,3), activation=\"relu\"))\nmodelcnn.add(Conv2D(64, (3,3), activation=\"relu\"))\nmodelcnn.add(MaxPooling2D(2,2))\nmodelcnn.add(Dropout(0.3))\n\nmodelcnn.add(Conv2D(32, (3,3), activation=\"relu\"))\nmodelcnn.add(MaxPooling2D((2,2)))\n\nmodelcnn.add(Flatten())\nmodelcnn.add(Dense(512, activation=\"relu\"))\nmodelcnn.add(Dropout(0.5))\nmodelcnn.add(Dense(1, activation=\"sigmoid\"))\n\nmodelcnn.compile(loss=\"binary_crossentropy\", \n optimizer=optimizers.RMSprop(lr=1e-4),\n metrics=[\"accuracy\"])","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-11-22T21:17:14.583014Z","iopub.execute_input":"2023-11-22T21:17:14.583214Z","iopub.status.idle":"2023-11-22T21:17:20.185355Z","shell.execute_reply.started":"2023-11-22T21:17:14.583191Z","shell.execute_reply":"2023-11-22T21:17:20.184693Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"modelcnn.summary()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-11-22T21:17:20.187081Z","iopub.execute_input":"2023-11-22T21:17:20.187389Z","iopub.status.idle":"2023-11-22T21:17:20.194120Z","shell.execute_reply.started":"2023-11-22T21:17:20.187352Z","shell.execute_reply":"2023-11-22T21:17:20.193335Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Lets fit our model with the training data and evaluate its performance\n","metadata":{}},{"cell_type":"code","source":"modelcnn.fit_generator(train_gen,\n epochs=epochs,\n validation_data=valid_gen)","metadata":{"execution":{"iopub.status.busy":"2023-11-22T21:17:20.195544Z","iopub.execute_input":"2023-11-22T21:17:20.195880Z","iopub.status.idle":"2023-11-22T21:36:30.153488Z","shell.execute_reply.started":"2023-11-22T21:17:20.195845Z","shell.execute_reply":"2023-11-22T21:36:30.152339Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"loss, accuracy = modelcnn.evaluate_generator(valid_gen, valid_gen.samples//batch_size, workers=12)\nprint(\"Validation: accuracy = %f ; loss = %f \" % (accuracy, loss))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-11-22T21:36:30.157917Z","iopub.execute_input":"2023-11-22T21:36:30.158181Z","iopub.status.idle":"2023-11-22T21:36:39.314749Z","shell.execute_reply.started":"2023-11-22T21:36:30.158147Z","shell.execute_reply":"2023-11-22T21:36:39.313984Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n## [2.2 Transfer Learning](#sec2.2)","metadata":{}},{"cell_type":"markdown","source":"In the following sections, we will be focusing on **Pre-trained** models. Pre-trained models are architectures are were previously trained using large datasets to solve similar problems that we are trying to solve; which is image classification. \nOne benefits of using pre-trained model is that because these models has been trained on huge datasets, they are used to learning very good discriminative features, enabling us to use it for transfer learning. \n**Transfer learning** is process of reusing a pretained model trained for a task and use it for another task. Two approaches for transfer learning are:\n1. Feature Extraction\n2. Fine Tuning","metadata":{}},{"cell_type":"markdown","source":"\n### [2.2.1 Feature Extraction](#sec2.2.1)","metadata":{}},{"cell_type":"markdown","source":"One of the approach for transfer learning is **feature extraction**, where we leverage on the pretrained models to predict a set of output features, and feeding these output features into a separate classifier algorithm to generate prediction. In this case, we will be using [VGG16](https://arxiv.org/pdf/1409.1556.pdf) as our feature extractor, and using **Logistic Regression** as our binary classifier","metadata":{}},{"cell_type":"markdown","source":"What we are doing here is to freeze the VGG16 layers up to the dense layer; output from the model will have a shape of 7x7x512. \n\nImage referenced from [here](https://appliedmachinelearning.blog/2019/07/29/transfer-learning-using-feature-extraction-from-trained-models-food-images-classification/)","metadata":{}},{"cell_type":"markdown","source":"By setting *include_top = False*, we essentially removed the fully connected layers from the pretrained VGG16 model. The output from the VGG16 output layer will be an array with shape(sample_size, 7,7,512)","metadata":{}},{"cell_type":"code","source":"vgg = VGG16(weights='imagenet',\n include_top=False,\n input_shape=(224, 224, 3))\n\nfor layers in vgg.layers:\n layers.trainable=False\n\nprint(vgg.output)","metadata":{"execution":{"iopub.status.busy":"2023-11-22T21:36:39.316397Z","iopub.execute_input":"2023-11-22T21:36:39.316807Z","iopub.status.idle":"2023-11-22T21:36:40.025501Z","shell.execute_reply.started":"2023-11-22T21:36:39.316766Z","shell.execute_reply":"2023-11-22T21:36:40.024794Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We will use training set as our data that will be further split into train-test, due to memory constraint. The shape of the output features will be (18750, 25088)","metadata":{}},{"cell_type":"code","source":"feature_list = []\nfor path in train_df['filename'].to_numpy():\n x = load_img(path,target_size=(img_size,img_size))\n img_array = img_to_array(x)\n img_array = np.expand_dims(img_array, axis=0)\n features = vgg.predict(img_array)\n feature_list.append(features)\n \nfeat_lst = np.reshape(feature_list,(-1,7*7*512))","metadata":{"execution":{"iopub.status.busy":"2023-11-22T21:36:40.026819Z","iopub.execute_input":"2023-11-22T21:36:40.027037Z","iopub.status.idle":"2023-11-22T21:39:19.170427Z","shell.execute_reply.started":"2023-11-22T21:36:40.027012Z","shell.execute_reply":"2023-11-22T21:39:19.169728Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del feature_list\nprint(feat_lst.shape)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-11-22T21:39:19.172041Z","iopub.execute_input":"2023-11-22T21:39:19.172347Z","iopub.status.idle":"2023-11-22T21:39:19.188076Z","shell.execute_reply.started":"2023-11-22T21:39:19.172308Z","shell.execute_reply":"2023-11-22T21:39:19.187395Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Logistic Regression\nFeatures generated from VGG will be fed into Logistic Regression Model for classification. The logistic regression will output whether the features represent a Dog or a Cat","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nfrom sklearn.linear_model import LogisticRegression\n\ny = train_df['category'].to_numpy() #convert df to numpy array with shape(18750,)\n\nX_train, X_test, y_train, y_test = train_test_split(feat_lst, y, test_size=0.2, random_state=2020)\n\nglm = LogisticRegression(C=0.1)\nglm.fit(X_train,y_train)","metadata":{"execution":{"iopub.status.busy":"2023-11-22T21:39:19.189473Z","iopub.execute_input":"2023-11-22T21:39:19.189728Z","iopub.status.idle":"2023-11-22T21:39:50.694217Z","shell.execute_reply.started":"2023-11-22T21:39:19.189702Z","shell.execute_reply":"2023-11-22T21:39:50.693367Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Accuracy on validation set using Logistic Regression: \",glm.score(X_test,y_test))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-11-22T21:39:50.695771Z","iopub.execute_input":"2023-11-22T21:39:50.696320Z","iopub.status.idle":"2023-11-22T21:39:51.092127Z","shell.execute_reply.started":"2023-11-22T21:39:50.696279Z","shell.execute_reply":"2023-11-22T21:39:51.091258Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n### [2.2.2 Fine Tuning](#sec2.2.2)","metadata":{}},{"cell_type":"markdown","source":"Fine tuning on a pretrained model involves training some of the layers instead of freezing all the weights.","metadata":{}},{"cell_type":"markdown","source":"In this example, we will be using the pretrained [ResNet](https://arxiv.org/pdf/1512.03385.pdf) for fine tuning, by doing the following: \n1. Freezing the first 171 layers and only training the last 4 layers in Resnet. \n2. Adding our own fully connected layer with 1024 neurons.\n3. Add a single neuron Dense layer with sigmoid activation as our classifier.\n","metadata":{}},{"cell_type":"code","source":"np.random.seed(2020)\n\nres = resnet.ResNet50(weights='imagenet',\n include_top=False,\n input_shape=(224, 224, 3))\n\nres_train_gen = generate_train_batch('resnet')\nres_valid_gen = generate_valid_batch('resnet')\n\n\nfor layer in res.layers[:171]:\n layer.trainable=False\n \n\nflat = Flatten()(res.output) #Flatten the output layer from our Resnet model\ndense = Dense(1024,activation='relu')(flat)\ndrop = Dropout(0.5)(dense)\nclassifier = Dense(1, activation='sigmoid')(drop)\n\n\nres_model = Model(res.input, classifier)\noptimizer=optimizers.Adam(1e-5)\n\n\nres_model.compile(optimizer= optimizer,\n loss='binary_crossentropy',\n metrics=['accuracy'])\n\nres_model.fit_generator(res_train_gen,\n epochs=epochs,\n validation_data=res_valid_gen,\n validation_steps=res_train_gen.samples//batch_size,\n steps_per_epoch = res_valid_gen.samples//batch_size)","metadata":{"execution":{"iopub.status.busy":"2023-11-22T21:39:51.093688Z","iopub.execute_input":"2023-11-22T21:39:51.094234Z","iopub.status.idle":"2023-11-22T21:52:14.992809Z","shell.execute_reply.started":"2023-11-22T21:39:51.094194Z","shell.execute_reply":"2023-11-22T21:52:14.992079Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"loss, accuracy = res_model.evaluate_generator(res_valid_gen, res_valid_gen.samples//batch_size, workers=12)\nprint(\"Validation: accuracy = %f ; loss = %f \" % (accuracy, loss))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-11-22T21:52:14.994284Z","iopub.execute_input":"2023-11-22T21:52:14.994516Z","iopub.status.idle":"2023-11-22T21:52:39.242107Z","shell.execute_reply.started":"2023-11-22T21:52:14.994491Z","shell.execute_reply":"2023-11-22T21:52:39.241174Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# [3. Results](#sec3)","metadata":{}},{"cell_type":"markdown","source":"We will make predictions on 12 images collected from the **test data**, using the 3 architectures we have built thus far; [Custom CNN](#sec2.1), [Feature Extraction](#sec2.2.1) and [Fine Tuning](#sec2.2.2).","metadata":{}},{"cell_type":"code","source":"# define function for evaluating model performance on test images\n\ntestdf = pd.DataFrame({'filename': test_x})\ntest_sample = testdf.sample(n=12, random_state=2020)\n\ndef test_img(model,name):\n result_lst = []\n for path in test_sample['filename'].to_numpy():\n full_path = '../test1/'+path\n x = load_img(full_path, target_size=(224,224))\n img_array = img_to_array(x)\n img_array = np.expand_dims(img_array, axis=0)\n if name == 'vgg':\n features = model.predict(img_array)\n features = np.reshape(features,(-1,7*7*512))\n result = glm.predict(features)\n else:\n result = model.predict(img_array)\n \n result = 'dog' if float(result) >0.5 else 'cat'\n \n result_lst.append(result)\n return result_lst","metadata":{"execution":{"iopub.status.busy":"2023-11-22T21:52:39.243730Z","iopub.execute_input":"2023-11-22T21:52:39.244051Z","iopub.status.idle":"2023-11-22T21:52:39.258548Z","shell.execute_reply.started":"2023-11-22T21:52:39.244010Z","shell.execute_reply":"2023-11-22T21:52:39.257396Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# get test predictions from all models\ncustom_cnn_result = test_img(modelcnn, 'cnn')\ntrflearn_result = test_img(vgg,'vgg')\nfinetune_result = test_img(res_model,'resnet')","metadata":{"execution":{"iopub.status.busy":"2023-11-22T21:52:39.261016Z","iopub.execute_input":"2023-11-22T21:52:39.261252Z","iopub.status.idle":"2023-11-22T21:52:42.813340Z","shell.execute_reply.started":"2023-11-22T21:52:39.261227Z","shell.execute_reply":"2023-11-22T21:52:42.812691Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The architecture using Transfer Learning and Fine tuning makes pretty good prediction, agreeing with each other on almost all the images, except one, which the Fine tuning method has incorrectly predicted a Dog as Cat. Our Custom CNN doesnt really work very well with alot of false positive (missclassifying Cats as Dogs).","metadata":{}},{"cell_type":"code","source":"# plotting images with prediction\npred_results = list(zip(custom_cnn_result,trflearn_result,finetune_result))\ntest_array = test_sample['filename'].to_numpy()\n\nplt.figure(figsize=(15, 15))\nfor i in range(0, 12):\n plt.subplot(4, 3, i+1)\n cust,tf,ft = pred_results[i]\n img = test_array[i]\n path = '../test1/' + img\n image = load_img(path, target_size=(256,256))\n plt.text(135, 200, 'Custom CNN: {}'.format(cust), color='lightgreen',fontsize= 11, bbox=dict(facecolor='black', alpha=0.9))\n plt.text(135, 225, 'Transfer Learn: {}'.format(tf), color='lightgreen',fontsize= 11, bbox=dict(facecolor='black', alpha=0.9))\n plt.text(135, 250, 'Fine Tune: {}'.format(ft), color='lightgreen',fontsize= 11, bbox=dict(facecolor='black', alpha=0.9))\n plt.imshow(image)\n\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-11-22T21:52:42.814523Z","iopub.execute_input":"2023-11-22T21:52:42.814794Z","iopub.status.idle":"2023-11-22T21:52:45.165442Z","shell.execute_reply.started":"2023-11-22T21:52:42.814766Z","shell.execute_reply":"2023-11-22T21:52:45.164419Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Learning Points\n1. Setting a learning rate small enough is very important in the case of Transfer Learning and Fine Tuning, as i realized that setting learning rate to be 1e-4 will result in very low accuracy ~50%. This could be due to the complex loss function, thus, causing divergence during gradient updates if learning rate is not small enough.\n2. Images passed into ResNet needs to be preprocessed via the specific preprocessing function provided *from keras.applications.resnet50 import preprocess_input* when generating mini-batches, as it will affect the performance of ResNet if standard preprocessing is done instead.\n3. Due to memory constraint from Kaggle Kernel (~13GB), there is possibility of the kernel crashing. What i did to mitigate this is to use the *del* function, to remove some of the variables; feature output object from VGG16. I believe there are much better ways to do it, any advice from the experienced Kagglers will be much appreciated :)","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"I hope you guys have enjoyed following through this notebook. Do ping me up for any mistakes or doubts:) \n\nCheers \n","metadata":{}},{"cell_type":"markdown","source":"#### References \n\nSome references i have taken from: [@bulentsiyah](https://www.kaggle.com/bulentsiyah) and [@serkanpeldek](https://www.kaggle.com/serkanpeldek)\n\nhttps://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html \nhttps://www.learnopencv.com/keras-tutorial-using-pre-trained-imagenet-models/ ","metadata":{}},{"cell_type":"markdown","source":"#### For Submission","metadata":{"trusted":true}},{"cell_type":"code","source":" test_generator = ImageDataGenerator(rescale=1./255)\n test_gen = test_generator.flow_from_dataframe(\n testdf,\n '../test1',\n x_col='filename',\n y_col=None,\n class_mode=None,\n batch_size=batch_size,\n target_size=(img_size, img_size),\n shuffle=False\n )","metadata":{"execution":{"iopub.status.busy":"2023-11-22T21:52:45.167287Z","iopub.execute_input":"2023-11-22T21:52:45.167567Z","iopub.status.idle":"2023-11-22T21:52:45.315847Z","shell.execute_reply.started":"2023-11-22T21:52:45.167535Z","shell.execute_reply":"2023-11-22T21:52:45.314694Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# predict = model.predict_generator(test_gen, steps=test_gen.samples/batch_size)\n threshold = 0.5\n #testdf['category'] = np.where(predict > threshold, 1,0)","metadata":{"execution":{"iopub.status.busy":"2023-11-22T21:52:45.317127Z","iopub.execute_input":"2023-11-22T21:52:45.317376Z","iopub.status.idle":"2023-11-22T21:52:45.801243Z","shell.execute_reply.started":"2023-11-22T21:52:45.317350Z","shell.execute_reply":"2023-11-22T21:52:45.798858Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### (Add-ons) Pytorch Implementation for Resnet Finetuning","metadata":{}},{"cell_type":"code","source":"import torch\nfrom torch.utils.data import Dataset, DataLoader\nimport matplotlib.pyplot as plt\nfrom PIL import Image\nimport os","metadata":{"execution":{"iopub.status.busy":"2023-11-22T21:53:36.064616Z","iopub.execute_input":"2023-11-22T21:53:36.064953Z","iopub.status.idle":"2023-11-22T21:53:36.069416Z","shell.execute_reply.started":"2023-11-22T21:53:36.064899Z","shell.execute_reply":"2023-11-22T21:53:36.068635Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class CatDogData(Dataset):\n def __init__(self, direct, df, transform):\n self.dir = direct\n self.df = df\n self.transform = transform\n \n def __getitem__(self, idx):\n img, label = self.df.iloc[idx]\n path = os.path.join(self.dir, img)\n img_array = Image.open(path)\n \n if self.transform:\n img_array = self.transform(img_array)\n label = torch.as_tensor(int(label))\n \n return img_array, label\n \n def __len__(self):\n return self.df.shape[0]","metadata":{"execution":{"iopub.status.busy":"2023-11-22T21:53:39.272555Z","iopub.execute_input":"2023-11-22T21:53:39.272907Z","iopub.status.idle":"2023-11-22T21:53:39.281424Z","shell.execute_reply.started":"2023-11-22T21:53:39.272870Z","shell.execute_reply":"2023-11-22T21:53:39.280359Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Augmentation","metadata":{}},{"cell_type":"code","source":"from torchvision import transforms\n\ntrain_trf = transforms.Compose([\n transforms.Resize((224, 224)),\n transforms.ToTensor(),\n transforms.Normalize(\n mean=[0.485, 0.456, 0.406],\n std=[0.229, 0.224, 0.225]\n )\n])\n\nvalid_trf = transforms.Compose([\n transforms.Resize((224, 224)),\n transforms.ToTensor(),\n transforms.Normalize(\n mean=[0.485, 0.456, 0.406],\n std=[0.229, 0.224, 0.225]\n )\n])","metadata":{"execution":{"iopub.status.busy":"2023-11-22T21:53:46.719336Z","iopub.execute_input":"2023-11-22T21:53:46.719658Z","iopub.status.idle":"2023-11-22T21:53:47.011297Z","shell.execute_reply.started":"2023-11-22T21:53:46.719622Z","shell.execute_reply":"2023-11-22T21:53:47.010337Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Create dataloader for mini batch training","metadata":{}},{"cell_type":"code","source":"train_data = CatDogData('/kaggle/working/temp/train', train_df, train_trf)\nvalid_data = CatDogData('/kaggle/working/temp/train', valid_df, valid_trf)\n\ntrain_loader = DataLoader(train_data, batch_size = 32, shuffle=True)\nvalid_loader = DataLoader(valid_data, batch_size=32)","metadata":{"execution":{"iopub.status.busy":"2023-11-22T21:53:54.444056Z","iopub.execute_input":"2023-11-22T21:53:54.444457Z","iopub.status.idle":"2023-11-22T21:53:54.449963Z","shell.execute_reply.started":"2023-11-22T21:53:54.444414Z","shell.execute_reply":"2023-11-22T21:53:54.449143Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Model","metadata":{}},{"cell_type":"code","source":"from torchvision.models import resnet50\nimport torch.nn as nn","metadata":{"execution":{"iopub.status.busy":"2023-11-22T21:53:58.152489Z","iopub.execute_input":"2023-11-22T21:53:58.152800Z","iopub.status.idle":"2023-11-22T21:53:58.156670Z","shell.execute_reply.started":"2023-11-22T21:53:58.152769Z","shell.execute_reply":"2023-11-22T21:53:58.155910Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = resnet50(pretrained=True)","metadata":{"execution":{"iopub.status.busy":"2023-11-22T21:54:02.229313Z","iopub.execute_input":"2023-11-22T21:54:02.229642Z","iopub.status.idle":"2023-11-22T21:54:04.885781Z","shell.execute_reply.started":"2023-11-22T21:54:02.229604Z","shell.execute_reply":"2023-11-22T21:54:04.885056Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pip install torchsummary","metadata":{"execution":{"iopub.status.busy":"2023-11-22T21:54:11.000611Z","iopub.execute_input":"2023-11-22T21:54:11.000911Z","iopub.status.idle":"2023-11-22T21:54:19.761572Z","shell.execute_reply.started":"2023-11-22T21:54:11.000882Z","shell.execute_reply":"2023-11-22T21:54:19.760723Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"c = 0\nfor child in model.children():\n if c < 9:\n for param in child.parameters():\n param.requires_grad = False\n c+=1","metadata":{"execution":{"iopub.status.busy":"2023-11-22T21:54:25.028530Z","iopub.execute_input":"2023-11-22T21:54:25.028853Z","iopub.status.idle":"2023-11-22T21:54:25.034881Z","shell.execute_reply.started":"2023-11-22T21:54:25.028823Z","shell.execute_reply":"2023-11-22T21:54:25.033854Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"output_layer = nn.Linear(1000, 2)\nmodel = nn.Sequential(model, output_layer)","metadata":{"execution":{"iopub.status.busy":"2023-11-22T21:54:35.575688Z","iopub.execute_input":"2023-11-22T21:54:35.575984Z","iopub.status.idle":"2023-11-22T21:54:35.580651Z","shell.execute_reply.started":"2023-11-22T21:54:35.575954Z","shell.execute_reply":"2023-11-22T21:54:35.579907Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')\nmodel = model.to(device)","metadata":{"execution":{"iopub.status.busy":"2023-11-22T21:54:42.445633Z","iopub.execute_input":"2023-11-22T21:54:42.445947Z","iopub.status.idle":"2023-11-22T21:54:42.521148Z","shell.execute_reply.started":"2023-11-22T21:54:42.445912Z","shell.execute_reply":"2023-11-22T21:54:42.520272Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class Averager:\n def __init__(self):\n self.current_total = 0.0\n self.iterations = 0.0\n\n def send(self, value):\n self.current_total += value\n self.iterations += 1\n\n @property\n def value(self):\n if self.iterations == 0:\n return 0\n else:\n return 1.0 * self.current_total / self.iterations\n\n def reset(self):\n self.current_total = 0.0\n self.iterations = 0.0","metadata":{"execution":{"iopub.status.busy":"2023-11-22T21:54:45.434761Z","iopub.execute_input":"2023-11-22T21:54:45.435080Z","iopub.status.idle":"2023-11-22T21:54:45.442205Z","shell.execute_reply.started":"2023-11-22T21:54:45.435049Z","shell.execute_reply":"2023-11-22T21:54:45.441194Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import tqdm","metadata":{"execution":{"iopub.status.busy":"2023-11-22T21:54:54.707262Z","iopub.execute_input":"2023-11-22T21:54:54.707546Z","iopub.status.idle":"2023-11-22T21:54:54.711608Z","shell.execute_reply.started":"2023-11-22T21:54:54.707518Z","shell.execute_reply":"2023-11-22T21:54:54.710663Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class Fitter:\n def __init__(self, model, device):\n self.model = model\n self.device = device\n \n param_list = [p for p in model.parameters() if p.requires_grad]\n \n self.optimizer = torch.optim.Adam(param_list, lr = 1e-5)\n \n self.criterion = nn.CrossEntropyLoss()\n \n def fit(self, model, train_loader, valid_loader):\n bst_loss = 10\n print(f'Model running with {self.device}')\n for epoch in range(5):\n self.train_one_epoch(model, train_loader)\n val_loss = self.validate(model, valid_loader)\n \n print(\"Previous loss: {} Current loss: {}\".format(bst_loss, val_loss))\n if val_loss < bst_loss:\n model.eval()\n torch.save(model, f'{epoch}.pth')\n bst_loss = val_loss\n \n def train_one_epoch(self, model, train_loader):\n itr = 1\n loss_hist = Averager()\n self.model.train()\n \n for images, targets in train_loader:\n images = images.to(self.device).float()\n targets = targets.to(self.device)\n\n output = self.model(images)\n loss = self.criterion(output, targets)\n loss_hist.send(loss.item())\n\n self.optimizer.zero_grad()\n loss.backward()\n self.optimizer.step()\n \n if itr % 100 == 0:\n print(f\"Iteration #{itr} loss: {loss.item()}\")\n\n\n itr += 1\n \n print(f\"Training Epoch loss: {loss_hist.value}\")\n loss_hist.reset()\n \n \n def validate(self, model, valid_loader):\n acc = 0\n val_loss = 0\n loss_hist = Averager()\n self.model.eval()\n \n for images, targets in valid_loader:\n with torch.no_grad():\n images = images.to(self.device).float()\n targets = targets.to(self.device)\n\n output = self.model(images)\n loss = self.criterion(output, targets)\n loss_hist.send(loss.item())\n\n #acc\n log = torch.sigmoid(output)\n pred = torch.argmax(log, 1)\n acc += (targets.cpu() == pred.cpu()).sum().item()\n \n\n val_loss = loss_hist.value \n print(f\"Validation Epoch loss: {loss_hist.value}\")\n loss_hist.reset()\n print(\"Validation Accuracy: {}\".format(acc/len(valid_data)))\n \n return val_loss","metadata":{"execution":{"iopub.status.busy":"2023-11-22T21:54:58.111550Z","iopub.execute_input":"2023-11-22T21:54:58.111879Z","iopub.status.idle":"2023-11-22T21:54:58.133014Z","shell.execute_reply.started":"2023-11-22T21:54:58.111849Z","shell.execute_reply":"2023-11-22T21:54:58.132214Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"run_model = Fitter(model, device)\nrun_model.fit(model, train_loader, valid_loader)","metadata":{"execution":{"iopub.status.busy":"2023-11-22T21:55:06.361339Z","iopub.execute_input":"2023-11-22T21:55:06.361650Z","iopub.status.idle":"2023-11-22T21:55:06.971874Z","shell.execute_reply.started":"2023-11-22T21:55:06.361617Z","shell.execute_reply":"2023-11-22T21:55:06.970528Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#base_model.trainable = True\n#model.summary()\n\nfrom tensorflow.keras.applications import ResNet50\n\n# Define the base model\nbase_model = ResNet50(weights='imagenet', include_top=False)\n\n# Set the base model to be trainable\nbase_model.trainable = True\n\n# Print the model summary\nbase_model.summary()","metadata":{"execution":{"iopub.status.busy":"2023-11-22T21:52:45.826194Z","iopub.status.idle":"2023-11-22T21:52:45.826541Z"},"trusted":true},"execution_count":null,"outputs":[]}]}