{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":70497,"databundleVersionId":7713244,"sourceType":"competition"}],"dockerImageVersionId":30646,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Segmentation-Assisted Transfer Learning: Enhancing Image Classification with YOLOv8 Segmentation, and ResNet and Inception Transfer Learning\nKerasTuner\n","metadata":{}},{"cell_type":"markdown","source":"## Table of Contents\n- [Introduction](#introduction)\n- [Importing Required Libraries](#importing-required-libraries)\n- [Installing YOLO and Loading YOLOv8 Segmentation Model](#installing-yolo-and-loading-yolov8-segmentation-model)\n- [Function to Isolate Objects on White Background](#function-to-isolate-objects-on-white-background)\n- [Data Generators](#data-generators)\n- [Image Batch Visualization](#image-batch-viz)\n- [Using ResNet101v2 as a Base Model](#using-resnet101v2-as-a-base-model)\n- [Model Architecture](#model-architecture)\n- [Training the Model](#training-the-model)\n- [Fine-tuning the Model](#fine-tuning-the-model)\n- [Evaluating Results and Visualization](#evaluating)\n- [Predicting the Results](#predicting-the-results)\n","metadata":{}},{"cell_type":"markdown","source":"<a id = \"introduction\"></a>\n\n# Introduction\n\nThis Kaggle notebook demonstrates a fruit classification task using a combination of ResNet101V2 as a base model and YOLOv8 segmentation for object detection. The goal is to classify fruits accurately based on images.","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport tensorflow as tf\nimport torch\nimport random\n\n# Set seed for NumPy\nnp.random.seed(42)\n\n# Set seed for TensorFlow\ntf.random.set_seed(42)\n\n# Set seed for PyTorch\ntorch.manual_seed(42)","metadata":{"execution":{"iopub.status.busy":"2024-03-12T05:46:35.118286Z","iopub.execute_input":"2024-03-12T05:46:35.118634Z","iopub.status.idle":"2024-03-12T05:47:01.011883Z","shell.execute_reply.started":"2024-03-12T05:46:35.118591Z","shell.execute_reply":"2024-03-12T05:47:01.010963Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"importing-required-libraries\"></a>\n\n# Importing Required Libraries\n\nThe notebook begins by importing necessary libraries including OpenCV, NumPy, Pandas, Matplotlib, TensorFlow, and Ultralytics for YOLOv8.","metadata":{}},{"cell_type":"code","source":"import os\nfrom pathlib import Path\n\nimport cv2\nimport numpy as np\nimport pandas as pd\n\nimport matplotlib.pyplot as plt\n\nfrom zipfile import ZipFile\nfrom os import listdir, walk\nfrom os.path import isfile, join\n\nfrom tensorflow.keras import Input, Model\nfrom tensorflow.keras.applications import ResNet101V2, Xception, InceptionV3\nfrom tensorflow.keras.applications.imagenet_utils import preprocess_input\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator, load_img, img_to_array\nfrom tensorflow.keras.layers import GlobalAveragePooling2D, Dense\nfrom tensorflow.keras.optimizers import Adam","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-03-12T05:47:01.013635Z","iopub.execute_input":"2024-03-12T05:47:01.014139Z","iopub.status.idle":"2024-03-12T05:47:01.724854Z","shell.execute_reply.started":"2024-03-12T05:47:01.014114Z","shell.execute_reply":"2024-03-12T05:47:01.724070Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"installing-yolo-and-loading-yolov8-segmentation-model\"></a>\n\n# Installing YOLO and Loading YOLOv8 Segmentation Model\nThe YOLOv8 segmentation model is installed using the Ultralytics library. After installation, the model is loaded for further use.","metadata":{}},{"cell_type":"code","source":"!pip install ultralytics --quiet\nimport ultralytics\nultralytics.checks()","metadata":{"execution":{"iopub.status.busy":"2024-03-12T05:47:01.725896Z","iopub.execute_input":"2024-03-12T05:47:01.726177Z","iopub.status.idle":"2024-03-12T05:47:23.917612Z","shell.execute_reply.started":"2024-03-12T05:47:01.726154Z","shell.execute_reply":"2024-03-12T05:47:23.916665Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load YOLOv8n-seg, predict an image with it\nfrom ultralytics import YOLO\nseg_model = YOLO('yolov8n-seg.pt')  # load a pretrained YOLOv8n segmentation model","metadata":{"execution":{"iopub.status.busy":"2024-03-12T05:47:23.920042Z","iopub.execute_input":"2024-03-12T05:47:23.920394Z","iopub.status.idle":"2024-03-12T05:47:24.613389Z","shell.execute_reply.started":"2024-03-12T05:47:23.920361Z","shell.execute_reply":"2024-03-12T05:47:24.612313Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"function-to-isolate-objects-on-white-background\"></a>\n\n# Function to Isolate Objects on White Background\nA Python function is defined to isolate objects detected by the YOLOv8 segmentation model on a white background. This function takes an input image and the segmentation model as input and returns an image with detected objects isolated on a white background.","metadata":{}},{"cell_type":"code","source":"def crop_square_around_mask(image, mask):\n    contours, _ = cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)\n    if contours:\n        x, y, w, h = cv2.boundingRect(contours[0])\n        size = max(w, h)\n        center_x = x + w // 2\n        center_y = y + h // 2\n        half_size = size // 2\n        x1 = max(center_x - half_size, 0)\n        y1 = max(center_y - half_size, 0)\n        x2 = min(center_x + half_size, image.shape[1])\n        y2 = min(center_y + half_size, image.shape[0])\n        cropped_image = image[y1:y2, x1:x2]\n        # Check if the cropped image is not square\n        if cropped_image.shape[0] != cropped_image.shape[1]:\n            # Determine the size of the padding\n            pad_size = max(cropped_image.shape[0], cropped_image.shape[1])\n            \n            # Create white background\n            white_bg = np.ones((pad_size, pad_size, 3), dtype=np.uint8) * 255\n            \n            # Calculate the position to place the cropped image\n            pos_x = (pad_size - cropped_image.shape[1]) // 2\n            pos_y = (pad_size - cropped_image.shape[0]) // 2\n            \n            # Place the cropped image on the white background\n            white_bg[pos_y:pos_y + cropped_image.shape[0], pos_x:pos_x + cropped_image.shape[1]] = cropped_image\n            \n            # Update the cropped image\n            cropped_image = white_bg\n        return cropped_image\n    return None\n\ndef isolate_object_on_white_bg(im_path, model, target_classes=[\"apple\", \"orange\", \"banana\"]):\n    img = cv2.imread(im_path, cv2.IMREAD_UNCHANGED)\n    \n    results = model(im_path)  # predict the image \n    for r in results:\n        if r.masks is not None:\n            result_img = np.ones_like(img) * 255  # White background\n            \n            for mask, c in zip(r.masks.xy, r.boxes.cls.cpu().numpy()):  # Move tensors to CPU and then convert to numpy arrays\n                class_name = r.names[int(c)] \n                if class_name in target_classes:\n                    b_mask = np.zeros(img.shape[:2], np.uint8)\n                    contour = mask.astype(np.int32).reshape(-1, 1, 2)\n                    _ = cv2.drawContours(b_mask, [contour], -1, (255, 255, 255), cv2.FILLED)\n                    mask3ch = cv2.cvtColor(b_mask, cv2.COLOR_GRAY2BGR)\n                    isolated = cv2.bitwise_and(mask3ch, img)\n                    isolated[np.where((isolated==[0,0,0]).all(axis=2))] = [255,255,255]\n                    result_img = cv2.bitwise_and(isolated, result_img)\n            \n            if np.all(result_img == 255):\n                return img\n            return crop_square_around_mask(result_img, b_mask)\n        \n    return img\n","metadata":{"execution":{"iopub.status.busy":"2024-03-12T05:52:27.514558Z","iopub.execute_input":"2024-03-12T05:52:27.515160Z","iopub.status.idle":"2024-03-12T05:52:27.531957Z","shell.execute_reply.started":"2024-03-12T05:52:27.515129Z","shell.execute_reply":"2024-03-12T05:52:27.531171Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To demonstrate the usage of the isolate_object_on_white_bg function, you can follow the example provided below. This example illustrates how to isolate objects detected by the YOLOv8 segmentation model on a white background using an input image.","metadata":{}},{"cell_type":"code","source":"# Example usage:\nimg_path = \"/kaggle/input/csc532-fruits-categorical-classification-2024/csc532_2024_validation/validation/DLAI5_Validation_fruits (106).jpg\"\nrimg = isolate_object_on_white_bg(img_path, seg_model)\n\nplt.imshow(cv2.cvtColor(rimg, cv2.COLOR_BGR2RGB))\n# plt.imshow(cv2.imread(img_path))","metadata":{"execution":{"iopub.status.busy":"2024-03-11T18:49:46.882155Z","iopub.execute_input":"2024-03-11T18:49:46.882930Z","iopub.status.idle":"2024-03-11T18:49:47.224695Z","shell.execute_reply.started":"2024-03-11T18:49:46.882895Z","shell.execute_reply":"2024-03-11T18:49:47.223616Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def display_original_and_isolated(img_path, seg_model):\n    # Load the original image\n    original_img = cv2.imread(img_path)\n    \n    # Isolate object on white background\n    isolated_img = isolate_object_on_white_bg(img_path, seg_model)\n    \n    # Plotting side by side\n    fig, axes = plt.subplots(1, 2, figsize=(10, 5))\n    axes[0].imshow(cv2.cvtColor(original_img, cv2.COLOR_BGR2RGB))\n    axes[0].set_title('Original Image')\n    axes[0].axis('off')\n    axes[1].imshow(cv2.cvtColor(isolated_img, cv2.COLOR_BGR2RGB))\n    axes[1].set_title('Isolated Object')\n    axes[1].axis('off')\n    plt.show()\n\n# Path to the directory containing images\ndirectory = \"/kaggle/input/csc532-fruits-categorical-classification-2024/csc532_2024_validation/validation/\"\n\n# Assuming you have already defined seg_model\n\n# Iterate over images in the directory\nfor filename in random.sample(os.listdir(directory), 5):\n    if filename.endswith(\".jpg\"):\n        img_path = os.path.join(directory, filename)\n        display_original_and_isolated(img_path, seg_model)","metadata":{"execution":{"iopub.status.busy":"2024-03-12T05:51:54.499919Z","iopub.execute_input":"2024-03-12T05:51:54.500732Z","iopub.status.idle":"2024-03-12T05:52:01.331554Z","shell.execute_reply.started":"2024-03-12T05:51:54.500699Z","shell.execute_reply":"2024-03-12T05:52:01.330575Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"data-generators\"></a>\n\n# Data Generators\nImageDataGenerators are created to preprocess the training and validation images.","metadata":{}},{"cell_type":"code","source":"IMG_SIZE = 224\n\n# datagen = ImageDataGenerator(\n#         samplewise_center=True,\n#         rescale=1/255\n# )\n\ndatagen = ImageDataGenerator(\n#     samplewise_center=True,  # set each sample mean to 0\n    rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)\n    zoom_range=0.3,  # Randomly zoom image\n    width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)\n    height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)\n    horizontal_flip=True,  # randomly flip images\n    vertical_flip=True,\n    rescale=1/255\n) \n","metadata":{"execution":{"iopub.status.busy":"2024-03-12T06:04:38.240273Z","iopub.execute_input":"2024-03-12T06:04:38.240692Z","iopub.status.idle":"2024-03-12T06:04:38.246266Z","shell.execute_reply.started":"2024-03-12T06:04:38.240646Z","shell.execute_reply":"2024-03-12T06:04:38.245349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_it = datagen.flow_from_directory('/kaggle/input/csc532-fruits-categorical-classification-2024/csc532_2024_train/train', \n                                       target_size=(IMG_SIZE, IMG_SIZE), \n                                       color_mode='rgb', \n                                       class_mode=\"categorical\")","metadata":{"execution":{"iopub.status.busy":"2024-03-12T06:04:41.319905Z","iopub.execute_input":"2024-03-12T06:04:41.320258Z","iopub.status.idle":"2024-03-12T06:04:45.784352Z","shell.execute_reply.started":"2024-03-12T06:04:41.320230Z","shell.execute_reply":"2024-03-12T06:04:45.783440Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"valid_it = datagen.flow_from_directory('/kaggle/input/csc532-fruits-categorical-classification-2024/csc532_2024_test/test', \n                                       target_size=(IMG_SIZE, IMG_SIZE), \n                                       color_mode='rgb', \n                                       class_mode=\"categorical\",\n                                       shuffle=False)","metadata":{"execution":{"iopub.status.busy":"2024-03-12T06:04:45.786224Z","iopub.execute_input":"2024-03-12T06:04:45.786701Z","iopub.status.idle":"2024-03-12T06:04:46.020829Z","shell.execute_reply.started":"2024-03-12T06:04:45.786664Z","shell.execute_reply":"2024-03-12T06:04:46.019916Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"files = [f for f in listdir('/kaggle/input/csc532-fruits-categorical-classification-2024/csc532_2024_validation/validation/') if isfile(join('/kaggle/input/csc532-fruits-categorical-classification-2024/csc532_2024_validation/validation', f))]\ntest_files = []\nfor file in files:\n    img_path = f'/kaggle/input/csc532-fruits-categorical-classification-2024/csc532_2024_validation/validation/{file}'\n    \n    image = isolate_object_on_white_bg(img_path, seg_model)\n    image = cv2.resize(image, (IMG_SIZE, IMG_SIZE))\n    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n    image = img_to_array(image)\n    image = image.reshape(IMG_SIZE, IMG_SIZE, 3)\n    image = image / 255.\n    test_files.append(image)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(test_files)","metadata":{"execution":{"iopub.status.busy":"2024-03-12T06:00:33.005669Z","iopub.execute_input":"2024-03-12T06:00:33.006052Z","iopub.status.idle":"2024-03-12T06:00:33.012327Z","shell.execute_reply.started":"2024-03-12T06:00:33.006024Z","shell.execute_reply":"2024-03-12T06:00:33.011441Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"image-batch-viz\"></a>\n\n# Image Batch Visualization\n","metadata":{}},{"cell_type":"markdown","source":"### Obtaining a Batch of Images and Labels\nTo obtain a batch of images and corresponding labels from the train and test datasets, we use the next() function on the generators (train_generator and test_generator). This function yields a batch of images along with their respective labels.","metadata":{}},{"cell_type":"code","source":"# Get a batch of images and labels from the train and test datasets\ntrain_images, train_labels = next(train_it)\ntest_images, test_labels = next(valid_it)","metadata":{"execution":{"iopub.status.busy":"2024-03-12T06:04:48.130150Z","iopub.execute_input":"2024-03-12T06:04:48.130518Z","iopub.status.idle":"2024-03-12T06:04:49.429917Z","shell.execute_reply.started":"2024-03-12T06:04:48.130489Z","shell.execute_reply":"2024-03-12T06:04:49.429107Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Displaying a Grid of Sample Images\nA function named show_images is defined to display a grid of sample images along with their corresponding class labels. This function takes three parameters:\n\nimages: A batch of images to display.\nlabels: The corresponding labels for the images.\ntitle: The title of the plot.\nThe function iterates over the images, plots them in a grid format using Matplotlib, and displays the class label for each image.","metadata":{}},{"cell_type":"code","source":"# Define a function to display a grid of sample images\ndef show_images(images, labels, title):\n    plt.figure(figsize=(10, 10))\n    for i in range(9):\n#         print(len(images))\n        plt.subplot(3, 3, i + 1)\n        plt.imshow(images[i])\n        plt.title(f'Class: {labels[i] if isinstance(labels[i], str) else labels[i].argmax()}')\n        plt.axis('off')\n    plt.suptitle(title, fontsize=16)\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2024-03-12T06:08:51.343111Z","iopub.execute_input":"2024-03-12T06:08:51.343534Z","iopub.status.idle":"2024-03-12T06:08:51.349759Z","shell.execute_reply.started":"2024-03-12T06:08:51.343503Z","shell.execute_reply":"2024-03-12T06:08:51.348855Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"type(\"appel\")","metadata":{"execution":{"iopub.status.busy":"2024-03-12T06:05:51.914719Z","iopub.execute_input":"2024-03-12T06:05:51.915077Z","iopub.status.idle":"2024-03-12T06:05:51.921331Z","shell.execute_reply.started":"2024-03-12T06:05:51.915051Z","shell.execute_reply":"2024-03-12T06:05:51.920389Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Sample Images from Train Dataset\nA grid of sample images from the train dataset is displayed using the show_images function.","metadata":{}},{"cell_type":"code","source":"# Display a grid of sample images from the train dataset\nshow_images(train_images, train_labels, 'Sample Images from Train Dataset')","metadata":{"execution":{"iopub.status.busy":"2024-03-12T06:07:35.833717Z","iopub.execute_input":"2024-03-12T06:07:35.834313Z","iopub.status.idle":"2024-03-12T06:07:36.658559Z","shell.execute_reply.started":"2024-03-12T06:07:35.834284Z","shell.execute_reply":"2024-03-12T06:07:36.657621Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Sample Images from Test Dataset\nSimilarly, a grid of sample images from the test dataset is displayed using the show_images function.","metadata":{}},{"cell_type":"code","source":"# Display a grid of sample images from the test dataset\nshow_images(test_images, test_labels, 'Sample Images from Test Dataset')","metadata":{"execution":{"iopub.status.busy":"2024-03-12T06:07:52.069160Z","iopub.execute_input":"2024-03-12T06:07:52.069840Z","iopub.status.idle":"2024-03-12T06:07:52.910529Z","shell.execute_reply.started":"2024-03-12T06:07:52.069809Z","shell.execute_reply":"2024-03-12T06:07:52.909649Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Display a grid of sample images from the test dataset\nshow_images(random.sample(test_files, 9), [\"unknown\"]*9, 'Sample Images from Test Dataset')","metadata":{"execution":{"iopub.status.busy":"2024-03-12T06:08:55.468853Z","iopub.execute_input":"2024-03-12T06:08:55.469199Z","iopub.status.idle":"2024-03-12T06:08:56.349318Z","shell.execute_reply.started":"2024-03-12T06:08:55.469173Z","shell.execute_reply":"2024-03-12T06:08:56.348465Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"using-resnet101v2-as-a-base-model\"></a>\n\n# Using ResNet101v2 as a Base Model\nThe ResNet101V2 model is used as a base model for feature extraction. Weights are initialized with pre-trained ImageNet weights and the top layer is removed.","metadata":{}},{"cell_type":"code","source":"# base_model = ResNet101V2(\n#     weights='imagenet',\n#     input_shape=(IMG_SIZE, IMG_SIZE, 3),\n#     include_top=False)\n\n# base_model.trainable = False","metadata":{"execution":{"iopub.status.busy":"2024-03-12T03:04:43.300342Z","iopub.execute_input":"2024-03-12T03:04:43.300724Z","iopub.status.idle":"2024-03-12T03:04:48.592323Z","shell.execute_reply.started":"2024-03-12T03:04:43.300692Z","shell.execute_reply":"2024-03-12T03:04:48.591486Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Using InceptionV3 as a Base Model","metadata":{}},{"cell_type":"code","source":"# base_model = InceptionV3(\n#     weights='imagenet',\n#     input_shape=(IMG_SIZE, IMG_SIZE, 3),\n#     include_top=False)\n\n# base_model.trainable = False","metadata":{"execution":{"iopub.status.busy":"2024-03-11T18:51:22.914851Z","iopub.execute_input":"2024-03-11T18:51:22.915238Z","iopub.status.idle":"2024-03-11T18:51:26.594941Z","shell.execute_reply.started":"2024-03-11T18:51:22.915207Z","shell.execute_reply":"2024-03-11T18:51:26.594138Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"model-architecture\"></a>\n\n# Model Architecture\nA new model architecture is created with ResNet101v2 as a base model and an output layer with 6 outputs corresponding to the 6 fruit classes. The model is compiled with categorical cross-entropy loss and Adam optimizer.","metadata":{}},{"cell_type":"code","source":"# # Define the input shape for the model, assuming IMG_SIZE x IMG_SIZE images with 3 color channels (RGB)\n# inputs = Input(shape=(IMG_SIZE, IMG_SIZE, 3))\n\n# # Connect the input to a pre-trained base model, with training set to False to freeze the base model's weights\n# x = base_model(inputs, training=False)\n\n# # Apply Global Average Pooling to reduce spatial dimensions and extract important features\n# x = GlobalAveragePooling2D()(x)\n\n# # Connect the output of the Global Average Pooling layer to a Dense layer with softmax activation\n# # The number of units in the Dense layer is set to the number of classes in the training dataset\n# outputs = Dense(len(train_it.class_indices), activation='softmax')(x)\n\n# model = Model(inputs, outputs)","metadata":{"execution":{"iopub.status.busy":"2024-03-12T03:04:48.593842Z","iopub.execute_input":"2024-03-12T03:04:48.594156Z","iopub.status.idle":"2024-03-12T03:04:49.532698Z","shell.execute_reply.started":"2024-03-12T03:04:48.594129Z","shell.execute_reply":"2024-03-12T03:04:49.531934Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# model.compile(loss = 'categorical_crossentropy' , optimizer='adam', metrics = [\"accuracy\"])","metadata":{"execution":{"iopub.status.busy":"2024-03-12T03:04:49.533787Z","iopub.execute_input":"2024-03-12T03:04:49.534100Z","iopub.status.idle":"2024-03-12T03:04:49.559579Z","shell.execute_reply.started":"2024-03-12T03:04:49.534074Z","shell.execute_reply":"2024-03-12T03:04:49.558883Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"training-the-model\"></a>\n\n# Training the Model\nThe model is trained for 20 epochs using the training and validation data generators.","metadata":{}},{"cell_type":"code","source":"# epochs = 10\n\n# history = model.fit(\n#     train_it,\n#     epochs=epochs,\n#     validation_data=valid_it,\n# )","metadata":{"execution":{"iopub.status.busy":"2024-03-12T03:04:53.263147Z","iopub.execute_input":"2024-03-12T03:04:53.263499Z","iopub.status.idle":"2024-03-12T03:38:20.936107Z","shell.execute_reply.started":"2024-03-12T03:04:53.263471Z","shell.execute_reply":"2024-03-12T03:38:20.934955Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"fine-tuning-the-model\"></a>\n\n# Fine-tuning the Model\nThe base model is fine-tuned by unfreezing some layers and re-compiling the model with a lower learning rate.","metadata":{}},{"cell_type":"code","source":"# base_model.trainable = True\n\n# fine_tune_at = 100\n\n# for layer in base_model.layers[:fine_tune_at]:\n#   layer.trainable =  False","metadata":{"execution":{"iopub.status.busy":"2024-03-12T03:38:20.938163Z","iopub.execute_input":"2024-03-12T03:38:20.938482Z","iopub.status.idle":"2024-03-12T03:38:20.963137Z","shell.execute_reply.started":"2024-03-12T03:38:20.938454Z","shell.execute_reply":"2024-03-12T03:38:20.962143Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# model.compile(loss = 'categorical_crossentropy' , optimizer=Adam(learning_rate=0.001/10), metrics = [\"accuracy\"])","metadata":{"execution":{"iopub.status.busy":"2024-03-12T03:38:20.964296Z","iopub.execute_input":"2024-03-12T03:38:20.964593Z","iopub.status.idle":"2024-03-12T03:38:20.998220Z","shell.execute_reply.started":"2024-03-12T03:38:20.964557Z","shell.execute_reply":"2024-03-12T03:38:20.997365Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # training again for 10 epochs\n# epochs = 10\n# history = model.fit(\n#     train_it,\n#     epochs=epochs,\n#     validation_data=valid_it,\n# )","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Utilizae KerasTuner to get the best search of hyperparameters","metadata":{}},{"cell_type":"code","source":"import kerastuner as kt\nfrom tensorflow.keras.applications import InceptionV3\nfrom tensorflow.keras.layers import GlobalAveragePooling2D, Dense\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras.optimizers import Adam\n\n# Assuming IMG_SIZE and train_it are defined appropriately\n\ndef build_model(hp):\n    base_model = InceptionV3(\n        weights='imagenet',\n        input_shape=(IMG_SIZE, IMG_SIZE, 3),\n        include_top=False)\n\n    base_model.trainable = False\n\n    inputs = Input(shape=(IMG_SIZE, IMG_SIZE, 3))\n    x = base_model(inputs, training=False)\n    x = GlobalAveragePooling2D()(x)\n\n    # Tune the number of units in the Dense layer\n    hp_units = hp.Int('units', min_value=32, max_value=512, step=32)\n    outputs = Dense(hp_units, activation='relu')(x)\n\n    outputs = Dense(len(train_it.class_indices), activation='softmax')(outputs)\n\n    model = Model(inputs, outputs)\n\n    # Tune the learning rate for the optimizer\n    hp_learning_rate = hp.Choice('learning_rate', values=[1e-2, 1e-3, 1e-4])\n\n    model.compile(optimizer=Adam(learning_rate=hp_learning_rate),\n                  loss='categorical_crossentropy',\n                  metrics=['accuracy'])\n\n    return model\n\n# Initialize the tuner (here we use the RandomSearch tuner)\ntuner = kt.RandomSearch(\n    build_model,\n    objective='val_accuracy',  # 'val_loss' or 'val_accuracy' based on your preference\n    max_trials=5,  # Number of hyperparameter combinations to try\n    directory='keras_tuner_dir',  # Directory to store the tuning results\n    project_name='my_image_classification'  # Name of the tuning project\n)\n\n# Define callbacks if needed, e.g., early stopping\n# callbacks = [tf.keras.callbacks.EarlyStopping(patience=3)]\n\n# Run the tuner search\ntuner.search(train_it, epochs=10, validation_data=valid_it)  # Assuming train_it and val_it are your training and validation data iterators\n\n# Get the optimal hyperparameters\nbest_hps=tuner.get_best_hyperparameters(num_trials=1)[0]\n\n# Build the model with the best hyperparameters\nmodel = tuner.hypermodel.build(best_hps)\nmodel.fit(train_it, epochs=10, validation_data=valid_it)\n","metadata":{"execution":{"iopub.status.busy":"2024-03-11T18:52:43.089181Z","iopub.execute_input":"2024-03-11T18:52:43.090008Z","iopub.status.idle":"2024-03-11T20:17:17.770487Z","shell.execute_reply.started":"2024-03-11T18:52:43.089972Z","shell.execute_reply":"2024-03-11T20:17:17.769509Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.save('rottenfruits_6_cls_inception_keras_tuner.h5')","metadata":{"execution":{"iopub.status.busy":"2024-03-05T08:36:32.757334Z","iopub.status.idle":"2024-03-05T08:36:32.757782Z","shell.execute_reply.started":"2024-03-05T08:36:32.757555Z","shell.execute_reply":"2024-03-05T08:36:32.757575Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"evaluating\"></a>\n\n# Evaluating the Results and Visulaization \n\n","metadata":{}},{"cell_type":"markdown","source":"### Evaluate Model on Validation Data\nTo assess the performance of the trained model on the validation dataset, we use the evaluate method. This method computes the loss and accuracy metrics for the validation data.","metadata":{}},{"cell_type":"code","source":"loss_, acc_ = model.evaluate(valid_it)\n# Print loss and accuracy\n\n# Define ANSI escape sequences for font styles\nbold = \"\\033[1m\"\nend = \"\\033[0m\"\n\n# Print loss and accuracy with larger font\nprint(bold + \"Loss: {:.4f}\".format(loss_) + end)\nprint(bold + \"Accuracy: {:.2f}%\".format(acc_ * 100) + end)","metadata":{"execution":{"iopub.status.busy":"2024-03-12T02:53:51.024310Z","iopub.status.idle":"2024-03-12T02:53:51.024631Z","shell.execute_reply.started":"2024-03-12T02:53:51.024474Z","shell.execute_reply":"2024-03-12T02:53:51.024488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Plot Confusion Matrix\nTo visualize the performance of the model in classifying different classes, we plot a confusion matrix. We define a function plot_confusion_matrix that prints and plots the confusion matrix.  \nThis function takes the confusion matrix (cm), the class names (classes), and other optional parameters to customize the plot. It visualizes the confusion matrix with options to normalize the values or not.","metadata":{}},{"cell_type":"code","source":"#Plot the confusion matrix. Set Normalize = True/False\ndef plot_confusion_matrix(cm, classes, normalize=True, title='Confusion matrix', cmap=plt.cm.Blues):\n    \"\"\"\n    This function prints and plots the confusion matrix.\n    Normalization can be applied by setting `normalize=True`.\n    \"\"\"\n    plt.figure(figsize=(10,10))\n    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n    plt.title(title)\n    plt.colorbar()\n    tick_marks = np.arange(len(classes))\n    plt.xticks(tick_marks, classes, rotation=45)\n    plt.yticks(tick_marks, classes)\n    if normalize:\n            cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n            cm = np.around(cm, decimals=2)\n            cm[np.isnan(cm)] = 0.0\n            print(\"Normalized confusion matrix\")\n    else:\n        print('Confusion matrix, without normalization')\n    thresh = cm.max() / 2.\n    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n        plt.text(j, i, cm[i, j],\n                 horizontalalignment=\"center\",\n                 color=\"white\" if cm[i, j] > thresh else \"black\")\n    plt.tight_layout()\n    plt.ylabel('True label')\n    plt.xlabel('Predicted label')","metadata":{"execution":{"iopub.status.busy":"2024-03-12T03:55:20.304837Z","iopub.execute_input":"2024-03-12T03:55:20.305784Z","iopub.status.idle":"2024-03-12T03:55:20.315356Z","shell.execute_reply.started":"2024-03-12T03:55:20.305748Z","shell.execute_reply":"2024-03-12T03:55:20.314337Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Generate Confusion Matrix and Classification Report\n\nTo further analyze the model's performance, we generate a confusion matrix and a classification report. We use the classification_report and confusion_matrix functions from scikit-learn library.","metadata":{}},{"cell_type":"code","source":"#Print the Target names\nfrom sklearn.metrics import classification_report, confusion_matrix\nimport itertools \n\n#shuffle=False\ntarget_names = []\nfor key in valid_it.class_indices:\n    target_names.append(key)\n\n#Confution Matrix\nY_pred = model.predict(valid_it)\ny_pred = np.argmax(Y_pred, axis=1)","metadata":{"execution":{"iopub.status.busy":"2024-03-12T03:55:24.245068Z","iopub.execute_input":"2024-03-12T03:55:24.245433Z","iopub.status.idle":"2024-03-12T03:55:44.289939Z","shell.execute_reply.started":"2024-03-12T03:55:24.245403Z","shell.execute_reply":"2024-03-12T03:55:44.288966Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Confusion Matrix')\ncm = confusion_matrix(valid_it.classes, y_pred)\nplot_confusion_matrix(cm, target_names, title='Confusion Matrix')","metadata":{"execution":{"iopub.status.busy":"2024-03-12T03:55:44.291726Z","iopub.execute_input":"2024-03-12T03:55:44.292384Z","iopub.status.idle":"2024-03-12T03:55:44.780466Z","shell.execute_reply.started":"2024-03-12T03:55:44.292351Z","shell.execute_reply":"2024-03-12T03:55:44.779552Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Print Classification Report\nprint('Classification Report')\nprint(classification_report(valid_it.classes, y_pred, target_names=target_names))","metadata":{"execution":{"iopub.status.busy":"2024-03-11T20:37:45.284202Z","iopub.execute_input":"2024-03-11T20:37:45.284504Z","iopub.status.idle":"2024-03-11T20:37:45.303470Z","shell.execute_reply.started":"2024-03-11T20:37:45.284478Z","shell.execute_reply":"2024-03-11T20:37:45.302603Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"predicting-the-results\"></a>\n\n# Predicting the Results\nThe trained model is used to predict the classes of fruit images in a validation set. The results are saved to a CSV file.","metadata":{}},{"cell_type":"code","source":"files = [f for f in listdir('/kaggle/input/csc532-fruits-categorical-classification-2024/csc532_2024_validation/validation/') if isfile(join('/kaggle/input/csc532-fruits-categorical-classification-2024/csc532_2024_validation/validation', f))]","metadata":{"execution":{"iopub.status.busy":"2024-03-12T03:55:49.560590Z","iopub.execute_input":"2024-03-12T03:55:49.561616Z","iopub.status.idle":"2024-03-12T03:55:49.855970Z","shell.execute_reply.started":"2024-03-12T03:55:49.561580Z","shell.execute_reply":"2024-03-12T03:55:49.854868Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"result_dict = {\n    'ID': [],\n    'Code': [],\n}\n\nfor file in files:\n#   image = load_img(f'../input/csc532-fruits-categorical-classification-2024/csc532_2024_validation/validation/{file}', target_size=(224, 224))\n    img_path = f'/kaggle/input/csc532-fruits-categorical-classification-2024/csc532_2024_validation/validation/{file}'\n    \n    image = isolate_object_on_white_bg(img_path, seg_model)\n#     image = cv2.imread(img_path)\n    image = cv2.resize(image, (IMG_SIZE, IMG_SIZE))\n    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n    image = img_to_array(image)\n    image = image.reshape(1, 224, 224, 3)\n    image = image / 255.\n\n    pred = np.argmax(model.predict(image)) + 1\n    result_dict['ID'].append(file)\n    result_dict['Code'].append(pred)","metadata":{"execution":{"iopub.status.busy":"2024-03-12T03:55:50.808865Z","iopub.execute_input":"2024-03-12T03:55:50.809275Z","iopub.status.idle":"2024-03-12T03:57:06.345247Z","shell.execute_reply.started":"2024-03-12T03:55:50.809230Z","shell.execute_reply":"2024-03-12T03:57:06.344386Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.DataFrame.from_dict(result_dict)\ndf.to_csv('result_inception_tuned_yoloseg.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2024-03-12T03:58:54.614114Z","iopub.execute_input":"2024-03-12T03:58:54.614512Z","iopub.status.idle":"2024-03-12T03:58:54.623005Z","shell.execute_reply.started":"2024-03-12T03:58:54.614481Z","shell.execute_reply":"2024-03-12T03:58:54.622060Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Reference:\nThis notebook was inspired by the Kaggle notebook [Fruits Categorical ResNet101V2](https://www.kaggle.com/code/stormcloser/fruits-categorical-resnet101v2) by stormcloser.\n","metadata":{}}]}