{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":67836,"databundleVersionId":7527339,"sourceType":"competition"}],"dockerImageVersionId":30648,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Training Notebook for DL Engima 1.0 by **DL_noobies**","metadata":{}},{"cell_type":"markdown","source":"# Competition Link [here](https://www.kaggle.com/competitions/dl-enigma-10-sust-cse-carnival-2024/overview)\n\n#### This notebook was inspired from [here](https://www.kaggle.com/code/ataullhasaim/mws-dl-enigma-1-0-starter-notebook)","metadata":{}},{"cell_type":"markdown","source":"# **Instructions for Running the Code**\n\nTo run the provided code, follow these steps:\n\n1. **Running Cells**: Click on the \"Run\" button located to the left of each cell in the Jupyter notebook. Alternatively, you can press `Ctrl+Enter` to run a single cell or `Shift+Enter` to run a cell and move to the next one.\n\n2. **Checking GPU Accelerator**: Ensure that your GPU Accelerator is enabled. It's recommended to use a GPU like the P100, but any GPU should work.\n\n3. **No External Libraries Needed**: You don't need to install any external libraries manually. The code utilizes libraries and functionalities already available in the Kaggle environment.\n\n4. **Step-by-Step Execution**: Run each cell one by one in sequential order. The code is organized in a way that each cell builds upon the previous one, so executing them sequentially ensures proper functionality.\n\n5. **Let Kaggle Do the Work**: Kaggle will handle the execution of the code and utilize the available GPU resources for accelerated processing.\n","metadata":{}},{"cell_type":"markdown","source":"# **Libraries Importation and Installation**","metadata":{}},{"cell_type":"code","source":"# basic libraries ...\n\nimport os\nimport numpy as np\nimport pandas as pd\nfrom tqdm import tqdm","metadata":{"execution":{"iopub.status.busy":"2024-02-10T10:17:32.921153Z","iopub.execute_input":"2024-02-10T10:17:32.922075Z","iopub.status.idle":"2024-02-10T10:17:33.267424Z","shell.execute_reply.started":"2024-02-10T10:17:32.922039Z","shell.execute_reply":"2024-02-10T10:17:33.266694Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The command installs or upgrades the ipywidgets library, which provides interactive HTML widgets for notebooks.\n\n!pip install -U ipywidgets","metadata":{"execution":{"iopub.status.busy":"2024-02-10T10:17:33.274372Z","iopub.execute_input":"2024-02-10T10:17:33.274670Z","iopub.status.idle":"2024-02-10T10:17:46.951675Z","shell.execute_reply.started":"2024-02-10T10:17:33.274648Z","shell.execute_reply":"2024-02-10T10:17:46.950761Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Installs Ultralytics for computer vision tasks.\n\n!pip install ultralytics","metadata":{"execution":{"iopub.status.busy":"2024-02-10T10:17:46.953986Z","iopub.execute_input":"2024-02-10T10:17:46.954300Z","iopub.status.idle":"2024-02-10T10:17:59.681806Z","shell.execute_reply.started":"2024-02-10T10:17:46.954270Z","shell.execute_reply":"2024-02-10T10:17:59.680846Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Importing YOLO model from Ultralytics library.\n\nfrom ultralytics import YOLO","metadata":{"execution":{"iopub.status.busy":"2024-02-10T10:17:59.683143Z","iopub.execute_input":"2024-02-10T10:17:59.683425Z","iopub.status.idle":"2024-02-10T10:18:05.446108Z","shell.execute_reply.started":"2024-02-10T10:17:59.683400Z","shell.execute_reply":"2024-02-10T10:18:05.445171Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Creating Working Directories for Dataset Organization**","metadata":{}},{"cell_type":"code","source":"# Deletes all files and directories in the current working directory recursively.\n\n!rm -rf *\n\n# Creates necessary directories for dataset organization.\n\n!mkdir /kaggle/working/datasets\n!mkdir /kaggle/working/datasets/badodd\n!mkdir /kaggle/working/datasets/badodd/labels\n!mkdir /kaggle/working/datasets/badodd/labels/train\n!mkdir /kaggle/working/datasets/badodd/images\n!mkdir /kaggle/working/datasets/badodd/images/train\n!mkdir /kaggle/working/datasets/badodd/images/test","metadata":{"execution":{"iopub.status.busy":"2024-02-10T10:18:05.447323Z","iopub.execute_input":"2024-02-10T10:18:05.447694Z","iopub.status.idle":"2024-02-10T10:18:12.993598Z","shell.execute_reply.started":"2024-02-10T10:18:05.447670Z","shell.execute_reply":"2024-02-10T10:18:12.992347Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Creates symbolic links to reference original files in another directory without duplicating their content.\n\ndef all_files_in_folder_symlink(source_dir, target_dir):\n    files = os.listdir(source_dir)\n    \n    for file in tqdm(files):\n        source_file = os.path.join(source_dir, file)\n        target_file = os.path.join(target_dir, file)\n        os.symlink(source_file, target_file)","metadata":{"execution":{"iopub.status.busy":"2024-02-10T10:18:12.995215Z","iopub.execute_input":"2024-02-10T10:18:12.995583Z","iopub.status.idle":"2024-02-10T10:18:13.001506Z","shell.execute_reply.started":"2024-02-10T10:18:12.995553Z","shell.execute_reply":"2024-02-10T10:18:13.000530Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Creating symbolic links for files in specified directories.\n\nall_files_in_folder_symlink(\"/kaggle/input/dl-enigma-10-sust-cse-carnival-2024/dlenigma1/BadODD/labels/train\",\"/kaggle/working/datasets/badodd/labels/train\")\nall_files_in_folder_symlink(\"/kaggle/input/dl-enigma-10-sust-cse-carnival-2024/dlenigma1/BadODD/images/train\",\"/kaggle/working/datasets/badodd/images/train\")\nall_files_in_folder_symlink(\"/kaggle/input/dl-enigma-10-sust-cse-carnival-2024/dlenigma1/BadODD/images/test\",\"/kaggle/working/datasets/badodd/images/test\")","metadata":{"execution":{"iopub.status.busy":"2024-02-10T10:18:13.002739Z","iopub.execute_input":"2024-02-10T10:18:13.002993Z","iopub.status.idle":"2024-02-10T10:18:14.073423Z","shell.execute_reply.started":"2024-02-10T10:18:13.002972Z","shell.execute_reply":"2024-02-10T10:18:14.072482Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Displaying the working directory structure ...\n\n!tree -d","metadata":{"execution":{"iopub.status.busy":"2024-02-10T10:18:14.074577Z","iopub.execute_input":"2024-02-10T10:18:14.074861Z","iopub.status.idle":"2024-02-10T10:18:14.078466Z","shell.execute_reply.started":"2024-02-10T10:18:14.074837Z","shell.execute_reply":"2024-02-10T10:18:14.077659Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training Yolov8 Model on Customized Data","metadata":{"execution":{"iopub.status.busy":"2024-02-10T10:18:14.081981Z","iopub.execute_input":"2024-02-10T10:18:14.082408Z","iopub.status.idle":"2024-02-10T10:18:14.088370Z","shell.execute_reply.started":"2024-02-10T10:18:14.082383Z","shell.execute_reply":"2024-02-10T10:18:14.087458Z"}}},{"cell_type":"code","source":"# Creating a configuration file for YOLO specifying dataset location and object classes.\n\nfile_content = \"\"\"\npath: /kaggle/working/datasets/badodd  # dataset root dir\ntrain: images/train  # train images (relative to 'path')\nval: images/train  # val images (relative to 'path')\ntest:  images/test\n\n# Classes\nnames:\n    0: auto_rickshaw\n    1: bicycle\n    2: bus\n    3: car\n    4: cart_vehicle\n    5: construction_vehicle\n    6: motorbike\n    7: person\n    8: priority_vehicle\n    9: three_wheeler\n    10: train\n    11: truck\n    12: wheelchair\n\"\"\"  \n\nwith open(\"yolov8.yaml\", mode=\"w\") as f:\n    f.write(file_content)","metadata":{"execution":{"iopub.status.busy":"2024-02-10T10:18:14.089562Z","iopub.execute_input":"2024-02-10T10:18:14.090136Z","iopub.status.idle":"2024-02-10T10:18:14.098344Z","shell.execute_reply.started":"2024-02-10T10:18:14.090090Z","shell.execute_reply":"2024-02-10T10:18:14.097385Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Initialize Weights & Biases (W&B) in disabled mode.\n\nimport wandb\nwandb.init(mode=\"disabled\")","metadata":{"execution":{"iopub.status.busy":"2024-02-10T10:18:14.129189Z","iopub.execute_input":"2024-02-10T10:18:14.129470Z","iopub.status.idle":"2024-02-10T10:18:15.857200Z","shell.execute_reply.started":"2024-02-10T10:18:14.129441Z","shell.execute_reply":"2024-02-10T10:18:15.855269Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Initializing YOLOv8 model with the provided weights.\n\nmodel = YOLO('yolov8l.pt')\n\n# Training the model for 40 epochs with specified parameters using accelerator GPU P100\nmodel.train(\n    \n    # Data & model parameters\n    data=\"/kaggle/working/yolov8.yaml\", \n    imgsz = 800,\n    \n    # Training parameters\n    epochs = 40,\n    batch=16,\n    val=True,\n    \n    # Augmentation\n    hsv_h = 0.03,\n    hsv_s=0.8,\n    hsv_v=0.8,\n    degrees = 0.6,\n    copy_paste=0.8,\n\n    # Optimization parameters\n    lr0=0.01,\n    lrf=0.01,\n    patience=10,\n    optimizer=\"SGD\",\n    momentum=0.85,\n    weight_decay=0.00055,\n)","metadata":{"execution":{"iopub.status.busy":"2024-02-10T10:18:15.859291Z","iopub.execute_input":"2024-02-10T10:18:15.860289Z","iopub.status.idle":"2024-02-10T17:36:40.398417Z","shell.execute_reply.started":"2024-02-10T10:18:15.860250Z","shell.execute_reply":"2024-02-10T17:36:40.397282Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Training Complete**\n## If you'd like to access the Inference Notebook, please click [HERE](https://www.kaggle.com/code/mobin555/dl-noobies-inference-notebook).\n\n## Alternatively, if you wish to continue with any other task, you're most welcome to do so.\n\n\n\n","metadata":{}},{"cell_type":"markdown","source":"# **Inference part** \n","metadata":{"execution":{"iopub.status.busy":"2024-02-10T17:36:40.399870Z","iopub.execute_input":"2024-02-10T17:36:40.400197Z","iopub.status.idle":"2024-02-10T17:36:40.404387Z","shell.execute_reply.started":"2024-02-10T17:36:40.400171Z","shell.execute_reply":"2024-02-10T17:36:40.403428Z"}}},{"cell_type":"code","source":"# Function to convert prediction boxes to the submission format string.\n\ndef get_prediction_string(boxes, scores, classes):\n    pred_strs = []\n    for i, score in enumerate(scores):\n        single_pred_str = \"\"\n        single_pred_str += str(float(classes[i])) + \" \" + str(float(score)) + \" \"\n        \n        x_center , y_center, width,height = boxes[i]\n        x1 = float(x_center) - (float(width) / 2)\n        y1 = float(y_center) - (float(height) / 2)\n        width = float(width)\n        height = float(height)\n        single_pred_str +=  f\"{x1} {y1} {width} {height}\"\n        \n        pred_strs.append(single_pred_str)\n    ans = ','.join(map(str, pred_strs))\n    if len(ans):\n        return ans\n#     the solution metrics failed in case of a NaN, '' (empty). So, return \"0 0 0 0 0 0\" for NaN, '' (empty) string\n    return \"0 0 0 0 0 0\"\n\n# Function to generate prediction entry in the required format.\n\ndef get_prediction_entry(i, filename, boxes, scores, classes):\n    return {\n        \"id\": i, # strating from 0 ...\n        \"ImageID\": filename.split('.')[0], # before the extension ...\n        \"PredictionString_pred\": get_prediction_string(boxes, scores, classes)\n    }\n\n# Directory path for test images.\ntest_directory = \"/kaggle/input/dl-enigma-10-sust-cse-carnival-2024/dlenigma1/BadODD/images/test\"\n\n# Loading the Trained best YOLO model.\nmodel = YOLO('/kaggle/working/runs/detect/train/weights/best.pt')\n\n\n# Function to perform inference on all test images and generate predictions.\n\ndef predict_all_files(test_directory):\n    predictions = []\n    for i,filename in tqdm(enumerate(os.listdir(test_directory))):\n        if filename.endswith(\".jpg\"):\n            filepath = os.path.join(test_directory, filename)\n            \n            results = model.predict(\n                source=filepath,\n                conf=0.40, # changed the confidence from default 0.25 to 0.4 for better prediction\n                verbose=False,\n            )\n            \n            boxes = results[0].boxes.xywhn\n            scores = results[0].boxes.conf\n            classes = results[0].boxes.cls\n            prediction = get_prediction_entry(i, filename, boxes, scores, classes)\n            predictions.append(prediction)\n#             to csv format ...\n    predictions_df = pd.DataFrame(predictions)\n    predictions_df.to_csv(\"submission.csv\", index=False)\n\n# Perform inference.\npredict_all_files(test_directory)","metadata":{"execution":{"iopub.status.busy":"2024-02-10T17:36:40.405857Z","iopub.execute_input":"2024-02-10T17:36:40.406125Z","iopub.status.idle":"2024-02-10T17:39:07.181438Z","shell.execute_reply.started":"2024-02-10T17:36:40.406101Z","shell.execute_reply":"2024-02-10T17:39:07.180414Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# loading the submission dataframe ....\n\nsubmission_df = pd.read_csv('/kaggle/working/submission.csv')","metadata":{"execution":{"iopub.status.busy":"2024-02-10T17:39:07.182947Z","iopub.execute_input":"2024-02-10T17:39:07.183413Z","iopub.status.idle":"2024-02-10T17:39:07.206778Z","shell.execute_reply.started":"2024-02-10T17:39:07.183376Z","shell.execute_reply":"2024-02-10T17:39:07.205895Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# dataframe shape ...\n\nsubmission_df.shape","metadata":{"execution":{"iopub.status.busy":"2024-02-10T17:39:07.207923Z","iopub.execute_input":"2024-02-10T17:39:07.208675Z","iopub.status.idle":"2024-02-10T17:39:07.214539Z","shell.execute_reply.started":"2024-02-10T17:39:07.208639Z","shell.execute_reply":"2024-02-10T17:39:07.213656Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# last few row of the dataframe\n\nsubmission_df.tail()","metadata":{"execution":{"iopub.status.busy":"2024-02-10T17:39:07.215722Z","iopub.execute_input":"2024-02-10T17:39:07.215962Z","iopub.status.idle":"2024-02-10T17:39:07.230946Z","shell.execute_reply.started":"2024-02-10T17:39:07.215941Z","shell.execute_reply":"2024-02-10T17:39:07.230117Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Checking for any null values \n\nsubmission_df.isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2024-02-10T17:39:07.232198Z","iopub.execute_input":"2024-02-10T17:39:07.232810Z","iopub.status.idle":"2024-02-10T17:39:07.240219Z","shell.execute_reply.started":"2024-02-10T17:39:07.232775Z","shell.execute_reply":"2024-02-10T17:39:07.239285Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## You can submit the following CSV file or the previously created \"submission.csv\" file to check the result.\n","metadata":{}},{"cell_type":"code","source":"# acceptable submission format ...\n\nsubmission_df.to_csv('Submission_final.csv',index=False)","metadata":{"execution":{"iopub.status.busy":"2024-02-10T17:39:07.241329Z","iopub.execute_input":"2024-02-10T17:39:07.241607Z","iopub.status.idle":"2024-02-10T17:39:07.311838Z","shell.execute_reply.started":"2024-02-10T17:39:07.241585Z","shell.execute_reply":"2024-02-10T17:39:07.311089Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Happy Ending","metadata":{}}]}