{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Let's start.\n\n## Loading Data","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0"}},{"cell_type":"markdown","source":"To show the result of CutMix augmentation we will use a set of images containing images of wheat field showing wheat heads. These images are takes from Kaggle's Glbal Wheat Detection competition dataset.","metadata":{}},{"cell_type":"code","source":"# Import glob to parse filenames\nimport glob\n\n# Import numpy\nimport numpy as np\n\n# Plot\nimport matplotlib.pyplot as plt\nplt.rcParams['figure.figsize'] = [10,10]\n\n# Opencv\nimport cv2\n\n# Path to data\ndata_folder = f\"/kaggle/input/global-wheat-detection/train/\"\n\n# Read filenames in the data folder\nfilenames = glob.glob(f\"{data_folder}*.jpg\")\n\n# Read first 10 filenames\nimage_paths = filenames[:10]\n\n# Display a sample image\nplt.imshow(cv2.cvtColor(cv2.imread(image_paths[0]), cv2.COLOR_BGR2RGB)); plt.show();\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Generating a Batch of Data","metadata":{}},{"cell_type":"markdown","source":"Now, we will create a set or \"batch\" of 4 images. Let's assume that this batch is a minibacth generated by a dataloader that loads minibatches during the training of the deep neural network model. In general, the bacth loaded by a dataloader also includes a batch of image labels if we are solving a classification problem. We will randomly assign a label to each image from from the set of {0,1,2}. We will further one-hot encode(https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html) these labels to {[1,0,0], [0,1,0], [0,0,1]}. Please follow this((https://stanford.edu/~shervine/blog/pytorch-how-to-generate-data-parallel)) to learn more about dataloaders in PyTorch.","metadata":{}},{"cell_type":"code","source":"# Create batch of 4 images\nimage_batch = []\nimage_batch_labels = []\n\nn_images = 4\n\nfor i in range(n_images):\n    image = cv2.cvtColor(cv2.imread(image_paths[i]), cv2.COLOR_BGR2RGB)\n    image_batch.append(image)\n    \n    label_temp = list(np.floor(np.random.rand(1)*2.99).astype(int))[0]\n    if label_temp == 0:\n        label = [1,0,0]\n    elif label_temp == 1:\n        label = [0,1,0]\n    else: # label_temp == 2\n        label = [0,0,1]\n    \n    image_batch_labels.append(label)\n\n# Convert image_batch to numpy array\nimage_batch = np.array(image_batch)\n# Conver image_batch_labels to numpy array\nimage_batch_labels = np.array(image_batch_labels)\n    \n# Print labels\nprint()\nprint(f\"Image labels: {image_batch_labels}\\n\")\n\n# Show images\nfor i in range(2):\n    for j in range(2):\n        plt.subplot(2,2,2*i+j+1)\n        plt.imshow(image_batch[2*i+j])\nplt.show()\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Generating Random Bounding Box\n\nHere is a function to generate a random bounding box in an image:","metadata":{}},{"cell_type":"code","source":"def rand_bbox(size, lamb):\n    \"\"\" Generate random bounding box \n    Args:\n        - size: [width, breadth] of the bounding box\n        - lamb: (lambda) cut ratio parameter\n    Returns:\n        - Bounding box\n    \"\"\"\n    W = size[0]\n    H = size[1]\n    cut_rat = np.sqrt(1. - lamb)\n    cut_w = np.int(W * cut_rat)\n    cut_h = np.int(H * cut_rat)\n\n    # uniform\n    cx = np.random.randint(W)\n    cy = np.random.randint(H)\n\n    bbx1 = np.clip(cx - cut_w // 2, 0, W)\n    bby1 = np.clip(cy - cut_h // 2, 0, H)\n    bbx2 = np.clip(cx + cut_w // 2, 0, W)\n    bby2 = np.clip(cy + cut_h // 2, 0, H)\n\n    return bbx1, bby1, bbx2, bby2","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, let's test the function by generating a random bounding box and crop an image using that bounding box.","metadata":{}},{"cell_type":"code","source":"# Read an image\nimage = cv2.cvtColor(cv2.imread(image_paths[0]), cv2.COLOR_BGR2RGB)\n\n# Crop a random bounding box\nlamb = 0.3\nsize = image.shape\nbbox = rand_bbox(size, lamb)\n\n# Draw bounding box on the image\nim = image.copy()\nx1 = bbox[0]\ny1 = bbox[1]\nx2 = bbox[2]\ny2 = bbox[3]\ncv2.rectangle(im, (x1, y1), (x2, y2), (255, 0, 0), 3)\nplt.imshow(im);\nplt.title('Original image with random bounding box')\nplt.show();\n\n# Show cropped image\nplt.imshow(image[y1:y2, x1:x2]);\nplt.title('Cropped image')\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## A CutMix Image Generation","metadata":{}},{"cell_type":"markdown","source":"In CutMix algorithm, for each image of batch a random region (in our case, a random bounding box region) is replaced with a patch from another training image. \n- The parameter ```lambda``` that determines the size of the bounding box is stochastically sampled from a Beta distribution(https://en.wikipedia.org/wiki/Beta_distribution).\n- The label of each resultant augmented image is estimated as a weighted sum of the original label and the label of the image from which the modified patch is borrowed, where weights are ```lambda``` and ```(1-lambda)``` respectively.","metadata":{}},{"cell_type":"code","source":"def generate_cutmix_image(image_batch, image_batch_labels, beta):\n    \"\"\" Generate a CutMix augmented image from a batch \n    Args:\n        - image_batch: a batch of input images\n        - image_batch_labels: labels corresponding to the image batch\n        - beta: a parameter of Beta distribution.\n    Returns:\n        - CutMix image batch, updated labels\n    \"\"\"\n    # generate mixed sample\n    lam = np.random.beta(beta, beta)\n    rand_index = np.random.permutation(len(image_batch))\n    target_a = image_batch_labels\n    target_b = image_batch_labels[rand_index]\n    bbx1, bby1, bbx2, bby2 = rand_bbox(image_batch[0].shape, lam)\n    image_batch_updated = image_batch.copy()\n    image_batch_updated[:, bbx1:bbx2, bby1:bby2, :] = image_batch[rand_index, bbx1:bbx2, bby1:bby2, :]\n    \n    # adjust lambda to exactly match pixel ratio\n    lam = 1 - ((bbx2 - bbx1) * (bby2 - bby1) / (image_batch.shape[1] * image_batch.shape[2]))\n    label = target_a * lam + target_b * (1. - lam)\n    \n    return image_batch_updated, label\n\n\n# Generate CutMix image\n# Let's use the first image of the batch as the input image to be augmented\ninput_image = image_batch[0]\nimage_batch_updated, image_batch_labels_updated = generate_cutmix_image(image_batch, image_batch_labels, 1.0)\n\n# Show original images\nprint(\"Original Images\")\nfor i in range(2):\n    for j in range(2):\n        plt.subplot(2,2,2*i+j+1)\n        plt.imshow(image_batch[2*i+j])\nplt.show()\n\n\n# Show CutMix images\nprint(\"CutMix Images\")\nfor i in range(2):\n    for j in range(2):\n        plt.subplot(2,2,2*i+j+1)\n        plt.imshow(image_batch_updated[2*i+j])\nplt.show()\n\n# Print labels\nprint('Original labels:')\nprint(image_batch_labels)\nprint('Updated labels')\nprint(image_batch_labels_updated)","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}