{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Beginner's Guide to Image Augmentation & Transforms\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Hello! If you are a beginner and learning to handle image data, then Image Augmentation & Transforms is often confusing. Some may wonder why it is needed and what is the use.\n\nWhat are the changes that occur and why is it necessary. We discuss these in brief in the notebook\n\nI will be using the Dog Breed database, as we are intuitively more familiar with dogs than human protiens :)\n\nWe use both Python as well as PyTorch in this notebook!\n\nHope this helps...","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Import Libraries","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"In the start always import the libraries that you feel you may use or need. Over time, build a list of libraries that you use and use it in all the notebooks you are working. You will naturally strat using some of the libraries that you are comfortable with and this will help.","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import os\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport seaborn as sns\n\nimport cv2\nimport random\nfrom random import randint\nimport time\n\n\nimport torch\nfrom torch.utils.data import Dataset, random_split, DataLoader\nimport torch.nn.functional as F\nimport torch.nn as nn\n\nfrom PIL import Image\nfrom scipy import ndimage\n\nimport torchvision\nimport torchvision.models as models\nimport torchvision.transforms as T\nfrom torchvision.utils import make_grid\nfrom torchvision.datasets.utils import download_url\nfrom torchvision.datasets import ImageFolder\n\nfrom tqdm.notebook import tqdm\n\nfrom sklearn.metrics import f1_score\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Read the data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"DATA_DIR = '../input/dog-breed-identification'\n\n\nTRAIN_DIR = DATA_DIR + '/train'                           \nTEST_DIR = DATA_DIR + '/test'                             \n\nTRAIN_CSV = DATA_DIR + '/labels.csv'                     \nTEST_CSV = DATA_DIR + '/submission.csv' ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Add more Data Fields","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"I will not dwelve into modeling here and focus on image transfors\n\nAs a result I will look at only the train file\n\nI will read and add some data fields that I find useful","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data_df = pd.read_csv(TRAIN_CSV)\ndata_df.head(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"What do you observe? a list of image names and breed. Lets add more details to the dataframe\n\nWe create a dictionary of all the breeds","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"labels_names=data_df[\"breed\"].unique()\nlabels_sorted=labels_names.sort()\n\nlabels = dict(zip(range(len(labels_names)),labels_names))\nlabels ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I like to use numbers instead of names for labels. Lets add the numbers as labels to the dataframe","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\nlbl=[]\nfor i in range(len(data_df[\"breed\"])):\n    temp=list(labels.values()).index(data_df.breed[i])\n    lbl.append(temp)\n\n    \ndata_df['lbl'] = lbl\n#data_df['lbl'] = data_df['lbl'].astype(str)\ndata_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Lets also add the path of each image to the file. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"path_img=[]\nfor i in range(len(data_df[\"id\"])):\n    temp=TRAIN_DIR + \"/\" + str(data_df.id[i]) + \".jpg\"\n    path_img.append(temp)\n\ndata_df['path_img'] =path_img\ndata_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Any other field you would like to add? Please make a note in comments. Thanks!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Exlporatory Data Analysis (EDA)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Let us look at the data and make some initial conclusions","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"num_images = len(data_df[\"id\"])\nprint('Number of images in Training file:', num_images)\nno_labels=len(labels_names)\nprint('Number of dog breeds in Training file:', no_labels)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Ok! we have over 10,000 images for 120 dog breeds.\n\nAre images equally distributed between all dog breeds?\n\nLet's plot a graph and see!","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"bar = data_df[\"breed\"].value_counts(ascending=True).plot.barh(figsize = (30,120))\nplt.title(\"Distribution of the Dog Breeds\", fontsize = 20)\nbar.tick_params(labelsize=16)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data_df[\"breed\"].value_counts(ascending=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We observe that the distribution is not equal. Scottish deerhound has 126 images\nwhile eskimo dog and briard breeds have 66 images","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Image Analysis","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Let us display 20 picture of the dataset with their labels","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(nrows=4, ncols=5, figsize=(15, 15),\n                        subplot_kw={'xticks': [], 'yticks': []})\n\nfor i, ax in enumerate(axes.flat):\n    ax.imshow(plt.imread(data_df.path_img[i]))\n    ax.set_title(data_df.breed[i])\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"What do you observe?\n\nAll images are of differnt sizes\n\nThe backgrounsd vary- some have humans, and other items in the backgrounds\n\nAlso some images are not vertical - e.g., the lakeland terrier in the lower night","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Image Transforms using Python ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Let us work on some image transforms using Python\nand later we will use Pytorch to do the same\n\nLets start with resizing images","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"random_img=randint(0,len(data_df.path_img))\nimg_path=data_df.path_img[random_img]\nimg= plt.imread(img_path)\n\nplt.imshow(img)\nplt.title(\"Original image\")\nplt.show()\n\nplt.imshow(cv2.resize(img, (150,150)))\nplt.title(\"After resizing\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Lets try to rotate the images...","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"random_img=randint(0,len(data_df.path_img))\nimg_path=data_df.path_img[random_img]\nimg= plt.imread(img_path)\n\nplt.imshow(img)\nplt.title(\"Original image\")\nplt.show()\n\n\n#rotation angle in degree\n\nrotated1 = ndimage.rotate(img, 90)\nplt.imshow(rotated1)\nplt.title(\"Image rotated 90 degrees\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As you can observe the originalt height of the image is retained while the width changes\nthis creates an issue of differnt sizes and lenghts of images\n\nLet us do both resize and rotation\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"random_img=randint(0,len(data_df.path_img))\nimg_path=data_df.path_img[random_img]\nimg= plt.imread(img_path)\n\nplt.imshow(img)\nplt.title(\"Original image\")\nplt.show()\n\n\nimg=cv2.resize(img, (150,150))\nturn =90\n\nfig, axes = plt.subplots(nrows=1, ncols=4, figsize=(16, 4),\n                        subplot_kw={'xticks': [], 'yticks': []})\n\nfor i, ax in enumerate(axes.flat):\n    ax.imshow(ndimage.rotate(img, i*90))\n    ax.set_title(\"After resizing rotated \"+ str(i*90) +\" degrees\")\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"What are the benefits?\nDeep Learning and Neural Networks need a lot of images\nby rotating images we are adding multiple extra images from one image\n\nNote that I have rotated by 90 degrees  but one may rotate by any random degree that that wants to rotate the image\n\nSimilarly, by blocking parts of images, cropping (removing part of images) and adding jitters, we can both augument images (add to the number of images) and transform them to make it more helpful for the neural network to classify","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Image transforms using PyTorch","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Now that we have a understanding of the transforms using Python, lets do the same using PyTorch","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#imagenet_stats = ([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])\n\ntrain_tfms = T.Compose([\n#this will resize the image \n    T.Resize(256),   \n   \n#Randomly change the brightness, contrast and saturation of an image\n#    T.ColorJitter(brightness=0.1, contrast=0.1, saturation=0.1, hue=0.1),    \n\n#this will remove parts (crop) the Image at a random location.   \n#    T.RandomCrop(32, padding=4, padding_mode='reflect'),   \n\n#Horizontally flip (rotate by 180 degree) the given image randomly; default is 50% of images\n    T.RandomHorizontalFlip(), \n    \n#Rotate the image by angle -here by 10%\n    T.RandomRotation(10),\n    \n#convert it to a tensor   \n    T.ToTensor()\n\n#Normalize a tensor image with mean and standard deviation - here with the Imagenet stats\n#    T.Normalize(*imagenet_stats,inplace=True), \n    \n#Randomly selects a rectangle region in an image and erases its pixels.    \n#    T.RandomErasing(inplace=True)\n])\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here I focus only on the train transform. Please make sure you make the same transforms in the validation set as well\n\nNote some of the commands are not running as they are as coments due to the # symbol\nremove the # symbol and see how the images below change. \n\nThis will help you visualise the impact of each transform\n\n[Read more about transforms here (click here)](https://pytorch.org/docs/stable/torchvision/transforms.html)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class DogDataset(Dataset):\n    def __init__(self, df, root_dir, transform=None):\n        self.df = df\n        self.transform = transform\n        self.root_dir = root_dir\n        \n    def __len__(self):\n        return len(self.df)    \n    \n    def __getitem__(self, idx):\n        row = self.df.loc[idx]\n        img_id, img_label = row['id'], row['lbl']\n        img_fname = self.root_dir + \"/\" + str(img_id) + \".jpg\"\n        img = Image.open(img_fname)\n        if self.transform:\n            img = self.transform(img)\n        return img, img_label","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data_ds = DogDataset(data_df, TRAIN_DIR, transform=train_tfms)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def show_sample(img, target, invert=True):\n    if invert:\n        plt.imshow(1 - img.permute((1, 2, 0)))\n    else:\n        plt.imshow(img.permute(1, 2, 0))\n    print('Labels:', labels[target])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# View Sample Images after Transform","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"show_sample(*data_ds[241])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"show_sample(*data_ds[149])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Do you notice that the transforms are most likely differnt for both images?\nThis is because the transforms are added randomly. In most liklihood each image will have some differnt rotation and.or flip and other transforms","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"***Try this out!***\n\nand try with using differnt transforms from the link and by removing the # in the code\n\n**Please share your comments and feedback.**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Thank you! Hope you like it!","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}