{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Image Augmentation examples \n\n---------------\n```\n이번 차종 이미지 분류 대회는 train set 이미지가 10,000개, 차종 class가 196개로 class당 대략 51개 정도의 이미지를 가지고 있습니다. 매우 작은 양의 이미지로 학습되기 때문에 과대적합을 피하고 일반화된 성능을 끌어내기 위해서는 Image Augmentation(이미지 증식)은 필수조건입니다.\nkeras에는 Image Augmentation을 위한 ImageDataGenerator 함수가 있으며 지금까지 찾아보고 실습한 내용을 정리해서 올립니다. 흔히 사용하는 rotate, shift, zoom, flip 뿐만아니라 조금 더 다양한 노이즈 적용방식을 구현해보고 샘플을 확인해보겠습니다. (기본적인 이미지로딩 과정과 뼈대는 기존 커널들을 참고하였습니다.)\n\n- 커널 참고\n    - Daehun Gwak, Keras, How to use pretrained model?\n    - Jang, Car Model Classification (Xception)\n- Augmentation 함수 관련\n    - https://keras.io/preprocessing/image/\n    - https://www.kaggle.com/safavieh/image-augmentation-using-skimage\n    - https://www.kaggle.com/gaborfodor/augmentation-methods \n    - https://machinelearningmastery.com/image-augmentation-deep-learning-keras/\n```\n---------------\n "},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import warnings\nimport matplotlib.pylab as plt\nimport PIL\nwarnings.filterwarnings('ignore')\nimage_size = 224","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# 이미지 폴더 경로\nDATA_PATH = '../input/'\nTRAIN_IMG_PATH = os.path.join(DATA_PATH, 'train')\n\n# CSV 파일 경로\ndf_train = pd.read_csv(os.path.join(DATA_PATH, 'train.csv'))\ndf_train = df_train.iloc[:50] # 편의상 50개까지만 이용 \ndf_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def crop_boxing_img(img_name, margin=16, size=(image_size, image_size)):\n    if img_name.split('_')[0] == 'train':\n        PATH = TRAIN_IMG_PATH\n        data = df_train\n    else:\n        PATH = TEST_IMG_PATH\n        data = df_test\n\n    img = PIL.Image.open(os.path.join(PATH, img_name))\n    pos = data.loc[data[\"img_file\"] == img_name, \\\n                   ['bbox_x1', 'bbox_y1', 'bbox_x2', 'bbox_y2']].values.reshape(-1)\n\n    width, height = img.size\n    x1 = max(0, pos[0] - margin)\n    y1 = max(0, pos[1] - margin)\n    x2 = min(pos[2] + margin, width)\n    y2 = min(pos[3] + margin, height)\n\n    return img.crop((x1, y1, x2, y2)).resize(size)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"TRAIN_CROP_PATH = \"./train_crop\"\n!mkdir {TRAIN_CROP_PATH}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# train_set의 모든 이미지에 대해 적용할 경우 시간이 좀 걸립니다.\nfor i, row in df_train.iterrows():\n    cropped = crop_boxing_img(row['img_file'])\n    cropped.save(f\"{TRAIN_CROP_PATH}/{row['img_file']}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\ndf_train[\"class\"] = df_train[\"class\"].astype('str')\ndf_train = df_train[['img_file', 'class']]\n\nX_val = df_train.copy()\n# its = np.arange(df_train.shape[0])\n# train_idx, val_idx = train_test_split(its, test_size = 0.8, shuffle= False)\n# X_train = df_train.iloc[train_idx, :]\n# X_val = df_train.iloc[val_idx, :]\n\n# print(X_train.shape)\nprint(X_val.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import keras \nfrom keras import models\nfrom keras import layers\nfrom keras import optimizers\nfrom keras.preprocessing.image import ImageDataGenerator","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"```\n먼저 rotaion, shift, zoom, flip에 대한 변형을 실행해보겠습니다. \n각 인자들에 대한 상세한 내용은 아래 사이트에서 살펴보시면 됩니다.\nhttps://keras.io/preprocessing/image/ \n\n그리고 중요한 사항으로 rescaled을 이용하여 이미지의 행렬값을 0~255에서 0~1로 변경하는 부분입니다. 신경망모델에서는 데이터의 특성에 대해서 스케일링한다음 학습하는 것이 매우 중요합니다.\n```"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_datagen = ImageDataGenerator(\n    rescale=1./255,\n    rotation_range=20,\n    width_shift_range=0.1,\n    height_shift_range=0.1,\n    zoom_range=0.1,\n    horizontal_flip=True,\n    fill_mode='nearest'\n)\n\nvalid_datagen = ImageDataGenerator(rescale=1./255) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"img_size = (image_size, image_size)\nbatch_size = 10\n\ntrain_generator = train_datagen.flow_from_dataframe(\n    dataframe = X_val, \n    directory = TRAIN_CROP_PATH,\n    x_col = 'img_file',\n    y_col = 'class',\n    target_size = img_size,\n    color_mode ='rgb',\n    class_mode ='categorical',\n    batch_size =batch_size,\n    shuffle =False\n)\n\nvalidation_generator = valid_datagen.flow_from_dataframe(\n    dataframe = X_val, \n    directory = TRAIN_CROP_PATH,\n    x_col ='img_file',\n    y_col ='class',\n    target_size = img_size,\n    color_mode ='rgb',\n    class_mode ='categorical',\n    batch_size =batch_size,\n    shuffle =False\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# 아래와 같이 augmentation 처리된 이미지들을 확인할 수 있습니다.\n\nfor data1, label1 in train_generator:\n    print('배치 데이터 크기:', data1.shape)\n    print('배치 레이블 크기:', label1.shape)\n    break\n    \nfor data2, label2 in validation_generator:\n    print('배치 데이터 크기:', data2.shape)\n    print('배치 레이블 크기:', label2.shape)\n    break    \n\n       ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# 왼쪽은 증식된(변형된)이미지, 오른쪽은 기존 이미지입니다.\n\nimport matplotlib.pyplot as plt\nfrom keras.preprocessing import image\n\nfor i in range(4):   \n    fig, axs = plt.subplots(ncols=2, figsize=(8,4), sharex=True, sharey=True)\n    axs[0].imshow(image.array_to_img(data1[i]))\n    axs[1].imshow(image.array_to_img(data2[i]))\n    fig.tight_layout()\n    plt.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"```\nfeaturewise_center\nsamplewise_center\nfeaturewise_std_normalization\nsamplewise_std_normalization\nzca_whitening\n\n해당 인자들을 True로 활성화하려면 train_datagen.fit(train_x)을 해줘야합니다.\n하지만 여기서는 flow_from_dataframe을 이용하기때문에 (이미지가 아닌 경로로 접근) 아래 코드와 같이 바로 설정하지는 못합니다. \npreprocessing_function을 이용하여 동일한 효과를 내는 별도의 함수를 만들어 직접 이미지 처리를 할수 밖에 없습니다.\n해당 인자들의 효과는 아래 사이트에서 확인바랍니다.\nhttps://machinelearningmastery.com/image-augmentation-deep-learning-keras/\n```"},{"metadata":{"trusted":true},"cell_type":"code","source":"# compute quantities required for featurewise normalization\n# (std, mean, and principal components if ZCA whitening is applied)\n# Only required if featurewise_center or featurewise_std_normalization or zca_whitening are set to True.\n# train_datagen.fit(train_features)\n\ntrain_datagen = ImageDataGenerator(\n    rescale=1./255,\n    featurewise_center= True,  # set input mean to 0 over the dataset\n    samplewise_center=False,  # set each sample mean to 0\n    featurewise_std_normalization= True,  # divide inputs by std of the dataset\n    samplewise_std_normalization=False,  # divide each input by its std\n    zca_whitening=False,  # apply ZCA whitening\n    #rotation_range=40, # randomly rotate images in the range (degrees, 0 to 180)\n    #zoom_range = 0.2, # Randomly zoom image \n    #shear_range=0.2,\n    #width_shift_range=0.2,  # randomly shift images horizontally (fraction of total width)\n    #height_shift_range=0.2,  # randomly shift images vertically (fraction of total height)\n    horizontal_flip=True)  # randomly flip images\n\n# train_datagen.fit() # 이 부분에서 실행되지 않는다.\n\ntrain_generator = train_datagen.flow_from_dataframe(\n    dataframe = X_val, \n    directory = TRAIN_CROP_PATH,\n    x_col = 'img_file',\n    y_col = 'class',\n    target_size = img_size,\n    color_mode ='rgb',\n    class_mode ='categorical',\n    batch_size =batch_size,\n    shuffle =False\n)\n\ntrain_generator.reset()\nvalidation_generator.reset()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"```\n별도함수를 구성하고 preprocessing_function를 이용해 노이즈 필터를 적용해보겠습니다.\n이미지 처리를 위해 skimage라는 패키지를 이용하였으며 random하게 필터의 설정값을 적용합니다.\n아래코드는 https://www.kaggle.com/safavieh/image-augmentation-using-skimage 을 참고하였습니다.\n```"},{"metadata":{"trusted":true},"cell_type":"code","source":"from skimage.transform import warp, AffineTransform, ProjectiveTransform\nfrom skimage.exposure import equalize_adapthist, equalize_hist, rescale_intensity, adjust_gamma, adjust_log, adjust_sigmoid\nfrom skimage.filters import gaussian\nfrom skimage.util import random_noise\nimport random\n\ndef randRange(a, b):\n    return np.random.rand() * (b - a) + a\n\ndef randomIntensity(im):\n    # rescales the intesity of the image to random interval of image intensity distribution\n    return rescale_intensity(im,\n                             in_range=tuple(np.percentile(im, (randRange(0,10), randRange(90,100)))),\n                             out_range=tuple(np.percentile(im, (randRange(0,10), randRange(90,100)))))\n\ndef randomGamma(im):\n    # Gamma filter for contrast adjustment with random gamma value.\n    return adjust_gamma(im, gamma=randRange(1, 2.5))\n\ndef randomGaussian(im):\n    # Gaussian filter for bluring the image with random variance.\n    return gaussian(im, sigma=randRange(0, 5))\n\ndef randomNoise(im):\n    # random gaussian noise with random variance.\n    var = randRange(0.005, 0.01)\n    return random_noise(im, var=var)\n\n# 위 4가지 함수를 넣어 random하게 적용시킨다.\ndef augment(im, Steps= [randomGamma, randomGaussian, randomNoise]):\n    \n    im /= 255. # 추가    \n    i= np.random.randint(3)\n    step = Steps[i]\n    im = step(im)\n    return im\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# augment 함수 내부적으로 스케일링을 하기 때문에 rescale을 뺀다.\ntrain_datagen = ImageDataGenerator(preprocessing_function=augment) \nvalid_datagen = ImageDataGenerator(rescale=1./255) \n\ntrain_generator = train_datagen.flow_from_dataframe(\n    dataframe = X_val, \n    directory = TRAIN_CROP_PATH,\n    x_col = 'img_file',\n    y_col = 'class',\n    target_size = img_size,\n    color_mode ='rgb',\n    class_mode ='categorical',\n    batch_size =batch_size,\n    shuffle =False\n)\n\ntrain_generator.reset()\nvalidation_generator.reset()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for data1, label1 in train_generator:\n    print('배치 데이터 크기:', data1.shape)\n    print('배치 레이블 크기:', label1.shape)\n    break\n    \nfor data2, label2 in validation_generator:\n    print('배치 데이터 크기:', data2.shape)\n    print('배치 레이블 크기:', label2.shape)\n    break    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nfrom keras.preprocessing import image\n\nfor i in range(8):   \n    fig, axs = plt.subplots(ncols=2, figsize=(8,4), sharex=True, sharey=True)\n    axs[0].imshow(image.array_to_img(data1[i]))\n    axs[1].imshow(image.array_to_img(data2[i]))\n    fig.tight_layout()\n    plt.show()\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"```\n위 사진을 보면 필터값이 너무 세서 아예 뭉개져버린 경우도 발생합니다. \n해당 필터함수내 파라메타값 조정이 필요해보입니다.\n\n커널하나 작성하는것도 시간이 만만치않게 소요되네요.\n이미지 색상변경 (color noise) , Fancy PCA \n와 같은 변형기법도 있는데 추후 시간이 되면 업데이트하도록 하겠습니다.\n```"},{"metadata":{"trusted":true},"cell_type":"code","source":"!rm -rf *_crop","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}