{"cells":[{"metadata":{"id":"Or9Wjd8_ebAV","colab_type":"text"},"cell_type":"markdown","source":"# 2019 3rd ML month with KaKR\n\n자동차 이미지 데이터셋을 이용한 자동차 차종 분류 <br>\n\n\n### Preprocessing \n\n- 이미지 ratio를 유지하면서 crop/resizing을 수행.\n  - 자동차의 형태가 판별에 도움이 될 것으로 생각해서 ratio를 유지하고 검은색 padding을 추가함.\n  - 그대로 resizing을 한 것에 비해 좋은 결과를 얻지 못하였음.\n\n### 사용 모델\n- Xception <br>\n- 5 Fold cross validation\n\n### Ensemble\n- weighted averaging model + TTA\n  - cv model들의 output들을 dstack으로 합한 값을 입력으로 받는 meta learner 모델을 만들고 train하여 각각의 모델마다 다른 가중치로 ensemble이 이루어지도록 한다.\n  - test prediction시 TTA(train용 generator를 사용)"},{"metadata":{"id":"TfUhx-D9c9nD","colab_type":"code","colab":{},"trusted":true},"cell_type":"code","source":"import sys\nIN_COLAB = 'google.colab' in sys.modules","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 상수 / 파라메터 설정"},{"metadata":{"trusted":true},"cell_type":"code","source":"# model-specific constants:\n\n# image w/h :\n#     ResNet50 : (224, 224)\n#     Xception : (299, 299)\nIMAGE_WIDTH, IMAGE_HEIGHT = (299, 299)","execution_count":null,"outputs":[]},{"metadata":{"id":"Hkb7Timl9ToV","colab_type":"code","colab":{},"trusted":true},"cell_type":"code","source":"# 전체적으로 문제없이 돌아가는지만 확인할 때 사용.\nTESTFLIGHT = False\n\n# train을 통해 weight를 만들려 할 때 사용\nGENERATE_WEIGHTS = False\n\nBATCH_SIZE = 32\nEPOCHS = 40\nK_FOLDS = 5\nPATIENCE = 6\n\n# 오류 없이 돌아가는지 확인하려 할 때 사용한다:\nif TESTFLIGHT:\n    EPOCHS = 3\n    K_FOLDS = 3\n\nASSIGNED_FOLD_JOBS = [x for x in range(K_FOLDS)]","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"fTY2tRgA_KPX"},"cell_type":"markdown","source":"# **1. 데이터 분석:**"},{"metadata":{"colab_type":"code","id":"4hfwgBAG_orv","colab":{},"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport os\nfrom pathlib import Path\nimport shutil\n\nDATA_PATH = \"../input\"\nCROPPED_DATA_PATH = \"../cropped\"\nMODEL_PATH = \"../models\"\n\nif IN_COLAB:\n    DATA_PATH = \"./input\"\n    CROPPED_DATA_PATH = \"./cropped\"\n    MODEL_PATH = \"./models\"\nelse:\n    # path 목록\n    DATA_PATH = \"../input\"            # input 데이터 경로\n    CROPPED_DATA_PATH = \"./cropped\"  # crop한 이미지를 저장할 경로\n    MODEL_PATH = \"./\"          # training 완료된 weight 파일이 저장될 경로\n    \n    if os.path.exists(MODEL_PATH) == False:\n        Path(MODEL_PATH).mkdir(parents=True, exist_ok=True)\n    \n    # 직접 사용하지 않는다:\n    PRE_MODEL_NAME = \"kakr-3rd-xception\"\n    INPUT_DATA_NAME = \"2019-3rd-ml-month-with-kakr\"\n    \n    pre_models_path = os.path.join(DATA_PATH, PRE_MODEL_NAME)\n    \n    # weight를 생성하려는 목적이 아니면 weight파일을 미리 복사해 둔다.\n\n    if os.path.exists(pre_models_path):\n        for fname in os.listdir(pre_models_path):\n            filepath = os.path.join(pre_models_path, fname)\n            if os.path.isfile(filepath):\n                if GENERATE_WEIGHTS == True:\n                    if fname.find(\"h5\") > 0:\n                        continue\n                destfilepath = os.path.join(MODEL_PATH, fname)\n                print(\"copy file \", filepath, \" to \", destfilepath)\n                shutil.copy(filepath, destfilepath)\n\n    if os.path.exists(os.path.join(DATA_PATH, INPUT_DATA_NAME)):\n        DATA_PATH = os.path.join(DATA_PATH, INPUT_DATA_NAME)\n        \nprint(\"Paths : \")\nprint(\"----------------------------------\")\nprint(\"DATA_PATH         : \", DATA_PATH)\nprint(\"CROPPED_DATA_PATH : \", CROPPED_DATA_PATH)\nprint(\"MODEL_PATH        : \", MODEL_PATH)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# colab에서 구동을 위한 것이므로 코드 참고 시에는 무시하면 됩니다.\ndef tmp_copy_weights_to_model_path():\n    for fname in os.listdir(\"./\"):\n\n        if fname.find(\"h5\") > -1:\n            dstpath = os.path.join(MODEL_PATH, fname)\n            shutil.copy(fname, dstpath)\n\nif IN_COLAB == True:\n    tmp_copy_weights_to_model_path()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# csv 파일을 읽어서 DataFrame을 생성합니다.\ndf_train = pd.read_csv(os.path.join(DATA_PATH, \"train.csv\"))\ndf_test = pd.read_csv(os.path.join(DATA_PATH, \"test.csv\"))\ndf_class = pd.read_csv(os.path.join(DATA_PATH, \"class.csv\"))\n\nclasses = df_class['id'].values.astype('str').tolist()\nnum_classes_out = len(classes)\n\nif os.path.exists(MODEL_PATH) == False:\n    Path(MODEL_PATH).mkdir(parents=True, exist_ok=True)\n    \n    \nimport zipfile\nz = zipfile.ZipFile(os.path.join(DATA_PATH, \"train.zip\"))\nz.extractall(\"./train\")\nz.close()\n\nimport zipfile\nz = zipfile.ZipFile(os.path.join(DATA_PATH, \"test.zip\"))\nz.extractall(\"./test\")\nz.close()","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"code","id":"v2d94WEkAbkJ","outputId":"879a317c-1214-47f0-ffd2-fa1a4811e1b2","colab":{"base_uri":"https://localhost:8080/","height":204},"trusted":true},"cell_type":"code","source":"df_train.head()","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"8d1dxCAeAwSp"},"cell_type":"markdown","source":"- img_file : 이미지 파일명, train.zip에 있는 이미지 파일 이름.\n- (bbox_x1, bbox_y1) : 자동차 위치 좌측 상단 좌표\n- (bbox_x2, bbox_y2) : 자동차 위치 우측 하단 좌표\n- class : 해당 이미지의 자동차 class index\n"},{"metadata":{"colab_type":"code","id":"Dg-kOXgHAfax","outputId":"8f34641e-d695-4669-94c0-48dfad0bb6ba","colab":{"base_uri":"https://localhost:8080/","height":204},"trusted":true},"cell_type":"code","source":"df_test.head()","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"code","id":"-0pz0JOiCkw0","outputId":"a5632ae3-b677-4c13-e605-390a3d57aa5d","colab":{"base_uri":"https://localhost:8080/","height":204},"trusted":true},"cell_type":"code","source":"df_class.head()","execution_count":null,"outputs":[]},{"metadata":{"id":"mA6uRFMonQNX","colab_type":"text"},"cell_type":"markdown","source":"class별 데이터 수 확인"},{"metadata":{"id":"agmKTft7ncHi","colab_type":"code","outputId":"ba53bccc-998a-4ab9-b177-c5aa63bcd9e0","colab":{"base_uri":"https://localhost:8080/","height":394},"trusted":true},"cell_type":"code","source":"import seaborn as sns\nimport matplotlib.pyplot as plt\n\nfig, ax = plt.subplots(2, 1, figsize=(24, 5))\ndef check_count_by_class():\n    df_merged = df_train.append(df_test, ignore_index=True, sort=False)\n    sns.countplot(x='class', data=df_merged, ax=ax[0])\n    ax[1].set_xlim([0, 300])\n    df_merged['class'].plot.kde(ax=ax[1])\n    print(\"max count = \", df_merged['class'].value_counts().index[0], \":\", df_merged['class'].value_counts()[df_merged['class'].value_counts().index[0]])\n    \n    a = df_merged['class'].value_counts()\n    print(\"MAX count : \", a.idxmax().astype('int'), a[a.idxmax()])\n    print(\"MIN count : \", a.idxmin().astype('int'), a[a.idxmin()])    \n\ncheck_count_by_class()\n","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"Ie8BnP3JhhnM"},"cell_type":"markdown","source":"이미지의 width/height min/max 값과 분포를 확인"},{"metadata":{"colab_type":"code","id":"rUSg_FuohgzJ","outputId":"087d9994-f3fb-49bb-e935-a6e128851c3e","colab":{"base_uri":"https://localhost:8080/","height":658},"trusted":true},"cell_type":"code","source":"from PIL import Image, ImageDraw\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport os\n\n\nwidths = []\nheights = []\n\nfor idx, row in df_train.iterrows():    \n    im = Image.open(os.path.join(\"./train\", row['img_file']))\n    w, h = im.size\n    widths.append(w)\n    heights.append(h)\n    \nwidths = np.array(widths)\nheights = np.array(heights)\nprint(\"width  min/max/mean : \", np.min(widths), np.max(widths), np.mean(widths))\nprint(\"height min/max/mean : \", np.min(heights), np.max(heights), np.mean(heights))\n\nbins = [x for x in range(-1, 6000, 200)]\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfig, ax = plt.subplots(2, 2, figsize=(16, 10))\n\n# draw as hist:\nax[0][0].hist(widths, bins=bins)\nax[0][0].set_title(\"width hist\")\nax[0][1].hist(heights, bins=bins)\nax[0][1].set_title(\"height hist\")\n\n# draw as kde:\nsns.kdeplot(np.array(widths), ax=ax[1][0])\nax[1][0].set_title(\"width kde\")\nsns.kdeplot(np.array(heights), ax=ax[1][1])\nax[1][1].set_title(\"height kde\")\n    \n","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"code","id":"HYvaQ7YYWKWw","outputId":"a92bd37c-7071-442f-998b-2095ab43797d","colab":{"base_uri":"https://localhost:8080/","height":1000},"trusted":true},"cell_type":"code","source":"from PIL import Image, ImageDraw\nimport matplotlib.pyplot as plt\nimport matplotlib.patches as patches\n\nn = 5\n\nfix, ax = plt.subplots(n, 1, figsize = (10, 40))\naxidx = 0\n\ndf_sample = df_train.sample(n)\nfor idx, row in df_sample.iterrows():\n    im = Image.open(os.path.join(\"./train\", row['img_file']))\n    \n    # 이미지 내에 박스를 그린다.\n    draw = ImageDraw.Draw(im)\n    #draw.rectangle((row['bbox_x1'], row['bbox_y1'], row['bbox_x2'], row['bbox_y2']), outline='blue', width=2)\n    \n    print(\"bbox:\", row['bbox_x1'], row['bbox_y1'], row['bbox_x2'], row['bbox_y2'])\n    width = row['bbox_x2'] - row['bbox_x1']\n    height = row['bbox_y2'] - row['bbox_y1']\n    rect = patches.Rectangle((row['bbox_x1'], row['bbox_y1']), width, height, linewidth=3, edgecolor='r',facecolor='none')\n    ax[axidx].add_patch(rect)\n    ax[axidx].imshow(im)\n    \n    axidx += 1\n\n","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"ZYcCHWvbbld8"},"cell_type":"markdown","source":"ratio의 분포를 학인합니다."},{"metadata":{"colab_type":"code","id":"7w3Vmde-WWcZ","outputId":"8e247eab-653c-49a3-f26a-9e0e58660d0e","colab":{"base_uri":"https://localhost:8080/","height":422},"trusted":true},"cell_type":"code","source":"def checkratio(df):\n    df['ratio'] = (df['bbox_x2'] - df['bbox_x1']) / (df['bbox_y2'] - df['bbox_y1'])\n    return df\n\nfig, ax = plt.subplots(1, 1, figsize=(12, 8))\n\ndf_merged = df_train.append(df_test, ignore_index=True, sort=False)\ndf_tmp = checkratio(df_merged)\ndf_tmp['ratio'].plot.kde()","execution_count":null,"outputs":[]},{"metadata":{"id":"S0hYlq-SBn68","colab_type":"text"},"cell_type":"markdown","source":"# **2. 데이터 편집:**\n\n이미지 내 자동차의 위치가 csv파일에 들어 있으므로, 위치에 맞게 crop하고 crop된 이미지의 width/height를 1:1로 맞춰야 합니다. <br>\n이미지를 그대로 resize하면 자동차의 형태가 변형되어 shape을 통한 판별에 문제가 생길 것으로 판단하였습니다. resize전에 먼저 이미지의 좌/우 혹은 위/아래에 padding을 추가하여 ratio를 유지하도록 코드를 작성하였습니다."},{"metadata":{"colab_type":"code","id":"wcga964JSp0C","outputId":"73e9f0cd-b47a-4aac-f168-b3a63e960a22","colab":{"base_uri":"https://localhost:8080/","height":51},"trusted":true},"cell_type":"code","source":"image_width, image_height = IMAGE_WIDTH, IMAGE_HEIGHT\n\n# crop해서 만들 이미지의 W/H 비율을 1:1로 한다.\n# 모델의 이미지 입력은 width,height가 동일하므로 1:1만이 가능함.\ndefault_ratio = 1.0\n\nfrom pathlib import Path\n    \ndef get_fixed_img(filename, area, ratio, output_size):\n    debug = False\n    im = Image.open(filename)\n    cropIm = im.crop(area)\n    \n    if debug:\n        print(\"crop w/h=\", cropIm.width, cropIm.height)\n    \n    w, h = cropIm.width, cropIm.height\n    # w : h = w/h ratio : 1\n    fixedW, fixedH = w, h\n    if w/h >= ratio:\n        fixedH = w / ratio\n    else:\n        fixedW = h * ratio\n    fixedW, fixedH = int(fixedW), int(fixedH)\n\n    if debug:\n        print(\"fixed w/h=\", fixedW, fixedH, \"ratio=\", fixedW/fixedH)\n    \n    newIm = Image.new(\"RGB\", (fixedW, fixedH))\n    newIm.paste(cropIm, ((fixedW - w)//2, (fixedH - h)//2))\n    \n    #newIm = newIm.resize(output_size, resample=Image.NEAREST)\n    return newIm\n\ndef img_from_row(row, path, ratio, w, h):\n    filepath = os.path.join(path, row['img_file'])\n    area = (row['bbox_x1'], row['bbox_y1'], row['bbox_x2'], row['bbox_y2'])\n    return get_fixed_img(filepath, area, ratio, (w, h))    \n\ndef make_crop_img(df, dirname):\n    # 저장될 디렉토리부터 생성\n    dirpath_crop = os.path.join(CROPPED_DATA_PATH, dirname)\n    if os.path.exists(dirpath_crop) == False:\n        Path(dirpath_crop).mkdir(parents=True, exist_ok=True)\n\n    for idx, row in df.iterrows():\n        src_path = dirname\n        target_img_path = os.path.join(dirpath_crop, row['img_file'])\n        \n        # 파일이 없거나, 사이즈가 0이면 새로 만든다.\n        isvalid = os.path.exists(target_img_path)\n        isvalid = (isvalid and (os.path.getsize(target_img_path) > 0))\n        if isvalid == False:\n            im = img_from_row(row, src_path, default_ratio, image_width, image_height)\n            im.save(target_img_path)\n\ndef make_cropped_imgs():\n    dirpath_crop = os.path.join(CROPPED_DATA_PATH, \"train\")\n    print(dirpath_crop)\n    make_crop_img(df_train, \"train\")\n    \n    dirpath_crop = os.path.join(CROPPED_DATA_PATH, \"test\")\n    print(dirpath_crop)\n    make_crop_img(df_test, \"test\")\n\nmake_cropped_imgs()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# show cropped image\ndef show_cropped_imgs():\n    dirpath_crop = os.path.join(CROPPED_DATA_PATH, \"train\")    \n    n = 4\n    fix, ax = plt.subplots(n, 1, figsize = (10, 40))\n    axidx = 0\n    df_sample = df_train.sample(n)\n    for idx, row in df_sample.iterrows():\n        im = Image.open(os.path.join(dirpath_crop, row['img_file']))\n\n        # 이미지 내에 박스를 그린다.\n        draw = ImageDraw.Draw(im)\n        ax[axidx].imshow(im)\n        axidx += 1\n    plt.show()\n    \nshow_cropped_imgs()","execution_count":null,"outputs":[]},{"metadata":{"id":"Cl0GUQTGcjKu","colab_type":"code","colab":{},"trusted":true},"cell_type":"code","source":"epochs = EPOCHS\nbatch_size = BATCH_SIZE\n\ndef get_total_batch(num_samples, batch_size):    \n    if (num_samples % batch_size) > 0 :\n        return (num_samples // batch_size) + 1\n    else :\n        return num_samples // batch_size","execution_count":null,"outputs":[]},{"metadata":{"id":"2Ndi8KrlAkFN","colab_type":"text"},"cell_type":"markdown","source":"해당 competition의 평가는 F1 Score를 사용합니다.\n\n    Precision : TP / (TP + FP)\n    Recall : TP / (TP + FN) \n    \n    F1 Score = (2 X Recall X Precision) / (Recall + Precision)\n    \nearly stopping 등의 결정을 위한 F1 Score 함수를 정의합니다."},{"metadata":{"id":"43HYqH-oAi6-","colab_type":"code","colab":{},"trusted":true},"cell_type":"code","source":"from keras import backend as K\ndef recall(y_target, y_pred):\n    # clip(t, clip_value_min, clip_value_max) : clip_value_min~clip_value_max 이외 가장자리를 깎아 낸다\n    # round : 반올림한다\n    y_target_yn = K.round(K.clip(y_target, 0, 1)) # 실제값을 0(Negative) 또는 1(Positive)로 설정한다\n    y_pred_yn = K.round(K.clip(y_pred, 0, 1)) # 예측값을 0(Negative) 또는 1(Positive)로 설정한다\n\n    # True Positive는 실제 값과 예측 값이 모두 1(Positive)인 경우이다\n    count_true_positive = K.sum(y_target_yn * y_pred_yn) \n\n    # (True Positive + False Negative) = 실제 값이 1(Positive) 전체\n    count_true_positive_false_negative = K.sum(y_target_yn)\n\n    # Recall =  (True Positive) / (True Positive + False Negative)\n    # K.epsilon()는 'divide by zero error' 예방차원에서 작은 수를 더한다\n    recall = count_true_positive / (count_true_positive_false_negative + K.epsilon())\n\n    # return a single tensor value\n    return recall\n\n\ndef precision(y_target, y_pred):\n    # clip(t, clip_value_min, clip_value_max) : clip_value_min~clip_value_max 이외 가장자리를 깎아 낸다\n    # round : 반올림한다\n    y_pred_yn = K.round(K.clip(y_pred, 0, 1)) # 예측값을 0(Negative) 또는 1(Positive)로 설정한다\n    y_target_yn = K.round(K.clip(y_target, 0, 1)) # 실제값을 0(Negative) 또는 1(Positive)로 설정한다\n\n    # True Positive는 실제 값과 예측 값이 모두 1(Positive)인 경우이다\n    count_true_positive = K.sum(y_target_yn * y_pred_yn) \n\n    # (True Positive + False Positive) = 예측 값이 1(Positive) 전체\n    count_true_positive_false_positive = K.sum(y_pred_yn)\n\n    # Precision = (True Positive) / (True Positive + False Positive)\n    # K.epsilon()는 'divide by zero error' 예방차원에서 작은 수를 더한다\n    precision = count_true_positive / (count_true_positive_false_positive + K.epsilon())\n\n    # return a single tensor value\n    return precision\n\n\ndef f1score(y_target, y_pred):\n    _recall = recall(y_target, y_pred)\n    _precision = precision(y_target, y_pred)\n    # K.epsilon()는 'divide by zero error' 예방차원에서 작은 수를 더한다\n    _f1score = ( 2 * _recall * _precision) / (_recall + _precision+ K.epsilon())\n    \n    # return a single tensor value\n    return _f1score","execution_count":null,"outputs":[]},{"metadata":{"id":"P44HYK78m36a","colab_type":"text"},"cell_type":"markdown","source":"CV를 위해 KFold를 생성합니다. <br>\n전체 데이터에서의 class 간 비율을 fold에서도 동일하게 유지하기 위해 StratifiedKFold를 사용합니다."},{"metadata":{"id":"67OC86KXiKTq","colab_type":"code","colab":{},"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import StratifiedKFold, KFold\nskfold = StratifiedKFold(n_splits=K_FOLDS, random_state=2019)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"다음 커널에서 random_eraser 함수를 가져와서 사용\n\nhttps://www.kaggle.com/seriousran/what-0-95121-model-failed"},{"metadata":{"trusted":true},"cell_type":"code","source":"#ref: https://github.com/yu4u/cutout-random-erasing/blob/master/cifar10_resnet.py\ndef get_random_eraser(p=0.5, s_l=0.02, s_h=0.4, r_1=0.3, r_2=1/0.3, v_l=0, v_h=255, pixel_level=False):\n    def eraser(input_img):\n        img_h, img_w, img_c = input_img.shape\n        p_1 = np.random.rand()\n\n        if p_1 > p:\n            return input_img\n\n        while True:\n            s = np.random.uniform(s_l, s_h) * img_h * img_w\n            r = np.random.uniform(r_1, r_2)\n            w = int(np.sqrt(s / r))\n            h = int(np.sqrt(s * r))\n            left = np.random.randint(0, img_w)\n            top = np.random.randint(0, img_h)\n\n            if left + w <= img_w and top + h <= img_h:\n                break\n\n        if pixel_level:\n            c = np.random.uniform(v_l, v_h, (h, w, img_c))\n        else:\n            c = np.random.uniform(v_l, v_h)\n\n        input_img[top:top + h, left:left + w, :] = c\n\n        return input_img\n\n    return eraser","execution_count":null,"outputs":[]},{"metadata":{"id":"OEEqftiEAYwP","colab_type":"text"},"cell_type":"markdown","source":"train/eval에 사용할 ImageDataGenerator를 생성한다. <br>\nvalidation generator의 경우 동일한 이미지로 평가를 해야 개선 여부 확인이 가능하므로 ImageDataGenerator의 augmentation을 모두 off 하였습니다."},{"metadata":{"id":"WYYN9Kuj8gQ7","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"outputId":"ba91374e-3314-4da5-b12a-200a31f6e62e","trusted":true},"cell_type":"code","source":"from keras.applications.resnet50 import ResNet50, preprocess_input\nfrom keras.preprocessing.image import ImageDataGenerator\n\ndatagen_train = ImageDataGenerator(\n    rescale=1./255,\n    featurewise_center=False,  # set input mean to 0 over the dataset\n    samplewise_center=False,  # set each sample mean to 0\n    featurewise_std_normalization=False,  # divide inputs by std of the dataset\n    samplewise_std_normalization=False,  # divide each input by its std\n    zca_whitening=False,  # apply ZCA whitening\n    rotation_range=20,  # randomly rotate images in the range (degrees, 0 to 180)\n    zoom_range = 0.3, # Randomly zoom image \n    width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)\n    height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)\n    horizontal_flip=True,  # randomly flip images\n    vertical_flip=False,  # randomly flip images\n    preprocessing_function = get_random_eraser(v_l=0, v_h=255),\n    )\n\ndatagen_val = ImageDataGenerator(\n    rescale=1./255,\n#     featurewise_center=False,  # set input mean to 0 over the dataset\n#     samplewise_center=False,  # set each sample mean to 0\n#     featurewise_std_normalization=False,  # divide inputs by std of the dataset\n#     samplewise_std_normalization=False,  # divide each input by its std\n#     zca_whitening=False,  # apply ZCA whitening\n#     rotation_range=15,  # randomly rotate images in the range (degrees, 0 to 180)\n#     zoom_range = 0.2, # Randomly zoom image \n#     width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)\n#     height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)\n#     horizontal_flip=True,  # randomly flip images\n#     vertical_flip=False,  # randomly flip images\n#     preprocessing_function = preprocess_input\n)\n\n\n# 아래 안해주면 에러남. categorical이어서 기준 col이 숫자값이면 안되는 것인듯.\ndf_train['class'] = df_train['class'].astype('str')","execution_count":null,"outputs":[]},{"metadata":{"id":"ZQGcf6k6MhJI","colab_type":"text"},"cell_type":"markdown","source":"<a id=\"modelling\"></a> <br> \n# **3. 모델 생성:**\n\nXception을 base model로 사용하고 끝에 간단한 fc 레이어들을 붙여서 사용할 모델을 생성합니다."},{"metadata":{"id":"J6wcCEmaBYnQ","colab_type":"code","colab":{},"trusted":true},"cell_type":"code","source":"from keras.applications.resnet50 import ResNet50, preprocess_input\nfrom keras.applications.xception import Xception\nfrom keras import optimizers\nfrom keras.models import Sequential, Model\nfrom keras.layers import Dense, Dropout, Flatten, Activation, Conv2D, GlobalAveragePooling2D\n\n__models = {}\n\ndef get_model(base_model, show_summary=False):\n    base_model = base_model(weights='imagenet', include_top=False, input_shape = (IMAGE_WIDTH, IMAGE_HEIGHT,3))\n\n    model = Sequential()\n    model.add(base_model)\n    model.add(GlobalAveragePooling2D())\n\n    # 중간 layer 추가(dropout)\n    model.add(Dense(2048, activation='relu', kernel_initializer='he_normal'))\n    model.add(Dropout(0.15))\n\n    model.add(Dense(196, activation='softmax', kernel_initializer='lecun_normal'))\n    if show_summary:\n        model.summary()\n\n    optimizer = optimizers.Nadam(lr=0.0002)\n    model.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['acc', recall, precision, f1score])\n    \n    return model\n    \n__models = {\"Xception\" : Xception}","execution_count":null,"outputs":[]},{"metadata":{"id":"Wg7MD-EjFXaz","colab_type":"text"},"cell_type":"markdown","source":"model 훈련시 사용할 callback 목록을 생성합니다."},{"metadata":{"id":"us-KG_QzFWkV","colab_type":"code","colab":{},"trusted":true},"cell_type":"code","source":"from keras.callbacks import ModelCheckpoint, EarlyStopping, ReduceLROnPlateau, LambdaCallback\n\ndef get_callbacks(model_save_filename, patience):\n    es = EarlyStopping(monitor='val_f1score', min_delta=0, patience=patience, verbose=1, mode='max')\n\n    rr = ReduceLROnPlateau(monitor = 'val_f1score', factor = 0.5, patience = patience / 2,\n                           min_lr=0.000001,\n                           verbose=1,\n                           mode='max')\n\n    mc = ModelCheckpoint(filepath=model_save_filename, monitor='val_f1score', verbose=1,\n                           save_best_only=True, mode='max')\n\n    return [es, rr, mc]\n    \n    ","execution_count":null,"outputs":[]},{"metadata":{"id":"g8nXPztkGW3I","colab_type":"text"},"cell_type":"markdown","source":"모델을 train 합니다.\n\nStratifiedKFold은 label의 구성비율을 유지하면서 fold를 만들어내므로 split() 호출 시, y(label)도 넘겨야 합니다."},{"metadata":{"id":"1SWsr8gVNzy_","colab_type":"code","outputId":"2e457433-34b1-4f47-c993-ed3a2f58ab5e","colab":{"base_uri":"https://localhost:8080/","height":1000},"trusted":true},"cell_type":"code","source":"#aaa = skfold.split(X=df_train['img_file'], y=df_train['class'])\nimport ssl\nfrom keras.models import model_from_json\n\nssl._create_default_https_context = ssl._create_unverified_context\nhistory_list = {}\nfor _m in __models:\n    print(\"Model : \", _m)\n        \n    # 미리 fold를 나누어 생생해 둔 dataframe 파일을 사용한다.\n    for fold_index in ASSIGNED_FOLD_JOBS:\n\n        # 모델 생성\n        model = get_model(__models[_m], fold_index == 0)\n\n        # weight를 저장할 파일명을 생성\n        model_save_filename = (\"%s_%d.h5\" % (_m , fold_index))\n        model_save_filepath = os.path.join(MODEL_PATH, model_save_filename)\n\n        # 나눠진 dataframe을 load\n        df_train_filename = (\"fold_%d_train.csv\" % fold_index)\n        df_val_filename = (\"fold_%d_val.csv\" % fold_index)\n\n        dataframe_train = pd.read_csv(os.path.join(MODEL_PATH, df_train_filename))\n        dataframe_val = pd.read_csv(os.path.join(MODEL_PATH, df_val_filename))\n\n        # 아래 안해주면 에러남. categorical이어서 기준 col이 숫자값이면 안되는 것인듯.\n        dataframe_train['class'] = dataframe_train['class'].astype('str')\n        dataframe_val['class'] = dataframe_val['class'].astype('str')\n\n        # ImageDataGenerator 생성(train/val)\n        datagen_train_flow = datagen_train.flow_from_dataframe(dataframe=dataframe_train,\n                                                   directory=os.path.join(CROPPED_DATA_PATH, \"train\"),\n                                                   x_col='img_file',\n                                                   y_col=\"class\",\n                                                   classes = classes,\n                                                   target_size=(IMAGE_WIDTH, IMAGE_HEIGHT),\n                                                   color_mode='rgb',\n                                                   class_mode='categorical',\n                                                   batch_size=batch_size,\n                                                   shuffle = True)\n\n        datagen_val_flow = datagen_val.flow_from_dataframe(dataframe=dataframe_val,\n                                                   directory=os.path.join(CROPPED_DATA_PATH, \"train\"),\n                                                   x_col='img_file',\n                                                   y_col=\"class\",\n                                                   classes = classes,\n                                                   target_size=(IMAGE_WIDTH, IMAGE_HEIGHT),\n                                                   color_mode='rgb',\n                                                   class_mode='categorical',\n                                                   batch_size=batch_size,\n                                                   shuffle = True)\n        \n        if GENERATE_WEIGHTS == True:\n            if os.path.exists(model_save_filepath) == True:\n                os.remove(model_save_filepath)\n\n        # 동일 이름의 weight 파일이 있으면 넘어간다.\n        if os.path.exists(model_save_filepath) == True:\n            print(\">>>\", model_save_filepath, \" already trained... skip!\")\n            continue\n        \n        train_steps = get_total_batch(dataframe_train.shape[0], batch_size)\n        val_steps = get_total_batch(dataframe_val.shape[0], batch_size)\n        \n        if TESTFLIGHT:\n            train_steps = 10\n            val_steps = 10\n            \n        history = model.fit_generator(datagen_train_flow,\n            epochs=epochs,\n            steps_per_epoch = train_steps,\n            validation_data = datagen_val_flow,\n            validation_steps = val_steps,\n            callbacks = get_callbacks(model_save_filepath, PATIENCE),\n            verbose=0)\n        \n        history_list[model_save_filename] = history\n        \n        model = None\n        \n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"training history 진행과정에 대해 그래프를 그려 본다."},{"metadata":{"id":"otW2BuTDRyBa","colab_type":"code","outputId":"0b5d576a-71c9-4b12-d87a-069e10a1a4fb","colab":{"base_uri":"https://localhost:8080/","height":934},"trusted":true},"cell_type":"code","source":"if GENERATE_WEIGHTS == True:\n    fig, ax = plt.subplots(1, 1, figsize=(12,8 * len(__models)))\n    from cycler import cycler\n\n    # set color cycle : 그래프 색깔을 알아서 cycling해준다.\n    x = np.linspace(0, 1, 10)\n    number = 5\n    cmap = plt.get_cmap('gnuplot')\n    colors = [cmap(i) for i in np.linspace(0, 1, number)]\n    ax.set_prop_cycle(cycler('color', colors))\n\n    for hname in history_list:\n        history = history_list[hname]\n        plot_label = \"val_score : \" + hname\n        ax.plot(history.history['val_f1score'], label=plot_label)        \n    ax.legend()        \n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **4. Weighted averaging을 통한 ensemble & submission**\n\n최대 F1 Score를 낸 모델들은 ModelCheckpoint callback으로 파일로 저장되므로, 저장된 weight를 load해서 모델을 준비한다."},{"metadata":{"trusted":true},"cell_type":"code","source":"datagen_submit = ImageDataGenerator(\n    rescale=1./255,\n)\n\ndef load_sub_models():\n    sub_models = []\n    for _m in __models:\n        print(\"Model \", _m, \" : \")\n        for _, _, filenames in os.walk(MODEL_PATH):\n            for fname in filenames:\n                if fname.find(_m) >= 0 and fname.find(\".h5\") >= 0:        \n                    print(\">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> loading \", fname)\n                    model = get_model(__models[_m])\n\n                    # model에 데이터파일을 올린다.\n                    # model 명이 있는 것들 중에서만 올린다.\n\n                    fpath = os.path.join(MODEL_PATH, fname)\n                    print(\"model weight fpath:\", fpath)\n                    model.load_weights(fpath)\n\n                    sub_models.append(model)\n    return sub_models\n\nsubmodels = load_sub_models()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Weighted averaging model\n\nsub-model 각각의 결과가 weight 값에 따라 다르게 사용되도록 하는 방법으로, 여기서는 기존 output 값의 합을 입력으로, 기존 label을 그대로 사용하는 별도의 모델을 생성해서 최종 결과를 구하는 데 사용합니다.<br>\nfold로 생성된 sub-model의 예측결과를 dstack으로 합하여 meta learner용 dataset으로 생성하고 meta learner 모델을 훈련시킵니다."},{"metadata":{"trusted":true},"cell_type":"code","source":"from numpy import dstack\n\ndef make_meta_learner_dataset(submodels, df, imgdirname, tta = False):\n    datagen_submit = ImageDataGenerator(rescale=1./255)\n    stackX = None\n    for model in submodels:\n\n\t\t# make prediction :\n        datagen_metalearner_flow = datagen_submit.flow_from_dataframe(\n            dataframe=df,\n            directory=os.path.join(CROPPED_DATA_PATH, imgdirname),\n            x_col='img_file',\n            y_col=None,\n            target_size= (IMAGE_WIDTH, IMAGE_HEIGHT),\n            color_mode='rgb',\n            class_mode=None,\n            batch_size=batch_size,\n            shuffle=False)\n        \n        datagen_metalearner_augmented_flow = datagen_train.flow_from_dataframe(\n            dataframe=df,\n            directory=os.path.join(CROPPED_DATA_PATH, imgdirname),\n            x_col='img_file',\n            y_col=None,\n            target_size=(IMAGE_WIDTH, IMAGE_HEIGHT),\n            color_mode='rgb',\n            class_mode=None,\n            batch_size=batch_size,\n            shuffle=False)\n        \n        datagen_flow = datagen_metalearner_flow\n        if tta == True:\n            print(\"select tta flow\")\n            datagen_flow = datagen_metalearner_augmented_flow\n\n        datagen_flow.reset()\n        pred = model.predict_generator(generator = datagen_flow,\n                                       steps = get_total_batch(df.shape[0], batch_size),\n                                       verbose=1)\n        \n        if stackX is None:\n            stackX = pred\n        else:\n            stackX = dstack((stackX, pred))\n   \n    print(\"stackX.shape = \", stackX.shape)\n        \n    # flatten predictions to [rows, members x probabilities]\n    stackX = stackX.reshape((stackX.shape[0], stackX.shape[1]*stackX.shape[2]))\n    return stackX","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import keras\nfrom keras import layers, models","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def make_meta_learner_model(input_shape, output_class_count, dropout):\n    print(input_shape)\n    print(output_class_count)\n    print(dropout)\n    print(input_shape[1] * 2)\n    \n    model = models.Sequential()\n\n    model.add(layers.Dense(units=input_shape[1] * 2, activation='relu', kernel_initializer='he_normal'))\n    model.add(layers.Dropout(dropout))\n    \n    model.add(layers.Dense(units=int(input_shape[1] / 2), activation='relu', kernel_initializer='he_normal'))\n    model.add(layers.Dropout(dropout))\n\n    model.add(layers.Dense(units=int(input_shape[1] / 4), activation='relu', kernel_initializer='he_normal'))\n    model.add(layers.Dropout(dropout))\n    \n    model.add(layers.Dense(units=output_class_count, activation='softmax', kernel_initializer='he_normal'))\n    \n    model.compile(loss=\"categorical_crossentropy\", optimizer='adam', metrics=['acc', recall, precision, f1score])\n    \n    return model\n\n\ndef get_callbacks_for_meta_learner(model_save_filename, patience):\n    es = EarlyStopping(monitor='f1score', min_delta=0, patience=patience, verbose=1, mode='max')\n\n    rr = ReduceLROnPlateau(monitor = 'f1score', factor = 0.5, patience = patience / 2,\n                           min_lr=0.000001,\n                           verbose=1, \n                           mode='max')\n\n    mc = ModelCheckpoint(filepath=model_save_filename, monitor='f1score', verbose=1,\n                           save_best_only=True, mode='max')\n\n    return [es, rr, mc]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.utils import to_categorical\n\nprint(\"Build dataset for meta-learner...\")\nmeta_train_X = None\n\nMETA_LEARNER_DATA_EPOCHS = 1\nfor epoch in range(META_LEARNER_DATA_EPOCHS):\n    print(\"Epoch \", epoch)\n    epochs_x = make_meta_learner_dataset(submodels, df_train, \"train\")\n    if meta_train_X is None:\n        meta_train_X = epochs_x\n    else:\n        meta_train_X = vstack([meta_train_X, epochs_x])\nprint(\"meta_train_X.shape=\", meta_train_X.shape)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from numpy import vstack\n\n# 모델이 훈련된 label값에 맞게 Y값을 만들어야 한다.\nmeta_train_Y = None\nlabels = (datagen_train_flow.class_indices)\ny = df_train['class'].values\ny = [labels[x] for x in y]\ny = to_categorical(y)\nfor i in range(META_LEARNER_DATA_EPOCHS):\n    if meta_train_Y is None:\n        meta_train_Y = y\n    else:\n        meta_train_Y = vstack([meta_train_Y, y])\nprint(\"meta_train_Y.shape=\", meta_train_Y.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def train_meta_learner(X, Y):\n    \n    meta_learner_model_save_filename = (\"meta_learner_model.h5\")\n    meta_learner_model_save_filepath = os.path.join(MODEL_PATH, meta_learner_model_save_filename)\n    \n    print(\"Training meta-learner model...\")\n    meta_learner_model = make_meta_learner_model(X.shape, len(classes), dropout=0.3)\n    meta_learner_model.fit(X, Y, epochs=20, verbose=1, batch_size=128, callbacks=get_callbacks_for_meta_learner(meta_learner_model_save_filepath, 6))\n    \n    meta_learner_model.load_weights(meta_learner_model_save_filepath)\n    \n    return meta_learner_model\n\nmeta_learner_model = train_meta_learner(meta_train_X, meta_train_Y)\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"train이 완료된 모델로 test dataset을 predict해서 submission을 만든다."},{"metadata":{"trusted":true},"cell_type":"code","source":"tta_len = 4\n\npreds = []\nfor i in range(tta_len):\n    meta_submit_X = make_meta_learner_dataset(submodels, df_test, \"test\", tta=True)\n    pred = meta_learner_model.predict(meta_submit_X, batch_size=32)\n    preds.append(pred)\n    \npred = np.mean(np.array(preds), axis=0)\n\nsubmit_Y = np.argmax(pred, axis=1)\nlabels = (datagen_train_flow.class_indices)\nlabels = dict((v,k) for k, v in labels.items())\n\npredictions = [labels[k] for k in submit_Y]\n\nsubmission = pd.DataFrame()\nsubmission['img_file'] = df_test['img_file']\nsubmission[\"class\"] = predictions\nsubmission.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"colab":{"name":"kakr_car_cv.ipynb","version":"0.3.2","provenance":[],"collapsed_sections":[],"toc_visible":true},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.5.2"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"accelerator":"GPU"},"nbformat":4,"nbformat_minor":1}