{"cells":[{"metadata":{},"cell_type":"markdown","source":"# 3rd ML Month : Car Class Classification\n안녕하세요! 두 번째 대회에 이어서 처음으로 이미지 대회에 참가해보는 4학년 재학중인 창원의 학부생입니다!<br>들어가기에 앞서 대회를 개최하고 관리해주신 관리자 분들과 지식을 공유해주신 모든 분들에게 깊은 감사를 드립니다!"},{"metadata":{},"cell_type":"markdown","source":"* 이번 대회에서 25위를 기록한 커널입니다.\n* 파이썬 스크립트로 작성한 것을 시간 관계상 훈련 과정은 빼고\n* 미리 훈련시켜놓은 가중치를 사용해 클래스를 예측하는 과정만 넣었습니다.\n* 참고한 코드는 다음과 같습니다.(많은 공부가 되었습니다. 모두에게 감사드립니다!)\n    * [3rd ML Month baseline code : TaeJin Kim님](https://www.kaggle.com/fulrose/3rd-ml-month-car-model-classification-baseline)\n    * [Car Image Cropping : Taemyung Heo님](https://www.kaggle.com/tmheo74/3rd-ml-month-car-image-cropping-updated-7-10)\n    * [Cutout Augmentation (on Keras EfficientNet) : Chanran Kim님](https://www.kaggle.com/seriousran/cutout-augmentation-on-keras-efficientnet)"},{"metadata":{},"cell_type":"markdown","source":"## 훈련 과정\n### Feature Engineering\n* bounding box 정보가 훈련, 시험 데이터에 다행히 다 있어서 사진들을 전부 차 크기정보만 남도록 Taemyung Heo님의 Cropping을 적용 했습니다.\n* Keras의 ImageDataGenerator를 사용해 data augmentation을 했습니다.\n* Chanran Kim 님의 Cutout Augmentation을 적용해보았습니다."},{"metadata":{},"cell_type":"markdown","source":"### Modeling\n* VGG19, Inception_v3, Resnet50, Xception, EfficientNetB3 모델을 사용해보았습니다.\n* 모델 모두 Imagenet으로 pretrained된 가중치를 사용했습니다.\n* 클래스별로 균등한 양을 추출해 학습하기위해서 Stratified 분리를 적용했습니다.\n* StratifiedKfold 모듈을 사용해 5개의 fold로 분리해 학습을 진행하려고하니 검증용 데이터의 양이 불필요하게 많아진다고 생각해 StratifiedShuffleSplit을 사용하여 train 데이터 90%, val 데이터 10% 비율로 나누도록 했습니다.\n* 게으름을 피운 탓에 많은 실험을 해보지 않았습니다..ㅠ 그 중 가장 성능이 좋았던 efficient_ent_B3 모델을 사용해서 5folds 앙상블링을 해 최종제출을 하였습니다."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# import libraries\nimport gc\nimport os\nimport sys\nimport PIL\nimport warnings\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport keras.backend as K\nfrom tqdm import tqdm\n\nwarnings.filterwarnings('ignore')\n\nfrom keras_preprocessing.image import ImageDataGenerator\nfrom keras import optimizers\nfrom keras.callbacks import ReduceLROnPlateau, EarlyStopping, ModelCheckpoint\nfrom sklearn.model_selection import train_test_split, StratifiedShuffleSplit\n\nfrom keras.layers import Conv2D, Dense, Flatten, GlobalAveragePooling2D, GlobalMaxPool2D, Dropout, MaxPooling2D\nfrom keras.models import Model\n!pip install efficientnet==0.0.4\nfrom keras.applications.resnet50 import ResNet50\nfrom keras.applications.inception_v3 import InceptionV3\nfrom keras.applications.xception import Xception\nfrom efficientnet import EfficientNetB3","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_model(model_name=None, include_top=False, input_shape=(256,256,3), fine_tuning=True, layer_to_freeze=None, load_pretrained : str = None, summary=False):\n    pmodel_name = model_name.strip().lower()\n    print(pmodel_name)\n\n    if pmodel_name == 'resnet50': base_model = ResNet50(include_top=include_top, input_shape=input_shape)\n    elif pmodel_name == 'inception_v3' : base_model = InceptionV3(include_top=include_top, input_shape=input_shape)\n    elif pmodel_name == 'xception' : base_model = Xception(include_top=include_top, input_shape=input_shape)\n    elif pmodel_name == 'efficient_net' : base_model = EfficientNetB3(include_top=include_top, input_shape=input_shape)\n    else : raise ValueError\n\n    if fine_tuning:\n        # Freese layers\n        assert layer_to_freeze != None, 'You must define layer\\'s name to freese.'\n        fr_layer_name = layer_to_freeze\n        set_trainable = False\n\n        for layer in base_model.layers:\n            if not layer.name == fr_layer_name:\n                set_trainable = True\n\n            layer.trainable = set_trainable\n\n    # change last layers\n    last_1dconv_1 = Conv2D(1024, 1, activation='relu')(base_model.output)\n    last_pool_1 = MaxPooling2D(pool_size=(2, 2), strides=(2, 2))(last_1dconv_1)\n    global_avg_pool = GlobalAveragePooling2D()(last_pool_1)\n#     last_Dense_1 = Dense(512, activation='relu')(global_avg_pool)\n    last_Dense_2 = Dense(196, activation='softmax')(global_avg_pool)\n\n    # compile\n    model = Model(base_model.input, last_Dense_2)\n\n    # summary\n    if summary:\n        model.summary()\n\n    # load pretrained weights\n    if load_pretrained:\n        model.load_weights(load_pretrained)\n\n    return model\n\ndef recall_m(y_true, y_pred):\n    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n    possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))\n    recall = true_positives / (possible_positives + K.epsilon())\n    return recall\n\ndef precision_m(y_true, y_pred):\n    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n    predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))\n    precision = true_positives / (predicted_positives + K.epsilon())\n    return precision\n\ndef f1_m(y_true, y_pred):\n    precision = precision_m(y_true, y_pred)\n    recall = recall_m(y_true, y_pred)\n    return 2 * ((precision * recall) / (precision + recall + K.epsilon()))\n\n# Define steps per epoch\ndef get_steps(num_samples, batch_size):\n    if (num_samples // batch_size) > 0:\n        return (num_samples // batch_size) + 1\n    else:\n        return num_samples // batch_size\n\n# https://www.kaggle.com/seriousran/cutout-augmentation-on-keras-efficientnet\ndef get_random_erazer(p=0.5, s_l=0.02, s_h=0.4, r_1=0.3, r_2=1/0.3, v_l=0, v_h=255, pixel_level=False):\n    def erazer(input_img):\n        img_h, img_w, img_c = input_img.shape\n        p_1 = np.random.rand()\n\n        if p_1 > p :\n            return input_img\n\n        while True:\n            s = np.random.uniform(s_l, s_h) * img_h * img_w\n            r = np.random.uniform(r_1, r_2)\n            w = int(np.sqrt(s/r))\n            h = int(np.sqrt(s*r))\n            left = np.random.randint(0, img_w)\n            top = np.random.randint(0, img_h)\n\n            if left + w <= img_w and top + h <= img_h:\n                break\n\n        if pixel_level:\n            c = np.random.uniform(v_l, v_h, (h, w, img_c))\n        else:\n            c = np.random.uniform(v_l, v_h)\n\n        input_img[top:top + h, left:left + w, :] = c\n\n        return input_img\n\n    return erazer\n\n\n# For prediction\ndef predict_class(model, num_samples, batch_size, train_generator, test_generator, output_name, DATA_PATH = '/kaggle/input/2019-3rd-ml-month-with-kakr', OUTPUT_PATH = './output'):\n\n    # Prediction\n    prediction = model.predict_generator(\n        generator=test_generator,\n        steps=get_steps(num_samples, batch_size),\n        verbose=1\n    )\n\n    predicted_indices = np.argmax(prediction, axis=1)\n    labels = (train_generator.class_indices)\n    labels = dict((v, k) for k, v in labels.items())\n    predictions = [labels[k] for k in predicted_indices]\n    \n    # Load submission form\n    submission = pd.read_csv(os.path.join(DATA_PATH, 'sample_submission.csv'))\n    submission['class'] = predictions\n    submission.to_csv(os.path.join(OUTPUT_PATH, '{}.csv'.format(output_name)), index=False)\n\ndef predict_class_ensemble(models, weights, num_samples, batch_size, train_generator, test_generator, DATA_PATH = '/kaggle/input/2019-3rd-ml-month-with-kakr', OUTPUT_PATH = './output'):\n    predictions = []\n\n\n    if not models == None:\n        num_models = len(models)\n\n        if not weights == None:\n            num_weights = len(weights)\n            for model_name in models:\n                for i, weight_name in enumerate(weights):\n                    print('=== predict {0} model - {1}\\'s split ==='.format(model_name, i))\n                    model = build_model(model_name=model_name, input_shape=(299,299,3), fine_tuning=False, summary=False)\n                    model.load_weights(os.path.join('/kaggle/input/3rd-ml-month-efficeintnet-5-folds', weight_name))\n\n                    test_generator.reset()\n\n                    prediction = model.predict_generator(\n                        generator=test_generator,\n                        steps=get_steps(num_samples, batch_size),\n                        verbose=1\n                    )\n\n                    predictions.append(prediction)\n\n        else:\n            for model_name in models:\n                print('=== predict {} model ==='.format(model_name))\n                model = build_model(model_name=model_name, input_shape=(299, 299, 3), fine_tuning=False, summary=False)\n                model.load_weights(os.path.join('/kaggle/input/3rd-ml-month-efficeintnet-5-folds', '{}_model_50_epochs.h5'.format(model_name)))\n\n                # 한번 예측 후에는 반드시 리셋 필수!\n                test_generator.reset()\n\n                prediction = model.predict_generator(\n                    generator=test_generator,\n                    steps=get_steps(num_samples, batch_size),\n                    verbose=1\n                )\n\n                print(np.argmax(prediction, axis=-1)[:10])\n\n                predictions.append(prediction)\n\n        print('Complete!')\n        predictions = np.array(predictions)\n        print(predictions.shape)\n\n        predictions = np.mean(predictions, axis=0)\n\n        # 제출 형식으로 변환\n        predict_indices = np.argmax(predictions, axis=-1)\n        labels = (train_generator.class_indices)\n        labels = dict((v,k) for k,v in labels.items())\n        prediction_ensemble = [labels[k] for k in predict_indices]\n\n        # 제출\n        output_name = 'ensemble_' + ','.join(models) + '_{}_splits.csv'.format(num_weights)\n        submission = pd.read_csv(os.path.join(DATA_PATH, 'sample_submission.csv'))\n        submission['class'] = prediction_ensemble\n        submission.to_csv(os.path.join(OUTPUT_PATH, output_name), index=False)\n        \ndef crop_boxing_img(img, pos, margin=16):\n    width, height = img.size\n    x1 = max(0, pos[0] - margin)\n    y1 = max(0, pos[1] - margin)\n    x2 = min(width, pos[2] + margin)\n    y2 = min(height, pos[3] + margin)\n\n    cropped_img = img.crop((x1,y1,x2,y2))\n    # plt.imshow(cropped_img)\n    # plt.show()\n    return cropped_img","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# set random seed\nRANDOM_SEED = 40\n\n# 데이터 경로 설정\nDATA_PATH = '/kaggle/input/2019-3rd-ml-month-with-kakr'\nOUTPUT_PATH = './cropped_images'\nos.listdir(DATA_PATH)\n\n# 이미지 경로 설정\nTRAIN_IMG_PATH = os.path.join(DATA_PATH, 'train')\nTEST_IMG_PATH = os.path.join(DATA_PATH, 'test')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# generate cropped images\nif not os.path.exists('./output'):\n    os.mkdir('./output')\nif not os.path.exists('./cropped_images'):\n    os.mkdir('./cropped_images')\n    \ndf_train = pd.read_csv(os.path.join(DATA_PATH, 'train.csv'))\ndf_test = pd.read_csv(os.path.join(DATA_PATH, 'test.csv'))\n\nif not os.path.exists('./cropped_images/train_crop'):\n    os.mkdir('./cropped_images/train_crop')\n\nif not os.path.exists('./cropped_images/test_crop'):\n    os.mkdir('./cropped_images/test_crop')\n\n# 훈련 이미지 자르기\nfor i, img_name in tqdm(enumerate(df_train['img_file'])):\n    img = PIL.Image.open(os.path.join(TRAIN_IMG_PATH, img_name))\n    pos = df_train.iloc[i][['bbox_x1', 'bbox_y1', 'bbox_x2', 'bbox_y2']].values.reshape(-1)\n    cropped_img = crop_boxing_img(img, pos)\n    cropped_img.save(os.path.join(OUTPUT_PATH, 'train_crop/'+img_name))\n\n# 시험 이미지 자르기\nfor i, img_name in tqdm(enumerate(df_test['img_file'])):\n    img = PIL.Image.open(os.path.join(TEST_IMG_PATH, img_name))\n    pos = df_test.iloc[i][['bbox_x1', 'bbox_y1', 'bbox_x2', 'bbox_y2']].values.reshape(-1)\n    cropped_img = crop_boxing_img(img, pos)\n    cropped_img.save(os.path.join(OUTPUT_PATH, 'test_crop/' + img_name))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# 데이터 경로 설정\nDATA_PATH = '/kaggle/input/2019-3rd-ml-month-with-kakr'\nOUTPUT_PATH = './output'\nos.listdir(DATA_PATH)\n\n# 이미지 경로 설정\nTRAIN_IMG_PATH = os.path.join('./cropped_images', 'train_crop')\nTEST_IMG_PATH = os.path.join('./cropped_images', 'test_crop')\n\n# 데이터 읽어오기\ndf_train = pd.read_csv(os.path.join(DATA_PATH, 'train.csv'))\ndf_test = pd.read_csv(os.path.join(DATA_PATH, 'test.csv'))\n\n# 모델링을 위한 데이터 준비\ndf_train['class'] = df_train['class'].astype('str')\n\ndf_train = df_train[['img_file', 'class']]\ndf_test = df_test[['img_file']]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Parameters\nimg_size = (299, 299)\nepochs = 50\nbatch_size = 16\nlearning_rate = 0.0002\nbase_model = 'efficient_net'\nload_pretrained = None\npatience = 5\nn_splits=1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Define Generator config\ntrain_datagen = ImageDataGenerator(\n    horizontal_flip = True,\n    vertical_flip = False,\n    zoom_range = 0.10,\n    rotation_range = 20,\n    width_shift_range = 0.2,\n    height_shift_range = 0.2,\n    shear_range=0.5,\n    brightness_range=[0.5, 1.5],\n    fill_mode='nearest',\n    rescale=1./255,\n    preprocessing_function=get_random_erazer(v_l=0, v_h=255)\n)\n\nval_datagen = ImageDataGenerator(rescale=1./255)\ntest_datagen = ImageDataGenerator(rescale=1./255)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Train, Test spllit\nsplitter = StratifiedShuffleSplit(n_splits=n_splits, train_size=0.9, test_size=0.1, random_state=RANDOM_SEED)\n\nscores = []","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# 학습\nfor i, (trn_idx, val_idx) in enumerate(splitter.split(df_train['img_file'], df_train['class'])):\n\n    print('============', '{}\\'th Split'.format(i), '============\\n')\n\n    X_train = df_train.iloc[trn_idx].copy()\n    X_val = df_train.iloc[val_idx].copy()\n    print('Train : {0} / Test : {1}'.format(X_train.shape, X_val.shape))\n\n    train_size = len(X_train)\n    val_size = len(X_val)\n\n    train_generator = train_datagen.flow_from_dataframe(\n        dataframe = X_train,\n        directory = TRAIN_IMG_PATH,\n        x_col = 'img_file',\n        y_col = 'class',\n        target_size = img_size,\n        color_mode = 'rgb',\n        class_mode = 'categorical',\n        batch_size = batch_size,\n        seed=RANDOM_SEED,\n        shuffle=True,\n        interploation = 'bicubic'\n    )\n\n    val_generator = val_datagen.flow_from_dataframe(\n        dataframe = X_val,\n        directory = TRAIN_IMG_PATH,\n        x_col = 'img_file',\n        y_col = 'class',\n        target_size = img_size,\n        color_mode = 'rgb',\n        class_mode = 'categorical',\n        batch_size = batch_size,\n        seed=RANDOM_SEED,\n        shuffle=True,\n        interploation = 'bicubic'\n    )\n\n    test_generator = test_datagen.flow_from_dataframe(\n        dataframe = df_test,\n        directory = TEST_IMG_PATH,\n        x_col = 'img_file',\n        y_col = None,\n        target_size = img_size,\n        color_mode = 'rgb',\n        class_mode = None,\n        batch_size = batch_size,\n        shuffle=False\n    )\n\n    # 훈련 루틴\n    \"\"\"\n    model = build_model(model_name=base_model, input_shape=(299, 299, 3), fine_tuning=False, summary=False)\n\n    # Load pretrained\n    if load_pretrained is not None:\n        print('Loading Pretrained networks {}'.format(load_pretrained))\n        model.load_weights(os.path.join(os.getcwd(), './weights/' + load_pretrained))\n\n\n    optimizer = optimizers.adam(lr=learning_rate)\n\n    model.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['acc', f1_m])\n\n    if not os.path.exists('./weights'):\n        os.mkdir('./weights')\n\n    # checkpoint_filename = 'inception_v3_model_{0}_epochs.h5'.format(epochs)\n    if n_splits > 1:\n        checkpoint_filename = '{0}_model_{1}_epochs_split_{2}.h5'.format(base_model,epochs, i)\n    else:\n        checkpoint_filename = '{0}_model_{1}_epochs.h5'.format(base_model, epochs)\n\n\n    # Define callbacks\n    reduce_lr = ReduceLROnPlateau(\n        monitor='val_loss',\n        factor=0.5,\n        patience=int(patience / 2),\n        verbose=1,\n        mode='min',\n        min_lr=0.0000001\n    )\n\n    early_stopping = EarlyStopping(\n        monitor='val_loss',\n        min_delta=0.000001,\n        patience=patience,\n        verbose=1,\n        mode='min'\n    )\n\n    model_checkpoint = ModelCheckpoint(\n        filepath=os.path.join('./weights', checkpoint_filename),\n        monitor='val_loss',\n        verbose=1,\n        save_best_only=True,\n        save_weights_only=True,\n        mode='min',\n        period=5\n    )\n\n    callbacks = [reduce_lr, early_stopping, model_checkpoint]\n\n    # 훈련\n    history = model.fit_generator(\n        generator = train_generator,\n        steps_per_epoch = get_steps(train_size, batch_size),\n        epochs = epochs,\n        callbacks = callbacks,\n        validation_data = val_generator,\n        validation_steps = get_steps(val_size, batch_size)\n    )\n    \"\"\"\n    ## 단일모델 예측\n    # Predict class\n    # predict_class(model, df_test.shape[0], batch_size, train_generator, test_generator, checkpoint_filename[:-3])\n\n    # Predict ensemble\n    # models = ['inception_v3', 'resnet50', 'xception', 'efficient_net']\n    weights = ['efficient_net_model_50_epochs_split_0.h5',\n               'efficient_net_model_50_epochs_split_1.h5',\n               'efficient_net_model_50_epochs_split_2.h5',\n               'efficient_net_model_50_epochs_split_3.h5',\n               'efficient_net_model_50_epochs_split_4.h5']\n    predict_class_ensemble(['efficient_net'], weights, df_test.shape[0], batch_size, train_generator, test_generator)\n\n    gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# output file이 지나지게 많아지기때문에 전처리한 사진은 삭제해주겠습니다.\n!rm -r cropped_images","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}