{"cells":[{"metadata":{},"cell_type":"markdown","source":"# <center>3rd ML Month - 18th solution </center>"},{"metadata":{},"cell_type":"markdown","source":"## <a id='0'><strong>운영진에 감사 말씀드립니다.</strong></a>\n좋은 데이터로 재미난 대회를 만들고 이끌어주신 운영진께 감사 말씀드립니다.\n\n덕분에 기술적으로 많은 것을 배웠고 특히 **<u>\"세상은 넓고 고수는 많다\"</u>**는 것을 다시 한번 느끼게 되었습니다.\n\n특히 얼마전 캐글 입문 강의로 수고해주신 태진님께 다시 한번 감사 말씀드립니다.      \n       \n앞으로도 이런 좋은 대회가 이어지면 좋겠습니다.\n       \n(비록 자랑할만한 등수는 아니지만 고수님들의 커널을 보고 배우며 삽질했던 내용을     \n공유하면 좋을 것 같아서 남깁니다.)\n"},{"metadata":{},"cell_type":"markdown","source":"### **Model Summary**\n- Resnext101 8 Folds로 최종 제출했고 대회 종료 이틀전에 큰 오류가 있음을 깨닫고 급하게 수정하고<br>\n  학습시키다보니 다른 모델을 제대로 앙상블 시킬 시간은 없었습니다.\n- Public 27위→Private 18위로 올라선 것은 TTA+8folds로 Generalization이 됐기 때문인듯합니다.  \n       ① 전처리  : Cropping , Histogram Equalization\n       ② Augmentation : cutout , rotation 30, horizonal=True, zoom:0.3, rescale :1/255\n       ③ Loss Function : categorical_crossentropy\n       ④ optimizer : Adam optimizer\n       ⑤ Basic Model : Resnext101(imagenet pretrained), 8 folds (평균 5.5hr/1fold 소요)\n       ⑥ Inference 시 TTA(5) 적용\n  "},{"metadata":{},"cell_type":"markdown","source":"## <strong>삽질의 기록</strong>\n**1.  Resnet50 + cropping + sigle fold** (public score 0.8345) <br>\n**2.  Resnet50 + cropping + 5folds** (public score 0.91439) <br>\n       - 학습이 제대로 되지 않고 있는 것으로 판단하고 뜯어보기 시작\n           . learning rate : 0.001 → 0.0001\n           . Dropout(0.25) 추가\n**3.  Resnet50 + cropping + 5folds** (public score 0.92061) <br>\n        - 깊은 모델(resnet50 → resnext101)로 변경하기로 결정\n        - resnet101,resnext101 모델은 keras.applications가 아닌 keras_applications 모듈에서\n          가져와야함. keras가 처음이라 이걸 몰라서 삽질을 오래함.\n        - keras_applications 에서 resnext101를 부르면 아래 에러가 생기는데,\n          모델을 부를 때 arguments를 넣어 줘야함(backend, layers, models, utils)         \n         \"AttributeError: 'NoneType' object has no attribute 'image_data_format'\"\n         →해결책 : http://donghao.org/2019/02/22/using-resnext-in-keras-2-2-4/\n        - 다음에 마주하게 되는 에러가 scaling과 관련된 에러인데 이는 preprocess_input을 활용하기\n          때문에 발생하는 것으로 Data Augmentation을 할 때 scale : 1/255로 변경하면 해결됨.\n**4.  Resnext101 + cropping + 5folds** (public score 0.93591) <br>    \n        - early stopping을 f1 score에 걸어두어서인지 val_loss가 충분히 떨어지지 않았는데\n          학습이 종료되는 현상이 발생함(underfitting) \n          . patience = 5 → 10 변경\n          . ReduceLROnPlateau(factor 0.5) 적용\n              → 뒤에 설명하겠지만 막판에 여기에 오류가 있다는 것을 알게되어 학습을 다시함.\n         - 학습 시간이 3배로 늘어나면서 Training / Inference kernel을 분리하기 시작.\n         \n**5.  Resnext101 + cropping + 5folds+ReduceLR** (public score 0.94787)          \n         - train Loss가 0에 가깝고 acc도 100에 가까운 반면, Val acc는 98% 수준으로 \n           Overfitting 되고 있다고 판단 (실제 Private score 0.943으로 Drop)\n           . Mixup 적용(0.2)했으나 오히려 Public score 0.01이상 drop\n           . overfitting 된 model 5개(+ Xception, resnet101, resnet50, Efficient_v3)의 결과물로\n             Voting을 수행하여 Public score 0.95074를 만들었으나 private score는 저조(0.946)\n             → Voting 보다 softmax 결과물들의 평균내어 argmax를 취하는게 더 성능이 좋음\n         - ** 5일간 여러 삽질을 하는 도중 두가지 큰 오류가 있음을 확인 **\n            . ReduceLROnPlateau 오류 : f1 score를 바라보게 해두고 mode를 default(min)으로 설정되어<br>\n            (patience 10) f1 score와 관계 없이 계속 절반씩 줄고 있었음. \n            . Earlystopping 오류 : monitoring을 val_f1_score가 아닌 f1_score로 설정해두어<br>\n              validation set이 아닌 train set의 f1 score가 더이상 개선이 없을때 끝나도록 되어 있었음<br>\n              사실상 Earlystopping 기능을 수행하지 못하고 Overfitting되고 있음\n**6.  (오류수정) Resnext101 + cropping + 8folds+ReduceLR** (public score 0.95217)\n        - 위 오류를 모두 수정하여 학습하기 시작, 대회 종료까지 하루밖에 남지 않은 상황이라<br>\n            한번에 학습 기회가 있을 듯하여 folds 수도 늘려서 조금이라도 overfitting을 줄여보고자함.\n        - 학습이 종료된 이후에 voting을 수행하여 score를 조금더 올렸으나 private score로 보면<br>\n          voting 하기 전이 더 잘 나옴\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"import gc\nimport os \nimport warnings\nwarnings.filterwarnings(action='ignore')\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom tqdm import tqdm # 진행 상태 표시as\nfrom keras import backend as K\nK.image_data_format() # 채널 first 인지 last인지 여부 판단\n# Image visualization\n\nimport PIL\nfrom PIL import ImageDraw","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"path = '../input/2019-3rd-ml-month-with-kakr'\nos.listdir(path)\n# 이미지 폴더 경로 \ntrain_img_path = os.path.join(path,'train')\ntest_img_path = os.path.join(path,'test')\n# csv 파일 경로\ndf_train = pd.read_csv(os.path.join(path,'train.csv'))\ndf_test = pd.read_csv(os.path.join(path,'test.csv'))\ndf_class = pd.read_csv(os.path.join(path,'class.csv'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#crop\n\ndef crop_boxing_img(img_name, margin=-4, size=(224,224)):\n    if img_name.split('_')[0] == 'train':\n        PATH = train_img_path\n        data = df_train\n    else:\n        PATH = test_img_path\n        data = df_test\n\n    img = PIL.Image.open(os.path.join(PATH, img_name))\n    pos = data.loc[data[\"img_file\"] == img_name, ['bbox_x1', 'bbox_y1', 'bbox_x2', 'bbox_y2']].values.reshape(-1)\n\n    width, height = img.size\n    x1 = max(0, pos[0] - margin)\n    y1 = max(0, pos[1] - margin)\n    x2 = min(pos[2] + margin, width)\n    y2 = min(pos[3] + margin, height)\n\n    return img.crop((x1, y1, x2, y2)).resize(size)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"TRAIN_CROPPED_PATH = '../cropped_train'\nTEST_CROPPED_PATH = '../cropped_test'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if (os.path.isdir(TRAIN_CROPPED_PATH) == False):\n    os.mkdir(TRAIN_CROPPED_PATH)\n\nif (os.path.isdir(TEST_CROPPED_PATH) == False):\n    os.mkdir(TEST_CROPPED_PATH)\n\nfor i, row in df_train.iterrows():\n    cropped = crop_boxing_img(row['img_file'])\n    cropped.save(os.path.join(TRAIN_CROPPED_PATH, row['img_file']))\n\nfor i, row in df_test.iterrows():\n    cropped = crop_boxing_img(row['img_file'])\n    cropped.save(os.path.join(TEST_CROPPED_PATH, row['img_file']))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Set Path of Preprocessed Train Images\nTRAIN_IMG_PREP_PATH = os.path.join('..', 'train_prep')\nif not os.path.exists(TRAIN_IMG_PREP_PATH):\n    os.makedirs(TRAIN_IMG_PREP_PATH, exist_ok=True)\n\n# Set Path of Preprocessed Test Images\nTEST_IMG_PREP_PATH = os.path.join('..', 'test_prep')\nif not os.path.exists(TEST_IMG_PREP_PATH):\n    os.makedirs(TEST_IMG_PREP_PATH, exist_ok=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def img_he_pad(img_file_name, add_padding=True):\n    if img_file_name.split('_')[0] == 'train':\n        IMG_CROP_PATH = TRAIN_CROPPED_PATH\n        data = df_train\n    elif img_file_name.split('_')[0] == 'test':\n        IMG_CROP_PATH = TEST_CROPPED_PATH\n        data = df_test\n        \n    # --- Histogram Equalization --- #\n    img = cv2.imread(os.path.join(IMG_CROP_PATH, img_file_name))\n    img_y_cr_cb = cv2.cvtColor(img, cv2.COLOR_BGR2YCrCb)\n    y, cr, cb = cv2.split(img_y_cr_cb)\n\n    # Equalize y (CLAHE (Contrast Limited Adaptive Histogram Equalization))\n    clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(6,6))\n    y_eq = clahe.apply(y)\n    img_y_cr_cb_eq = cv2.merge((y_eq, cr, cb))\n    img_bgr_eq = cv2.cvtColor(img_y_cr_cb_eq, cv2.COLOR_YCR_CB2BGR)\n    img_prep = img_bgr_eq\n\n    # --- Convert BGR To RGB (Just On cv2) ---\n    # b, g, r = cv2.split(img_bgr_eq)\n    # img_prep = cv2.merge((r,g,b))\n    \n    # -------- Add Padding --------- #\n    if add_padding:\n        img_prep_h, img_prep_w = img_prep.shape[0], img_prep.shape[1]  # (height, width)\n        ratio = float(IMG_SIZE) / max(img_prep_h, img_prep_w)\n        shape_no_padding = (int(img_prep_h * ratio), int(img_prep_w * ratio))\n\n        img_prep_no_padding = cv2.resize(img_prep, shape_no_padding[::-1])\n        \n        size_h = IMG_SIZE - shape_no_padding[0]\n        size_w = IMG_SIZE - shape_no_padding[1]\n        \n        top, bottom = size_h // 2, size_h - (size_h // 2)\n        left, right = size_w // 2, size_w - (size_w // 2)\n\n        PADDING_COLOR = (0, 0, 0)  # black\n        img_prep = cv2.copyMakeBorder(\n            img_prep_no_padding,\n            top,\n            bottom,\n            left,\n            right,\n            cv2.BORDER_CONSTANT,\n            value=PADDING_COLOR\n        )\n    \n    return img_prep","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import cv2\nIMG_SIZE = 224\n# Save Preprocessed Train Images (Path: ../train_prep)\nif not os.listdir(TRAIN_IMG_PREP_PATH):  # If PATH_IMG_TRAIN_PREP is empty\n    for idx, row in df_train.iterrows():\n        img_file_name = row['img_file']\n        img_prep = img_he_pad(img_file_name, add_padding=True)\n        cv2.imwrite(os.path.join(TRAIN_IMG_PREP_PATH, img_file_name), img_prep)\n\n# Save Preprocessed Test Images (Path: ../test_prep)\nif not os.listdir(TEST_IMG_PREP_PATH):\n    for idx, row in df_test.iterrows():\n        img_file_name = row['img_file']\n        img_prep = img_he_pad(img_file_name, add_padding=True)\n        cv2.imwrite(os.path.join(TEST_IMG_PREP_PATH, img_file_name), img_prep)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Define Function For Test\ndef test_he_padding(img_file_name):\n    # Show Cropped Image\n    img_crop = PIL.Image.open(os.path.join(TRAIN_CROPPED_PATH, img_file_name))\n    plt.figure(figsize=(12, 9))\n    plt.subplot(1, 2, 1)\n    plt.title(f'Cropped Image - {img_file_name}')\n    plt.imshow(img_crop)\n    plt.axis('off')\n\n    # Show Preprocessed Image\n    img_he_pad = PIL.Image.open(os.path.join(TRAIN_IMG_PREP_PATH, img_file_name))\n    plt.subplot(1, 2, 2)\n    plt.title(f'Historgram Equalized Cropped Image(Add Padding) - {img_file_name}')\n    plt.imshow(img_he_pad)\n    plt.axis('off')\n    \n    # Show Result\n    plt.show()\n    \n# Test Histogram Equalization & Add Padding\ntest_he_padding(img_file_name=df_train['img_file'].iloc[114])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import f1_score\n\ndef f1_metric(y_true, y_pred):\n\n    def recall(y_true, y_pred):\n        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n        possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))\n        recall = (true_positives + K.epsilon()) / (possible_positives + K.epsilon())\n        return recall\n\n    def precision(y_true, y_pred):\n        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n        predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))\n        precision = (true_positives + K.epsilon()) / (predicted_positives + K.epsilon())\n        return precision\n\n    precision = precision(y_true, y_pred)\n    recall = recall(y_true, y_pred)\n    return 2*((precision*recall)/(precision+recall+K.epsilon()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras_applications.resnext import ResNeXt101#, preprocess_input\n# from keras.applications.resnet_v2. import ResNet50, preprocess_input\n# from keras_applications.resnet import ResNet101, preprocess_input\nfrom keras.preprocessing.image import ImageDataGenerator\n\n# Parameter\nimg_size = (224, 224)\nepochs = 70\nbatch_size =16\n\n# Define Generator config\ntrain_datagen = ImageDataGenerator(\n    rotation_range=30,\n    horizontal_flip = True, \n    vertical_flip = False,\n    #zoom_range=0.30,\n    #width_shift_range=0.2,\n    #height_shift_range=0.2,\n    #shear_range=0.5,\n    brightness_range=[0.5, 1.5],\n    fill_mode='nearest',\n    rescale=1./255)\n    #preprocessing_function=preprocess_input)\n\nval_datagen = ImageDataGenerator(rescale=1./255)#preprocessing_function=preprocess_input)\ntest_datagen = ImageDataGenerator(rescale=1./255)#preprocessing_function=preprocess_input)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.models import Sequential, Model\nfrom keras.layers import Dense, Dropout, Flatten, Activation, Conv2D, GlobalAveragePooling2D\nfrom keras import layers, models, optimizers, utils, backend,regularizers\n# import keras\ndef get_model(model_name='ResNeXt101'):\n    resNet_model = ResNeXt101(include_top= False, input_shape = (224,224,3)\n                            , backend =backend, layers=layers, models = models,\n                             utils = utils\n                            )\n    # resNet_model.summary()\n    \n    model = Sequential()\n    model.add(resNet_model)\n    model.add(GlobalAveragePooling2D())\n    model.add(layers.Dropout(0.25))  # 과적합 줄여보기\n    model.add(Dense(196, activation='softmax', kernel_initializer='he_normal', kernel_regularizer=regularizers.l2(l=0.01)))\n#     model.add(Dense(196, activation='softmax', kernel_initializer='he_normal'))\n#     model.add(Dense(196, kernel_initializer='he_normal', kernel_regularizer=regularizers.l2(l=0.01)))\n#     model.add(LeakyReLU(alpha=0.01))    \n    model.summary()\n    adam = optimizers.Adam(lr=0.0001, beta_1=0.9, beta_2=0.999, epsilon=1e-8)\n    model.compile(optimizer = adam,loss = 'categorical_crossentropy', metrics=['acc',f1_metric])\n    # compile 할때 넣어줘야지 아래에서 early stopping 할때 사용 가능하다\n    return model\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_steps(num_samples,batch_size):\n    if (num_samples % batch_size)>0:\n        return (num_samples // batch_size) + 1\n    else :\n        return num_samples // batch_size","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import StratifiedKFold\nk_folds = 5\nkfold =StratifiedKFold(n_splits = k_folds, random_state = 1990)\ndf_train['class'] = df_train['class'].astype('str')\ndf_train= df_train[['img_file','class']]\ndf_test = df_test[['img_file']]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nfile=[]\nfrom keras.callbacks import ModelCheckpoint, EarlyStopping,ReduceLROnPlateau\nfor idx,(train_index, valid_index) in enumerate(kfold.split(\n                                df_train['img_file'],df_train['class'])):\n    if idx != 0 : continue # 여러번 나눠서 돌리기 위함\n   # if idx == 1 : continue\n    #if idx == 2 : continue\n    #if idx == 3 : continue    \n    traindf = df_train.iloc[train_index,:].reset_index()\n#     validdf = df_train.iloc[valid_index,:].reset_index()\n#     traindf.to_csv('%s_traindf'%idx,index=False)\n#     validdf.to_csv('%s_validdf'%idx,index=False)\n    \n    nb_train_samples = len(traindf)\n#     nb_validation_samples = len(validdf)\n#     nb_test_samples = len(df_test)\n    # Make Generator\n    train_generator = train_datagen.flow_from_dataframe(\n        dataframe=traindf, \n        directory=TRAIN_CROPPED_PATH,#'../input/train/',\n        x_col = 'img_file',\n        y_col = 'class',\n        target_size = img_size,\n        color_mode='rgb',\n        class_mode='categorical',\n        batch_size=batch_size,\n        seed=42\n    )\n\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ntest_generator = test_datagen.flow_from_dataframe(\n            dataframe=df_test,\n            directory=TEST_IMG_PREP_PATH,#TEST_CROPPED_PATH,#'../input/test',\n            x_col='img_file',\n            y_col=None,\n            target_size= img_size,\n            color_mode='rgb',\n            class_mode=None,\n            batch_size=batch_size,\n            shuffle=False\n            )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"path = '../input/models10/'\nlst = os.listdir(path)\nprint(lst)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\n%%time\ntta_steps = 5 #models5 기준 1회 0.950 ,5회 0.949 , 10회 0.9507\nprediction = []\nfor i, name in enumerate(lst):\n    preds =[]\n    print(name)\n    model = get_model()\n    model.load_weights(os.path.join(path,name))\n    for j in tqdm(range(tta_steps)):                \n        test_generator.reset()\n        nb_test_samples = len(df_test)\n        pred = model.predict_generator(\n            generator = test_generator,\n            steps = get_steps(nb_test_samples, batch_size),\n            verbose=1\n            )\n        preds.append(pred)\n        \n        gc.collect()\n#         print(np.mean(preds,axis=0))    \n    pd.DataFrame(np.mean(preds,axis=0)).to_csv('%s.csv'%i, index= False)\n#     prediction.append(np.mean(preds,axis=0)) \n    del preds\n    gc.collect()\nfor i, name in enumerate(lst):\n    prediction.append(np.array(pd.read_csv('%s.csv'%i)))\n# print(prediction)\ny_pred = np.mean(prediction,axis=0)\nprint(y_pred)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"path = '../input/2019-3rd-ml-month-with-kakr'\npreds_class_indices = np.argmax(y_pred,axis =1)\nlabels = (train_generator.class_indices)\nlabels = dict((v,k) for k,v in labels.items())\nfinal_pred = [labels[k] for k in preds_class_indices]\n\nsubmission = pd.read_csv(os.path.join(path, 'sample_submission.csv'))\nsubmission[\"class\"] = final_pred\nsubmission.to_csv(\"submission_rev03.csv\", index=False)\nsubmission.head()\n              ","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}