{"cells":[{"metadata":{},"cell_type":"markdown","source":"<h1><b>2019 3rd ML month with KaKR Solution (LB 13th)</b></h1>\n<br>\n3차 대회를 참가하여 진행했던 내용들을 정리해보았습니다.<br>\n태명님이 제공해주신 crop 데이터셋과 아래 커널들을 참고했습니다.<br>\n이상하거나 잘못된 내용이 있을수 있습니다. 의견 주시면 감사하겠습니다 ^^<br>\n<br>\n*참고커널 <br>\nhttps://www.kaggle.com/fulrose/3rd-ml-month-car-model-classification-baseline <br>\nhttps://www.kaggle.com/tmheo74/3rd-ml-month-car-image-cropping-updated-7-10 <br>\nhttps://www.kaggle.com/easter3163/3rd-ml-month-keras-efficientnet <br>\n"},{"metadata":{},"cell_type":"markdown","source":"# 1. Overview\n## 1.1 Network Model"},{"metadata":{},"cell_type":"markdown","source":"처음에 어떤 모델을 골라야 할지 몰라서 이 사이트를 참조하였습니다.<br>\nhttps://paperswithcode.com/sota/image-classification-on-imagenet <br>\n<img src=\"http://cruiserx.cafe24.com/kaggle/1.1.imagenet-sota.png\"><br>\n모델 종류도 참 많고 각각 특성이 있겠지만 EfficientNet이 상대적으로 메모리를 적게쓰면서도 성능이 괜찮은것 같습니다. <br>\n그래서 이번 대회에서는 EfficientNet중에서 B3,B4,B5를 사용했습니다. B7을 쓰고 싶었지만 단계마다 시간이 두배씩 증가하더군요. <br>\n여력이 있다면 한가지 모델만 시도하는 것보다 두세개 모델을 시험해보는 것이 좋겠고 앙상블 하는것이 좋은것 같습니다."},{"metadata":{},"cell_type":"markdown","source":"## 1.2 Preprocessing\n주어진 이미지셋에서 박스로 crop하고, 그 안에서도 차 이외의 요소를 제거하는걸 고민해서 아래처럼 전처리를 했었습니다.<br>\nhttps://www.kaggle.com/cruiserx/3rd-ml-month-car-image-segmentation-crop\n"},{"metadata":{},"cell_type":"markdown","source":"<img src=\"http://cruiserx.cafe24.com/kaggle/1.2.image-preprocessing.png?123\">"},{"metadata":{},"cell_type":"markdown","source":"그런데 점수를 비교해보니 box로 crop만 한 경우에 비해서 주변을 잘라낸게 점수가 더 낮게 나옵니다. <br>\n차 주변의 요소들도 모델의 분류 예측력에 영향을 미치는 것 같습니다.<br>\n<br>\n비율을 유지하도록 패딩을 하는 것도 안하는 것에 비해서 점수가 낮게 나옵니다.<br>\n패딩 색상에 따라서도 점수가 다르게 나오는데요, 랜덤/검은색/흰색 보다 회색으로 패딩하는게 점수가 더 높았습니다.<br>\n생각나는 이런저런 전처리를 시도해보았지만 점수가 나아지지 않아서 그냥 태명님의 crop dataset을 이용하였습니다.<br>\n\n아래 그림들은 이런저런 전처리를 해봤던 내용입니다. 등수에 도움은 안되었지만요..<br>\n<img src=\"http://cruiserx.cafe24.com/kaggle/1.2.image-preprocessing2.png\"><br>\n<img src=\"http://cruiserx.cafe24.com/kaggle/1.2.image-preprocessing3.png\"><br>"},{"metadata":{},"cell_type":"markdown","source":"## 1.3 Stratified Split/OverSampling\n\n- 층화추출(Stratified Sampling)<br>\n train-set과 validation-set을 70:30으로 나누기로 정하고<br>\n 각각 클래스별로 같은 비율로 뽑히도록 stratify 옵션을 주었습니다.<br>\n 이 옵션을 주지 않으면 어떤 클래스는 많고 어떤 클래스는 적게 추출됩니다.<br>\n\n>  \n    train_test_split(df_train, df_train['class'], test_size=validation_ratio, random_state=1005, \\\n                     stratify=df_train['class'])\n\n- 오버샘플링(Over-Sampling)<br>\n<img src=\"http://cruiserx.cafe24.com/kaggle/1.3.imbalance-data.png\"><br>\n 클래스 별로 데이터 분포를 보면 데이터가 적은 클래스와 많은 클래스간에 차이가 있는 imbalance dataset입니다.<br>\n 처리하는데는 오버샘플링도 있고 언더샘플링도 있고 여러가지 방법들이 있습니다만, 저는 오버샘플링 방법을 이용하였습니다.<br>\n 데이터를 버리기엔 좀 아깝다는 생각도 들었고, train시 제너레이터가 Augmentation을 해주므로,<br>\n 동일한 데이터를 채워넣는것에 비해서는 나쁘진 않겠다 생각했습니다."},{"metadata":{},"cell_type":"markdown","source":"## 1.4 Augmentation\n- Train Time<br>\n 처음 참고했던 커널의 제너레이터 파라미터를 기준점으로 놓고 범위를 조금씩 바꿔가면서 점수가 높게 나오는걸로<br>\n 골랐습니다. 시험해본것들 중에서 가장 나았던건 아래와 같습니다.<br>\n\n>  \n    aug_datagen = ImageDataGenerator( \n                        rotation_range=10,  \n                        width_shift_range=0.1,  \n                        height_shift_range=0.1, \n                        horizontal_flip=True,  \n                        vertical_flip=False,  \n                        zoom_range=0.1,  \n                        shear_range=0.2,  \n                        fill_mode='nearest'  \n                  )  \n\n- Test Time<br>\n 학습이 완료된 모델로 test데이터를 가공하는데 augmentation을 수행한 결과물들을 조합하면 점수가 오릅니다.<br>\n 수동으로 어떤 부분을 변형할지 지정하는 방법도 있지만, 이미 학습할때 썼던 제너레이터가 있으니 SEED를 바꿔가며 활용하였습니다.<br>\n\n>  \n    test_generator2 = aug_datagen.flow_from_dataframe(\n                           dataframe=df_test,\n                           directory=TEST_IMG_PATH,\n                           x_col='img_file',\n                           y_col=None,\n                           target_size= (image_size,image_size),\n                           color_mode='rgb',\n                           class_mode=None,\n                           batch_size=batch_size,\n                           shuffle=False,\n                           seed=2011 + seq\n                     )   \n\n"},{"metadata":{},"cell_type":"markdown","source":"## 1.5 Model Training\n\n> \nEfficientNet B3, B4, B5를 시드를 바꿔가서 train set을 만들어서 학습시켰습니다. <br>\n대체로 B4가 잘 맞는것 같습니다만, 메모리 한계로 B5는 배치사이즈를 줄여서 돌렸으니 <br>\n정확한 비교는 안될것 같습니다. <br>\n<br>\n\n\nOptimizer는 처음에는 rmsprop을 사용하다가 몇가지 시험해보고 그중에 점수가 잘 나오던 adamax로 바꿨습니다.<br>\n<br>\n기준 metric이 f1이라서 아래처럼 함수를 정의해서 썼지만,<br>\n\n>\ndef f1_m(y_true, y_pred):\n    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n    possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))\n    predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))\n    recall = true_positives / (possible_positives + K.epsilon())\n    precision = true_positives / (predicted_positives + K.epsilon())\n    return 2*((precision*recall)/(precision+recall+K.epsilon()))\n\n<br>\n내장함수가 아니었기 때문인지..학습이 훨씬 더 오래 걸리는 문제가 있었습니다. <br>\nf1에 맞춰서 돌리면 점수가 좀 낫다는 글을 보긴 했지만, 시간상 할수없이accuracy를 사용했습니다.<br>\n<br>\n아래는 모델들을 학습시키면서 기록해두었던 점수와 수행시간입니다.<br>\n<br>\n- <b>EfficientNet B3 (10 set)</b>\n<br>\n<br>\n    - SplitSeed 1000 : val_acc= 0.9032 val_f1_m= 0.9066 (7182s)\n    - SplitSeed 1001 : val_acc= 0.9169 val_f1_m= 0.9189 (8044s)\n    - SplitSeed 1002 : val_acc= 0.9102 val_f1_m= 0.912  (8856s)\n    - SplitSeed 1003 : val_acc= 0.9129 val_f1_m= 0.9165 (6844s)\n    - SplitSeed 1004 : val_acc= 0.9126 val_f1_m= 0.9148 (7685s)\n    - SplitSeed 1005 : val_acc= 0.9146 val_f1_m= 0.916  (7274s)\n    - SplitSeed 1006 : val_acc= 0.9209 val_f1_m= 0.924  (9083s)\n    - SplitSeed 1007 : val_acc= 0.9166 val_f1_m= 0.9199 (8838s)\n    - SplitSeed 1008 : val_acc= 0.9146 val_f1_m= 0.9173 (8148s)\n    - SplitSeed 1009 : val_acc= 0.9069 val_f1_m= 0.9094 (5886s)\n<br>\n<br>\n- <b>EfficientNet B4 (20 set)</b>\n<br>\n<br>\n    - SplitSeed 1000 : val_acc= 0.9109 val_f1_m= 0.9141 (8100s)\n    - SplitSeed 1001 : val_acc= 0.9243 val_f1_m= 0.9278 (10902s)\n    - SplitSeed 1002 : val_acc= 0.9266 val_f1_m= 0.9283 (14297s)\n    - SplitSeed 1003 : val_acc= 0.9239 val_f1_m= 0.9265 (10237s)\n    - SplitSeed 1004 : val_acc= 0.9196 val_f1_m= 0.9218 (12780s)\n    - SplitSeed 1005 : val_acc= 0.9266 val_f1_m= 0.9287 (13675s)\n    - SplitSeed 1006 : val_acc= 0.9343 val_f1_m= 0.9364 (14473s)\n    - SplitSeed 1007 : val_acc= 0.9299 val_f1_m= 0.9323 (15496s)\n    - SplitSeed 1008 : val_acc= 0.9246 val_f1_m= 0.9267 (13735s)\n    - SplitSeed 1009 : val_acc= 0.9249 val_f1_m= 0.9261 (15751s)\n    - SplitSeed 1010 : val_acc= 0.9306 val_f1_m= 0.933  (13088s)\n    - SplitSeed 1011 : val_acc= 0.9243 val_f1_m= 0.9274 (11652s)\n    - SplitSeed 1012 : val_acc= 0.9229 val_f1_m= 0.9238 (13940s)\n    - SplitSeed 1013 : val_acc= 0.9276 val_f1_m= 0.9306 (14321s)\n    - SplitSeed 1014 : val_acc= 0.9249 val_f1_m= 0.9263 (14511s)\n    - SplitSeed 1015 : val_acc= 0.9206 val_f1_m= 0.9222 (13666s)\n    - SplitSeed 1016 : val_acc= 0.9152 val_f1_m= 0.9198 (11838s)\n    - SplitSeed 1017 : val_acc= 0.9213 val_f1_m= 0.9231 (12300s)\n    - SplitSeed 1018 : val_acc= 0.9243 val_f1_m= 0.9276 (15369s)\n    - SplitSeed 1019 : val_acc= 0.9253 val_f1_m= 0.9272 (10911s)\n<br>\n<br>\n- <b>EfficientNet B5 (10 set)</b>\n<br>\n<br>\n    - SplitSeed 1000 : val_acc= 0.9183 val_f1_m= 0.9213 (26633s)\n    - SplitSeed 1001 : val_acc= 0.9229 val_f1_m= 0.925  (30866s)\n    - SplitSeed 1002 : val_acc= 0.9266 val_f1_m= 0.929  (31631s)\n    - SplitSeed 1003 : val_acc= 0.9152 val_f1_m= 0.9174 (25085s)\n    - SplitSeed 1004 : val_acc= 0.9223 val_f1_m= 0.9247 (25701s)\n    - SplitSeed 1005 : val_acc= 0.9273 val_f1_m= 0.9292 (28102s)\n    - SplitSeed 1006 : val_acc= 0.9156 val_f1_m= 0.9182 (23569s)\n    - SplitSeed 1007 : val_acc= 0.9246 val_f1_m= 0.9265 (26997s)\n    - SplitSeed 1008 : val_acc= 0.9193 val_f1_m= 0.9198 (25223s)\n    - SplitSeed 1009 : val_acc= 0.9279 val_f1_m= 0.9289 (22363s)\n    "},{"metadata":{},"cell_type":"markdown","source":"## 1.6 Model Ensemble\n생성한 모델은 총 40개, 각 모델별로 test-time에 augmentation하여 얻어진 prediction 10개.<br>\n전부 합치면 400개의 prediction이 나옵니다.<br>\n<br>\n구해진 prediction들을 단순평균하여 최종 prediction을 계산하였고, <br>\n<b>avg_submission_40set.csv</b><br>\n<br>\n이를 DNN에 넣어서 최종 prediction을 조합해보기도 하였습니다.<br>\n<b>dnn_submission_40set.csv</b><br>\n<br>\n좀 더 다양한 모델들을 결합하였다면 더 나은 점수를 얻지 않았을까하는 아쉬움이 있습니다.  "},{"metadata":{},"cell_type":"markdown","source":"## 1.7 Partial Kernel\n정리하는 차원에서 커널을 하나로 작성했습니다만, 전처리 따로 학습은 모델별로 각각 인퍼런스도 각각 앙상블도 각각 작은 커널을 만들어서 진행하였습니다.<br>\n캐글환경만 사용하다 보니 9시간 제약도 있고 길게 돌다가 에러가 나면 중간결과물을 전부 잃어버리게 되니 타격이 너무 큽니다.<br>\n그래서 과정별로 학습한 weight, train prediction, test prediction, augmented test prediction은 모두 파일로 저장하여 재활용했습니다.<br>\n"},{"metadata":{},"cell_type":"markdown","source":"## 1.8 Commit Monitoring\n커널 코드를 작성해서 커밋하는데 길게는 9시간까지 돌아야하는데, 얼마나 돌고 있는지 알수없는건 참 답답한 일이었습니다<br>\n구글링으로 찾은 로깅방법을 활용해서 모니터링 정보를 커밋로그에 남기는 팁을 디스커션에 남겨놓았습니다.<br>\n다들 활용하셨으면 좋겠습니다.<br>\nhttps://www.kaggle.com/c/2019-3rd-ml-month-with-kakr/discussion/103725#latest-597942\n<img src=\"http://cruiserx.cafe24.com/kaggle/1.8.commit-monitoring.png\">"},{"metadata":{},"cell_type":"markdown","source":"# 2. Common Task"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Efficient Net 설치\n!pip3 install keras_efficientnets","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Import\nimport pandas as pd\nimport numpy as np\nimport warnings\nimport seaborn as sns\nimport matplotlib.pylab as plt\nimport PIL\nimport os\nimport h5py\nimport datetime\nfrom pytz import timezone, utc\nfrom keras import backend as K\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.applications import Xception\nfrom keras.callbacks import EarlyStopping, ReduceLROnPlateau, ModelCheckpoint, Callback\nfrom keras import layers, models, optimizers, utils\nfrom sklearn.model_selection import train_test_split\n\nwarnings.filterwarnings('ignore')\n\n# Efficient Net\nfrom keras_efficientnets import EfficientNetB3\nfrom keras_efficientnets import EfficientNetB4\nfrom keras_efficientnets import EfficientNetB5","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Utility functions\n\n# commit 메시지에 로그를 남길수 있게 합니다\ndef print2(string):  \n    os.system(f'echo \\\"{string}\\\"')\n    print(string)\n\n# dataframe을 한줄에 여러개 출력하게 합니다.\nfrom IPython.display import display_html\ndef display_side_by_side(*args):\n    html_str=''\n    for df in args:\n        html_str+=df.to_html()\n    display_html(html_str.replace('table','table style=\"display:inline\"'),raw=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# cropped 데이터셋 확인\n!ls -l ../input/3rd-ml-month-car-image-cropping-dataset","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"IMG_PATH = '../input/3rd-ml-month-car-image-cropping-dataset'\nTRAIN_IMG_PATH = os.path.join(IMG_PATH, 'train_crop')\nTEST_IMG_PATH = os.path.join(IMG_PATH, 'test_crop')\n\nDATA_PATH = '../input/2019-3rd-ml-month-with-kakr'\ndf_train = pd.read_csv(os.path.join(DATA_PATH, 'train.csv'))\ndf_test = pd.read_csv(os.path.join(DATA_PATH, 'test.csv'))\ndf_class = pd.read_csv(os.path.join(DATA_PATH, 'class.csv'))\n\ndf_train['class'] = df_train['class'].astype('str')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"display_side_by_side(df_train.head(), df_test.head() )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Model Training\n40개의 모델을 트레이닝할떄 아래코드들에서 모델클래스와 변수,랜덤 시드만 바꿔가며 실행했습니다.<br>\n시간이 많이 걸리므로 이 커널에서는 중간에 끝나도록 처리해두었습니다. "},{"metadata":{"trusted":true},"cell_type":"code","source":"#학습할 모델에 대한 정보\nmodel_class = EfficientNetB3\nimage_size = 299\nbatch_size = 16\n#model_class = EfficientNetB4\n#image_size = 380\n#batch_size = 16\n#model_class = EfficientNetB5\n#image_size = 456\n#batch_size = 8\nRND_SEED = 1005\nbasename = 'b3d2ovs'\n\n#실제 실행시에는 False로\nJUST_TEST = True","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#train_test_split\nfrom sklearn.model_selection import train_test_split\n\nvalidation_ratio = 0.3\nnb_train_data = df_train.shape[0]\nnb_test_data = df_test.shape[0]\n\nnb_validation_sample = nb_train_data * validation_ratio\nnb_train_sample = nb_train_data - nb_validation_sample\n\nX_train, X_valid, y_train, y_valid = \\\n    train_test_split(df_train, df_train['class'], test_size=validation_ratio, \\\n                     random_state=RND_SEED, \\\n                     stratify=df_train['class'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"min_sample = min(X_train.groupby('class')['img_file'].count())\nmax_sample = max(X_train.groupby('class')['img_file'].count())\nprint(min_sample,',',max_sample)\n\n## over sampling.\n\nmax_samples = max(X_train.groupby('class')['img_file'].count())\nXX_train = None\ndisplay(X_train.describe())\n#display(X_train['class'].unique())\nfor cls in X_train['class'].unique():\n    df = X_train[X_train['class']==cls]\n    df_size = df['img_file'].count()\n\n    #부족한 정수배만큼 복제하여 채움\n    for idx in range(max_samples // df_size):\n        if (XX_train is None):\n            XX_train = df\n        else:\n            XX_train = XX_train.append(df)\n    \n    #나머지는 비복원 랜덤추출\n    if (max_samples % df_size > 0):\n        dfx = df.sample(n=max_samples % df_size, replace=False, random_state=int(cls)+1000)\n        if (XX_train is None):\n            XX_train = dfx\n        else:\n            XX_train = XX_train.append(dfx)\n\ndisplay(XX_train.describe())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"min_sample = min(XX_train.groupby('class')['img_file'].count())\nmax_sample = max(XX_train.groupby('class')['img_file'].count())\nprint(min_sample, ',', max_sample)\n\nX_train = XX_train\ny_train = XX_train['class']\nprint(nb_train_sample)\nnb_train_sample = X_train.shape[0]\nprint(nb_train_sample)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# split count check\ndisplay_side_by_side(df_train[['img_file','class']].groupby('class').count().head(3), \\\n                    X_train[['img_file','class']].groupby('class').count().head(3), \\\n                    X_valid[['img_file','class']].groupby('class').count().head(3) )\n\n# split data check\ndisplay_side_by_side(df_train[['img_file','class']].groupby('class').max().head(3), \\\n                      X_train[['img_file','class']].groupby('class').max().head(3), \\\n                      X_valid[['img_file','class']].groupby('class').max().head(3) )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_datagen = ImageDataGenerator(\n    rotation_range=10,\n    width_shift_range=0.1,\n    height_shift_range=0.1,\n    horizontal_flip=True,\n    vertical_flip=False,\n    zoom_range=0.1,\n    shear_range=0.2,\n    fill_mode='nearest'\n    )\nvalid_datagen = ImageDataGenerator()\ntest_datagen = ImageDataGenerator()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# 분할된 train_set 학습에 쓰는 generator\ntrain_generator = train_datagen.flow_from_dataframe( \n    dataframe=X_train, \n    directory=TRAIN_IMG_PATH, \n    x_col=\"img_file\", y_col=\"class\", \n    target_size=(image_size, image_size),\n    batch_size=batch_size,\n    class_mode='categorical',\n    seed=2019,\n    color_mode='rgb'\n)\n# 분할된 validation_set 학습에 쓰는 generator\nvalidation_generator = train_datagen.flow_from_dataframe( \n    dataframe=X_valid, \n    directory=TRAIN_IMG_PATH, \n    x_col=\"img_file\", y_col=\"class\", \n    target_size=(image_size, image_size),\n    batch_size=batch_size,\n    class_mode='categorical',\n    seed=2019,\n    color_mode='rgb'\n)\n# test_data predict에 쓰는 generator\ntest_generator = test_datagen.flow_from_dataframe(\n    dataframe=df_test,\n    directory=TEST_IMG_PATH,\n    x_col='img_file',\n    y_col=None,\n    target_size= (image_size,image_size),\n    color_mode='rgb',\n    class_mode=None,\n    batch_size=batch_size,\n    shuffle=False\n)\n# train_data predict에 쓰는 generator\nfulltrain_generator = test_datagen.flow_from_dataframe(\n    dataframe=df_train,\n    directory=TRAIN_IMG_PATH,\n    x_col='img_file',\n    y_col=\"class\",\n    target_size=(image_size, image_size),\n    color_mode='rgb',\n    class_mode='categorical',\n    batch_size=batch_size,\n    shuffle=False\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def f1_m(y_true, y_pred):\n    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n    possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))\n    predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))\n    recall = true_positives / (possible_positives + K.epsilon())\n    precision = true_positives / (predicted_positives + K.epsilon())\n    return 2*((precision*recall)/(precision+recall+K.epsilon()))\n\ndef get_model():\n    base_model = model_class(weights='imagenet', include_top=False, input_shape=(image_size, image_size, 3))\n    # base_model.trainable = False\n\n    model = models.Sequential()\n    model.add(base_model)\n    model.add(layers.GlobalAveragePooling2D())\n    model.add(layers.Dense(1024, activation='relu'))\n    model.add(layers.Dropout(0.5))\n    model.add(layers.Dense(1024, activation='relu'))\n    model.add(layers.Dropout(0.5))\n    model.add(layers.Dense(196, activation='softmax'))\n    model.summary()\n\n    #optimizer = optimizers.SGD(lr=1e-4, decay=1e-6, momentum=0.9, nesterov=True)\n    #optimizer = optimizers.RMSprop(lr=0.0001)\n    optimizer = optimizers.Adamax()\n    model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['acc', f1_m])\n\n    return model\n\nmodel = get_model()\nmodel_path = './{}{}.hdf5'.format(basename, RND_SEED)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_steps(num_samples, batch_size):\n    if (num_samples % batch_size) > 0:\n        return (num_samples // batch_size) + 1\n    else:\n        return num_samples // batch_size\n    \n# Train중에 로그를 출력하는 CallBack\nKST = timezone('Asia/Seoul')\nclass EpochLogWrite(Callback):\n    def on_train_begin(self, logs={}):\n        None\n    def on_batch_begin(self, batch, logs={}):\n        None\n    def on_epoch_begin(self, epoch, logs={}):\n        tmx = utc.localize(datetime.datetime.utcnow()).astimezone(KST).time()\n        print2('Epoch #{} begins at {}'.format(epoch+1, tmx))\n    def on_train_end(self, logs={}):\n        None\n    def on_batch_end(self, batch, logs={}):\n        None\n    def on_epoch_end(self, epoch, logs={}):\n        tmx = utc.localize(datetime.datetime.utcnow()).astimezone(KST).time()\n        print2('Epoch #{} ends at {}  acc={} val_acc={} val_f1={}'.format(epoch+1, tmx, round(logs['acc'],4), round(logs['val_acc'],4), round(logs['val_f1_m'],4) ))\n\npatient = 2\ncallbacks1 = [\n    EpochLogWrite(),\n    EarlyStopping(monitor='val_acc', patience=patient, mode='max', verbose=1),\n    ReduceLROnPlateau(monitor = 'val_acc', factor = 0.5, patience = patient / 2, min_lr=0.00001, verbose=1, mode='max'),\n    ModelCheckpoint(filepath=model_path, monitor='val_acc', verbose=1, save_best_only=True, mode='max'),\n    ]\n\ncallbacks2 = [\n    EpochLogWrite(),\n    EarlyStopping(monitor='val_f1_m', patience=patient, mode='max', verbose=1),\n    ReduceLROnPlateau(monitor = 'val_f1_m', factor = 0.5, patience = patient / 2, min_lr=0.00001, verbose=1, mode='max'),\n    ModelCheckpoint(filepath=model_path, monitor='val_f1_m', verbose=1, save_best_only=True, mode='max'),\n    ]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if JUST_TEST==True:\n    history = model.fit_generator(\n        train_generator,\n        steps_per_epoch=10, #get_steps(nb_train_sample, batch_size),\n        epochs=3,\n        validation_data=validation_generator,\n        validation_steps=10, #get_steps(nb_validation_sample, batch_size),\n        verbose=1,\n        callbacks = callbacks1\n    )    \nelse:\n    history = model.fit_generator(\n        train_generator,\n        steps_per_epoch=get_steps(nb_train_sample, batch_size),\n        epochs=100,\n        validation_data=validation_generator,\n        validation_steps=get_steps(nb_validation_sample, batch_size),\n        verbose=1,\n        callbacks = callbacks1\n    )\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('val_acc=', np.round(history.history['val_acc'][-1], 4))\nprint('val_f1_m=', np.round(history.history['val_f1_m'][-1], 4))\n\nacc = history.history['acc']\nval_acc = history.history['val_acc']\nloss = history.history['loss']\nval_loss = history.history['val_loss']\n\nepochs = range(1, len(acc) + 1)\n\nplt.plot(epochs, acc, 'bo', label='Training Acc')\nplt.plot(epochs, val_acc, 'b', label='Validation Acc')\nplt.title('Training and validation accuracy')\nplt.legend()\nplt.show()\n\nplt.plot(epochs, loss, 'bo', label='Traing loss')\nplt.plot(epochs, val_loss, 'b', label='Validation loss')\nplt.title('Trainging and validation loss')\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#학습한 모델로 test_data를 prediction하고 싱글모델 제출파일을 만듭니다. \nmodel.load_weights(model_path)\ntest_generator.reset()\n\nprediction = model.predict_generator(\n    generator=test_generator,\n    steps = get_steps(nb_test_data, batch_size),\n    verbose=1\n)\n\npredicted_class_indices=np.argmax(prediction, axis=1)\n\n# Generator class dictionary mapping\nlabels = (train_generator.class_indices)\nlabels = dict((v,k) for k,v in labels.items())\npredictions = [labels[k] for k in predicted_class_indices]\n\nsub0=pd.DataFrame({'img_file': df_test['img_file'], 'class':predictions})\nif JUST_TEST==False:\n    sub0.to_csv('./single_submit_{}{}.csv'.format(basename, RND_SEED), index=False)\n\nsub0.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#나중에 앙상블에 쓰기 위해서 학습한 모델로 train_data전체를 prediction합니다. \nfulltrain_prediction = model.predict_generator(\n    generator=fulltrain_generator,\n    steps = get_steps(nb_train_data, batch_size),\n    verbose=1\n)\n\npredicted_class_indices=np.argmax(fulltrain_prediction, axis=1)\n\n# Generator class dictionary mapping\nlabels = (fulltrain_generator.class_indices)\nlabels = dict((v,k) for k,v in labels.items())\npredictions = [labels[k] for k in predicted_class_indices]\n\nmidout=pd.DataFrame({'img_file': df_train.img_file, 'class':df_train['class'], 'predicted':predictions})\nmidout.to_csv('./fulltrain_{}{}.csv'.format(basename, RND_SEED), index=False)\n\nmidout.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#prediction 결과를 h5파일에 저장합니다.\nhf = h5py.File('./{}{}_data.h5'.format(basename,RND_SEED), 'w')\nhf.create_dataset('full_pred', data=fulltrain_prediction)\nhf.create_dataset('test_pred0', data=prediction)\nhf.close()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. Augmented Test Prediction\n윗단계에서 학습할때 생성한 웨이트 파일을 불러와서 Augmentation된 Test Prediction을 9세트를 생성합니다.\n트레이닝과 마찬가지로 시간이 꽤 걸리므로 일부만 처리하도록 해두었습니다. "},{"metadata":{"trusted":true},"cell_type":"code","source":"WEIGHT_PATH = '../input/3rd-ml-month-efficientnet-weights'\n\n# weight path 확인\n!ls -l ../input/3rd-ml-month-efficientnet-weights | head -6","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"aug_datagen = ImageDataGenerator(\n    rotation_range=10,\n    width_shift_range=0.1,\n    height_shift_range=0.1,\n    horizontal_flip=True,\n    vertical_flip=False,\n    zoom_range=0.1,\n    shear_range=0.2,\n    fill_mode='nearest'\n    )\norg_datagen = ImageDataGenerator()\n\narr_test_generator = []\ntest_generator = org_datagen.flow_from_dataframe(\n    dataframe=df_test,\n    directory=TEST_IMG_PATH,\n    x_col='img_file',\n    y_col=None,\n    target_size= (image_size,image_size),\n    color_mode='rgb',\n    class_mode=None,\n    batch_size=batch_size,\n    shuffle=False\n)\narr_test_generator.append(test_generator)\n\nfor idx in range(9):\n  test_generator2 = aug_datagen.flow_from_dataframe(\n      dataframe=df_test,\n      directory=TEST_IMG_PATH,\n      x_col='img_file',\n      y_col=None,\n      target_size= (image_size,image_size),\n      color_mode='rgb',\n      class_mode=None,\n      batch_size=batch_size,\n      shuffle=False,\n      seed=2011 + idx\n  )\n  arr_test_generator.append(test_generator2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = get_model()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if JUST_TEST == True:\n    model_seeds = [1000]\nelse:\n    model_seeds = range(1000,1010)\n\nfor idx in model_seeds:\n    fname = '{}/{}{}.hdf5'.format(WEIGHT_PATH,basename,idx)\n    fname_out = './{}{}_data2.h5'.format(basename,idx)\n    print2('#Model: {} => {}'.format(fname,fname_out))\n    tmx = utc.localize(datetime.datetime.utcnow()).astimezone(KST).time()\n    print2('  - Load begins at {}'.format(tmx))\n    model.load_weights(fname)\n    tmx = utc.localize(datetime.datetime.utcnow()).astimezone(KST).time()\n    print2('  - Load ends at {}'.format(tmx))\n  \n    arr_predictions = []\n    for idx2 in range(1,10):\n        print2('  - Make test prediction #{}'.format(idx2+1) )\n        arr_test_generator[idx2].reset()\n        \n        if JUST_TEST==True:        \n            prediction = model.predict_generator(\n                generator=arr_test_generator[idx2],\n                steps = 4, #get_steps(nb_test_data, batch_size),\n                verbose=1\n            )\n        else:\n            prediction = model.predict_generator(\n                generator=arr_test_generator[idx2],\n                steps = get_steps(nb_test_data, batch_size),\n                verbose=1\n            )\n            \n        arr_predictions.append(prediction)\n    \n    print2('  - Write dataset')\n    hf = h5py.File(fname_out, 'w')\n    for idx2 in range(9):\n        datasetname = 'test_pred{}'.format(idx2+1)\n        hf.create_dataset(datasetname, data=arr_predictions[idx2])\n    hf.close()\n  \n    print(' ')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 5. Average Ensemble Model \n트레이닝시에 생성한 원본test prediction, 윗단계에 생성한 augmented prediction을 불러와서 앙상블합니다.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"PREDICTION_PATH1 = '../input/3rd-ml-month-efficientnet-predictions-1'\nPREDICTION_PATH2 = '../input/3rd-ml-month-efficientnet-predictions-2'\n\n# weight path 확인\n!ls -l ../input/3rd-ml-month-efficientnet-predictions-1 | head -6\n!ls -l ../input/3rd-ml-month-efficientnet-predictions-2 | head -6\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#처리해놓은 파일들을 불러온다.\narr_predictions = []\nfor basenm, rng in [('b3d2ovs',range(1000,1010)), ('b4d2ovs',range(1000,1020)), ('b5d2ovs', range(1000,1010))]:\n    for idx in rng:\n        fname_out = '{}/{}{}_data.h5'.format(PREDICTION_PATH1, basenm, idx)\n        print('load: ', fname_out)\n        hf = h5py.File(fname_out, 'r')\n        #print('  keys = ',list(hf.keys()))\n        n1 = hf.get('test_pred0')\n        n1 = np.array(n1)\n        arr_predictions.append(n1)\n        hf.close()\n\n        fname_out = '{}/{}{}_data2.h5'.format(PREDICTION_PATH2, basenm, idx)\n        print('load: ', fname_out)\n        hf = h5py.File(fname_out, 'r')\n        #print('  keys = ',list(hf.keys()))\n        for idx2 in range(9):\n            datanm = 'test_pred{}'.format(idx2+1)\n            n1 = hf.get(datanm)\n            n1 = np.array(n1)\n            #print(datanm,n1.shape)\n            arr_predictions.append(n1)\n        hf.close()        \n        ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# prediction average\navg_prediction = None\nfor idx in range(len(arr_predictions)):\n    #print(idx)\n    if (avg_prediction is None):\n        avg_prediction = arr_predictions[idx]\n    else:\n        avg_prediction = avg_prediction + arr_predictions[idx]\navg_prediction = avg_prediction / len(arr_predictions)   ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"avg_predicted_class_indices=np.argmax(avg_prediction, axis=1)\n\n# Generator class dictionary mapping\nlabels = (fulltrain_generator.class_indices)\nlabels = dict((v,k) for k,v in labels.items())\navg_predictions = [labels[k] for k in avg_predicted_class_indices]\n\nsub0=pd.DataFrame({'img_file': df_test['img_file'], 'class':avg_predictions})\nsub0.to_csv('avg_submission_40set.csv', index=False)\n\nsub0.head()\n!ls -l *.csv","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 6. DNN Ensemble Model \n트레이닝시에 모델별로 생성해두었던 40set의 full-train prediction을 hstack하여 입력으로 하여 클래스를 예측하는 모델을 만들어 훈련시킵니다.<br>\n이 모델에 test prediction을 hstack한 입력, augmented set별로 prediction을 hstack한 입력을 각각 넣어 출력된 prediction 10개를 평균하여 최종 prediction을 구합니다."},{"metadata":{"trusted":true},"cell_type":"code","source":"#처리해놓은 파일들을 불러온다.\n\narr_fullpredictions = []\nfor basenm, rng in [('b3d2ovs',range(1000,1010)), ('b4d2ovs',range(1000,1020)), ('b5d2ovs', range(1000,1010))]:\n    for idx in rng:\n        fname_out = '{}/{}{}_data.h5'.format(PREDICTION_PATH1, basenm, idx)\n        print('load: ', fname_out)\n        hf = h5py.File(fname_out, 'r')\n        n1 = hf.get('full_pred')\n        n1 = np.array(n1)\n        arr_fullpredictions.append(n1)\n        #print(n1.shape)\n        hf.close()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# 열방향으로 데이터 합침\nstacked_set = np.hstack(arr_fullpredictions)\nprint(stacked_set.shape)\nstacked_set2 = stacked_set.reshape((-1, len(arr_fullpredictions), 196))\nprint(stacked_set2.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train2 = df_train.copy()\ndf_train2['class'] = df_train2['class'].astype(str)\n\ntest_datagen = ImageDataGenerator()\nfulltrain_generator = test_datagen.flow_from_dataframe(\n    dataframe=df_train2,\n    directory=TRAIN_IMG_PATH,\n    x_col='img_file',\n    y_col=\"class\",\n    target_size=(image_size, image_size),\n    color_mode='rgb',\n    class_mode='categorical',\n    batch_size=batch_size,\n    shuffle=False\n)\n\ndef batch_generator(X, y, batch_size=16):\n    #Return a random row from X, y\n\n    labels = (fulltrain_generator.class_indices)\n    meta_train_Y = y\n    meta_train_Y = [labels[x] for x in meta_train_Y]\n    meta_train_Y = utils.to_categorical(meta_train_Y)\n    \n    while True:\n        # choose batch_size random rows / labels from the data\n        idx = np.random.randint(0, X.shape[0], batch_size)\n        rows = X[idx]\n        enc_labels = meta_train_Y[idx]\n        \n        yield rows, enc_labels\n        \n#print(fulltrain_generator.class_indices)        ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras import backend\n\ndef get_model(obj):\n  # Create a keras model\n  \n  '''\n  #cnn model\n  input = layers.Input(shape=(obj.shape[1], obj.shape[2],))\n  input2 = layers.Reshape((len(arr_fullpredictions),196,1)) (input)\n\n  node1 = layers.Conv2D(1, (len(arr_fullpredictions), 1), data_format='channels_last') (input2)\n  \n  drop1 = layers.Dropout(0.5)(node1)\n  drop1 = layers.Flatten()(drop1)\n\n  node2 = layers.Dense(196, activation='relu')(drop1)\n  node2 = layers.Dropout(0.5)(node2)\n  node3 = layers.Dense(196, activation='softmax')(node2)\n  \n  model = models.Model(inputs=input, outputs=node3)\n  \n  model.summary()\n  '''\n\n\n  #'''\n  #dnn model\n  input = layers.Input(shape=(obj.shape[1], obj.shape[2],))\n  input2 = layers.Flatten()(input)\n  node1 = layers.Dense(196, activation='selu')(input2)\n  drop1 = layers.Dropout(0.5)(node1)\n  node2 = layers.Dense(196, activation='selu')(drop1)\n  drop2 = layers.Dropout(0.5)(node2)\n  node3 = layers.Dense(196, activation='softmax')(drop2)\n\n  model = models.Model(inputs=input, outputs=node3)\n \n  model.summary()\n  #'''\n\n  \n  '''\n  #dnn model2\n  input = layers.Input(shape=(obj.shape[1], obj.shape[2],))\n  input2 = layers.Flatten()(input)\n  node1 = layers.Dense(196*4, activation='relu')(input2)\n  drop1 = layers.Dropout(0.5)(node1)\n  node2 = layers.Dense(196*4, activation='relu')(drop1)\n  drop2 = layers.Dropout(0.5)(node2)\n  node3 = layers.Dense(196, activation='softmax')(drop2)\n\n  model = models.Model(inputs=input, outputs=node3)\n \n  model.summary()\n  '''\n  \n  optimizer = optimizers.Adamax()\n  model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['acc', f1_m])  #\n\n  return model\n\nmodel = get_model(stacked_set2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Train중에 로그를 출력하는 CallBack\nKST = timezone('Asia/Seoul')\nclass EpochLogWrite(Callback):\n    def on_train_begin(self, logs={}):\n        None\n    def on_batch_begin(self, batch, logs={}):\n        None\n    def on_epoch_begin(self, epoch, logs={}):\n        tmx = utc.localize(datetime.datetime.utcnow()).astimezone(KST).time()\n        print2('Epoch #{} begins at {}'.format(epoch+1, tmx))\n    def on_train_end(self, logs={}):\n        None\n    def on_batch_end(self, batch, logs={}):\n        None\n    def on_epoch_end(self, epoch, logs={}):\n        tmx = utc.localize(datetime.datetime.utcnow()).astimezone(KST).time()\n        print2('Epoch #{} ends at {}  acc={} val_acc={} val_f1={}'.format(epoch+1, tmx, round(logs['acc'],4), round(logs['val_acc'],4), round(logs['val_f1_m'],4) ))\n        \npatient = 2\ncallbacks1 = [\n    EpochLogWrite(),\n    EarlyStopping(monitor='val_acc', patience=patient, mode='max', verbose=1),\n    ReduceLROnPlateau(monitor = 'val_acc', factor = 0.5, patience = patient / 2, min_lr=0.00001, verbose=1, mode='max'),\n    ModelCheckpoint(filepath='./dnn_model.hdf5', monitor='val_acc', verbose=1, save_best_only=True, mode='max'),\n    ]\n\ncallbacks2 = [\n    EpochLogWrite(),\n    EarlyStopping(monitor='val_f1_m', patience=patient, mode='max', verbose=1),\n    ReduceLROnPlateau(monitor = 'val_f1_m', factor = 0.5, patience = patient / 2, min_lr=0.00001, verbose=1, mode='max'),\n    ModelCheckpoint(filepath='./dnn_model.hdf5', monitor='val_f1_m', verbose=1, save_best_only=True, mode='max'),\n    ]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(y_valid.values.shape)\nprint(np.zeros(df_test.shape[0]).shape )\nprint(df_test.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nX_train, X_valid, y_train, y_valid = \\\n    train_test_split(stacked_set2, df_train2['class'], test_size=validation_ratio, \\\n                     random_state=1005, \\\n                     stratify=df_train['class'])\n\n\nprint(X_train.shape)\nprint(y_train.shape)\nprint(X_valid.shape)\nprint(y_valid.shape)\n\ntrain_generator = batch_generator(X_train, y_train.values, batch_size=batch_size)\nvalid_generator = batch_generator(X_valid, y_valid.values, batch_size=batch_size)\n#test_generator = testbatch_generator(stacked_set, np.zeros(df_test.shape[0]), batch_size=batch_size)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model.fit_generator(\n    train_generator,\n    steps_per_epoch = get_steps(nb_train_sample, batch_size),\n    epochs=100,\n    validation_data=valid_generator,\n    validation_steps = get_steps(nb_validation_sample, batch_size),\n    verbose=1,\n    callbacks = callbacks1\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.load_weights('./dnn_model.hdf5')\n\narr_gross_predictions = []\nfor midx in range(10):\n    print('### Round: #{}'.format(midx))\n  \n    arr_predictions = []\n    for basenm, rng in [('b3d2ovs',range(1000,1010)), ('b4d2ovs',range(1000,1020)), ('b5d2ovs', range(1000,1010))]:\n        for idx in rng:\n            if (midx==0):\n                fname_out = '{}/{}{}_data.h5'.format(PREDICTION_PATH1, basenm, idx)\n                #print('load: ', fname_out)\n                hf = h5py.File(fname_out, 'r')\n                n1 = hf.get('test_pred0')\n                n1 = np.array(n1)\n                arr_predictions.append(n1)\n                hf.close()\n            else:\n                fname_out = '{}/{}{}_data2.h5'.format(PREDICTION_PATH2, basenm, idx)\n                #print('load: ', fname_out)\n                hf = h5py.File(fname_out, 'r')\n                n1 = hf.get('test_pred{}'.format(midx))\n                n1 = np.array(n1)\n                arr_predictions.append(n1)\n                hf.close()\n\n    # 열방향으로 데이터 합침\n    stacked_tset = np.hstack(arr_predictions)\n    print(stacked_tset.shape)\n    stacked_tset2 = stacked_tset.reshape((-1, len(arr_predictions), 196))\n    print(stacked_tset2.shape)\n\n    dnn_prediction = model.predict(stacked_tset2)\n    arr_gross_predictions.append(dnn_prediction)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"avg_prediction = None\nfor idx in range(len(arr_gross_predictions)):\n    print('### Round: #{}'.format(idx))\n    if (avg_prediction is None):\n        avg_prediction = arr_gross_predictions[idx]\n    else:\n        avg_prediction = avg_prediction + arr_gross_predictions[idx]\navg_prediction = avg_prediction / len(arr_gross_predictions)   ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"avg_predicted_class_indices=np.argmax(avg_prediction, axis=1)\n\n# Generator class dictionary mapping\nlabels = (fulltrain_generator.class_indices)\nlabels = dict((v,k) for k,v in labels.items())\navg_predictions = [labels[k] for k in avg_predicted_class_indices]\n\nsub0=pd.DataFrame({'img_file': df_test['img_file'], 'class':avg_predictions})\nsub0.to_csv('dnn_submission_40set.csv', index=False)\n\nsub0.head()\n","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}