{"cells":[{"metadata":{},"cell_type":"markdown","source":"# <center>3rd ML Month - 14th solution </center>"},{"metadata":{},"cell_type":"markdown","source":"# 14th solution - [CNN-Stacking](https://www.kaggle.com/kimtaegwan/14th-solution?scriptVersionId=18985950)\n- [inference discussion](https://www.kaggle.com/c/imaterialist-challenge-fashion-2018/discussion/57934#latest-535771)\n\n### [Model composition Notebooks](https://www.kaggle.com/kimtaegwan/14th-plasc-solution?scriptVersionId=18841530) \n#### [](http://)\n\n    EfficientNetB4[Public Score = 0.95600] \n     - Model training(v143, v144, v145, v146, v147)\n    Xception[Public Score = 0.94787]\n     - Model training(v230, v231, v233, v234, v239)\n    Resnet50[Public Score = 0.92682]\n     - Model training(v247, 249, v249)\n \n### CNN structure\n    Input(shape=(None, number of models, number of classes,1))    \n    Conv2D(filters=8,  kernelsize=(1, 1)) \n    Conv2D(filters=16, kernelsize=(1, 1)) \n    Dropout(0.5)\n    Conv2D(filters=32, kernelsize=(2, 1)) \n    Dropout(0.5)\n    Conv2D(filters=64, kernelsize=(2, 1)) \n    Dropout(0.5)\n    Flatten() \n    Dense(1024) \n    Dense(196, activation='softmax')\n \n### 5-fold\n - validation set shape - (14594, 3, 196, 1)\n - Training this model with Kfold (k=5) on the validation set,\n - Optimizer - AdamAccumulate[Public Score = 0.95791]\n - Optimizer = adam [Public Score = 0.96174]"},{"metadata":{},"cell_type":"markdown","source":"# Notebooks update\n### [Version1](https://www.kaggle.com/kimtaegwan/keras-baseline-0-924?scriptVersionId=17248066)[Public Score = 0.811] \n 1. EfficientNet_basic\n\n\n### [Version3](https://www.kaggle.com/kimtaegwan/keras-baseline-0-924?scriptVersionId=17253347)[Public Score = 0.828] \n 1. Data Augmentation \n\n\n### [Version8](https://www.kaggle.com/kimtaegwan/keras-baseline-0-924?scriptVersionId=17280984)[Public Score = 0.924] \n 1. Callbacklist(EarlyStopping,  ReduceLROnPlateau,  ModelCheckpoint)\n 2. ayers.Dense(1024) -> layers.Dense(2048)\n\n\n### [Version9](https://www.kaggle.com/kimtaegwan/keras-baseline-0-950?scriptVersionId=17364507)[Public Score = 0.950] - Weighted average ensemble through cross validation\n    1. **fold1** : [Public 0.903]\n    2. **fold2** : [Public 0.923]\n    3. **fold3** : [Public 0.916]\n    4. **fold4** : [Public 0.926]\n    5. **fold5** : [Public 0.926]\n    - **Ensemble** [Public 0.950]\n    \n### [Version19](https://www.kaggle.com/kimtaegwan/keras-baseline-0-951?scriptVersionId=17546328)[Public Score = 0.951] \n 1. 5fold -> 6fold\n\n\n### [Version25](https://www.kaggle.com/kimtaegwan/keras-efficientnet-0-954?scriptVersionId=17638584)[Public Score = 0.954]  \n 1. Cutout Augmentation\n 2. TTA\n\n\n### [Version122](https://www.kaggle.com/kimtaegwan/keras-efficientnet-0-954?scriptVersionId=18301737)[Public Score = 0.955]  \n 1. Model(EfficientNetB3 -> EfficientNetB4)\n 2. imgsize(299 -> 380)\n 3. Version119, Version120  Model training\n\n\n### [Version135](https://www.kaggle.com/kimtaegwan/keras-efficientnet-0-954?scriptVersionId=18466939)[Public Score = 0.956] - Semi-supervised Learning\n   1. [Pseudo Label dataset](https://www.kaggle.com/kimtaegwan/keras-efficientnet-0-954/output?scriptVersionId=18353645) \n   2. Version132, Version133 Model training\n    3. [Keras Semi-supervised Learning Notebooks](https://www.kaggle.com/kimtaegwan/keras-semi-supervised-learning)\n    \n    \n### [Version135](https://www.kaggle.com/kimtaegwan/keras-efficientnet-0-954?scriptVersionId=18567231)[Public Score = 0.958] \n 1. Transfer Learning, \n 2. Optimizer( adam -> AdamAccumulate)\n \n \n### [Version314](https://www.kaggle.com/kimtaegwan/14th-solution?scriptVersionId=18985950)[Public Score = 0.961] - CNN-stacking"},{"metadata":{},"cell_type":"markdown","source":"# Try(update...)\n\n### 1. [xgboost-stacking](https://www.kaggle.com/kimtaegwan/14th-plasc-solution?scriptVersionId=18869365)[Public Score = 0.94882] \n    EfficientNetB4[Public Score = 0.95600] \n    Xception[Public Score = 0.94787]\n    Resnet50[Public Score = 0.92682]\n\n   1. 위 세개 모델의 output을 feature사용 train_shape - (14594, 588)\n   2. if class ==1, 1 else 0 이런식으로 총 196번 훈련\n   3. 5-fold를 사용해 학습\n   4. predict 결과를 argmax를 통해 class할당\n    \n- class 196개를 한번에 돌리면 학습이 안됨\n- 196번 훈련 및 예측으로 인하여 class가 두,세번 예측되어 196을 넘어가는 경우가 많음(f1 스코어를 통해 최적화하는 방법을 시도하지 못했습니다.\n\n### 2. Mixup\n- Mixup 시도하였으나 f1스코어가 떨어졌습니다. class별 훈련데이터가 적어서 그렇지않을까 생각합니다.\n\n### 3. Ensemble\n    EfficientNetB4[Public Score = 0.95600] \n    Xception[Public Score = 0.94787]\n    Resnet50[Public Score = 0.92682]\n위의 세개의 모델로 가중평균 앙상블 을 시도했습니다. Public Score 는 0.95695로 낮게 나왔지만,\n최종 Private Score는 0.95663로 최종점수보다 약2% 높은 점수를 얻었습니다.\n앙상블의 힘이 대단하다는 것을 다시 한번 느낍니다"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.6"},"widgets":{"application/vnd.jupyter.widget-state+json":{"state":{"3715af98687648c086b811c9c67bbc4e":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"IntProgressModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"IntProgressModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"ProgressView","bar_style":"success","description":"","description_tooltip":null,"layout":"IPY_MODEL_7281aa9ea74a478fbb4c090d017837d3","max":1,"min":0,"orientation":"horizontal","style":"IPY_MODEL_4f5621e8edfe44858d7681b17fca45d0","value":1}},"43a1ed6fdff44558888e19713c690c5c":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HTMLModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_ac5241846ee84cc496b8cea1587ecc58","placeholder":"​","style":"IPY_MODEL_569bd41910724805838b2f5e1ec48168","value":"5/|/| 5/? [00:07&lt;00:00,  1.54s/it]"}},"4f5621e8edfe44858d7681b17fca45d0":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"ProgressStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"ProgressStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","bar_color":null,"description_width":""}},"516a14f3d623425ea473026de832921f":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"569bd41910724805838b2f5e1ec48168":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"DescriptionStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}},"7281aa9ea74a478fbb4c090d017837d3":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"ac5241846ee84cc496b8cea1587ecc58":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"e833bcd8488c4ed7a8a700fc3e7050e3":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HBoxModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HBoxModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HBoxView","box_style":"","children":["IPY_MODEL_3715af98687648c086b811c9c67bbc4e","IPY_MODEL_43a1ed6fdff44558888e19713c690c5c"],"layout":"IPY_MODEL_516a14f3d623425ea473026de832921f"}}},"version_major":2,"version_minor":0}}},"nbformat":4,"nbformat_minor":1}