{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Modifications\n\n## **2019/07/28**\n- kfold 4로 변경\n    - 이유 : kaggle gpu session수 제한 4\n    - [3rd ML month] various model k-fold_1 ~ 4 부터 적용\n- epoc 낮추기\n    - 대부분 20 epoc에서 high score 등장\n    - 이거 정정해야됨 kfold-v3~v5 확인해보면 resnet50은 best epoch이 40, 60, 100도 있고 제각각임\n    \n## **2019/07/29**\n- [3rd ML month] various model k-fold\n    - finetune 모드 추가\n    - resnet50으로 0.92의 점수를 달성하신 것이 놀랍습니다! rough annealing과 fine tuning으로 두 단계에 걸쳐서 학습을 시키시는 것이 흥미롭네요. 우선 lr을 크게 잡아서 여러 local minimum의 후보들을 찾고 그 중 가장 가능성이 높은 곳 하나를 뽑은 뒤 lr을 작게 두고 완벽히 local minima까지 내려가는 것으로 이해했는데 맞나요? 실제 효과를 보셨는지도 궁금합니다. 감사합니다 ㅎㅎ\n        - https://www.kaggle.com/yangsaewon/pytorch-baseline-updated-7-10\n        \n## **2019/07/30**\n- 그동안 inference 코드에 seed 고정이 안되서 결과가 바뀌었나?\n    - tta 때 test image 변경하는게 계속 바뀌었을듯\n    - [3rd ML month] k-fold ensemble inference 에서 시드 고정으로 바꿈\n\n## **2019/08/03**\nAugmentation\n- train : flip, rotation, policy\n- valid : crop, flip\n- test  : crop\n\nvalid가 best model 찾는 조건인데 이 조건이 standard랑 다름\n\n## **2019/08/04**\n이 이후의 training은 validation 에서 augmentation을 안함\n평가도 tta 안붙이고함\n\n## **2019/08/05**\nSGDR 기존 코드 약간 이상한듯? (t_0=1, t_mult=2 일때나 다른떄나 요상함)\n- 파이토치 홈피 코드로 바꿨음"},{"metadata":{},"cell_type":"markdown","source":"# TODO\n\n- mixup\n    - 언제 잘되나\n        - 왜 잘 안됬나\n    - 원래 논문 읽고 얘기\n- [Auto augmentation](https://www.kaggle.com/hominlee/ka-kr-sillim-autoaugmentation) 이미 쓰고 있지만 공부해보기\n    - 2019/07/27 에는 cifar policy를 쓰고있음\n- SGDR\\: [Stochastic Gradient Descent with Warm Restarts] (https://arxiv.org/abs/1608.03983)\n    - 이것도 이미 쓰고 있지만..\n    - learning rate 바꿔봐야하나\n- 다른 architecture로 바꿔보기\n    - 2019/07/27 에는 res50\n- WSL network?\n    - https://www.kaggle.com/c/2019-3rd-ml-month-with-kakr/discussion/97814#latest-567186\n- 2nd 1st kaggle korea 도 참고하기\n\n- fold 할 때 이 방법이 좋으려나?\n    - 저 같은 경우에는 fold를 나눈 csv 파일을 미리 만들어놓고, kfold 4로 한다고하면 커널 4개를 만들어서 각각 다른 폴드로 학습을 합니다. 한번에 한 커널에서 다 진행하면 학습할 수 없어요 :)\n        - sewonHwang, https://www.kaggle.com/seriousran/what-0-95121-model-failed\n- def get_random_eraser\n    - https://www.kaggle.com/seriousran/what-0-95121-model-failed 에서 쓰네 \n- 다른 모델 간 ensemble\n    - https://www.kaggle.com/c/2019-3rd-ml-month-with-kakr/discussion/96491#latest-561184\n    - 모델 효과적으로 쓸 수 있는 코드?\n- 마지막에 fc 하나만 더했음 지금까진\n    - 그럼 앞에는 freeze 안되있는거네 지금?\n    \n**기존코드 training, test 때 data transform 어떻게 할 지 생각해봐야겠네**\n- 지금은 test 때 randomsized crop 있긴한데 기능을 안하네? size가 같아서\n"},{"metadata":{},"cell_type":"markdown","source":"# Procedure"},{"metadata":{},"cell_type":"markdown","source":"## **2019/07/26**\n\n### kernel : pytorch baseline-v4\n\nmodel : https://www.kaggle.com/yangsaewon/pytorch-baseline-updated-7-10 를 그대로 따른 모델\n\ndescription : mixup_loss = true\n\n**accuracy : 0.90196** \n<br/>\n<br/>\n<br/>\n### kernel : pytorch baseline-v5\n\nmodel : https://www.kaggle.com/yangsaewon/pytorch-baseline-updated-7-10 를 그대로 따른 모델\n\nmodel location : https://www.kaggle.com/ikheeshin/pytorch-baseline-v5\n\ndescription : mixup_loss = false\n\n**accuracy : 0.93161**\n<br/>\n<br/>\n<br/>\n### kernel : tta pytorch inference-v1\n\nmodel : pytorch baseline-v5\n\nmodel location : https://www.kaggle.com/ikheeshin/pytorch-baseline-v5\n\ndescription : [tta](https://www.kaggle.com/yangsaewon/pytorch-inference-tta) 를 이용한 inference, tta=3\n\n**accuracy : 0.93687**\n<br/>\n<br/>\n<br/>\n### kernel : tta pytorch inference-v2\n\nmodel : [3rd ML month] kfold-v3\n\nmodel location : https://www.kaggle.com/ikheeshin/3rd-ml-moth-kfold\n\ndescription : kfold-v3을 확인해보면 5 fold 중 1,2 는 트레이닝이 끝났지만 3은 중간에 끝난 것을 알 수 있음(시간초과). fold 1,2,3 을 사용해서 inference한 결과임 (+tta)\n\n**accuracy : 0.94643**\n<br/>\n<br/>\n<br/>"},{"metadata":{},"cell_type":"markdown","source":"## **2019/07/27**\n\n### kernel : tta pytorch inference-v3\n\nmodel : [3rd ML month] kfold-v3, v4\n\nmodel location : https://www.kaggle.com/ikheeshin/res50-kfold\n\ndescription : kfold-v4는 5 fold 중 4, 5를 트레이닝함. kfold-v3 + kfold-v4 => fold 1~5 사용해서 inference\n\n**accuracy : 0.94452**\n<br/>\n<br/>\n<br/>"},{"metadata":{},"cell_type":"markdown","source":"## **2019/07/28**\n\n**today to do** : [Uploading & Updating Datasets with the Kaggle API](https://codelabs.developers.google.com/codelabs/upload-update-data-kaggle-api/index.html?index=..%2F..index#4)\n    -  **done** : https://www.kaggle.com/ikheeshin/4f_weights\n\n### kernel : tta pytorch inference-v4\n\nmodel : [3rd ML month] various model k-fold_1 ~ 4 ( resnext50, batch size = 64, epoc 50)\n\nmodel location : https://www.kaggle.com/ikheeshin/res101-4fold\n\ndescription : 앗 resnext50_32x4d 쓴거였음, 4fold\n\n**accuracy : 0.95026**\n<br/>\n<br/>\n<br/>\n### kernel : tta pytorch inference-v5\n\nmodel : [3rd ML month] various model k-fold_1 ~ 4 ( resnext101, batch size = 48, epoc 60)\n\nmodel location : https://www.kaggle.com/ikheeshin/resnext101-4fold\n\ndescription : resnext101, 4fold, best는 11epoch 꺼네 epoch 50으로 더 줄여도 될듯\n\n**accuracy : 0.95361**\n<br/>\n<br/>\n<br/>\n### kernel : tta pytorch inference-v6\n\nmodel : [3rd ML month] various model k-fold_1 ~ 4 ( resnet101, batch size = 40, epoc 50)\n\nmodel location : https://www.kaggle.com/ikheeshin/res101-re-4fold\n\ndescription : resnet101, 4fold\n\n**accuracy : 0.91917**\n<br/>\n<br/>\n<br/>"},{"metadata":{},"cell_type":"markdown","source":"## **2019/07/29**\n\n\n### kernel : tta pytorch inference-v7\n\nmodel : [3rd ML month] various model k-fold_1 ~ 4 ( vgg19_bn, batch size = 42, epoc 100)\n\nmodel location : https://www.kaggle.com/ikheeshin/vgg19bn-4fold\n\ndescription : 243s\n\n**accuracy : 0.92922**\n<br/>\n<br/>\n<br/>\n### kernel : tta pytorch inference-v8\n\nmodel : [3rd ML month] various model k-fold_1 ~ 4 ( resnet50, batch size = 128, epoc 100)\n\nmodel location : https://www.kaggle.com/ikheeshin/res50-4fold\n\ndescription : 트레이닝 과정을 보면 epoc이 12, 22 등이 될 때 resnet50의 경우는 score가 확 떨어지지 않네 resnext는 score가 12때 확 떨어져서 그게 최대 score였는데. learning rate 등 바꾸면 개선 여지 있을듯?\n\n**accuracy : 0.94165**\n<br/>\n<br/>\n<br/>\n### kernel : [3rd ML month] k-fold ensemble inference-v2\n\nmodel : [3rd ML month] various model k-fold_1 ~ 4 ( resnet50, resnext50, resnext101 )\n\nmodel location : https://www.kaggle.com/ikheeshin/4f_weights\n\ndescription : res50, resnext50, resnetx101 ensemble, tta=3\n\n**accuracy : 0.94882**\n<br/>\n<br/>\n<br/>\n### kernel : [3rd ML month] k-fold ensemble inference-v3\n\nmodel : [3rd ML month] various model k-fold_1 ~ 4 ( resnext50, resnext101 )\n\nmodel location : https://www.kaggle.com/ikheeshin/4f_weights\n\ndescription : resnext50, resnetx101 ensemble, tta=3\n\n**accuracy : 0.95265**\n<br/>\n<br/>\n<br/>\n### kernel : [3rd ML month] k-fold ensemble inference-v4\n\nmodel : [3rd ML month] various model k-fold_1 ~ 4 (resnext101)\n\nmodel location : https://www.kaggle.com/ikheeshin/4f_weights\n\ndescription : resnetx101, tta=5\n\n**accuracy : 0.95169**\n<br/>\n<br/>\n<br/>\n## Discussion\n- ensenble을 했는데 성능이 떨어지는 이유는?\n    - https://www.kaggle.com/c/2019-3rd-ml-month-with-kakr/discussion/101851#latest-586698\n    - 지금 생각엔 시드가 고정 안되서 떨어진건가 싶기도하네\n- tta는 제 기능을 하고있는가?\n    - 보통 파이토치에서 test transform 할 때 flip 추가해서 prediction 하고 평균내긴 하는데 개인적인 의견으로는 그렇게 해도 제가 말한거랑 비슷한 효과라고 생각되네요 정확한건 일두님만 아시겠지만..\n        - 지금 내코드에선 flip 안함\n    - tta=5가 tta=3 보다 성능이 낮았네\n    - k-fold ensemble inference 코드 제대로된 것 맞나?\n- 8 fold랑 앙상블 해야할 것 같음\n- 참고\n    - https://www.kaggle.com/c/2019-3rd-ml-month-with-kakr/discussion/98673#latest-581516\n        - mixup 사용?\n    - https://www.kaggle.com/c/2019-3rd-ml-month-with-kakr/discussion/101851#latest-586698\n        - held-out set을 따로 만들어서 앙상블 모델을 만들어야죠. 안그러면 오버피팅이 나겠죠.\n- finetuning으로 cv score 올라가는걸 보니 작은 모델(res50)에서 mixup + adam 쓰면 더 높을수도?\n<br/>\n<br/>\n<br/>"},{"metadata":{},"cell_type":"markdown","source":"## **2019/07/30**\n\n**today to do** : 표로 한눈에 볼 수 있게 정리하기. fine tune 결과도 비교할 수 있도록, **mixup 다시 써보기**\n- **작은모델로 (res50) 이런저런 방법 하는게 나을수도? + 8fold**\n<br/>\n<br/>\n<br/>\n\n### kernel : [3rd ML month] k-fold ensemble inference-v5\n\nmodel : [3rd ML month] various model k-fold_1 ~ 4 (fresnext101)\n\nmodel location : https://www.kaggle.com/ikheeshin/4f_weights\n\ndescription : resnetx101 finetune한 것. 결과가 똑같음. 그동안 inference 코드에 시드 고정을 안한게 문제일수도 있다고 생각해서 추가함\n\n**accuracy : 0.95361**\n<br/>\n<br/>\n<br/>\n\n### kernel : [3rd ML month] k-fold ensemble inference-v6\n\nmodel : [3rd ML month] various model k-fold_1 ~ 4 (fresnext101)\n\nmodel location : https://www.kaggle.com/ikheeshin/4f_weights\n\ndescription : resnetx101 finetune한 것. 시드 고정하니까 내려가네ㅋㅋㅋ\n\n**accuracy : 0.95265**\n<br/>\n<br/>\n<br/>"},{"metadata":{},"cell_type":"markdown","source":"## 정리해야되는 것: 표로\n- [3rd ml month] various model k-fold를 썼을 때 각 모델별 accuracy, 몇 epoch, batch, 1 epoch 당 몇초 등등 한눈에 보이게\n- 지금 inference 코드에서 tta 할 때 어떤게 랜덤인거지?\n    - 더 변화 줄 수 있나?\n    - tta 도 하나 파서 눈으로 볼 수 있겠네\n- data analysis 나도 한번 해보자\n    - tta 할 때 사진 어떻게 변하나, policy 할 때 어떻게 변하나 이거 전부다 kernel 파서 확인해보자\n- prediction 한걸 모델별로 matrice로 저장해두면 바로 가능하네?\n    - 혹은 세 모델 앙상블이면 inference한 후 가능한 모든조합에 대해서 csv 파일 뱉으면되네"},{"metadata":{},"cell_type":"markdown","source":"## <center/>  **Cifar policy, no mixup**  </center>\n\n| Model | location | Size(MB) | fold1 | fold2 | fold3 | fold4 | lb score | sec/epoc | \n|:--------|:--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|\n|**ResNext50** | 4f_weights | 93.9 | 0.9412/20 | 0.9413/20 | 0.9316/99 | 0.9347/20 | 0.95026| 165 | | | | | |\n|**ResNext101** | 4f_weights | 349.5 | 0.9416/11 | 0.9433/11 | 0.9328/11 | 0.9362/11 | **0.95361** | 308 | | | | | |\n|**ResNet-101** | 4f_weights | 172.1 | 0.9062/10 | 0.8980/10 | 0.8834/11 | 0.9043/10 | 0.91917 | 175 | | | | | |\n|**VGG19-bn** | 4f_weights | 561.6 | 0.9078/11 | 0.9049/10 | 0.8949/10 | 0.9032/11 | 0.92922 | 245 | | | | | |\n|**ResNet-50** | 4f_weights | 95.9 | 0.9408/99 | 0.9381/50 | 0.9233/11 | 0.9319/39 | 0.94165 | 124 | | | | | |\n|**fResNext-101** | 4f_weights | 349.5 | 0.9495/22 | 0.9539/28 | 0.9386/22 | 0.9417/30 | 0.95361 | 325 | | | | | |\n\n\nparameters\n- lr : 0.01\n- Tmax=10\n\n위의 표 시드 고정하기전임..\n\n고정한 후\n- ResNext101 : 95217\n- fResNext101 : 95265\n- ResNext50 : 95121\n- ResNet-50 : 94213"},{"metadata":{},"cell_type":"markdown","source":"## <center/>  **Play with ResNet50**  </center>\n\n| Kernel | Optimizer | Mixup | Policy | lr | Sched | Validation | lb\n|:--------|:--------|:--------:|--------:|--------:|--------:|--------:|--------:|\n| k_fold_1 v9 |**SGDR** | o | x | 1e-2 | T_max = 10| 0.92253/60 |\n| k_fold_1 v10 |**SGDR** | o | Cifar | 1e-2 | T_max = 10 | 0.90425/110 |\n| k_fold_1 v11 |**SGDR** | x | Cifar | 1e-2 | T_max = 10 | 0.9408/100 |\n| k_fold_2 v12 |**SGDR** | x | Cifar | 5e-3 | T_max = 10 | 0.94319/110 |\n| k_fold_3 v21 |**SGDR** | x | Cifar | 5e-3 | T_max = 15 |  0.94438/88 |\n| k_fold_3 v22 |**SGDR** | x | Cifar | 5e-3 | T_max = 20 |  0.94358/101 |\n| k_fold_3 v11 |**SGDR** | **x** | **Cifar** | **5e-3** | **T_max = 20** | **0.94795/118** |\n| k_fold_1 v18 |**SGDR** | x | Imagenet | 5e-3 | T_max = 20 | 0.94279/118 |\n| k_fold_2 v20 |**SGDR** | x | SVHN | 5e-3 | T_max = 20 | 0.93802/97 |\n| k_fold_3 v20 |**SGDR** | x | Cifar | 5e-3 | T_max = 25 | 0.94319/97 |\n| k_fold_4 v10 |**SGDR** | x | Cifar | 1e-2 | T_max = 20 | 0.94319/80 |\n| k_fold_1 v17 |**SGDR** | x | Cifar | 4e-3 | T_max = 20 | 0.93961/118 |\n| k_fold_2 v19 |**SGDR** | x | Cifar | 3e-3 | T_max = 20 | 0.93762/120 |\n| k_fold_3 v19 |**SGDR** | x | Cifar | 2e-3 | T_max = 20 | 0.93246/120 |\n| k_fold_4 v19 |**SGDR** | x | Cifar | 1e-3 | T_max = 20 | 0.89273/118 |\n| k_fold_2_v11 |**AdamW** | o | x | 1e-3 | Plateau | 0.90187/118\n| k_fold_3_v10 |**AdamW** | o | Cifar | 1e-3 | Plateau | 0.90942/110\n| k_fold_4_v9 | **AdamW** | x | Cifar | 1e-3 | Plateau| 0.9269/97\n\n- ~120 epoch, batch_size = 128, 1 fold of 4 fold (whichfold = 1)\n    - 120epoc 까지 볼땐 learning rate 4e-3 ~ 1e-3 가 작은듯\n    - **resnext101 에선 다를 수도 있지 않을까?**\n\n- platour? 이거 valid score에 걸꺼면 mode='max'로 해야되는거아님? - 맞음\n- 확실히 SGDR이 정말 빠르네\n    - restart 제대로 조절하면 resnext도 성능 올라가려나? - 그럴듯?\n    - **learning rate 더 낮춰봐도 될듯, T max도 더 키워도 될수도ㅋㅋ**\n- adamW 에 weight decay 0.1 하고 해볼까 - training 안됨"},{"metadata":{},"cell_type":"markdown","source":"## **2019/07/31**\n**today to do** : ResNet50 4fold, 8fold, ensemble of (4fold, 8fold)\n\n\n## <center/>  **4 fold**  </center>\n| Model | Kernel | Optimizer | Mixup | Policy | lr | Sched | fold0 | fold1 | fold2 | fold3 | lb |\n|:--------|:--------|:--------|:--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|\n| ResNet-50 | v(12,13,12,12)  |**SGDR** | x | Cifar | 1e-2 | T_max = 20 | 0.93743/40 | 0.94319/80 | 0.94094/61 | 0.92628/120 |  |\n| ResNet-50 | v(13,14,13,14)  |**SGDR** | x | Cifar | 5e-3 | T_max = 20 | 0.93743/119 | 0.94795/118 | 0.94741/139 | 0.93657/120 | 0.94643/v10,**aResNet-50** |\n| ResNext101 | v(14,15,15,15) |**SGDR** | x | Cifar | 5e-3 | T_max = 20 | 0.9452/38 | **0.9535/39** | 0.9527/38 | 0.9452/40 | 0.95313/v11,**aResNext101** |\n| ResNext101 | |**SGDR** | x | Cifar | 4e-3 | T_max = 20 | | 0.95352/39 |\n| ResNext101 | |**SGDR** | x | Cifar | 3e-3 | T_max = 20 | | 0.95312/39 |\n| ResNext101 | |**SGDR** | x | Cifar | 2e-3 | T_max = 20 | | 0.94835/59 |\n| ResNext101 | v(20, , , ) |**SGDR** | x | Cifar | 5e-3 | T_max = 25 | | 0.95074/24 |\n| ResNext101 | v( ,22, , ) |**SGDR** | x | Cifar | 5e-3 | T_max = 30 | | 0.95153/30 |\n| ResNext101 | v( , , 24, ) |**SGDR** | x | Cifar | 5e-3 | T_max = 10/T_mult = 2 | | 0.95272/30 |\n\nResNext101 은 ~61epoc (T_mult=2 일땐 71)"},{"metadata":{},"cell_type":"markdown","source":"## **2019/08/01**\n## <center/>  **8 fold**  </center>\n| Model | Optimizer | Mixup | Policy | lr | Sched | fold0 | fold1 | fold2 | fold3 | fold4 | fold5 | fold6 | fold7 | lb |\n|:--------|:--------|:--------|:--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|\n| ResNext101 |**SGDR** | x | Cifar | 5e-3 | T:20 | 0.9411/61 | 0.9516/20 | 0.9582/39 | 0.9551/20 | 0.9583/41 | 0.9547/18 | 0.9448/21 | 0.9499/40 | 0.95648/**bResNext101** |\n\n실수로 8fold 중에 0~3 fold 만 사용한건 0.95743 나왔음ㅋㅋㅋㅋㅋ\n- [3rd ML month] k-fold ensemble inference, v13\n- 확실히 4fold 보다 8fold가 더 좋은듯?\n    \n## <center/>  **Models Ensemble**  </center>\n| Model | descript | lb |\n|:--------|--------:|--------:|\n| aResNext101 + bResNext101 | tta=3 | 0.95743/v1 |\n| aResNext101 + bResNext101 | tta=5 | 0.95791/v2 |\n| aResNext101 + bResNext101 | tta=5, (F,R,P) | 0.95839/v4 |\n| aResNext101 + bResNext101 | tta=5, (F,R) | 0.95743/v5 |\n| aResNext101 + bResNext101 | tta=7, (F,R,P) |  0.95791/v6 |\n\n- 4fold + 8fold ensemble하면 조금 오르네\n- tta 높아도 올라가는구나\n\n\n추가로 알아야하는 것\n- **어떤 tta가 더 좋은가?**\n    - tta 횟수가 성능 향상하는지? - 그런듯\n- resnet50에서 learning rate 더 낮춰볼 수 있나?\n    - ~120 epoc으로 하면 트레이닝 되다 말음\n- **model ensemble은 어떤 식으로 해야하나?**\n- imagenet policy는 어떠려나?"},{"metadata":{},"cell_type":"markdown","source":"## **2019/08/02**\n**today to do** : tta 횟수별 알아보기, learning rate 낮춘결과, augmentation 관찰해보기\n\nCifar policy가 최고인듯\n\n\n## <center/>  **Play with tta of aResNet-50**  </center>\n| Model | kernel | n_tta | R-crop | H-flip | R-Rot | policy | lb |\n|:--------|--------:|--------:|--------:|--------:|--------:|--------:|--------:|\n|**aResNet-50** | test_1/v2 | 3 | o | x | x | x |  |\n|**aResNet-50** | test_1/v1 | 3 | o | o | x | x | 0.94835 |\n|**aResNet-50** | test_2/v1 | 3 | o | o | o | x | 0.94500 |\n|**aResNet-50** | test_3/v1 | 3 | o | o | o | o | 0.94643 |\n|**aResNet-50** | test_4/v1 | 3 | x | o | o | o | 0.94882 |\n|**aResNet-50** | test_2/v2 | 3 | x | x | o | o | |\n|**aResNet-50** | test_3/v2 | 3 | x | o | x | o | 0.94882 |\n|**aResNet-50** | test_4/v2 | 3 | x | o | o | x | 0.94978 |\n\nn_tta=3이기 때문에 policy 들어간게 낮은가?\n- 5로 해서도 봐야하나\n\nAugmentation\n- train : flip, rotation, policy\n- valid : crop, flip\n- test  : crop\n- 지금까지 train, valid, test 전부 augmentation 기준이 다르네?\n    - cv에서 best model 고를 때 valid loader의 기준이 train loader랑 다르고\n    - TTA 할 때도 다르네 음..\n\n원칙은 valid에는 어떤 augmentation도 하지 않는거네"},{"metadata":{},"cell_type":"markdown","source":"## **2019/08/03**\n**today to do** : valid에 augment 넣지 않은 거랑 비교해봐야함\n\n\n시도해볼만한 것들\n- Tmax = n_epoch\n- restart 전 epoch들 ensemble\n- valid 바꾸기\n- WSN(weakly) 이거 pre-train으로 써보기\n<br/>\n<br/>\n<br/>\n<br/>\n<br/>\n<br/>\n<br/>\n<br/>"},{"metadata":{},"cell_type":"markdown","source":"\n--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n\n**Note : 여기 전까진 valid에 transform이 되어있음**\n- 아래 실험들은 valid에 transform 되어있지 않음 이게 맞는듯\n\n## **2019/08/04**\n\n\n## <center/>  **4 fold revisit**  </center>\n| Model | Kernel | Optimizer | Mixup | Policy | lr | <center/>Sched</center> | fold0 | fold1 | fold2 | fold3 | lb |\n|:--------|:--------|:--------|:--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|\n| ResNet-50 | v(,,,)  |**SGDR** | x | Cifar | 5e-3 | T_max=20/T_mult=1 | |  |  |  | 0.94835/aResNet-50\n| ResNet-50 | v(22,24,26,22)  |**SGDR** | x | Cifar | 5e-3 | T_max=10/T_mult=2 | 0.93937/144| 0.95074/145 | 0.94984/144 | 0.94069/142 | 0.94978/bResNet-50\n| ResNet-50 | v(23,25,27,23)  |**SGDR** | x | Cifar | 5e-3 | T_max=20/T_mult=1 | 0.94054/138 | 0.94795/119 | 0.94903/121 | 0.93369/119 | 0.95074/cResNet-50\n| ResNet-50 | v(24,26,28,24)  |**SGDR** | x | Cifar | 5e-3 | T_max=150/T_mult=1 | 0.94015/144 | 0.95312/136 | 0.95024/140 | 0.93781/147 | 0.95408/dResNet-50\n| ResNet-50 | v(25,27,29,25)  |**SGDR** | x | Cifar | 5e-3 | T_max=1/T_mult=2 |  |  |  |  | \n\n맨 위는 전에 ~120epoch, valid에 transform \n- 그럼 tta 안한게 오히려 좋았던거야?\n- 이미 valid를 뽑을 때 transform한걸 뽑으니까 앵간한 tta로는 안되네\n\n**달라진 점!**\n- validation에서 augmentation 안함\n- **inference할 때 tta 안쓰고 비교함**\n    - 그래야 train하는 과정과 tta 과정 분리해서 생각할 수 있을듯\n- ~151epoc : SGDR에서 봤듯 30, 70, 150"},{"metadata":{},"cell_type":"markdown","source":"## **2019/08/05~**\n\n~151 epoch, valid no transform, no tta, **SGDR changed** (pytorch source에 나와있는 코드가 더 맞는듯)\n\n\n## <center/>  **4 fold last**  </center>\n| Model | Kernel | Optimizer | Mixup | Policy | lr | <center/>Sched</center> | fold0 | fold1 | fold2 | fold3 | lb | tta |\n|:--------|:--------|:--------|:--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|\n| ResNet-50 | v(,,,)  |**SGDR** | x | Cifar | 5e-3 | T_max=20/T_mult=1 | |  |  |  | 0.94835/aResNet-50\n| ResNet-50 | v(26,28,30,26)  |**SGDR** | x | Cifar | 5e-3 | T_max=10/T_mult=2 | 0.94287/145 | 0.95193/137 | 0.95024/144 | 0.94028/140 | 0.95074/eResNet-50\n| ResNet-50 | v(27,29,31,27)  |**SGDR** | x | Cifar | 5e-3 | T_max=150/T_mult=1 | 0.94365/134 | 0.95312/139 | 0.95267/142 | 0.9411/143 |  **0.95408/fResNet-50** | 0.95313/5,F,R |\n| ResNext101 | v(28,30,32,28)  |**SGDR** | x | Cifar | 5e-3 | T_max=100/T_mult=1 | 0.9479/98 | 0.9571/95 | 0.95348/95 | 0.94069/95 |  0.95217/cResNext101 |  |\n| ResNext101 | v(44,44,44,49)  |**SGDR** | x | Cifar | 5e-3 | T_max=10/T_mult=2 | 0.9479/31 |  0.9559/31 | 0.9551/71 | 0.9448/30 | 0.95648/dResNext101 |\n| ResNext50 | v(33,33,35,31)  |**SGDR** | x | Cifar | 5e-3 | T_max=150/T_mult=1 | 0.94209/124 | 0.95153/114 | 0.95267/148 | 0.94316/147 | 0.95552/aResNext50 |  |\n| ResNext50 | v(43,43,43,48)  |**SGDR** | x | Cifar | 5e-3 | **T_max=10/T_mult=2** | 0.9440/149 | 0.9578/147 | 0.9526/147 | 0.9439/141 | **0.95695/hResNext50** |\n| ResNext101_wsl (8d) | v(36,37,39,47)  |**SGDR** | x | Cifar | 5e-3 | **T_max=10/T_mult=2** | 0.9483/70 | 0.9574/31 | 0.9559/28 | 0.9481/28 | **0.96030/aResNext101_32x8d_wsl** |\n| ResNext101_wsl (16d) | v(42,42,42,37)  |**SGDR** | x | Cifar | 5e-3 | **T_max=10/T_mult=2** | 0.94753/31 | 0.9551/31 | 0.9563/30 | 0.9452/30 |**0.95600/aResNext101_32x16d_wsl** |\n\nfResNet-50 tta\n- 3HRC 0.95074\n- 3H 0.95408\n- tta 하나마나네? 왜그러지\n\nResNext101_wsl (16d)는 31epoc 까지만 돌림 (batch size = 32)\n- 결과 보니까 더 돌리면 성능 올라갈 것 같긴한데..\n\n음... resnext50 해볼까..\n\n\ntraining augmentation에서 resized crop 해도 괜찮을듯?\n- 지금까지는 전부 resize flip rotation policy\n\nResNext101_8d_wsl : batch size = 48, ~71epoch\n\n\n## <center/>  **8 fold last**  </center>\n| Model | Optimizer | Mixup | Policy | lr | Sched | fold0 | fold1 | fold2 | fold3 | fold4 | fold5 | fold6 | fold7 | lb |\n|:--------|:--------|:--------|:--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|\n| ResNet-50 |**SGDR** | x | Cifar | 5e-3 | 150/1 | 0.9381/141 | 0.9485/139 | 0.9582/128 | 0.9559/143 | 0.9551/133 | 0.9579/144 | 0.9491/142 | 0.9447/142 | 0.9540/gResNet-50\n| ResNext50 |**SGDR** | x | Cifar | 5e-3 | 150/1 | 0.9388/127 | 0.9516/139 | 0.9613/129 | 0.9566/128 | 0.9527/130 | 0.9596/133 | 0.9431/137 | 0.9555/132 | 0.9564/bRexNext50 |\n| ResNext101_wsl (8d) |**SGDR** | x | Cifar | 5e-3 | 10/2 | 0.9501/29 | 0.9585/27 | 0.9590/29 | 0.9590/31 | 0.9583/69 | 0.9621/30 | 0.9508/30 | 0.9559/31 | **0.96221/bResNext101_wsl** |\n\n\n\n\n## <center/>  **Models Ensemble last**  </center>\n| Model | lb | tta |\n|:--------|--------:|--------:|\n| fResNet-50 + gResNet-50 | 0.95648 |\n| cResNext101 + fResNet-50 + gResNet-50 | 0.95695 |\n| aResNext50 + cResNext101 + fResNet-50 + gResNet-50 | **0.95982** |\n| bResNext50 + aResNext50 + cResNext101 + fResNet-50 + gResNet-50 | 0.95887 |\n| bResNext50 + aResNext50 + fResNet-50 + gResNet-50 | 0.95743 |\n| bResNext50 + cResNext101 + fResNet-50 + gResNet-50 | 0.95887 |\n| bResNext50 + aResNext50 | 0.95695 |\n| bResNext101_32x8d_wsl + aResNext101_32x8d_wsl| 0.96221 |\n| aResNext50 + bResNext101_32x8d_wsl + aResNext101_32x8d_wsl | **0.96413** | 2F 그대로, 3C 96078 감소, 3P 그대로\n| fResNet-50 + aResNext50 + bResNext101_32x8d_wsl + aResNext101_32x8d_wsl | 0.96269 |\n| cResNext101 + aResNext50 + bResNext101_32x8d_wsl + aResNext101_32x8d_wsl | 0.96221 | \n| aResNext101_32x16d + aResNext50 + bResNext101_32x8d_wsl + aResNext101_32x8d_wsl | 0.96269 |\n| aResNext101_32x16d + bResNext101_32x8d_wsl + aResNext101_32x8d_wsl | 0.96221 |\n| hResNext50 + bResNext101_32x8d_wsl + aResNext101_32x8d_wsl | 0.96174 | \n| bResNext50 + bResNext101_32x8d_wsl + aResNext101_32x8d_wsl | 0.96221 |\n| dResNext101 + bResNext101_32x8d_wsl + aResNext101_32x8d_wsl | 0.96269 |\n| dResNext101 + bResNext101_32x8d_wsl | 0.96269 |\n| hResNext50 + dResNext101 + bResNext101_32x8d_wsl | 0.96126 |\n\nWSL 가져와서?\n\nsegmentation 된 데이터로?\n\ntrain augmentation할 때 resize가 아니라 random crop을 했다면?\n\n\n\n| Model | fold0 | fold1 | fold2 | fold3 | fold4 | fold5 | fold6 | fold7 | fold8 | fold9 | fold10 | fold11 | lb |\n|:--------|:--------|:--------|:--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|--------:|\n| | 0.9461/30 | 0.9485/30 | 0.9580/30 | 0.9650/69 | 0.9596/71 | 0.9643/27 | 0.9617/11 | 0.9621/67 | 0.9601/30 | 0.9502/30 | 0.9470/10 | 0.9546/68| 0.96221 |\n"},{"metadata":{"trusted":false},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.5.2"}},"nbformat":4,"nbformat_minor":1}