{"cells":[{"metadata":{"_uuid":"5097a48e76b225ba6239c176856d289ab431421e"},"cell_type":"raw","source":"fastai函式庫是第一個可以為視覺、文字、表格資料、時間序列以及協同過濾等深度學習應用，提供單一一致介面的深度學習資料庫。也就是說，當開發者能夠以fastai函式庫實作電腦視覺模型，則表示具有能力以相同的方法創建自然語言處理模型，或者任何應用模型。"},{"metadata":{"_uuid":"a5b7f6054774949f1d16bc92abfd210e7bdf15bc"},"cell_type":"markdown","source":"* DeepLearning : https://medium.com/@intheblackworld/deep-learning-tutorial-%E5%BF%83%E5%BE%97-b1f7f84a497d\n* Fast.ai Lesson Note : https://medium.com/@hiromi_suenaga/deep-learning-2-part-1-lesson-2-eeae2edd2be4\n* Fast.ai Lesson Note : https://medium.com/@easonlalala/fast-ai-lesson-1-%E7%AD%86%E8%A8%98-6d696a89b0b0\n\n![Fast.ai 課程結構](https://cdn-images-1.medium.com/max/800/1*D0WqPCX7RfOL47TOEfkzYg.png)"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"%reload_ext autoreload\n%autoreload 2\n%matplotlib inline\n\nfrom fastai.imports import *\nfrom fastai.transforms import *\nfrom fastai.conv_learner import *\nfrom fastai.model import *\nfrom fastai.dataset import *\nfrom fastai.sgdr import *\nfrom fastai.plots import *\n# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\nimport os\nprint(os.listdir(\"../input/newdogscats/dogscats/dogscats\"))\nPATH = \"../input/newdogscats/dogscats/dogscats/\"\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4fb719f39f95c420b23349e677c735441c5727f7"},"cell_type":"code","source":"os.listdir(PATH)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"torch.cuda.is_available()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a006cd8e225a45cecb0fb0cc485190a2fe56a11d"},"cell_type":"code","source":"torch.backends.cudnn.enabled\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4db50bdaff218c4ee433bb50b91e7cd4951665d2"},"cell_type":"code","source":"os.listdir(f'{PATH}valid')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7c5ea15d31ca20849fa481576dd77812468d12a5"},"cell_type":"code","source":"files = os.listdir(f'{PATH}valid/cats')[:5]\nfiles","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4c1a7142aedba1d34f8661729be2d92e10098315"},"cell_type":"code","source":"img = plt.imread(f'{PATH}valid/cats/{files[0]}')\nplt.imshow(img);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"36f6ec5aa63329e7aa6d3998b7bfff84113bb6cf"},"cell_type":"code","source":"arch=resnet34\nsz=224\ndata = ImageClassifierData.from_paths(PATH, tfms=tfms_from_model(arch, sz))\nTMP_PATH = \"/tmp/tmp\"\nMODEL_PATH = \"/tmp/model\"\nlearn = ConvLearner.pretrained(arch, data, precompute=True, tmp_name=TMP_PATH, models_name=MODEL_PATH)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e6b0905750ba447b9b441c2e07884db61d4d0c19"},"cell_type":"markdown","source":"* Resnet34，全名是Residual Network with 34 layers，模型本身透過1.3億張含有2萬種標籤的照片訓練後得到CNN模型\n* sz代表照片尺寸，理論上模型可以使用任何尺寸的照片重新縮放後進行訓練\n* 因為Resnet訓練用的照片大多是224x224 or 299x299，因此我們把所有照片縮放到224x224 pixels 來訓練\n* ImageClassifierData.from_paths讀取資料夾內的訓練照片，tfms_from_model這個函式表示說我們如何把照片轉成pre-trained model可以取用的規格\n* ConvLearner.pretrained 把資料丟到Resnet 中，並且將加入符合訓練資料的輸出層，並且開啟precompute"},{"metadata":{"trusted":true,"_uuid":"bf993fd6397f5d9818ce6dea5e2be03749e09c58"},"cell_type":"code","source":"lrf=learn.lr_find()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"afef809eee7d364f029ece65dc8c184ac14be585"},"cell_type":"markdown","source":"[Learning rate](https://itw01.com/VUYJLEX.html)\n* Gradient Descent\n* GD我們是一次用全部訓練極的數據去計算損失函數的梯度就更新一次參數\n* Stochastic Gradient Descent\n* SGD就是一次跑一個樣本或是小批次(mini-batch)樣本然後算出一次梯度或是小批次梯度的平均後就更新一次，那這個樣本或是小批次的樣本是隨機抽取\n![Gradient Descent](https://cdn-images-1.medium.com/max/800/1*cWCG1LQgGZ268J9tdlyOpA.png)![Learning rate](https://cdn-images-1.medium.com/max/400/1*X5QbtfhszYNBSV6DZzbT7A.jpeg)"},{"metadata":{"_uuid":"5567ca841be390563138dd2f2f452aa83f1e0f96"},"cell_type":"markdown","source":"![Cyclical Learning Rates (CLR)](https://img.itw01.com/images/2018/03/19/10/3454_t2xA7Z_VUYJLEX.jpg!r1024x0.jpg)\n* Cyclical Learning Rates （CLR)\n* CLR希望透過可變的LR來增加訓練的準確度。通常我們希望隨著訓練過程越來越接近最低點，LR隨著越小越好，這樣一來可以很快找到最佳化的方向，並且兼顧在最低點附近微調。\n1.  設定最大學習率, Maximum Learning Rate, MLR\n2. 根據特定函數逐步減少Learning Rate，稱為LR Annealing\n![Cosine](https://github.com/apachecn/fastai-ml-dl-notes-zh/raw/master/img/1_xmIlOee7PWLc6fa7xdfRkA.png)\n3. 每完成一個循環Cycle，重啟Learning Rate(SGD with restart)\n![CLR](https://cdn-images-1.medium.com/max/600/1*roAnB_YqpfqHttLgF4tj4Q.png)"},{"metadata":{"trusted":true,"_uuid":"af3a495a429c62365c3c57d32134d2cbb3dd3c7c"},"cell_type":"code","source":"learn.sched.plot_lr()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0f6471f3ca57486f1b7114777ca04aef49984514"},"cell_type":"code","source":"learn.sched.plot()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"70c7e8db096fc43278b0e318206513a875e5be42"},"cell_type":"code","source":"learn.fit(1e-2, 1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"beeab3d40a4259d8a3036167b9e8c31cabf2b121"},"cell_type":"markdown","source":"* 最後learn.fit給定Learning Rate = 0.01 與Learning Cycles (Epochs) = 3 這兩個參數後就可以開始訓練了。\n* Learning Rate代表在Gradient Descent 中每一步的大小，Learning Cycles則是代表我們要讓模型學習所有的資料多少次。"},{"metadata":{"_uuid":"6547903413b2c9a3e47bd379b6e85bf562cc8056"},"cell_type":"markdown","source":"[Epoch Batch Iteration](https://kknews.cc/zh-tw/other/r8v66nx.html)\n* 每個 Epoch 要訓練的圖片數量： 5000\n* 訓練集具有的 Batch 個數： 5000 /256 = 195 + 1 = 196\n* 每個 Epoch 需要完成的 Batch 個數： 196\n* 每個 Epoch 具有的 Iteration 個數： 196\n* 每個 Epoch 中發生模型權重更新的次數： 196\n* 訓練 10 Epochs，模型權重更新的次數： 196 * 10"},{"metadata":{"_uuid":"fc2fad53deb5ed624cd617476913f7ed0d30fb23"},"cell_type":"markdown","source":"* fast.ai使用的是Cosine函數，讓LR隨著訓練過程逐步減少。每一個Cycle都從MLR逐步變小\n* 直到一個Cycle結束後又重回最大值重新訓練，也就是restart"},{"metadata":{"trusted":true,"_uuid":"d9ff0a5e755dc7d5465c88caed5e211876ff3905"},"cell_type":"code","source":"learn.fit(1e-2, 3, cycle_len=1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2915022872a741ab07b59099b98b2a475121a6a4"},"cell_type":"markdown","source":"* Underfitting，因為參數過少連Training set都會有頗大的預測誤差\n* Overfitting，因為參數過多導致過度符合Training set的資料特性，使得其無法預測較為普遍的資料集"},{"metadata":{"trusted":true,"_uuid":"bf99f9f44c4899e67af31596d6b1b89e0895c718"},"cell_type":"code","source":"learn.sched.plot_lr()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"58670d1bbbd9918eccde357ca28fbb8982363c1d"},"cell_type":"markdown","source":"* Data Augmentation\n* 在訓練模型時資料的大小可以說是最關鍵的因素之一。\n* 為了使有限的資料達到更精確的模型我們使用Data Augmentation來增加Training Set的大小。\n* 原則上各種轉換要能夠保留訓練目標的特徵，常見的轉換有鏡像轉換Mirror、翻轉Flip、縮放Scale、隨機剪裁Random Crop、旋轉Rotation 或是扭曲Shearing。\n* Data Augmentation又可以分成Train Time Data Augmentation 與 Test Time Data Augmentation。\n* 前者在訓練時加入擴增影像來增加資料大小，提高模型準確度。後者則是在測試階段將測試樣本轉換成數個影像讓模型判別再取平均，降低誤判的機率。"},{"metadata":{"trusted":true,"_uuid":"0f584a5f9c7c33cac74da286d499fd63defe7a2a"},"cell_type":"code","source":"tfms = tfms_from_model(resnet34, sz, aug_tfms=transforms_side_on, max_zoom=1.1)\ndata = ImageClassifierData.from_paths(PATH, tfms=tfms)\nlearn = ConvLearner.pretrained(arch, data, precompute=True, tmp_name=TMP_PATH, models_name=MODEL_PATH)\nlearn.fit(1e-2, 1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ce07e2fbaf14bceaf7dc82f6f9beb682b54df24a"},"cell_type":"markdown","source":"* Now we created a new data object that includes augmentation. Initially, the augmentations actually do nothing because of precompute=True.\n* We are using a pre-trained network which has already learned to recognize features (i.e. we do not want to change hyper parameters it learned), so what we can do is to pre-compute activations for hidden layers and just train the final linear portion.\n* To use data augmentation, we have to do learn.precompute=False"},{"metadata":{"trusted":true,"_uuid":"f9d5a7c414974ee80d929c3f3c35783dd7873794"},"cell_type":"code","source":"learn.precompute=False\nlearn.fit(1e-2, 3, cycle_len=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3cdbbe31b383711ab4357824adda2155c0ccba1c"},"cell_type":"code","source":"learn.sched.plot_lr()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1d1585b6b14c39660b2cf4dbf814ee92e535e719"},"cell_type":"markdown","source":"Saving model"},{"metadata":{"trusted":true,"_uuid":"0552b267315efeb92a9de5015efb9aa04c0fcb74"},"cell_type":"code","source":"learn.save('224_lastlayer')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"02ad12c8b7bbe5086c0431a1339fc4a50ff348ba"},"cell_type":"code","source":"print(os.listdir(\"/tmp/\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b871618d31cf0150dac2c1da2ea5a8986a5398a4"},"cell_type":"code","source":"print(os.listdir(\"/tmp/tmp\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d4015afa5c88209977d34df820e54cf4c2ca387d"},"cell_type":"code","source":"print(os.listdir(\"/tmp/model\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3aa3e6c2d9c9f21010e237bb455f0e4e50ad3942"},"cell_type":"code","source":"learn.load('224_lastlayer')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b30b0813570a3aea4a5f394faa621c68e520f778"},"cell_type":"markdown","source":"Fine Tuning and Differential Learning Rate\n* Pre-trained model跟我們的訓練的目標有高度的相似性，例如說狗或貓。假使是衛星空拍影像或是病理切片這種可能跟ImageNet中沒那麼相似的話訓練效果可能就會大打折扣\n* ConvLearner.pretrained 函式預設在Pre-trained model的最後加入兩層Dense Neuron Net使得Output符合我們要訓練的目標。\n* 並且預設trainable weight只有包含最後那兩層，換句話說前面所有的weight都是frozen不會隨著訓練做改變。\n* learn.unfreeze() 可以釋放所有先前pretrained weight重新調整、訓練。在架構不變的情況下，微調其中的參數可以讓transfer model 的準確度大大提升。在unfreeze的情況下，所有模型中的參數都會隨著訓練集而改變。\n* learn.unfreeze(n) 可以輸入你想要unfreeze的layer"},{"metadata":{"trusted":true,"_uuid":"d6ab6a952a0074b22759a5cabb4726eab14a8bf4"},"cell_type":"code","source":"learn.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"68ac3fa2fa67d8a011e200a0534a03e249a416ae"},"cell_type":"code","source":"learn.unfreeze()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"35676c2d005229f2697af582f69a22a25d63da55"},"cell_type":"code","source":"learn.summary()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e8467acc13d49a9924272d94f6195d763b7e295e"},"cell_type":"markdown","source":"* Earlier we said 3 is the number of epochs, but it is actually cycles. \n* So if cycle_len=2 , it will do 3 cycles where each cycle is 2 epochs (i.e. 6 epochs). \n* cycle_mult=2 : this multiplies the length of the cycle after each cycle (1 epoch + 2 epochs + 4 epochs = 7 epochs)."},{"metadata":{"trusted":true,"_uuid":"dc03eb2f8b5d4a3c4d65170f561d7a90229ce2ef"},"cell_type":"code","source":"lr=np.array([1e-4,1e-3,1e-2])\nlearn.fit(lr, 3, cycle_len=1, cycle_mult=2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fd256c622a63b776d7a3b1e343e61efb54435ed5"},"cell_type":"code","source":"learn.sched.plot_lr()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8ccc5ab5aea590950ffda8fadc4dade75631c305"},"cell_type":"code","source":"learn.save('224_all')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2e91d9e1ed345b9a96ca77367f3bf21e42c27c86"},"cell_type":"code","source":"learn.load('224_all')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6fa4eec8a518061361c45a98ff9d86b921fc2ee3"},"cell_type":"markdown","source":"Test Time Data Augmentation\n* What this means is that we are going to take 4 data augmentations at random as well as the un-augmented original (center-cropped). \n* We will then calculate predictions for all these images, take the average, and make that our final prediction. \n* Note that this is only for validation set and/or test set."},{"metadata":{"trusted":true,"_uuid":"b52f3ac603179763fb7a8e4500e587e081291836"},"cell_type":"code","source":"log_preds,y = learn.TTA()\nprobs = np.mean(np.exp(log_preds),0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"68ee707278bc47a513e3e27b2c721103e73901eb"},"cell_type":"code","source":"accuracy_np(probs, y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1e0b4a878b246b5a412a260468d46339adc9e7f7"},"cell_type":"code","source":"preds = np.argmax(probs, axis=1)\nprobs = probs[:,1]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9cd121c56da93575beddb841374e95ab1ad61b54"},"cell_type":"markdown","source":"Confusion matrix"},{"metadata":{"trusted":true,"_uuid":"b71691f6e6c22ad51cb4d863aae015cfe81c8384"},"cell_type":"code","source":"from sklearn.metrics import confusion_matrix\ncm = confusion_matrix(y, preds)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1e47c48e4244cc71a49e79c99100e26d53e4cffe","scrolled":true},"cell_type":"code","source":"plot_confusion_matrix(cm, data.classes)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8ef681c3c814615cfd819cf5c81a911a5f5f1447"},"cell_type":"markdown","source":"1. Use lr_find() to find highest learning rate where loss is still clearly improving\n1. Train last layer with data augmentation (i.e. precompute=False) for 2–3 epochs with cycle_len=1\n1. Unfreeze all layers\n1. Set earlier layers to 3x-10x lower learning rate than next higher layer. \n1. Train full network with cycle_mult=2 until over-fitting"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}