{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":5441,"databundleVersionId":38425,"sourceType":"competition"},{"sourceId":9900,"sourceType":"datasetVersion","datasetId":6209},{"sourceId":110819,"sourceType":"datasetVersion","datasetId":57539},{"sourceId":110930,"sourceType":"datasetVersion","datasetId":57578}],"dockerImageVersionId":11543,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Tutorial Keras: Transfer Learning with ResNet50 for image classification on Cats & Dogs dataset\n\n### Suni Kumar","metadata":{"_uuid":"40733c39647bac5056ba0dada8409d6184fefb68"}},{"cell_type":"markdown","source":"This kernel is intended to be a tutorial on Keras around image files handling for Transfer Learning using pre-trained weights from ResNet50 convnet.\n\nThough loading all train & test images resized (224 x 224 x 3) in memory would have incurred ~4.9GB of memory, the plan was to batch source image data during the training, validation & testing pipeline. Keras ImageDataGenerator supports batch sourcing image data for all training, validation and testing. Actually, it is quite clean and easy to use Keras ImageDataGenerator except few limitations (listed at the end).\n\nKeras ImageDataGenerator expects labeled training images to be available in certain folder heirarchy, 'train' data was manually split into 10k for training & 2.5k for validation and re-arranged into the desired folder hierarchy. Even 'test' images had to rearranged due to a known issue in flow_from_directory.","metadata":{"_uuid":"0182259264d23ccfb3c27d530ed10fb4ba7a35da"}},{"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.applications import ResNet50\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras.layers import Dense, Flatten, GlobalAveragePooling2D\nimport os","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2025-02-13T03:30:34.073693Z","iopub.execute_input":"2025-02-13T03:30:34.073971Z","iopub.status.idle":"2025-02-13T03:30:34.082465Z","shell.execute_reply.started":"2025-02-13T03:30:34.073926Z","shell.execute_reply":"2025-02-13T03:30:34.081909Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Global Constants","metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false,"jupyter":{"outputs_hidden":true}}},{"cell_type":"code","source":"train_dir = \"/kaggle/input/catsdogs-trainvalid-80pc-prepd/trainvalidfull4keras/train/\"\nvalid_dir = \"/kaggle/input/catsdogs-trainvalid-80pc-prepd/trainvalidfull4keras/valid\"\ntest_dir = \"/kaggle/input/test-files-prepd/test4keras/test\"","metadata":{"trusted":true,"_uuid":"d157dc1311928d1ada13ed2ef4a34c4ea5c0b538","execution":{"iopub.status.busy":"2025-02-13T03:30:34.083283Z","iopub.execute_input":"2025-02-13T03:30:34.083464Z","iopub.status.idle":"2025-02-13T03:30:34.096604Z","shell.execute_reply.started":"2025-02-13T03:30:34.083431Z","shell.execute_reply":"2025-02-13T03:30:34.095936Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Image parameters\nIMG_SIZE = (224, 224)\nBATCH_SIZE = 100\n\ntrain_datagen = ImageDataGenerator(rescale=1./255,)\nvalid_datagen = ImageDataGenerator(rescale=1./255)\n\ntrain_generator = train_datagen.flow_from_directory(\n    train_dir,\n    target_size=IMG_SIZE,\n    batch_size=BATCH_SIZE,\n    class_mode='categorical',\n    shuffle=True  \n)\n\nvalid_generator = valid_datagen.flow_from_directory(\n    valid_dir,\n    target_size=IMG_SIZE,\n    batch_size=BATCH_SIZE,\n    class_mode='categorical'\n)","metadata":{"trusted":true,"_uuid":"ee0da10690e2848537102cb0c74f9fe19a50431b","execution":{"iopub.status.busy":"2025-02-13T03:30:34.097899Z","iopub.execute_input":"2025-02-13T03:30:34.098199Z","iopub.status.idle":"2025-02-13T03:30:40.092835Z","shell.execute_reply.started":"2025-02-13T03:30:34.098135Z","shell.execute_reply":"2025-02-13T03:30:40.092318Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### ResNet50\n* Notice that resnet50 folder has 2 pre-trained weights files... xyz_tf_kernels.h5 & xyz_tf_kernels_NOTOP.h5\n* The xyz_tf_kernels.h5 weights is useful for pure prediction of test image and this prediction will rely completely on ResNet50 pre-trained weights, i.e., it does not expected any training from our side\n* Out intention in this kernel is Transfer Learning by using ResNet50 pre-trained weights except its TOP layer, i.e., the xyz_tf_kernels_NOTOP.h5 weights... Use this weights as initial weight for training new layer using train images","metadata":{"_uuid":"b651539464d36c7443a601e914479f0d2153f435"}},{"cell_type":"code","source":"from tensorflow.keras import layers, models, regularizers\nfrom tensorflow.keras.applications import ResNet50\nfrom tensorflow.keras.layers import BatchNormalization, Activation, GlobalAveragePooling2D, Dense, Dropout\nfrom tensorflow.keras.optimizers import Adam, SGD\nfrom tensorflow.keras.applications.resnet50 import preprocess_input\nfrom tensorflow.keras.regularizers import l2\n\n# Define constants\nIMG_SHAPE = (224, 224, 3) \nNUM_CLASSES = 2  \n\nresnet_base = ResNet50(include_top = False, pooling = 'avg', weights = '/kaggle/input/resnet50/resnet50_weights_tf_dim_ordering_tf_kernels_notop.h5')\n\n# for layer in resnet_base.layers[:-10]:  \n#     layer.trainable = False\n\nmodel = models.Sequential()\n\nmodel.add(resnet_base)\n\n# model.add(Dense(512, activation='relu'))\n# model.add(Dropout(0.5))  \nmodel.add(Dense(2, activation='softmax', kernel_regularizer=l2(0.02)))\n\nmodel.layers[0].trainable = False\n\nmodel.summary()\n","metadata":{"trusted":true,"_uuid":"2fc47c3f887e6713b5a2be38c02b38a8ec80bf97","execution":{"iopub.status.busy":"2025-02-13T03:30:40.094198Z","iopub.execute_input":"2025-02-13T03:30:40.094419Z","iopub.status.idle":"2025-02-13T03:30:49.842918Z","shell.execute_reply.started":"2025-02-13T03:30:40.094372Z","shell.execute_reply":"2025-02-13T03:30:49.842264Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Define Our Transfer Learning Network Model Consisting of 2 Layers\n\nHere, we are preparing specification or blueprint of the TensorFlow DAG (directed acyclcic graph) for just the MODEL part.","metadata":{"_uuid":"4ebad39fff164624f5616a83cd8831308431da7d"}},{"cell_type":"code","source":"from tensorflow.keras.callbacks import EarlyStopping\nfrom tensorflow.keras.optimizers import Adam\nimport tensorflow as tf\n\n\nmodel.compile(optimizer=SGD(lr = 0.0001, decay = 1e-5, momentum = 0.9, nesterov = True),\n              loss='categorical_crossentropy',\n              metrics=['accuracy'])\n\nhistory = model.fit_generator(\n    train_generator, \n    steps_per_epoch=10,\n    validation_data=valid_generator,  \n    validation_steps=10,\n    epochs=150,\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-13T03:40:56.670239Z","iopub.execute_input":"2025-02-13T03:40:56.670505Z","iopub.status.idle":"2025-02-13T03:54:01.052898Z","shell.execute_reply.started":"2025-02-13T03:40:56.670466Z","shell.execute_reply":"2025-02-13T03:54:01.052205Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Compile Our Transfer Learning Model","metadata":{"_uuid":"3587981a5b6b1d6ff8221738d51d78cafb2dd02c"}},{"cell_type":"code","source":"","metadata":{"trusted":true,"_uuid":"670238611770f43a056332cc06efff44e20ad124"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Prepare Keras Data Generators\n\nKeras *ImageDataGenerator(...)* generates batches of tensor image data with real-time data augmentation. The data will be looped over (in batches). It is useful with large dataset to source, pre-process (resize, color conversion, image augmentation, batch normalize) & supply resulting images in batches to downstream Keras modeling components, namely *fit_generator(...)* & *predict_generator(...)* -vs- *fit(...)* & *predict(...)* for small dataset.\n\nKaggle competition rule expects Dog & Cat to be labeled as 1 & 0. Keras >> ImageDataGenerator >> flow_from_directory takes in 'classes' list for mapping it to LABEL indices otherwise treats sub-folders enumerated classes in alphabetical order, i.e., Cat is 0 & Dog is 1.","metadata":{"_uuid":"8d003a99fe4ff58b8905c7f6b5286d97a852cb7a"}},{"cell_type":"code","source":"","metadata":{"trusted":true,"_uuid":"ac9ebd909fee81c8c8d9b9fccb6590944d8106eb"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Max number of steps that these generator will have opportunity to process their source content\n# len(train_generator) should be 'no. of available train images / BATCH_SIZE_TRAINING'\n# len(valid_generator) should be 'no. of available train images / BATCH_SIZE_VALIDATION'\n","metadata":{"trusted":true,"_uuid":"5014f91b6aeae661f0bc5db5d546089a07ca5b9d","execution":{"iopub.status.busy":"2025-02-13T03:39:21.681692Z","iopub.execute_input":"2025-02-13T03:39:21.681983Z","iopub.status.idle":"2025-02-13T03:39:21.685710Z","shell.execute_reply.started":"2025-02-13T03:39:21.681923Z","shell.execute_reply":"2025-02-13T03:39:21.684783Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Train Our Model With Cats & Dogs Train (splitted) Data Set","metadata":{"_uuid":"f56e7cf9eae4c86c72468e322e7f00859a2ce9cd"}},{"cell_type":"code","source":"# Early stopping & checkpointing the best model in ../working dir & restoring that as our model for prediction\n","metadata":{"trusted":true,"_uuid":"2dfad78129d725f42110cde0270c32d7373d6d1d","execution":{"iopub.status.busy":"2025-02-13T03:39:21.686888Z","iopub.execute_input":"2025-02-13T03:39:21.687390Z","iopub.status.idle":"2025-02-13T03:39:21.697971Z","shell.execute_reply.started":"2025-02-13T03:39:21.687133Z","shell.execute_reply":"2025-02-13T03:39:21.697265Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true,"_uuid":"d108d3598210930bab431f43c6865488b01d467b"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true,"_uuid":"cf85fe3c0653aa56503b7da058ea8acf445eec6e"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Training Metrics\n\nOne of the default callbacks that is registered when training all deep learning models is the History callback. It records training metrics (training accuracy, training loss, validation loss & validation accuracy) for each epoch. Note that training accuracy & loss during epoch steps are somewhat incomplete information and they are not recorded in history.\n\nObserve that training uses early stopping, hence metrics is available for epochs run, not for NUM_EPOCHS.","metadata":{"_uuid":"1ef4dd95f6d12f9277255576645e8bfce5270b81"}},{"cell_type":"code","source":"","metadata":{"trusted":true,"_uuid":"383d7a0aa87e1508a46fcebfcd5a80d7aafb840f"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true,"_uuid":"00e51568f7e6022c6738c0e1a41080bc7152c257"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true,"_uuid":"3941e7c5d983e2ad6ae6d6da0a1415ea27e1db7e"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Observe Prediction Time With Different Batch Size\n\nWith GPU, 97s for full prediction with batch_size=100 -vs- 264s with 1. But note that to avoid ImageDataGenerator iterator repeatability, we need to use 1 as batch_size.","metadata":{"trusted":true,"_uuid":"8c6e7f35552f87b522aa32397ffb71ea732ce80b"}},{"cell_type":"code","source":"","metadata":{"trusted":true,"_uuid":"9911ff38ec4b82f5282f078aae63b2a510a9a6cc"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true,"_uuid":"e54bc22bd1a2af7d704f9164b624b79b8cbd4f56"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true,"_uuid":"39bac850385ba7129c97c1c0d389b36e7f2f0dfe"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Keras Limitations\n\n* [10/02/2018] The *validation_split* is not supported in *fit_generator*, hence its expects ImageDataGenerator for pre-splitted train & valid.\n* [10/02/2018] Model learning through *fit_generator* is not compatible for Sklearn *GridSearchCV* again *mostly* due to no support for *validation_split*.","metadata":{"_uuid":"2e5600300a330725b29c74c813135144a22cee56"}},{"cell_type":"markdown","source":"### Followup Plan\n\n1. Scale and pad and avoid aspect ratio change of original image through Keras ImageDataGenerator pre-processing insfrastructure\n2. Image augmentation\n3. Pipeline\n4. Distributed ML for Grid Search on Spark Cluster","metadata":{"_uuid":"60a34703bd652a314983948d93ee2dfe6d760ec3"}},{"cell_type":"markdown","source":"### References\n\n1. [Transfer Learning by Dan B](https://www.kaggle.com/dansbecker/transfer-learning)","metadata":{"_uuid":"422beefc8f408582e85292cec3fe8225228f2ab5"}}]}