{"cells":[{"metadata":{},"cell_type":"markdown","source":"## FEATURE EXTRACTION\n\n* The main concept used here is to extract features from the images using pretrained Models and train on them. \n* To increase the generalization we can extract features using many different models, concatenate them and use them together. \n* In this way, we can achieve high accuracy even without using high end GPU's\n* I have written the main code in the form of functions, so that you can use them easily."},{"metadata":{},"cell_type":"markdown","source":"### IMPORTING LIBRARIES"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# for garbage collection\nimport gc\n\n# for warnings\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\n# utility libraries\nimport os\nimport copy\nimport tqdm\nimport numpy as np \nimport pandas as pd \nimport cv2, random, time, shutil, csv\nimport tensorflow as tf\nimport math\n\n# keras libraries\nimport keras\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.models import Model\nfrom keras.layers import BatchNormalization, Dense, GlobalAveragePooling2D, Lambda, Dropout, InputLayer, Input\nfrom keras.utils import to_categorical\nfrom keras import backend as K","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### READING IMAGES\n\n* To preserve RAM, training is done on 6000 images from the dataset. We can achieve even higher accuracy using full dataset.\n* The Image size is set to 331, because NASNET requires the image size to be fixed.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# set image size here\nimg_size = 331\ndata_dir = '../input/dog-breed-identification'\ndata_df = pd.read_csv(os.path.join(data_dir, 'labels.csv'))\nclass_names = sorted(data_df['breed'].unique())\nprint(f\"No. of classes read - {len(class_names)}\")\ntime.sleep(1)\n\nimages_list = sorted(os.listdir(os.path.join(data_dir, 'train')))\nX = []\nY = []\ni = 0\nfor image in tqdm.tqdm(images_list[:6000]):\n    cls_name = data_df[data_df['id'] == image[:-4]].iloc[0,1]\n    cls_index = int(class_names.index(cls_name)) \n\n    # Reading RGB Images\n    image_path = os.path.join(data_dir, 'train',image)\n    orig_image = cv2.cvtColor(cv2.imread(image_path), cv2.COLOR_BGR2RGB)\n    res_image = cv2.resize(orig_image,(img_size, img_size))\n    X.append(res_image)\n    Y.append(cls_index)\n    i+=1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Converting the lists to arrays and deleting the list for memory saving"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Converting to arrays\nprint(len(X), len(Y))\nXarr = np.array(X)\nYarr = np.array(Y).reshape(-1,1)\n\ndel(X)\nprint(Xarr.shape, Yarr.shape)\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"One Hot encoding the classes"},{"metadata":{"trusted":true},"cell_type":"code","source":"# converting labels to one hot\nYarr_hot = to_categorical(Y)\nprint(Xarr.shape, Yarr_hot.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here we will prepare the training feature extraction pipeline.\n* I have made separate functions for feature extraction for training and validation to experiment with augmentations for training.\n* For now I have included random flip and brightness augmentations, feel free to add further."},{"metadata":{"trusted":true},"cell_type":"code","source":"# FEATURE EXTRACTION OF TRAINING ARRAYS\nAUTO = tf.data.experimental.AUTOTUNE\ndef get_features(model_name, data_preprocessor, data):\n    '''\n    1- Create a feature extractor to extract features from the data.\n    2- Returns the extracted features and the feature extractor.\n\n    '''\n    dataset = tf.data.Dataset.from_tensor_slices(data)\n\n\n    def preprocess(x):\n        x = tf.image.random_flip_left_right(x)\n        x = tf.image.random_brightness(x, 0.5)\n        return x\n\n    ds = dataset.map(preprocess, num_parallel_calls=AUTO).batch(64)\n\n    input_size = data.shape[1:]\n    #Prepare pipeline.\n    input_layer = Input(input_size)\n    preprocessor = Lambda(data_preprocessor)(input_layer)\n\n    base_model = model_name(weights='imagenet', include_top=False,\n                                input_shape=input_size)(preprocessor)\n\n    avg = GlobalAveragePooling2D()(base_model)\n    feature_extractor = Model(inputs = input_layer, outputs = avg)\n\n\n    #Extract feature.\n    feature_maps = feature_extractor.predict(ds, verbose=1)\n    print('Feature maps shape: ', feature_maps.shape)\n    \n    # deleting variables\n    del(feature_extractor, base_model, preprocessor, dataset)\n    gc.collect()\n    return feature_maps","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# FEATURE EXTRACTION OF VALIDAION AND TESTING ARRAYS\ndef get_valfeatures(model_name, data_preprocessor, data):\n    '''\n    Same as above except not image augmentations applied.\n    Used for feature extraction of validation and testing.\n    '''\n\n    dataset = tf.data.Dataset.from_tensor_slices(data)\n\n    ds = dataset.batch(64)\n\n    input_size = data.shape[1:]\n    #Prepare pipeline.\n    input_layer = Input(input_size)\n    preprocessor = Lambda(data_preprocessor)(input_layer)\n\n    base_model = model_name(weights='imagenet', include_top=False,\n                                input_shape=input_size)(preprocessor)\n\n    avg = GlobalAveragePooling2D()(base_model)\n    feature_extractor = Model(inputs = input_layer, outputs = avg)\n    #Extract feature.\n    feature_maps = feature_extractor.predict(ds, verbose=1)\n    print('Feature maps shape: ', feature_maps.shape)\n    return feature_maps","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As explained previously, we will use several Pretrained models to extract features and concatenate them to make the final feature vector. \n* This function takes the models, preprocessors and the training/validation tag and calls the above functions.\n* We can even add custom features here like, mean, variance and other statistical measures.\n* This helps keep the pipeline neat"},{"metadata":{"trusted":true},"cell_type":"code","source":"# RETURNING CONCATENATED FEATURES USING MODELS AND PREPROCESSORS\ndef get_concat_features(feat_func, models, preprocs, array):\n\n    print(f\"Beggining extraction with {feat_func.__name__}\\n\")\n    feats_list = []\n\n    for i in range(len(models)):\n        \n        print(f\"\\nStarting feature extraction with {models[i].__name__} using {preprocs[i].__name__}\\n\")\n        # applying the above function and storing in list\n        feats_list.append(feat_func(models[i], preprocs[i], array))\n\n    # features concatenating\n    final_feats = np.concatenate(feats_list, axis=-1)\n    # memory saving\n    del(feats_list, array)\n    gc.collect()\n\n    return final_feats\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* We are using 4 models here, InceptionV3, Xception, NASNetLarge and InceptionResnetV2. We are importing their model architecture and their preprocessors.\n* Feel free to test with other models"},{"metadata":{"trusted":true},"cell_type":"code","source":"# DEFINING models and preprocessors imports \n\nfrom keras.applications.inception_v3 import InceptionV3, preprocess_input\ninception_preprocessor = preprocess_input\n\nfrom keras.applications.xception import Xception, preprocess_input\nxception_preprocessor = preprocess_input\n\nfrom keras.applications.nasnet import NASNetLarge, preprocess_input\nnasnet_preprocessor = preprocess_input\n\nfrom keras.applications.inception_resnet_v2 import InceptionResNetV2, preprocess_input\ninc_resnet_preprocessor = preprocess_input\n\nmodels = [InceptionV3,  InceptionResNetV2, Xception, ]\npreprocs = [inception_preprocessor,  inc_resnet_preprocessor, \n            xception_preprocessor, ]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally the feature extraction part"},{"metadata":{"trusted":true},"cell_type":"code","source":"# calculating features of the data\n\nfinal_train_features = get_concat_features(get_features, models, preprocs, Xarr)\n\n#del(x_train, )\ngc.collect()\nprint('Final feature maps shape', final_train_features.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## MODEL TRAINING\n* We are using a STRATIFIED 3 FOLD Split to train a small Deep Neural network.\n* The DNN consists of the Input Layer, a Dropout for regularization and the Output Dense layer.\n* The models are trained for 80 epochs with Early stopping callback and adam optimizer.\n* The Models are stored in a list for prediction"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.callbacks import EarlyStopping\nEarlyStop_callback = keras.callbacks.EarlyStopping(monitor='val_accuracy', patience=10, restore_best_weights=True,\n                                                   verbose=0)\n\nmy_callback=[EarlyStop_callback]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import StratifiedKFold\n\nsplits = list(StratifiedKFold(n_splits=3, shuffle=True, random_state=10).split(final_train_features, Y))\n\ntrained_models = []\naccuracy = []\nlosses = []\n\n#Prepare And Train DNN model\n\nfor i, (train_idx, valid_idx) in enumerate(splits): \n\n    print(f\"\\nStarting fold {i+1}\\n\")\n    x_train_fold = final_train_features[train_idx, :]\n    y_train_fold = Yarr_hot[train_idx, :]\n    x_val_fold = final_train_features[valid_idx]\n    y_val_fold = Yarr_hot[valid_idx, :]\n\n    dnn = keras.models.Sequential([\n        InputLayer(final_train_features.shape[1:]),\n        Dropout(0.7),\n        Dense(120, activation='softmax')\n    ])\n\n    dnn.compile(optimizer='adam',\n                loss='categorical_crossentropy',\n                metrics=['accuracy'])\n\n    print(\"Training...\")\n    #Train simple DNN on extracted features.\n    h = dnn.fit(x_train_fold, y_train_fold,\n                batch_size=128,\n                epochs=80,\n                verbose=0,\n                validation_data = (x_val_fold, y_val_fold),\n                callbacks=my_callback)  # max 95.07\n\n    print(\"Evaluating model ...\")\n    model_res = dnn.evaluate(x_val_fold, y_val_fold)\n\n    accuracy.append(model_res[1])\n    losses.append(model_res[0])\n    trained_models.append(dnn)\n\nprint('\\n CV Score -')\nprint(f\"\\nAccuracy - {sum(accuracy)/len(accuracy)}\")\nprint(f\"\\nLoss - {sum(losses)/len(losses)}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# SAVING RAM\n\ndel(final_train_features, Y, Yarr_hot, Xarr)\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prediction\n* Uncomment the code for prediction"},{"metadata":{"trusted":true},"cell_type":"code","source":"# TEST IMAGES\n#test_images_list = sorted(os.listdir(os.path.join(data_dir, 'test')))\n#X = []\n#i = 0\n#for image in tqdm.tqdm(test_images_list):\n\n#    image_path = os.path.join(data_dir, 'test',image)\n#    orig_image = cv2.cvtColor(cv2.imread(image_path), cv2.COLOR_BGR2RGB)\n#    res_image = cv2.resize(orig_image,(img_size, img_size))\n#    X.append(res_image)\n#    i+=1\n\n#Xtesarr = np.array(X)\n\n#del(X)\n#gc.collect()\n\n#Xtesarr.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# FEATURE EXTRACTION OF TEST IMAGES\n#test_features = get_concat_features(get_valfeatures, models, preprocs, Xtesarr)\n\n#del(Xtesarr)\n#gc.collect()\n#print('Final feature maps shape', test_features.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#y_pred_norm = trained_models[0].predict(test_features, batch_size=128)/3\n#for dnn in trained_models[1:]:\n#    y_pred_norm += dnn.predict(test_features, batch_size=128)/3\n\n#y_pred_norm.shape\n\n#df.iloc[:, 1:] = y_pred_norm\n#df.to_csv('submission.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## The END\n\nIf you have any suggestions or advice for me or for the notebook, do comment. I will be glad to hear you.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}