{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
\n
\n\n

Dog breed identification\n

\n
","metadata":{}},{"cell_type":"markdown","source":"1. [Introduction](#introduction)\n1. [Data Collection](#datacollection)\n1. [Data Analysis](#dataanalysis)\n 1. [Explore Labels](#explorelabels)\n 1. [Explore Images](#exploreimages)\n 1. [Data Visualization](#datavisualization)\n1. [Data Preprocessing](#datapreprocessing)\n 1. [Data Augmentation](#dpdataaugmentation)\n 1. [Labels encoding](#labelsencoding)\n 1. [Create Dataset](#createdataset)\n1. [Model Training and Evaluation](#trainandevaluate)\n 1. [Search best models](#searchbestmodels)\n 1. [Stacking classifier](#stackingclassifier)\n1. [Submission](#submission)\n1. [Conclusion](#conclusion)","metadata":{}},{"cell_type":"markdown","source":"# Introduction ","metadata":{}},{"cell_type":"markdown","source":"In this notebook, for the task of dog breed identification, I will try to evaluate different prelearned models from Keras Applications and Keras implementation of ViT (Vision Transformer), and then build a stacking classifier from the best of them. It is also interesting to check how well models based on CNNs perform compared to Vision Transformer.\n\nVision Transformer is a very interesting architecture for image classification that uses a Transformer-like architecture over patches of the image. I will not cover this architecture in my notebook (there are a lot of resources in web described this topic), I will only check its performance for this classification task.","metadata":{}},{"cell_type":"code","source":"%matplotlib inline\nimport warnings\nwarnings.simplefilter(action='ignore', category=(RuntimeWarning, FutureWarning, UserWarning))\n\nimport time\nimport os\nimport numpy as np\nimport pandas as pd\nimport math\nimport gc\nimport matplotlib as mpl\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\nfrom PIL import Image\nimport itertools\nfrom itertools import islice\nimport tensorflow as tf\nimport keras_tuner as kt\nimport keras\nfrom keras import layers, Model\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Activation, Dropout, MaxPooling2D, GlobalAvgPool2D, Input, concatenate, Average, RandomZoom, RandomTranslation, Rescaling, Concatenate, GlobalAveragePooling2D\nfrom keras.callbacks import ReduceLROnPlateau, EarlyStopping\nfrom keras.optimizers import SGD, Adam, Nadam, RMSprop\nfrom keras.utils import to_categorical, plot_model\nfrom keras.losses import categorical_crossentropy, SparseCategoricalCrossentropy\nfrom keras.metrics import categorical_accuracy\nfrom keras.applications import Xception, ResNet50, ResNet101, InceptionV3, InceptionResNetV2, MobileNet, DenseNet121, DenseNet201, NASNetMobile, EfficientNetB0, EfficientNetB1, EfficientNetB2, EfficientNetB3, EfficientNetB4, EfficientNetB5\nfrom IPython.display import Markdown, display\nfrom sklearn.preprocessing import LabelEncoder\n\n# to make this notebook's output stable across runs\nnp.random.seed(42)\ntf.random.set_seed(42)\n\nON_KAGGLE = os.getenv(\"KAGGLE_KERNEL_RUN_TYPE\") is not None\n\nIMAGE_SIZE = (448, 448)\nBATCH_SIZE = 32\n\ndef download_from_kaggle(competition):\n command = \"kaggle competitions download -c \"\n filepath = Path(\"data/\" + competition + \".zip\")\n\n if not filepath.is_file():\n subprocess.run((command + competition).split())\n Path(\"data\").mkdir(parents=True, exist_ok=True)\n shutil.unpack_archive(competition + \".zip\", \"data\")\n shutil.move(competition + \".zip\", \"data\")","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-06-09T11:41:01.495858Z","iopub.execute_input":"2023-06-09T11:41:01.496205Z","iopub.status.idle":"2023-06-09T11:41:10.329050Z","shell.execute_reply.started":"2023-06-09T11:41:01.496175Z","shell.execute_reply":"2023-06-09T11:41:10.328083Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install --quiet vit-keras\nfrom vit_keras import vit\nfrom vit_keras import visualize","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:41:10.330880Z","iopub.execute_input":"2023-06-09T11:41:10.331522Z","iopub.status.idle":"2023-06-09T11:41:25.357340Z","shell.execute_reply.started":"2023-06-09T11:41:10.331468Z","shell.execute_reply":"2023-06-09T11:41:25.356258Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data Collection ","metadata":{}},{"cell_type":"code","source":"competition = \"dog-breed-identification\"\nif not ON_KAGGLE:\n download_from_kaggle(competition)\n labels_path = \"labels.csv\"\n submission_path = \"sample_submission.csv\"\nelse:\n labels_path = f\"/kaggle/input/{competition}/labels.csv\"\n submission_path = f\"/kaggle/input/{competition}/sample_submission.csv\"\n \nlabels_df = pd.read_csv(labels_path)\nsubmission_df = pd.read_csv(submission_path)","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:41:25.360011Z","iopub.execute_input":"2023-06-09T11:41:25.360396Z","iopub.status.idle":"2023-06-09T11:41:25.938508Z","shell.execute_reply.started":"2023-06-09T11:41:25.360359Z","shell.execute_reply":"2023-06-09T11:41:25.937539Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_folder = f\"/kaggle/input/{competition}/train/\"\ntest_folder = f\"/kaggle/input/{competition}/test/\"\ntrain_folder, test_folder","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:41:25.941419Z","iopub.execute_input":"2023-06-09T11:41:25.941792Z","iopub.status.idle":"2023-06-09T11:41:25.950706Z","shell.execute_reply.started":"2023-06-09T11:41:25.941757Z","shell.execute_reply":"2023-06-09T11:41:25.949640Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Analysis ","metadata":{}},{"cell_type":"markdown","source":"## Explore Labels ","metadata":{}},{"cell_type":"code","source":"print(f'Labels shape:\\t {labels_df.shape}',\n f'\\nSubmission shape:\\t {submission_df.shape}')","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:41:25.952151Z","iopub.execute_input":"2023-06-09T11:41:25.952711Z","iopub.status.idle":"2023-06-09T11:41:25.959425Z","shell.execute_reply.started":"2023-06-09T11:41:25.952677Z","shell.execute_reply":"2023-06-09T11:41:25.958471Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"labels_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:41:25.960723Z","iopub.execute_input":"2023-06-09T11:41:25.961309Z","iopub.status.idle":"2023-06-09T11:41:25.981451Z","shell.execute_reply.started":"2023-06-09T11:41:25.961204Z","shell.execute_reply":"2023-06-09T11:41:25.980647Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:41:25.982899Z","iopub.execute_input":"2023-06-09T11:41:25.983222Z","iopub.status.idle":"2023-06-09T11:41:26.011218Z","shell.execute_reply.started":"2023-06-09T11:41:25.983192Z","shell.execute_reply":"2023-06-09T11:41:26.010432Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Explore images ","metadata":{}},{"cell_type":"code","source":"def get_images_info(folder):\n size_images = dict()\n for file_name in os.listdir(folder):\n if file_name.endswith('.jpg'):\n file_path = os.path.join(folder, file_name)\n with Image.open(file_path) as img:\n width, heigth = img.size\n resolution = width*heigth\n root_resolution = math.sqrt(resolution)\n ratioHW = heigth / width\n size_images[file_name] = {'width': width, 'heigth': heigth, 'resolution': resolution, \n 'root_resolution': root_resolution, 'ratioHW': ratioHW}\n return size_images","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-06-09T11:41:26.012429Z","iopub.execute_input":"2023-06-09T11:41:26.012786Z","iopub.status.idle":"2023-06-09T11:41:26.020350Z","shell.execute_reply.started":"2023-06-09T11:41:26.012754Z","shell.execute_reply":"2023-06-09T11:41:26.019407Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Evaluate image sizes. This information is useful for selecting image sizes for model learning.","metadata":{}},{"cell_type":"code","source":"images_info_dict = get_images_info(train_folder)\nimages_info_pd = pd.DataFrame.from_dict(images_info_dict, orient='index')\nimages_info_pd.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:41:26.021955Z","iopub.execute_input":"2023-06-09T11:41:26.022416Z","iopub.status.idle":"2023-06-09T11:42:14.270315Z","shell.execute_reply.started":"2023-06-09T11:41:26.022385Z","shell.execute_reply":"2023-06-09T11:42:14.269389Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"images_info_pd.describe()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:42:14.274630Z","iopub.execute_input":"2023-06-09T11:42:14.274924Z","iopub.status.idle":"2023-06-09T11:42:14.304104Z","shell.execute_reply.started":"2023-06-09T11:42:14.274900Z","shell.execute_reply":"2023-06-09T11:42:14.303098Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Data Visualization ","metadata":{}},{"cell_type":"code","source":"def plot_image_from_path(image_path, label):\n img = mpimg.imread(image_path)\n plt.imshow(img)\n plt.axis('off')\n plt.title(label)\n plt.show()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:42:14.305729Z","iopub.execute_input":"2023-06-09T11:42:14.306059Z","iopub.status.idle":"2023-06-09T11:42:14.310942Z","shell.execute_reply.started":"2023-06-09T11:42:14.306029Z","shell.execute_reply":"2023-06-09T11:42:14.310013Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Plot random image.","metadata":{}},{"cell_type":"code","source":"image_item = labels_df.iloc[0]\npath_to_image = os.path.join(train_folder, f\"{image_item.id}.jpg\")\nplot_image_from_path(path_to_image, image_item.breed)","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:42:14.312437Z","iopub.execute_input":"2023-06-09T11:42:14.313130Z","iopub.status.idle":"2023-06-09T11:42:14.620114Z","shell.execute_reply.started":"2023-06-09T11:42:14.313092Z","shell.execute_reply":"2023-06-09T11:42:14.619319Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Preprocessing ","metadata":{}},{"cell_type":"markdown","source":"### Data augmentation ","metadata":{}},{"cell_type":"markdown","source":"Create a data augmentation block for further use in the learning process.","metadata":{}},{"cell_type":"code","source":"data_augmentation = keras.Sequential([\n layers.RandomFlip(\"horizontal\"), \n layers.RandomRotation(0.05),\n RandomZoom(0.1),\n], name='data_augmentation')","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:42:14.620997Z","iopub.execute_input":"2023-06-09T11:42:14.621284Z","iopub.status.idle":"2023-06-09T11:42:17.530448Z","shell.execute_reply.started":"2023-06-09T11:42:14.621258Z","shell.execute_reply":"2023-06-09T11:42:17.529526Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Labels encoding ","metadata":{}},{"cell_type":"markdown","source":"Encode breeds with a value between 0 and 119 using sklearn.preprocessing.LabelEncoder and then convert these values to one-hot encoding.","metadata":{}},{"cell_type":"code","source":"labels_df['label'] = LabelEncoder().fit_transform(labels_df.breed)\nlabels_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:42:17.531892Z","iopub.execute_input":"2023-06-09T11:42:17.532344Z","iopub.status.idle":"2023-06-09T11:42:17.546581Z","shell.execute_reply.started":"2023-06-09T11:42:17.532307Z","shell.execute_reply":"2023-06-09T11:42:17.545718Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"labels_list = labels_df['label'].to_numpy().tolist()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:42:17.548043Z","iopub.execute_input":"2023-06-09T11:42:17.548588Z","iopub.status.idle":"2023-06-09T11:42:17.559572Z","shell.execute_reply.started":"2023-06-09T11:42:17.548555Z","shell.execute_reply":"2023-06-09T11:42:17.558275Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_full_oh = tf.keras.utils.to_categorical(labels_list, num_classes=120, dtype='float32')\ny_full_oh[:2]","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:42:17.560937Z","iopub.execute_input":"2023-06-09T11:42:17.561302Z","iopub.status.idle":"2023-06-09T11:42:17.577659Z","shell.execute_reply.started":"2023-06-09T11:42:17.561270Z","shell.execute_reply":"2023-06-09T11:42:17.576538Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Create an additional dictionary for identifying breed labels from numeric values.","metadata":{}},{"cell_type":"code","source":"labels_breeds_df = labels_df[['label','breed']].copy()\nlabels_breeds_df.drop_duplicates(inplace=True)\nlabels_breeds_df.set_index('label',drop=True,inplace=True)\nindex_to_breed = labels_breeds_df.to_dict()['breed']\nlist(islice(index_to_breed.items(), 10))","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:42:17.578695Z","iopub.execute_input":"2023-06-09T11:42:17.578937Z","iopub.status.idle":"2023-06-09T11:42:17.599281Z","shell.execute_reply.started":"2023-06-09T11:42:17.578915Z","shell.execute_reply":"2023-06-09T11:42:17.598479Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Complete list of all breeds in the dataset.","metadata":{}},{"cell_type":"code","source":"dog_breeds = sorted(list(index_to_breed.values()))\ndog_breeds[:10]","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:42:17.600461Z","iopub.execute_input":"2023-06-09T11:42:17.600963Z","iopub.status.idle":"2023-06-09T11:42:17.608343Z","shell.execute_reply.started":"2023-06-09T11:42:17.600931Z","shell.execute_reply":"2023-06-09T11:42:17.607395Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(dog_breeds)","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:42:17.609705Z","iopub.execute_input":"2023-06-09T11:42:17.610749Z","iopub.status.idle":"2023-06-09T11:42:17.619239Z","shell.execute_reply.started":"2023-06-09T11:42:17.610714Z","shell.execute_reply":"2023-06-09T11:42:17.618506Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Create Dataset ","metadata":{}},{"cell_type":"markdown","source":"Create functions for creating tf.data.Dataset. These functions will be used below in the Model Training section to create datasets with different parameters. To create datasets, I use image_dataset_from_directory function. I also tried from_tensor_slices function with additional image loading and preprocessing functions (including resize_with_pad and CenterCrop), but without better results.","metadata":{}},{"cell_type":"code","source":"def prepare_ds(ds, prep_input_fx, cache_ds=False, shuffle=False, batch_size=32, prefetch=True):\n ds_has_y = isinstance(ds.element_spec, tuple)\n if ds_has_y:\n ds = ds.map(lambda x, y: (prep_input_fx(x), y))\n else:\n ds = ds.map(lambda x: prep_input_fx(x))\n if cache_ds:\n ds = ds.cache()\n if shuffle:\n ds = ds.shuffle(128)\n ds = ds.batch(batch_size)\n if prefetch:\n ds = ds.prefetch(tf.data.AUTOTUNE)\n return ds","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:42:17.620610Z","iopub.execute_input":"2023-06-09T11:42:17.621275Z","iopub.status.idle":"2023-06-09T11:42:17.628602Z","shell.execute_reply.started":"2023-06-09T11:42:17.621239Z","shell.execute_reply":"2023-06-09T11:42:17.627925Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def create_raw_ds(train_folder, labels_list, split_size=0.1, image_size=(224,224)):\n train_ds_raw, val_ds_raw = tf.keras.utils.image_dataset_from_directory(\n train_folder,\n# seed=SEED,\n labels=labels_list,\n label_mode='categorical',\n validation_split=0.1,\n shuffle=False,\n subset=\"both\",\n image_size=image_size,\n batch_size=None,\n crop_to_aspect_ratio=False\n )\n return train_ds_raw, val_ds_raw","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:42:17.629803Z","iopub.execute_input":"2023-06-09T11:42:17.630420Z","iopub.status.idle":"2023-06-09T11:42:17.638905Z","shell.execute_reply.started":"2023-06-09T11:42:17.630381Z","shell.execute_reply":"2023-06-09T11:42:17.638219Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Create final datasets for training after using the Keras tuner.","metadata":{}},{"cell_type":"code","source":"train_ds_raw, val_ds_raw = create_raw_ds(train_folder, labels_list, image_size=IMAGE_SIZE)","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:42:17.640209Z","iopub.execute_input":"2023-06-09T11:42:17.640879Z","iopub.status.idle":"2023-06-09T11:42:22.999203Z","shell.execute_reply.started":"2023-06-09T11:42:17.640846Z","shell.execute_reply":"2023-06-09T11:42:22.998204Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_ds_raw = tf.keras.utils.image_dataset_from_directory(\n test_folder,\n# seed=SEED,\n labels=None,\n shuffle=False,\n image_size=IMAGE_SIZE,\n batch_size=None,\n crop_to_aspect_ratio=False\n)","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:42:23.000561Z","iopub.execute_input":"2023-06-09T11:42:23.001364Z","iopub.status.idle":"2023-06-09T11:42:45.942901Z","shell.execute_reply.started":"2023-06-09T11:42:23.001327Z","shell.execute_reply":"2023-06-09T11:42:45.941988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I do not use the .cache function here because it is very memory consuming and can quickly overflow all RAM with large enough image sizes.","metadata":{}},{"cell_type":"code","source":"train_ds = train_ds_raw.batch(BATCH_SIZE).prefetch(buffer_size=tf.data.AUTOTUNE)\nval_ds = val_ds_raw.batch(BATCH_SIZE)\ntest_ds = test_ds_raw.batch(BATCH_SIZE)\ntrain_ds.element_spec, val_ds.element_spec, test_ds.element_spec","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:42:45.944315Z","iopub.execute_input":"2023-06-09T11:42:45.944665Z","iopub.status.idle":"2023-06-09T11:42:45.959541Z","shell.execute_reply.started":"2023-06-09T11:42:45.944632Z","shell.execute_reply":"2023-06-09T11:42:45.958452Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Data Visualization","metadata":{}},{"cell_type":"markdown","source":"Review and visualize images in the created dataset.","metadata":{}},{"cell_type":"code","source":"def plot_ds_images(ds):\n plt.figure(figsize=(10, 10))\n for images, labels in ds.take(1):\n lbl=labels.numpy()\n for i in range(9):\n ax = plt.subplot(3, 3, i + 1)\n plt.imshow(images[i].numpy().astype('uint8'))\n plt.title(index_to_breed[lbl[i].argmax()])\n plt.axis(\"off\")","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:42:45.960814Z","iopub.execute_input":"2023-06-09T11:42:45.961237Z","iopub.status.idle":"2023-06-09T11:42:45.969510Z","shell.execute_reply.started":"2023-06-09T11:42:45.961204Z","shell.execute_reply":"2023-06-09T11:42:45.967512Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_ds_images(train_ds)","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:42:45.970542Z","iopub.execute_input":"2023-06-09T11:42:45.971031Z","iopub.status.idle":"2023-06-09T11:42:47.568761Z","shell.execute_reply.started":"2023-06-09T11:42:45.970997Z","shell.execute_reply":"2023-06-09T11:42:47.567930Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_ds_images(val_ds)","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:42:47.570081Z","iopub.execute_input":"2023-06-09T11:42:47.570665Z","iopub.status.idle":"2023-06-09T11:42:49.120281Z","shell.execute_reply.started":"2023-06-09T11:42:47.570631Z","shell.execute_reply":"2023-06-09T11:42:49.119484Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Visualize the attention map from the Vision Transformer model.","metadata":{}},{"cell_type":"code","source":"def plot_attention_map(original_image, attention_image):\n fig, (ax1, ax2) = plt.subplots(ncols = 2)\n ax1.axis('off')\n ax2.axis('off')\n ax1.set_title('Original')\n ax2.set_title('Attention Map')\n _ = ax1.imshow(original_image)\n _ = ax2.imshow(attention_image)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-06-09T11:42:49.131321Z","iopub.execute_input":"2023-06-09T11:42:49.131892Z","iopub.status.idle":"2023-06-09T11:42:49.137970Z","shell.execute_reply.started":"2023-06-09T11:42:49.131857Z","shell.execute_reply":"2023-06-09T11:42:49.137116Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vit_model_test = vit.vit_b32(\n image_size = IMAGE_SIZE,\n activation = 'softmax',\n pretrained = True,\n include_top = True,\n pretrained_top = False,\n classes = 120\n)","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:42:49.139298Z","iopub.execute_input":"2023-06-09T11:42:49.140290Z","iopub.status.idle":"2023-06-09T11:42:57.792022Z","shell.execute_reply.started":"2023-06-09T11:42:49.140257Z","shell.execute_reply":"2023-06-09T11:42:57.790990Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for image, _ in train_ds_raw.take(1):\n image_example = image.numpy().astype('uint8')\n break\nattention_map = visualize.attention_map(model = vit_model_test, image = image_example)\nplot_attention_map(image_example, attention_map)","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:42:57.793701Z","iopub.execute_input":"2023-06-09T11:42:57.794063Z","iopub.status.idle":"2023-06-09T11:43:11.486224Z","shell.execute_reply.started":"2023-06-09T11:42:57.794028Z","shell.execute_reply":"2023-06-09T11:43:11.485397Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del vit_model_test\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:43:11.487696Z","iopub.execute_input":"2023-06-09T11:43:11.488278Z","iopub.status.idle":"2023-06-09T11:43:11.778612Z","shell.execute_reply.started":"2023-06-09T11:43:11.488246Z","shell.execute_reply":"2023-06-09T11:43:11.777527Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model Training and Evaluation ","metadata":{}},{"cell_type":"code","source":"early_stopping = EarlyStopping(\n monitor='val_categorical_accuracy', \n patience=10,\n verbose=1,\n restore_best_weights = True)\nreduce_lr_on_plateau = ReduceLROnPlateau(\n monitor='val_categorical_accuracy', \n patience=3, verbose=1, factor=0.3\n)","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:43:11.780620Z","iopub.execute_input":"2023-06-09T11:43:11.780997Z","iopub.status.idle":"2023-06-09T11:43:11.787662Z","shell.execute_reply.started":"2023-06-09T11:43:11.780963Z","shell.execute_reply":"2023-06-09T11:43:11.786680Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def grid_search_report(grid_search, res_num=10):\n tuner_report_df = pd.DataFrame()\n trials = grid_search.oracle.get_best_trials(num_trials=15)\n for trial in trials:\n tuner_report_df = tuner_report_df.append(\n trial.hyperparameters.get_config()[\"values\"] | {\"Score\": trial.score}, \n ignore_index=True)\n display(tuner_report_df.head(res_num))","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:43:11.788923Z","iopub.execute_input":"2023-06-09T11:43:11.789326Z","iopub.status.idle":"2023-06-09T11:43:11.798046Z","shell.execute_reply.started":"2023-06-09T11:43:11.789294Z","shell.execute_reply":"2023-06-09T11:43:11.797034Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_loss_and_acc(y_true, y_pred):\n cat_ce = tf.keras.losses.CategoricalCrossentropy()\n loss_value = cat_ce(y_true, y_pred).numpy()\n cat_acc = tf.keras.metrics.CategoricalAccuracy()\n cat_acc.update_state(y_true, y_pred)\n acc_value = cat_acc.result().numpy()\n return [loss_value, acc_value]","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Search best models ","metadata":{}},{"cell_type":"markdown","source":"Let's evaluate different pre-trained models from official Keras applications and Keras implementation of ViT (Vision Transformer) using Keras GridSearch tuner.\n\nIn model search I use not very big image size and only part of images in training dataset. It's not so crusial because main goal of this section is to compare performance of models and to choose best ones (of course it can be something different on full set and other image sizes).\n\nEach application will be created without top layers (added manually), with pre-trained weights on imagenet dataset. All base model layers are frozen (used transfer learning). For other hyperparameters like learning rate, optimizer and so on I chose standart values.","metadata":{}},{"cell_type":"code","source":"IMAGE_SIZE_search = (288, 288)","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:43:11.799478Z","iopub.execute_input":"2023-06-09T11:43:11.799910Z","iopub.status.idle":"2023-06-09T11:43:11.810215Z","shell.execute_reply.started":"2023-06-09T11:43:11.799877Z","shell.execute_reply":"2023-06-09T11:43:11.809196Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Create a helper dictionary for use in the Keras tuner for building appropriate models and preprocessing images. Keys of the dictionary are model names, values are model classes and preprocessing functions.","metadata":{}},{"cell_type":"code","source":"models_search = {\n 'Xception': [Xception, keras.applications.xception.preprocess_input],\n 'ResNet101': [ResNet101, keras.applications.resnet.preprocess_input],\n 'InceptionV3': [InceptionV3, keras.applications.inception_v3.preprocess_input],\n 'InceptionResNetV2': [InceptionResNetV2, keras.applications.inception_resnet_v2.preprocess_input],\n 'MobileNetV2': [MobileNet, keras.applications.mobilenet.preprocess_input],\n 'DenseNet201': [DenseNet201, keras.applications.densenet.preprocess_input],\n 'NASNetLarge': [NASNetMobile, keras.applications.nasnet.preprocess_input],\n 'EfficientNetB5': [EfficientNetB5, keras.applications.efficientnet.preprocess_input],\n 'EfficientNetB4': [EfficientNetB4, keras.applications.efficientnet.preprocess_input],\n 'EfficientNetB3': [EfficientNetB3, keras.applications.efficientnet.preprocess_input],\n 'vit_b16': [vit.vit_b16, vit.preprocess_inputs],\n 'vit_b32': [vit.vit_b32, vit.preprocess_inputs],\n 'vit_l16': [vit.vit_l16, vit.preprocess_inputs],\n 'vit_l32': [vit.vit_l32, vit.preprocess_inputs],\n }","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:43:11.811296Z","iopub.execute_input":"2023-06-09T11:43:11.812047Z","iopub.status.idle":"2023-06-09T11:43:11.824505Z","shell.execute_reply.started":"2023-06-09T11:43:11.812013Z","shell.execute_reply":"2023-06-09T11:43:11.823535Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Create functions for building models in Keras Tuner.","metadata":{}},{"cell_type":"code","source":"def get_keras_model(app_class, shape, base_locked=True, learning_rate=0.001):\n base_model = app_class(\n include_top=False,\n weights='imagenet',\n input_shape=shape,\n )\n inputs = keras.Input(shape=shape)\n aug_inputs = data_augmentation(inputs)\n x = base_model(aug_inputs)\n x = keras.layers.GlobalAveragePooling2D()(x)\n x = Dropout(0.7)(x)\n outputs = Dense(120, activation='softmax')(x)\n model = keras.Model(inputs, outputs)\n base_model.trainable = not base_locked\n optimizer = Adam(learning_rate=learning_rate)\n model.compile(loss=categorical_crossentropy, \n optimizer=optimizer,\n metrics=[categorical_accuracy]\n )\n return model","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:43:11.825752Z","iopub.execute_input":"2023-06-09T11:43:11.826289Z","iopub.status.idle":"2023-06-09T11:43:11.835908Z","shell.execute_reply.started":"2023-06-09T11:43:11.826256Z","shell.execute_reply":"2023-06-09T11:43:11.834958Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_vit_model(app_class, shape, base_locked=True, learning_rate=0.001):\n base_model = app_class(\n image_size = shape[:2],\n activation = 'softmax',\n pretrained = True,\n include_top = False,\n pretrained_top = False,\n classes = 120\n )\n inputs = keras.Input(shape=shape)\n aug_inputs = data_augmentation(inputs)\n x = base_model(aug_inputs)\n x = Dropout(0.7)(x)\n outputs = Dense(120, activation='softmax')(x)\n model = keras.Model(inputs, outputs)\n base_model.trainable = not base_locked\n optimizer = Adam(learning_rate=learning_rate)\n model.compile(loss=categorical_crossentropy, \n optimizer=optimizer,\n metrics=[categorical_accuracy]\n )\n return model","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:43:11.838229Z","iopub.execute_input":"2023-06-09T11:43:11.839156Z","iopub.status.idle":"2023-06-09T11:43:11.847162Z","shell.execute_reply.started":"2023-06-09T11:43:11.839102Z","shell.execute_reply":"2023-06-09T11:43:11.846322Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Create main builder function.","metadata":{}},{"cell_type":"code","source":"def model_app_builder(hp):\n models_names = list(models_search.keys())\n model_type = hp.Choice('model_type', values=models_names)\n model_class = models_search[model_type][0]\n if model_type.startswith('vit'):\n model = get_vit_model(model_class, IMAGE_SIZE_search + (3,))\n else:\n model = get_keras_model(model_class, IMAGE_SIZE_search + (3,))\n return model","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:43:11.848456Z","iopub.execute_input":"2023-06-09T11:43:11.849188Z","iopub.status.idle":"2023-06-09T11:43:11.860933Z","shell.execute_reply.started":"2023-06-09T11:43:11.849157Z","shell.execute_reply":"2023-06-09T11:43:11.859903Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I create a subclass of GridSearch with redefined \"run_trial\" method, because for each model I need slightly different datasets with own processing function.","metadata":{}},{"cell_type":"code","source":"class CustomGridTuner(kt.tuners.GridSearch):\n def run_trial(self, trial, train_ds, *args, **kwargs):\n val_ds = kwargs['validation_data']\n hp = trial.hyperparameters\n model_type = hp.Choice('model_type', values = list(models_search.keys()))\n prep_input = models_search[model_type][1]\n train_ds = prepare_ds(train_ds, prep_input)\n val_ds = prepare_ds(val_ds, prep_input, prefetch=False)\n kwargs['validation_data'] = val_ds\n return super(CustomGridTuner, self).run_trial(trial, train_ds, *args, **kwargs)","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:43:11.862111Z","iopub.execute_input":"2023-06-09T11:43:11.862613Z","iopub.status.idle":"2023-06-09T11:43:11.872060Z","shell.execute_reply.started":"2023-06-09T11:43:11.862581Z","shell.execute_reply":"2023-06-09T11:43:11.871030Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Create training and validation datasets with relatively small image sizes.","metadata":{}},{"cell_type":"code","source":"train_ds_raw, val_ds_raw = create_raw_ds(train_folder, labels_list, image_size=IMAGE_SIZE_search)","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:43:11.873256Z","iopub.execute_input":"2023-06-09T11:43:11.873710Z","iopub.status.idle":"2023-06-09T11:43:17.006466Z","shell.execute_reply.started":"2023-06-09T11:43:11.873620Z","shell.execute_reply":"2023-06-09T11:43:17.005559Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_ds_raw.element_spec, val_ds_raw.element_spec","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:43:17.007812Z","iopub.execute_input":"2023-06-09T11:43:17.008419Z","iopub.status.idle":"2023-06-09T11:43:17.016136Z","shell.execute_reply.started":"2023-06-09T11:43:17.008383Z","shell.execute_reply":"2023-06-09T11:43:17.015151Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Specify only the part of the training dataset to be used in the Keras tuner to speed up the grid search.","metadata":{}},{"cell_type":"code","source":"train_ratio_search = 0.4\ntrain_len = int(train_ds_raw.cardinality())\ntrain_len_search = int(round(train_len * train_ratio_search, 0))\nprint(f'Searhing ds length: {train_len_search}')","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:43:17.017790Z","iopub.execute_input":"2023-06-09T11:43:17.018572Z","iopub.status.idle":"2023-06-09T11:43:17.030935Z","shell.execute_reply.started":"2023-06-09T11:43:17.018540Z","shell.execute_reply":"2023-06-09T11:43:17.030060Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Create a CustomGridTuner object and start searching.","metadata":{}},{"cell_type":"code","source":"grid_search_tuner = CustomGridTuner(\n hypermodel=model_app_builder,\n objective=\"val_categorical_accuracy\",\n seed=42,\n overwrite=True\n)\ngrid_search_tuner.search(\n train_ds_raw.take(train_len_search),\n validation_data=val_ds_raw,\n epochs=7,\n)","metadata":{"execution":{"iopub.status.busy":"2023-06-09T11:43:17.032085Z","iopub.execute_input":"2023-06-09T11:43:17.035697Z","iopub.status.idle":"2023-06-09T13:02:23.401143Z","shell.execute_reply.started":"2023-06-09T11:43:17.035672Z","shell.execute_reply":"2023-06-09T13:02:23.400191Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"grid_search_report(grid_search_tuner, 14)","metadata":{"execution":{"iopub.status.busy":"2023-06-09T13:02:23.402803Z","iopub.execute_input":"2023-06-09T13:02:23.403182Z","iopub.status.idle":"2023-06-09T13:02:23.433328Z","shell.execute_reply.started":"2023-06-09T13:02:23.403137Z","shell.execute_reply":"2023-06-09T13:02:23.431960Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we see, Vision Transformer Large models are best, some of Keras applications also has good performance.","metadata":{}},{"cell_type":"code","source":"gc.collect()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-06-09T13:02:23.434744Z","iopub.execute_input":"2023-06-09T13:02:23.435313Z","iopub.status.idle":"2023-06-09T13:02:25.542484Z","shell.execute_reply.started":"2023-06-09T13:02:23.435280Z","shell.execute_reply":"2023-06-09T13:02:25.541461Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Stacking classifier ","metadata":{}},{"cell_type":"markdown","source":"Select some best models for further stacking and training.\n\nI use here only transfer learning without fine-tuning (I also tried it, but without better results). Therefore I use here technique for feature extraction from base stacking model (without top classifier layer) and pass them for training top MLP model (again not only NN but other models like logistic regression can be used). This approach also reduces the training time.","metadata":{}},{"cell_type":"code","source":"models_for_stacking = ['vit_l32', 'EfficientNetB3', 'InceptionResNetV2']","metadata":{"execution":{"iopub.status.busy":"2023-06-09T13:02:25.549289Z","iopub.execute_input":"2023-06-09T13:02:25.549612Z","iopub.status.idle":"2023-06-09T13:02:25.728649Z","shell.execute_reply.started":"2023-06-09T13:02:25.549586Z","shell.execute_reply":"2023-06-09T13:02:25.727560Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class MCDropout(tf.keras.layers.Dropout):\n def call(self, inputs, training=False):\n return super().call(inputs, training=True)","metadata":{"execution":{"iopub.status.busy":"2023-06-09T13:30:18.919880Z","iopub.execute_input":"2023-06-09T13:30:18.920275Z","iopub.status.idle":"2023-06-09T13:30:18.926179Z","shell.execute_reply.started":"2023-06-09T13:30:18.920245Z","shell.execute_reply":"2023-06-09T13:30:18.925009Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_keras_model_feat(app_class, shape, inputs, prep_input_fx):\n base_model = app_class(\n include_top=False,\n weights='imagenet',\n input_shape=shape,\n )\n x = prep_input_fx(inputs)\n x = base_model(x)\n outputs = keras.layers.GlobalAveragePooling2D()(x)\n return outputs","metadata":{"execution":{"iopub.status.busy":"2023-06-09T13:02:25.730360Z","iopub.execute_input":"2023-06-09T13:02:25.730755Z","iopub.status.idle":"2023-06-09T13:02:26.499427Z","shell.execute_reply.started":"2023-06-09T13:02:25.730719Z","shell.execute_reply":"2023-06-09T13:02:26.498177Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_vit_model_feat(app_class, shape, inputs, prep_input_fx):\n base_model = app_class(\n image_size = shape[:2],\n activation = 'softmax',\n pretrained = True,\n include_top = False,\n pretrained_top = False,\n classes = 120\n ) \n x = prep_input_fx(inputs)\n outputs = base_model(x)\n return outputs","metadata":{"execution":{"iopub.status.busy":"2023-06-09T13:02:26.502434Z","iopub.execute_input":"2023-06-09T13:02:26.503383Z","iopub.status.idle":"2023-06-09T13:02:26.687480Z","shell.execute_reply.started":"2023-06-09T13:02:26.503345Z","shell.execute_reply":"2023-06-09T13:02:26.686467Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def build_feat_model(models_names, shape, aug_layer=None):\n all_outputs = []\n inputs = keras.Input(shape=shape)\n if aug_layer != None:\n aug_inputs = aug_layer(inputs)\n else:\n aug_inputs = inputs\n for model_type in models_names:\n model_class = models_search[model_type][0]\n model_prep_input = models_search[model_type][1]\n if model_type.startswith('vit'):\n model_outputs = get_vit_model_feat(model_class, shape, aug_inputs, model_prep_input)\n else:\n model_outputs = get_keras_model_feat(model_class, shape, aug_inputs, model_prep_input)\n all_outputs.append(model_outputs)\n concat_outputs = Concatenate()(all_outputs)\n model = Model(inputs, concat_outputs)\n return model","metadata":{"execution":{"iopub.status.busy":"2023-06-09T13:02:26.688995Z","iopub.execute_input":"2023-06-09T13:02:26.689460Z","iopub.status.idle":"2023-06-09T13:02:27.463610Z","shell.execute_reply.started":"2023-06-09T13:02:26.689426Z","shell.execute_reply":"2023-06-09T13:02:27.462666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_features = build_feat_model(models_for_stacking, IMAGE_SIZE + (3,), data_augmentation)\nmodel_features.summary()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T13:02:27.465479Z","iopub.execute_input":"2023-06-09T13:02:27.465870Z","iopub.status.idle":"2023-06-09T13:03:04.049818Z","shell.execute_reply.started":"2023-06-09T13:02:27.465838Z","shell.execute_reply":"2023-06-09T13:03:04.048685Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_features_from_model(model, ds, count=1):\n features = []\n ds_has_y = isinstance(ds.element_spec, tuple)\n if ds_has_y:\n labels = []\n y = np.concatenate([y for x, y in ds], axis=0)\n for i in range(count):\n predictions = model.predict(ds)\n features.append(predictions)\n if ds_has_y:\n labels.append(y)\n if ds_has_y:\n return np.concatenate(features), np.concatenate(labels)\n else:\n return np.concatenate(features)","metadata":{"execution":{"iopub.status.busy":"2023-06-09T13:03:04.053557Z","iopub.execute_input":"2023-06-09T13:03:04.054315Z","iopub.status.idle":"2023-06-09T13:03:04.068146Z","shell.execute_reply.started":"2023-06-09T13:03:04.054271Z","shell.execute_reply":"2023-06-09T13:03:04.066638Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"With param count we can multiply dataset and each count will get slightly different features because of using data augmentation layer.","metadata":{}},{"cell_type":"code","source":"features_train, y_train = get_features_from_model(model_features, train_ds, count=2)\nfeatures_val, y_val = get_features_from_model(model_features, val_ds)\nprint(features_train.shape, y_train.shape, features_val.shape, y_val.shape)","metadata":{"execution":{"iopub.status.busy":"2023-06-09T13:03:04.072978Z","iopub.execute_input":"2023-06-09T13:03:04.075330Z","iopub.status.idle":"2023-06-09T13:18:53.078395Z","shell.execute_reply.started":"2023-06-09T13:03:04.075290Z","shell.execute_reply":"2023-06-09T13:18:53.077401Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After extracting features from the base model, create Dense top model and train it on those features. I use MCDropout here instead of regular dropout. It allows to slightly increase the performance on the test dataset.","metadata":{}},{"cell_type":"code","source":"dnn_model = keras.models.Sequential([\n Input(features_train.shape[1:]),\n MCDropout(0.7),\n Dense(120, activation='softmax')\n])\ndnn_model.compile(\n optimizer=keras.optimizers.Adam(0.001),\n loss=categorical_crossentropy,\n metrics=[\"categorical_accuracy\"]\n)","metadata":{"execution":{"iopub.status.busy":"2023-06-09T13:30:39.192422Z","iopub.execute_input":"2023-06-09T13:30:39.193391Z","iopub.status.idle":"2023-06-09T13:30:39.280010Z","shell.execute_reply.started":"2023-06-09T13:30:39.193356Z","shell.execute_reply":"2023-06-09T13:30:39.278988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history_dnn = dnn_model.fit(\n features_train, \n y_train,\n validation_data=(features_val, y_val),\n batch_size=128,\n epochs=50,\n callbacks=[early_stopping, reduce_lr_on_plateau]\n)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-06-09T13:30:47.724344Z","iopub.execute_input":"2023-06-09T13:30:47.725348Z","iopub.status.idle":"2023-06-09T13:31:01.967700Z","shell.execute_reply.started":"2023-06-09T13:30:47.725310Z","shell.execute_reply":"2023-06-09T13:31:01.966649Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_val_probas = np.stack([dnn_model.predict(features_val)\n for sample in range(50)])\nval_predictions = y_val_probas.mean(axis=0)","metadata":{"_kg_hide-output":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"val_loss, val_accuracy = get_loss_and_acc(y_val, val_predictions)\nprint(f\"Validation loss: {val_loss}\", \n f\"\\nValidation accuracy: {val_accuracy}\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gc.collect()","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submission ","metadata":{}},{"cell_type":"code","source":"features_test = get_features_from_model(model_features, test_ds)\nfeatures_test.shape","metadata":{"execution":{"iopub.status.busy":"2023-06-09T13:19:09.951985Z","iopub.execute_input":"2023-06-09T13:19:09.952351Z","iopub.status.idle":"2023-06-09T13:26:45.799471Z","shell.execute_reply.started":"2023-06-09T13:19:09.952318Z","shell.execute_reply":"2023-06-09T13:26:45.798468Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_probas = np.stack([dnn_model.predict(features_test)\n for sample in range(80)])\npredictions = y_probas.mean(axis=0)","metadata":{"execution":{"iopub.status.busy":"2023-06-09T13:40:59.650810Z","iopub.execute_input":"2023-06-09T13:40:59.651193Z"},"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_df.loc[:, dog_breeds] = predictions\nsubmission_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-06-09T13:40:13.014886Z","iopub.execute_input":"2023-06-09T13:40:13.015247Z","iopub.status.idle":"2023-06-09T13:40:13.453361Z","shell.execute_reply.started":"2023-06-09T13:40:13.015218Z","shell.execute_reply":"2023-06-09T13:40:13.452444Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_df.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2023-06-09T13:40:15.249816Z","iopub.execute_input":"2023-06-09T13:40:15.250961Z","iopub.status.idle":"2023-06-09T13:40:17.741699Z","shell.execute_reply.started":"2023-06-09T13:40:15.250912Z","shell.execute_reply":"2023-06-09T13:40:17.740402Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Conclusion ","metadata":{}},{"cell_type":"markdown","source":"Comparison of CNN and ViT models showed that the latter (large versions with patch equal to 32) get better performance (I also tested them on full dataset with different image sizes). Staking the models allowed to increase the individual scores. The achieved loss is not very low (compared to the top leaderboard positions with losses of 0 and little more), but good enough.","metadata":{}}]}