{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"dockerImageVersionId":30746,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import math\n\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\nfrom sklearn.model_selection import train_test_split\n\nimport tensorflow as tf\n\nfrom transformers import ViTImageProcessor, TFViTForImageClassification","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-08-13T11:14:45.023544Z","iopub.execute_input":"2024-08-13T11:14:45.024302Z","iopub.status.idle":"2024-08-13T11:15:00.385665Z","shell.execute_reply.started":"2024-08-13T11:14:45.024265Z","shell.execute_reply":"2024-08-13T11:15:00.384597Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Introduction","metadata":{}},{"cell_type":"markdown","source":"Vision Transformers (ViT) were introduced in the paper [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://arxiv.org/abs/2010.11929) by a Google Brain research team. In this paper, they showed that a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks.\n\nThey also proved that to achieve SOTA results on classification tasks, the ViT needs pre-training on a LARGE dataset and then fine-tuning on your specific task dataset.\n\nIn this notebook, you will find an implementation of fine-tuning a pre-trained ViT on the MNIST dataset. The pre-trained ViT is `google/vit-base-patch16-224-in21k` and is pulled from Hugging Face. The training is using a small subset of the records available in `train.csv` to avoid OOM and long training times. Though this is not an optimal configuration, the results observed on the validation set are  promising.  ","metadata":{}},{"cell_type":"markdown","source":"# Hyperparameter Definition","metadata":{}},{"cell_type":"markdown","source":"Let's set the hyperparameters that we'll use in this notebook: ","metadata":{}},{"cell_type":"code","source":"# sample_frac is the fraction of train.csv records that we will consider\nsample_frac=0.1\n\n# train_size represents the proportion of the dataset to include in the train split.\ntrain_size=0.8\n\n# we'll be fine-tuning the pre-trained ViT \"google/vit-base-patch16-224-in21k model\"\n# the model name shows that this ViT was trained on images from ImageNet-21k with resolution 224x224 and a patch size of 16x16\nmodel_id = 'google/vit-base-patch16-224-in21k'\n\n# batch size used during the fine-tuning process\nbatch_size = 32\n\n# parameters of the AdamW optimizer\nlearning_rate = 0.001\nweight_decay = 0.0001\n\n# number of epochs used during the model fine-tuning\nepochs = 100\n\n# early stopping parameter representing the number of epochs with no improvement after which training will be stopped. \npatience = math.ceil(epochs / 10)","metadata":{"execution":{"iopub.status.busy":"2024-08-13T11:15:00.387380Z","iopub.execute_input":"2024-08-13T11:15:00.387990Z","iopub.status.idle":"2024-08-13T11:15:00.393585Z","shell.execute_reply.started":"2024-08-13T11:15:00.387961Z","shell.execute_reply":"2024-08-13T11:15:00.392704Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Dataset Preparation","metadata":{}},{"cell_type":"markdown","source":"Let's start by loading a sample of the training data. The ViTModel requires a preprocessing step ViTImageProcessor that requires a huge amount of memory. The notebook resources cannot handle the training data, hence using a subset.","metadata":{}},{"cell_type":"code","source":"df_train = pd.read_csv('/kaggle/input/digit-recognizer/train.csv').sample(frac=sample_frac, axis=0, ignore_index=True, random_state=42)\ndf_train.head()","metadata":{"execution":{"iopub.status.busy":"2024-08-13T11:15:04.736631Z","iopub.execute_input":"2024-08-13T11:15:04.737478Z","iopub.status.idle":"2024-08-13T11:15:08.422702Z","shell.execute_reply.started":"2024-08-13T11:15:04.737447Z","shell.execute_reply":"2024-08-13T11:15:08.421752Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_train = df_train.pop('label')","metadata":{"execution":{"iopub.status.busy":"2024-08-13T11:15:08.424325Z","iopub.execute_input":"2024-08-13T11:15:08.424617Z","iopub.status.idle":"2024-08-13T11:15:08.457813Z","shell.execute_reply.started":"2024-08-13T11:15:08.424593Z","shell.execute_reply":"2024-08-13T11:15:08.456751Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's create a train and validation sets using `train_test_split`. With Keras `model.fit()`, there's a convenient way to split the train data automatically using the `validation_split` argument. Unfortunatly, this argument is not supported when the input data is a `tf.dataset`. As we'll use this type of input data, we have to create the sets: ","metadata":{}},{"cell_type":"code","source":"x_train, x_val, y_train, y_val = train_test_split(df_train, y_train, train_size=train_size)","metadata":{"execution":{"iopub.status.busy":"2024-08-13T11:15:09.736584Z","iopub.execute_input":"2024-08-13T11:15:09.737315Z","iopub.status.idle":"2024-08-13T11:15:09.811936Z","shell.execute_reply.started":"2024-08-13T11:15:09.737283Z","shell.execute_reply":"2024-08-13T11:15:09.810867Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The ViTImageProcessor expects 3D images. Let's reshape the images from the training and validation sets accordingly:","metadata":{}},{"cell_type":"code","source":"def convert_3D(df):\n    x = df.to_numpy() # x is an array of shape (784,)\n    x = x.reshape((df.shape[0],28,28,1)) # x is an array of shape (28,28,1)\n    x = np.repeat(x, 3, axis=3) # x is an array of shape (28,28,3). Each pixel has 3 times the same value (same value in each channel) \n    return x\n\nx_train = convert_3D(x_train)\nx_val = convert_3D(x_val)","metadata":{"execution":{"iopub.status.busy":"2024-08-13T11:15:11.523785Z","iopub.execute_input":"2024-08-13T11:15:11.524509Z","iopub.status.idle":"2024-08-13T11:15:11.580301Z","shell.execute_reply.started":"2024-08-13T11:15:11.524474Z","shell.execute_reply":"2024-08-13T11:15:11.579483Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"processor = ViTImageProcessor.from_pretrained(model_id)","metadata":{"execution":{"iopub.status.busy":"2024-08-13T11:15:12.310759Z","iopub.execute_input":"2024-08-13T11:15:12.311782Z","iopub.status.idle":"2024-08-13T11:15:13.218795Z","shell.execute_reply.started":"2024-08-13T11:15:12.311745Z","shell.execute_reply":"2024-08-13T11:15:13.217986Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x_train = processor(x_train)['pixel_values'] # x_train is now a list of 3360 elements, each element being an array of shape (224,224,3)\nx_val = processor(x_val)['pixel_values'] # x_val is now a list of 840 elements, each element being an array of shape (224,224,3)","metadata":{"execution":{"iopub.status.busy":"2024-08-13T11:15:13.354648Z","iopub.execute_input":"2024-08-13T11:15:13.355035Z","iopub.status.idle":"2024-08-13T11:15:24.157356Z","shell.execute_reply.started":"2024-08-13T11:15:13.355005Z","shell.execute_reply":"2024-08-13T11:15:24.156471Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we have to convert the training data into a `tf.data.Dataset`","metadata":{}},{"cell_type":"code","source":"tf_train = tf.data.Dataset.from_tensor_slices((x_train, y_train))\ntf_val = tf.data.Dataset.from_tensor_slices((x_val, y_val))","metadata":{"execution":{"iopub.status.busy":"2024-08-13T11:15:24.159285Z","iopub.execute_input":"2024-08-13T11:15:24.160121Z","iopub.status.idle":"2024-08-13T11:17:48.620832Z","shell.execute_reply.started":"2024-08-13T11:15:24.160082Z","shell.execute_reply":"2024-08-13T11:17:48.619879Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, we can check the transformation to ensure we did not corrupt the training data. To do this, we are going to display a few images with their labels.","metadata":{}},{"cell_type":"code","source":"fig=plt.figure()\n\ni=1\n\nfor x, y in tf_train.take(9):\n    fig.add_subplot(3, 3, i)\n    # x is a tensor, x.numpy() gives a ndarray of shape (3,224,224). \n    # plt.imshow() expects one image in (H,W,C)\n    # Hence, using transpose() to get an image of shape (224,224,3).\n    img = x.numpy().transpose((1,2,0)) \n    plt.imshow(img)\n    plt.title(f'label = {y}')\n    plt.axis('off')\n    if i==9 :\n        break\n    else:\n        i = i + 1","metadata":{"execution":{"iopub.status.busy":"2024-08-13T11:18:08.726385Z","iopub.execute_input":"2024-08-13T11:18:08.727066Z","iopub.status.idle":"2024-08-13T11:18:10.955735Z","shell.execute_reply.started":"2024-08-13T11:18:08.727032Z","shell.execute_reply":"2024-08-13T11:18:10.954656Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Last data preparation step will consist in creating batches and prepare the dataset for performance:","metadata":{}},{"cell_type":"code","source":"tf_train = tf_train.batch(batch_size).prefetch(tf.data.AUTOTUNE)\ntf_val = tf_val.batch(batch_size).prefetch(tf.data.AUTOTUNE)","metadata":{"execution":{"iopub.status.busy":"2024-08-13T11:18:13.329470Z","iopub.execute_input":"2024-08-13T11:18:13.329872Z","iopub.status.idle":"2024-08-13T11:18:13.343372Z","shell.execute_reply.started":"2024-08-13T11:18:13.329843Z","shell.execute_reply":"2024-08-13T11:18:13.342087Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model Fine-Tuning","metadata":{}},{"cell_type":"code","source":"# Load pre-trained ViT model\nmodel = TFViTForImageClassification.from_pretrained(model_id, num_labels=10)","metadata":{"execution":{"iopub.status.busy":"2024-08-13T11:18:15.950540Z","iopub.execute_input":"2024-08-13T11:18:15.951351Z","iopub.status.idle":"2024-08-13T11:18:19.465899Z","shell.execute_reply.started":"2024-08-13T11:18:15.951304Z","shell.execute_reply":"2024-08-13T11:18:19.464815Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.summary()","metadata":{"execution":{"iopub.status.busy":"2024-08-13T11:18:19.468221Z","iopub.execute_input":"2024-08-13T11:18:19.468535Z","iopub.status.idle":"2024-08-13T11:18:19.508639Z","shell.execute_reply.started":"2024-08-13T11:18:19.468509Z","shell.execute_reply":"2024-08-13T11:18:19.507692Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Define optimizer\noptimizer = tf.keras.optimizers.AdamW(learning_rate=learning_rate, weight_decay=weight_decay)\n\n# Define loss\nloss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)\n \n# Define metrics\nmetrics=tf.keras.metrics.SparseCategoricalAccuracy(name=\"accuracy\"),\n \n# Compile model\nmodel.compile(optimizer=optimizer, loss=loss, metrics=metrics)","metadata":{"execution":{"iopub.status.busy":"2024-08-13T11:18:21.872053Z","iopub.execute_input":"2024-08-13T11:18:21.873015Z","iopub.status.idle":"2024-08-13T11:18:21.982112Z","shell.execute_reply.started":"2024-08-13T11:18:21.872979Z","shell.execute_reply":"2024-08-13T11:18:21.981302Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"early_stopping = tf.keras.callbacks.EarlyStopping(monitor=\"val_loss\",patience=patience,restore_best_weights=True)","metadata":{"execution":{"iopub.status.busy":"2024-08-13T11:18:23.522214Z","iopub.execute_input":"2024-08-13T11:18:23.522948Z","iopub.status.idle":"2024-08-13T11:18:23.527773Z","shell.execute_reply.started":"2024-08-13T11:18:23.522911Z","shell.execute_reply":"2024-08-13T11:18:23.526725Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history = model.fit(tf_train, validation_data=tf_val, epochs=epochs, callbacks=[early_stopping])","metadata":{"execution":{"iopub.status.busy":"2024-08-13T11:18:24.454669Z","iopub.execute_input":"2024-08-13T11:18:24.455583Z","iopub.status.idle":"2024-08-13T13:00:29.408516Z","shell.execute_reply.started":"2024-08-13T11:18:24.455552Z","shell.execute_reply":"2024-08-13T13:00:29.407644Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# item : \"loss\" or \"accuracy\"\ndef plot_history(item):\n    plt.plot(history.history[item], label=item)\n    plt.plot(history.history[\"val_\" + item], label=\"val_\" + item)\n    plt.xlabel(\"Epochs\")\n    plt.ylabel(item)\n    plt.title(\"Train and Validation {} Over Epochs\".format(item), fontsize=14)\n    plt.legend()\n    plt.grid()\n    plt.show()\n\nplot_history(\"loss\")\nplot_history(\"accuracy\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}