{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":3004,"databundleVersionId":861823,"sourceType":"competition"}],"dockerImageVersionId":30840,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Purpose of this Notebook","metadata":{}},{"cell_type":"markdown","source":"This notebook is just a basic one to show how one might train a classifier on the mnist dataset with tensorflow. The chosen classifier is a CNN, as we try to classify images with simple patterns (no need to go to VIT architecture).\n\nThe goal is to showcase the basic components of model training (from scratch) in Tensorflow : no regularization, no data augmentation, no Transfer learning, no learning rate scheduling, no hyper-parameters tuning.\n\nI refer you to later notebooks for fancier techniques.","metadata":{}},{"cell_type":"markdown","source":"## Import necessary libraries","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow.keras import layers\n\nfrom tensorflow.keras.datasets import mnist\n\nimport matplotlib.pyplot as plt\nimport numpy as np","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2025-01-28T17:11:14.735285Z","iopub.execute_input":"2025-01-28T17:11:14.735618Z","iopub.status.idle":"2025-01-28T17:11:14.739672Z","shell.execute_reply.started":"2025-01-28T17:11:14.735594Z","shell.execute_reply":"2025-01-28T17:11:14.738756Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Load Dataset","metadata":{}},{"cell_type":"code","source":"(X_train, Y_train), (X_test, Y_test) = mnist.load_data()\n\n\nprint('X_train shape: ',X_train.shape)\nprint('Y_train shape: ',Y_train.shape)\n\nX_train = X_train.astype('float32')/255\nX_test = X_test.astype('float32')/255","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-01-28T17:11:14.740884Z","iopub.execute_input":"2025-01-28T17:11:14.741163Z","iopub.status.idle":"2025-01-28T17:11:15.121232Z","shell.execute_reply.started":"2025-01-28T17:11:14.741134Z","shell.execute_reply":"2025-01-28T17:11:15.120397Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Define model","metadata":{}},{"cell_type":"code","source":"def basic_CNN(X):\n    # input dim is (28,28)\n    x = layers.Lambda(lambda x: tf.expand_dims(x, axis=-1))(X)\n    # x dim is now (28, 28, 1)\n    x = layers.Conv2D(filters=32, kernel_size=3, padding=\"same\")(x) # (28,28,32)\n    x = layers.LeakyReLU()(x)\n    x = layers.Conv2D(filters=32, kernel_size=3, padding=\"same\")(x) # (28,28,32)\n    x = layers.LeakyReLU()(x)\n    x = layers.AveragePooling2D(pool_size=2)(x) # (14,14,32)\n    x = layers.Conv2D(filters=64, kernel_size=3, padding=\"same\")(x) # (14,14,64)\n    x = layers.LeakyReLU()(x)\n    x = layers.Conv2D(filters=64, kernel_size=3, padding=\"same\")(x) # (14,14,64)\n    x = layers.LeakyReLU()(x)\n    x = layers.AveragePooling2D(pool_size=2)(x) # (7,7,64)\n    x = layers.Conv2D(filters=128, kernel_size=3, padding=\"same\")(x) # (7,7,128)\n    x = layers.LeakyReLU()(x)\n    x = layers.Conv2D(filters=128, kernel_size=3, padding=\"same\")(x) # (7,7,128)\n    x = layers.LeakyReLU()(x)\n    x = layers.Flatten()(x) # (6272)\n    x = layers.Dense(units=64)(x) # (64)\n    x = layers.LeakyReLU()(x)\n    x = layers.Dense(units=10, activation=\"softmax\")(x) # (10) we have our digits predictions\n    \n    return x\n\nX = tf.keras.layers.Input(shape=(28,28))\npred = basic_CNN(X)\n    \nmodel = tf.keras.Model(inputs=X, outputs=pred)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-01-28T17:11:15.122652Z","iopub.execute_input":"2025-01-28T17:11:15.122898Z","iopub.status.idle":"2025-01-28T17:11:15.192439Z","shell.execute_reply.started":"2025-01-28T17:11:15.122878Z","shell.execute_reply":"2025-01-28T17:11:15.191644Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Compile model","metadata":{}},{"cell_type":"code","source":"model.compile(loss=tf.keras.losses.SparseCategoricalCrossentropy(),\n              optimizer='adam',\n              metrics=['accuracy'])\nmodel.summary()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-01-28T17:11:15.193679Z","iopub.execute_input":"2025-01-28T17:11:15.193892Z","iopub.status.idle":"2025-01-28T17:11:15.221438Z","shell.execute_reply.started":"2025-01-28T17:11:15.193873Z","shell.execute_reply":"2025-01-28T17:11:15.220766Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Fit model","metadata":{}},{"cell_type":"code","source":"history = model.fit(X_train, Y_train, validation_data=(X_test, Y_test), epochs=20, batch_size=256)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-01-28T17:11:15.222081Z","iopub.execute_input":"2025-01-28T17:11:15.222295Z","iopub.status.idle":"2025-01-28T17:12:40.721897Z","shell.execute_reply.started":"2025-01-28T17:11:15.222277Z","shell.execute_reply":"2025-01-28T17:12:40.720980Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plt.plot(np.log(history.history['loss']))\nplt.plot(np.log(history.history['val_loss']))\nplt.title('model log-loss')\nplt.ylabel('loss')\nplt.xlabel('epoch')\nplt.legend(['train', 'val'], loc='upper left')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-01-28T17:12:40.722893Z","iopub.execute_input":"2025-01-28T17:12:40.723193Z","iopub.status.idle":"2025-01-28T17:12:40.910961Z","shell.execute_reply.started":"2025-01-28T17:12:40.723144Z","shell.execute_reply":"2025-01-28T17:12:40.910083Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Why plot the log-loss instead of the loss first ?","metadata":{}},{"cell_type":"markdown","source":"This task is very easy (i.e. the entropy of the distribution of our data is very low), therefore the training loss goes toward 0.\nIf we only plot the loss, we will see a flat line at 0 (see next graph), that will be very close to the validation loss.\nWe want to see if the validation loss hits a plateau, or increase, while the training loss continues decreasing.\nTherefore, we need to scale the loss so that small variations at 0 are still easily visible.\nThe log-loss meets this objective.","metadata":{}},{"cell_type":"code","source":"plt.plot(history.history['loss'])\nplt.plot(history.history['val_loss'])\nplt.title('model loss')\nplt.ylabel('loss')\nplt.xlabel('epoch')\nplt.legend(['train', 'val'], loc='upper left')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-01-28T17:12:40.912232Z","iopub.execute_input":"2025-01-28T17:12:40.912513Z","iopub.status.idle":"2025-01-28T17:12:41.091419Z","shell.execute_reply.started":"2025-01-28T17:12:40.912492Z","shell.execute_reply":"2025-01-28T17:12:41.090546Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plt.plot(history.history['accuracy'])\nplt.plot(history.history['val_accuracy'])\nplt.title('model accuracy')\nplt.ylabel('accuracy')\nplt.xlabel('epoch')\nplt.legend(['train', 'val'], loc='upper left')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-01-28T17:12:41.092860Z","iopub.execute_input":"2025-01-28T17:12:41.093064Z","iopub.status.idle":"2025-01-28T17:12:41.274083Z","shell.execute_reply.started":"2025-01-28T17:12:41.093047Z","shell.execute_reply":"2025-01-28T17:12:41.273106Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Results","metadata":{}},{"cell_type":"markdown","source":"We see that our model is very powerful for classifying the digits in the mnist dataset, with a small training time. \\\nAlmost all the images are classified correctly. Longer training should help us reach 100% training accuracy, and maybe 100% validation accuracy. \\\nSome other techniques can help us increase validation accuracy, and will be seen in later notebooks.","metadata":{}}]}