{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### Introduction","metadata":{}},{"cell_type":"markdown","source":"MNIST is the famous database of handwritten digits. In this notebook I tried to explain the famous LeNet-5 CNN architecture, it is known as one of the first and famous CNN algorithms, which was developed to recognize digits in US Postal Service. Lenet-5 was developed by the French computer scientist Yann LeCun and was in Neural Computation under the name of \"Backpropagation Applied to Handwritten Zip Code Recognition\". The detailed explanation of the LeNet architecture can be found in the next sections.","metadata":{}},{"cell_type":"markdown","source":"### Data proccessing","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport warnings\nwarnings.filterwarnings('ignore')","metadata":{"execution":{"iopub.status.busy":"2021-10-20T09:14:58.331451Z","iopub.execute_input":"2021-10-20T09:14:58.331815Z","iopub.status.idle":"2021-10-20T09:14:58.337293Z","shell.execute_reply.started":"2021-10-20T09:14:58.331775Z","shell.execute_reply":"2021-10-20T09:14:58.33655Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data = pd.read_csv('../input/digit-recognizer/train.csv')\ntrain_data.head()","metadata":{"execution":{"iopub.status.busy":"2021-10-20T09:14:58.342654Z","iopub.execute_input":"2021-10-20T09:14:58.343103Z","iopub.status.idle":"2021-10-20T09:15:02.20088Z","shell.execute_reply.started":"2021-10-20T09:14:58.343062Z","shell.execute_reply":"2021-10-20T09:15:02.199985Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_data = pd.read_csv('../input/digit-recognizer/test.csv')\ntest_data.head()","metadata":{"execution":{"iopub.status.busy":"2021-10-20T09:15:02.203029Z","iopub.execute_input":"2021-10-20T09:15:02.203391Z","iopub.status.idle":"2021-10-20T09:15:04.416431Z","shell.execute_reply.started":"2021-10-20T09:15:02.203346Z","shell.execute_reply":"2021-10-20T09:15:04.415624Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data.shape","metadata":{"execution":{"iopub.status.busy":"2021-10-20T09:15:04.417561Z","iopub.execute_input":"2021-10-20T09:15:04.417783Z","iopub.status.idle":"2021-10-20T09:15:04.423657Z","shell.execute_reply.started":"2021-10-20T09:15:04.417755Z","shell.execute_reply":"2021-10-20T09:15:04.422753Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_data.shape","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2021-10-20T09:15:04.425365Z","iopub.execute_input":"2021-10-20T09:15:04.425596Z","iopub.status.idle":"2021-10-20T09:15:04.435816Z","shell.execute_reply.started":"2021-10-20T09:15:04.425569Z","shell.execute_reply":"2021-10-20T09:15:04.435245Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data['label'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2021-10-20T09:15:04.437079Z","iopub.execute_input":"2021-10-20T09:15:04.437563Z","iopub.status.idle":"2021-10-20T09:15:04.455208Z","shell.execute_reply.started":"2021-10-20T09:15:04.437532Z","shell.execute_reply":"2021-10-20T09:15:04.454382Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Seems like our data is pretty clean with somewhat equally distributed labels.","metadata":{}},{"cell_type":"markdown","source":"### Model Preparation","metadata":{}},{"cell_type":"code","source":"X = train_data.drop(['label'], axis = 1)\ny = train_data['label']","metadata":{"execution":{"iopub.status.busy":"2021-10-20T09:15:04.456417Z","iopub.execute_input":"2021-10-20T09:15:04.456777Z","iopub.status.idle":"2021-10-20T09:15:04.574306Z","shell.execute_reply.started":"2021-10-20T09:15:04.456742Z","shell.execute_reply":"2021-10-20T09:15:04.573425Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"![LeNet architecture](https://i.imgur.com/Ppd1JK9.png)","metadata":{}},{"cell_type":"markdown","source":"![LeNet architecture](https://i.imgur.com/XsaWqnL.png)","metadata":{}},{"cell_type":"markdown","source":"As we can see, the original LeNet architecture has 5 layers.\nThe first one Convolutional layer with input of 32x32 but we will use 28x28. We also have pooling layers after each convolution layers. In the 1990s the activation function used was tanh and the pooling function was Average Pooling but as the GPUs became stronger we realized relu function and Max Pooling performed even better! Lets start with reshaping our input and defining the model in keras!","metadata":{}},{"cell_type":"code","source":"X.shape","metadata":{"execution":{"iopub.status.busy":"2021-10-20T09:15:04.576133Z","iopub.execute_input":"2021-10-20T09:15:04.576376Z","iopub.status.idle":"2021-10-20T09:15:04.584779Z","shell.execute_reply.started":"2021-10-20T09:15:04.57635Z","shell.execute_reply":"2021-10-20T09:15:04.583826Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X = X.to_numpy()\nX = X.reshape(-1, 28, 28, 1)","metadata":{"execution":{"iopub.status.busy":"2021-10-20T09:15:04.585974Z","iopub.execute_input":"2021-10-20T09:15:04.586225Z","iopub.status.idle":"2021-10-20T09:15:04.592718Z","shell.execute_reply.started":"2021-10-20T09:15:04.586191Z","shell.execute_reply":"2021-10-20T09:15:04.591875Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X.shape","metadata":{"execution":{"iopub.status.busy":"2021-10-20T09:15:04.593756Z","iopub.execute_input":"2021-10-20T09:15:04.595264Z","iopub.status.idle":"2021-10-20T09:15:04.605742Z","shell.execute_reply.started":"2021-10-20T09:15:04.595231Z","shell.execute_reply":"2021-10-20T09:15:04.604711Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.imshow(X[0])","metadata":{"execution":{"iopub.status.busy":"2021-10-20T09:15:04.610267Z","iopub.execute_input":"2021-10-20T09:15:04.610514Z","iopub.status.idle":"2021-10-20T09:15:04.848104Z","shell.execute_reply.started":"2021-10-20T09:15:04.610483Z","shell.execute_reply":"2021-10-20T09:15:04.847215Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from tensorflow import keras\nfrom tensorflow.keras import layers\ndef define_LeNet():\n    model = keras.Sequential([\n    layers.Conv2D(filters = 6, kernel_size = (5,5), activation = 'relu', input_shape=(28, 28, 1), padding = \"same\"),\n    layers.MaxPool2D(pool_size = (2,2)),\n    layers.Conv2D(filters = 16, kernel_size = (5,5), activation = 'relu', padding=\"same\"),\n    layers.MaxPool2D(pool_size = (2,2)),\n    layers.Conv2D(filters = 120, kernel_size = (5,5), activation = 'relu', padding=\"same\"), \n    layers.Flatten(),\n    layers.Dense(120, activation = 'relu'),\n    layers.Dense(84, activation = 'relu'),\n    layers.Dropout(0.5), \n    layers.Dense(10, activation = \"softmax\")\n    ])\n    model.compile(optimizer='adam',\n              loss='categorical_crossentropy', # Loss\n              metrics=['accuracy'])\n    return model","metadata":{"execution":{"iopub.status.busy":"2021-10-20T09:15:04.849229Z","iopub.execute_input":"2021-10-20T09:15:04.849448Z","iopub.status.idle":"2021-10-20T09:15:10.739931Z","shell.execute_reply.started":"2021-10-20T09:15:04.849415Z","shell.execute_reply":"2021-10-20T09:15:10.738702Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have to one hot encode our labels to make our model output and are the labels in dataset compatible.","metadata":{}},{"cell_type":"code","source":"from keras.utils.np_utils import to_categorical\ny = to_categorical(y, num_classes = 10)","metadata":{"execution":{"iopub.status.busy":"2021-10-20T09:15:10.741175Z","iopub.execute_input":"2021-10-20T09:15:10.741467Z","iopub.status.idle":"2021-10-20T09:15:10.747077Z","shell.execute_reply.started":"2021-10-20T09:15:10.741427Z","shell.execute_reply":"2021-10-20T09:15:10.746208Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.15, random_state = 51, stratify = y)","metadata":{"execution":{"iopub.status.busy":"2021-10-20T09:15:10.748288Z","iopub.execute_input":"2021-10-20T09:15:10.748516Z","iopub.status.idle":"2021-10-20T09:15:12.045998Z","shell.execute_reply.started":"2021-10-20T09:15:10.748488Z","shell.execute_reply":"2021-10-20T09:15:12.045118Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To prevent our model from overfitting defining a callback function is pretty easy and helpful!","metadata":{}},{"cell_type":"code","source":"early_stopping = keras.callbacks.EarlyStopping(\n    min_delta = 0.001, # minimium amount of change to count as an improvement\n    patience = 20, # how many epochs to wait before stopping\n    restore_best_weights = True, # Will return best parameters incase of overfittting\n)","metadata":{"execution":{"iopub.status.busy":"2021-10-20T09:15:12.047173Z","iopub.execute_input":"2021-10-20T09:15:12.047389Z","iopub.status.idle":"2021-10-20T09:15:12.052526Z","shell.execute_reply.started":"2021-10-20T09:15:12.047362Z","shell.execute_reply":"2021-10-20T09:15:12.051462Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Increasing the our training data by data augmentation can be very helpful, even by changing the rotation and shifting it a little bit can increase the score by 0.01! ","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.preprocessing.image import ImageDataGenerator\ndatagen = ImageDataGenerator(\n        rotation_range = 15,\n        width_shift_range = 0.2, \n        height_shift_range = 0.2\n        )\ndatagen.fit(X_train)","metadata":{"execution":{"iopub.status.busy":"2021-10-20T09:15:12.054212Z","iopub.execute_input":"2021-10-20T09:15:12.054442Z","iopub.status.idle":"2021-10-20T09:15:12.187512Z","shell.execute_reply.started":"2021-10-20T09:15:12.05441Z","shell.execute_reply":"2021-10-20T09:15:12.186565Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Model Fitting","metadata":{}},{"cell_type":"code","source":"model = define_LeNet()\nhistory = model.fit_generator(datagen.flow(X_train,y_train, batch_size = 256),\n                              validation_data = (X_test,y_test),\n                              verbose = 2, \n                              steps_per_epoch = X_train.shape[0] // 256,\n                              epochs = 30,  \n                              callbacks = [early_stopping])","metadata":{"execution":{"iopub.status.busy":"2021-10-20T09:15:12.189029Z","iopub.execute_input":"2021-10-20T09:15:12.189352Z","iopub.status.idle":"2021-10-20T09:25:16.786823Z","shell.execute_reply.started":"2021-10-20T09:15:12.18931Z","shell.execute_reply":"2021-10-20T09:25:16.785931Z"},"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history_df = pd.DataFrame(history.history)\nhistory_df.loc[:, ['loss', 'val_loss']].plot();\nprint(\"Minimum validation loss: {}\".format(history_df['val_loss'].min()))","metadata":{"execution":{"iopub.status.busy":"2021-10-20T09:25:16.78828Z","iopub.execute_input":"2021-10-20T09:25:16.788503Z","iopub.status.idle":"2021-10-20T09:25:17.03659Z","shell.execute_reply.started":"2021-10-20T09:25:16.788476Z","shell.execute_reply":"2021-10-20T09:25:17.035985Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Looks like our model fit without underfitting/overfitting, time the save apply it on our test data and save, thanks for reading!","metadata":{}},{"cell_type":"code","source":"test_data = test_data.to_numpy()\ntest_data = test_data.reshape(-1, 28, 28, 1)","metadata":{"execution":{"iopub.status.busy":"2021-10-20T09:25:17.037687Z","iopub.execute_input":"2021-10-20T09:25:17.038398Z","iopub.status.idle":"2021-10-20T09:25:17.04269Z","shell.execute_reply.started":"2021-10-20T09:25:17.038362Z","shell.execute_reply":"2021-10-20T09:25:17.042019Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predicted = model.predict(test_data)\npredicted_real = np.argmax(predicted, axis = 1) ","metadata":{"execution":{"iopub.status.busy":"2021-10-20T09:25:17.043687Z","iopub.execute_input":"2021-10-20T09:25:17.043879Z","iopub.status.idle":"2021-10-20T09:25:21.58943Z","shell.execute_reply.started":"2021-10-20T09:25:17.043854Z","shell.execute_reply":"2021-10-20T09:25:21.588689Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = pd.read_csv('../input/digit-recognizer/sample_submission.csv')\nsubmission['Label'] = predicted_real\nsubmission.to_csv('my_submission.csv', index = False)","metadata":{"execution":{"iopub.status.busy":"2021-10-20T09:25:21.590423Z","iopub.execute_input":"2021-10-20T09:25:21.590741Z","iopub.status.idle":"2021-10-20T09:25:21.681438Z","shell.execute_reply.started":"2021-10-20T09:25:21.590713Z","shell.execute_reply":"2021-10-20T09:25:21.680637Z"},"trusted":true},"execution_count":null,"outputs":[]}]}