{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import tensorflow as tf\nimport matplotlib.pyplot as plt\nimport pandas as pd\nimport numpy as np\nfrom tensorflow.keras import datasets, layers, models, losses\nfrom keras.utils.np_utils import to_categorical\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-04-09T13:43:10.156601Z","iopub.execute_input":"2022-04-09T13:43:10.157421Z","iopub.status.idle":"2022-04-09T13:43:10.162712Z","shell.execute_reply.started":"2022-04-09T13:43:10.157371Z","shell.execute_reply":"2022-04-09T13:43:10.162189Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Train and test data\n\nLoad the data (Mnist handwritten)","metadata":{}},{"cell_type":"code","source":"pd_train = pd.read_csv('../input/digit-recognizer/train.csv')\n\n\nx_train = pd_train.drop(['label'], axis=1)\ny_train = pd_train['label']","metadata":{"execution":{"iopub.status.busy":"2022-04-09T13:43:10.210057Z","iopub.execute_input":"2022-04-09T13:43:10.210361Z","iopub.status.idle":"2022-04-09T13:43:12.997731Z","shell.execute_reply.started":"2022-04-09T13:43:10.210328Z","shell.execute_reply":"2022-04-09T13:43:12.996989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Scaling\n\nOriginal LeNet model receives 32 by 32 images, thus 28 by 28 MNIST images are padded with zeros and 8-bit (0–255 range) pixel values are scaled between 0 and 1.","metadata":{}},{"cell_type":"code","source":"x_train = x_train / 255.0\nx_train.shape","metadata":{"execution":{"iopub.status.busy":"2022-04-09T13:43:12.999435Z","iopub.execute_input":"2022-04-09T13:43:13.000066Z","iopub.status.idle":"2022-04-09T13:43:13.063392Z","shell.execute_reply.started":"2022-04-09T13:43:13.000017Z","shell.execute_reply":"2022-04-09T13:43:13.062766Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Most CNN’s accept 4-dimensional tensors as inputs having the dimensions of batch size, height, width, and channel. Since MNIST images are grayscale, the last dimension does not necessarily exist. We need to expand the tensor and create a dummy dimension at axis number 3. (Recall that the tensor initially had axis 0, 1, and 2.)","metadata":{}},{"cell_type":"markdown","source":"### Create the validation set\nThe last 2000 samples of the training set are reserved for the validation set. Used for hyperparam tuning\n","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\nX_train, X_val, Y_train, Y_val = train_test_split(x_train, y_train, test_size=0.2, shuffle=True, stratify=y_train, random_state=26)\n\n\nX_train = np.array(X_train).reshape(-1,28,28,1)\nX_val = np.array(X_val).reshape(-1,28,28,1)","metadata":{"execution":{"iopub.status.busy":"2022-04-09T13:43:13.064250Z","iopub.execute_input":"2022-04-09T13:43:13.064434Z","iopub.status.idle":"2022-04-09T13:43:13.415096Z","shell.execute_reply.started":"2022-04-09T13:43:13.064408Z","shell.execute_reply":"2022-04-09T13:43:13.414159Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n## The Model\n\nThere are 3 convolutional layers each having 5 by 5 kernels with 6, 16, and 120 feature maps, respectively. In between, there are 2 subsampling layers as average pooling. All these 5 layers use stride 1 and average pooling layers use 2 by 2 kernels as default settings. After the convolutional layers, hyperbolic tangent nonlinear activation is used, whereas subsampling layers are followed by sigmoid nonlinearity. After the last convolutional layer, the activations are flattened and fed into fully connected layers having 84 and 10 neurons. The output of the last layer (after softmax operation) represents the probabilities of the classes (from 0 to 9) for the input image.\n\nNowadays, tanh and sigmoid activations are rarely used due to saturation problems. Instead, ReLU and Leaky ReLU are much more popular. Also, using two consecutive 3 by 3 layers instead of a single 5 by 5 layer is more preferable, since the same receptive field size is obtained by a fairly lower number of trainable parameters.\n","metadata":{}},{"cell_type":"code","source":"model = models.Sequential()\nmodel.add(layers.Conv2D(filters=64, kernel_size = (3,3), activation=\"relu\", input_shape=(28,28,1)))\nmodel.add(layers.Conv2D(filters=64, kernel_size = (3,3), activation=\"relu\"))\nmodel.add(layers.MaxPooling2D(pool_size=(2,2)))\nmodel.add(layers.BatchNormalization())\n\nmodel.add(layers.Conv2D(filters=128, kernel_size = (3,3), activation=\"relu\"))\nmodel.add(layers.Conv2D(filters=128, kernel_size = (3,3), activation=\"relu\"))\nmodel.add(layers.MaxPooling2D(pool_size=(2,2)))\nmodel.add(layers.BatchNormalization())    \n\nmodel.add(layers.Conv2D(filters=256, kernel_size = (3,3), activation=\"relu\"))\nmodel.add(layers.MaxPooling2D(pool_size=(2,2)))\nmodel.add(layers.BatchNormalization())\n    \nmodel.add(layers.Flatten())\nmodel.add(layers.Dense(512,activation=\"relu\"))\n    \nmodel.add(layers.Dense(10,activation=\"softmax\"))\n    \nmodel.compile(loss=\"categorical_crossentropy\", optimizer=\"adam\", metrics=[\"accuracy\"])","metadata":{"execution":{"iopub.status.busy":"2022-04-09T13:43:13.416922Z","iopub.execute_input":"2022-04-09T13:43:13.417155Z","iopub.status.idle":"2022-04-09T13:43:13.545620Z","shell.execute_reply.started":"2022-04-09T13:43:13.417129Z","shell.execute_reply":"2022-04-09T13:43:13.544806Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Model training\n\nThe model is set to be optimized by Adam optimizer[7]. Sparse categorical cross-entropy measures the negative of the natural logarithm of the true class probability. For instance, if the final output of the model is a vector such as [0.03, 0.78, …, 0.05] and the true class of the input image is 1; the loss for this instance will be -ln(0.78) = 0.248. Accuracy metric is reported for each epoch. history object has a property named history which is useful for keeping track of the training phase.","metadata":{}},{"cell_type":"code","source":"model_path = './model_out.h5'\ncheckpointer = tf.keras.callbacks.ModelCheckpoint(\n    filepath=model_path,\n    monitor='val_accuracy',\n    verbose=1,\n    mode='max',\n    save_best_only=True)\n\nmodel.compile(optimizer='adam', loss=losses.sparse_categorical_crossentropy, metrics=['accuracy'])\n\nhistory = model.fit(X_train, Y_train, batch_size=16, epochs=10, validation_data=(X_val,Y_val))","metadata":{"execution":{"iopub.status.busy":"2022-04-09T13:43:13.547008Z","iopub.execute_input":"2022-04-09T13:43:13.547311Z","iopub.status.idle":"2022-04-09T14:02:15.159610Z","shell.execute_reply.started":"2022-04-09T13:43:13.547271Z","shell.execute_reply":"2022-04-09T14:02:15.158987Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Testing resuming training from saved model and SUBMISSION\n","metadata":{}},{"cell_type":"code","source":"# Save the weights\nmodel.save_weights('./checkpoints/my_checkpoint')\n\n\n# Save the entire model as a SavedModel.\n!mkdir -p saved_model\nmodel.save('model_out.h5')\n\nfrom tensorflow.keras.models import load_model\nmodel = load_model('model_out.h5')","metadata":{"execution":{"iopub.status.busy":"2022-04-09T14:06:10.787504Z","iopub.execute_input":"2022-04-09T14:06:10.787842Z","iopub.status.idle":"2022-04-09T14:06:11.916420Z","shell.execute_reply.started":"2022-04-09T14:06:10.787805Z","shell.execute_reply":"2022-04-09T14:06:11.915406Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.summary()","metadata":{"execution":{"iopub.status.busy":"2022-04-09T14:06:15.284932Z","iopub.execute_input":"2022-04-09T14:06:15.285230Z","iopub.status.idle":"2022-04-09T14:06:15.296811Z","shell.execute_reply.started":"2022-04-09T14:06:15.285199Z","shell.execute_reply":"2022-04-09T14:06:15.296089Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd_test = pd.read_csv('../input/digit-recognizer/test.csv')\nx_test = pd_test\nx_test = x_test/255.0\nX_test = np.array(x_test).reshape(-1,28,28,1)","metadata":{"execution":{"iopub.status.busy":"2022-04-09T14:06:18.727715Z","iopub.execute_input":"2022-04-09T14:06:18.727996Z","iopub.status.idle":"2022-04-09T14:06:21.156343Z","shell.execute_reply.started":"2022-04-09T14:06:18.727962Z","shell.execute_reply":"2022-04-09T14:06:21.155378Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = load_model('./model_out.h5')\ny_pred = model.predict(X_test)\ny_pred = np.argmax(y_pred,1)\npd_submit = pd.read_csv('../input/digit-recognizer/sample_submission.csv')\npd_submit['Label'] = y_pred\npd_submit.to_csv('submit.csv', header=True, index=False)","metadata":{"execution":{"iopub.status.busy":"2022-04-09T14:06:23.286924Z","iopub.execute_input":"2022-04-09T14:06:23.287593Z","iopub.status.idle":"2022-04-09T14:06:46.895784Z","shell.execute_reply.started":"2022-04-09T14:06:23.287545Z","shell.execute_reply":"2022-04-09T14:06:46.895047Z"},"trusted":true},"execution_count":null,"outputs":[]}]}