{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# MNIST Classification Challenge","metadata":{"id":"KEHvRGD2MEtS"}},{"cell_type":"markdown","source":"**Introduction:** \n\nThere are many ways to slice this problem, as well as the argument between the trade-off of more complex networks with negligible increases in performance. Therefore, this notebook's goal is to showcase how to identify handwritten digits from the MNIST dataset using a simple single hidden layer fully connected network. While greater performance can be achieved with other network architectures or additional preprocessing measure, this notebook is meant to be kept more simplistic in nature and to explore more of the possibilities of simple networks.\n\n**Methods:** \n\nThe notebook starts out by importing the CSV files that consist of training data and respective class labels, and test data without labels. These were loaded into a Pandas DataFrame for quick review. A sanity check was done to ensure there are no null values in the image pixel values. The 'label' column from the train data was then pulled out of the DataFrame, converted to a NumPy matrix, and stored in a new variable. This same column was dropped from the DataFrame.\n\nTo visualize the data we're working with, a random number was plotted along with its respective label. We also visualize the count of labels within the dataset, which allows us to see the balance within the dataset and which values have the most and least occurrences. \n\nPrior to training we normalize the pixel values, converting each value by 255 (the maximum potential pixel value). This is a simple method to reduce the data variance, aiding in model performance. \n\nKeeping the provided test data as 'test' data, we then split the training data into a training and validation set, with 90% kept for training and 10% for validation. This results in 37,800 images for training and 4,200 for validation. These sets will be used during the network's training process, with the provided test set withheld to test the model's ability to generalize.\n\nTo keep with the purpose of this notebook, a simple architecture is defined using a single hidden layer consisting of 60 nodes. While the node count is more of a random choice, it also helps showcase performance with a lower node count than commonly seen in FC layers. Further, an early stopping callback is added to the training process, monitoring the validation accuracy and halting training if performance does not increase over two epochs. This allows us to choose an arbitrarily higher number of epochs and letting the callback stop when necessary, reducing the risk of overfitting. Model performance is then plotted using the plot_keras_history library, which is more of a convenience but not necessary. \n\nThe trained model is then used to predict the labels for the test dataset, and an example image and its predicted label shown visually. This is also done for the first 15 images from the test dataset, as a point of reference for visual review of images against predicted labels. \n\nOne additional step is taken to review the hidden layer's node's activation values. This is an attempt to include some model transparency to the process, showing us what input images causing each node to activate and with what activation values.\n\n**Results:**\n\nOverall, a single hidden layer with 60 nodes was able to obtain accuracy of about 96%, showing the potential of a simple network in this specific use case. Additional steps can be taken to further increase model performance such as tuning hyperparameters or working with input data, including techniques like image augmentation or simply shifting train/test/validation splits. ","metadata":{"id":"I9r_a_EXtdvo"}},{"cell_type":"code","source":"!pip install kaggle plot-keras-history -q","metadata":{"id":"eBdWzKmptDKd","execution":{"iopub.status.busy":"2021-08-10T18:12:33.132166Z","iopub.execute_input":"2021-08-10T18:12:33.132558Z","iopub.status.idle":"2021-08-10T18:12:40.272971Z","shell.execute_reply.started":"2021-08-10T18:12:33.132527Z","shell.execute_reply":"2021-08-10T18:12:40.27177Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport collections\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport tensorflow as tf\nimport numpy as np\nimport dill\nfrom tensorflow.keras import models, backend\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Dropout, Dense, Flatten\nfrom sklearn.model_selection import train_test_split\nfrom plot_keras_history import plot_history\nplt.style.use('seaborn-ticks')","metadata":{"id":"FtbKvH1gKD8T","execution":{"iopub.status.busy":"2021-08-10T18:12:41.88686Z","iopub.execute_input":"2021-08-10T18:12:41.887195Z","iopub.status.idle":"2021-08-10T18:12:48.594772Z","shell.execute_reply.started":"2021-08-10T18:12:41.887164Z","shell.execute_reply":"2021-08-10T18:12:48.593685Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Read in the CSV files to a Pandas DataFrame\ntraindf = pd.read_csv('../input/digit-recognizer/train.csv')\ntestdf = pd.read_csv('../input/digit-recognizer/test.csv')\nprint('Train data shape:',traindf.shape,'\\n','Test data shape:',testdf.shape,'\\n')","metadata":{"id":"P1_Fw4THMa1U","outputId":"9963ad8b-2c3c-4ad3-9551-a29868a41fea","execution":{"iopub.status.busy":"2021-08-10T18:13:16.989013Z","iopub.execute_input":"2021-08-10T18:13:16.989368Z","iopub.status.idle":"2021-08-10T18:13:22.94978Z","shell.execute_reply.started":"2021-08-10T18:13:16.989339Z","shell.execute_reply":"2021-08-10T18:13:22.948612Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Analysis/Review","metadata":{"id":"yuS794GatrMW"}},{"cell_type":"code","source":"# Sanity check for any null values\nprint('Null values exist in training data?:',traindf.isnull().values.any())\nprint('Null values exist in test data?:',testdf.isnull().values.any())","metadata":{"id":"UKj7bglTmxI1","outputId":"053d9d93-916c-4122-c625-68421a64c4b4"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Extract the labels column from the train dataset, then drop it from the dataframe\nlabels = np.asarray(traindf['label'])\ntraindf.drop(columns=['label'], inplace=True)\n\nprint(labels[:20])\nprint(traindf.shape)","metadata":{"id":"ekWA5BNuN150","outputId":"0a381981-dd55-45ac-dddc-f4ad4b41cff3"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# View an example image and its associated label\nnum = 100\nplt.imshow(np.reshape(traindf.iloc[num,:].values,(28,28)))\nplt.title('The label of the image is:'+str(labels[num]))","metadata":{"id":"itqz1ihgLWfS","outputId":"be2ef090-e043-4e8a-d04a-2f002e0a2c65"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(16,8))\nplt.hist(labels, bins=(np.arange(11)-.5),ec='black',alpha=.8)\nplt.title('Histogram of Labels')\nplt.xlabel('Label')\nplt.ylabel('Count of Label')\nplt.xticks(range(max(labels)+1))\nplt.plot();","metadata":{"id":"1zl5mhhxvM0e","outputId":"ec2666a6-e960-47de-a319-01124ccfee62"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Normalize the data by dividing each pixel value by 255\n# This reduces the variance from 0 - 255 to 0 - 1\ntrain_norm = traindf.iloc[:,:]/255","metadata":{"id":"7SWH57awRwEE"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train/Validation Split","metadata":{"id":"C2ZT0OBXSji_"}},{"cell_type":"code","source":"# Keeping the test.csv file as the test set, splitting a validation set off the training set\n# Stratifying to help with ensuring we have balanced data\nX_train, X_val, y_train, y_val = train_test_split(train_norm, labels, test_size=.10, stratify=labels)","metadata":{"id":"K3yccSrZSi2c"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Reviewing shapes of training and validation \nprint('X_Train shape:',X_train.shape,\n      '\\n','X_val shape:',X_val.shape,\n      '\\n','Y_train shape:',y_train.shape,\n      '\\n','y_val shape:',y_val.shape)","metadata":{"id":"HBw56elcS5FH","outputId":"7f03c61c-a487-439f-82ef-67db991baa11"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model Training","metadata":{"id":"7AX4xZhzUS2v"}},{"cell_type":"code","source":"# Defining a sequential model architecture using single hidden layer\n# Goal is to showcase performance from a simple model architecture\nmodel = Sequential()\nmodel.add(Dense(60, activation='relu', input_shape=(X_train.iloc[1].shape))) # Single hidden layer\nmodel.add(Dropout(.3)) # Good practice to help with reducing overfitting risk\nmodel.add(Dense(10, activation='softmax')) # Softmax with 10 nodes for the 10 possible labels\n\n# Generate review of the model architecture\nmodel.summary()","metadata":{"id":"y_DX8knGUR0V","outputId":"1cd7bc9a-2c32-47a4-9948-c2d2adab1fd0"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Compile the model \nmodel.compile(optimizer=tf.keras.optimizers.Adam(1e-4),\n              loss=tf.keras.losses.SparseCategoricalCrossentropy(),\n              metrics=['accuracy'])","metadata":{"id":"JrOx19fpUv2x"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Fit the model. Setting 60 epochs but using early stopping\noutput = model.fit(X_train,\n                   y_train,\n                   epochs = 60,\n                   validation_data=(X_val,y_val),\n                   callbacks=[tf.keras.callbacks.EarlyStopping(monitor='val_accuracy', patience=2)]\n                   )","metadata":{"id":"GegS1jVkUxOB","outputId":"4f2761e4-54c3-4ff2-c2b1-9dcc42031428"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Review Model Performance","metadata":{"id":"Oz63WDQqv3lZ"}},{"cell_type":"code","source":"# Plot model performance using plot_history\nplot_history(output.history);","metadata":{"id":"uPAhRzCYrYiY","outputId":"893b8737-23a7-4fba-b8f0-a04613243c26"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create predicted labels from the original test dataset\npred = np.argmax(model.predict(testdf),axis=1)","metadata":{"id":"tJIdlymMVA8j"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Show an example from the test dataset, including its predicted label\nnum = 900\nplt.imshow(np.reshape(testdf.iloc[num,:].values,(28,28)))\nplt.title('The PREDICTED label of the image is:'+str(pred[num]));","metadata":{"id":"-nbVn8mmXmt5","outputId":"974a419e-5da1-4818-94ca-a571d2eab7a1"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create subplots for first 15 numbers from the test dataset\nplt.figure(figsize=(12,6))\nfor im in range(15):\n  plt.subplot(3, 5, im + 1)\n  plt.imshow(np.reshape(testdf.iloc[im,:].values,(28,28)))\n  plt.xticks([])\n  plt.yticks([])\n  plt.xlabel(pred[im])\nplt.show();","metadata":{"id":"sFJsFrbNpF3U","outputId":"c96a1670-c5e4-40fe-c6a8-e52a9e176ff4"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Reviewing Hidden Layer Activation Values","metadata":{"id":"S5c-b54mmRcJ"}},{"cell_type":"code","source":"# Extracts the outputs of the hidden layer:\nlayer_outputs = [layer.output for layer in model.layers]\n\n# Creates a model that will return these outputs, given the model input:\nactivation_model = models.Model(inputs=model.input, outputs=layer_outputs)\n\nprint(f\"There are {len(layer_outputs)} layers\",'\\n')\nlayer_outputs","metadata":{"id":"08nVQsnphI5U","outputId":"93c75493-bbb7-457e-9a38-da092cf0f60f"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.layers[0]","metadata":{"id":"C7K9xVU7FATK","outputId":"0f692026-8cb5-4083-8717-c5ff86fc2779"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"activations = activation_model.predict(X_train) # Predicting the training classes to generate activation values\nhidden_layer_activation = activations[0] # Hidden layer\noutput_layer_activations = activations[1] # Hidden layer's output\nhidden_layer_activation.shape   #  Each of the 60 hidden nodes has one activation value per training image\noutput_layer_activations.shape # Checking shape","metadata":{"id":"uDX26FGDhrk3","outputId":"af127840-6b60-479e-e693-76779d9cc9ae"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"The maximum activation value of the hidden nodes in the hidden layer is: {hidden_layer_activation.max()}\")","metadata":{"id":"wFbI1ssoh_CZ","outputId":"f0d96255-9c60-4078-857e-ddd44d518b02"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"activation_data = {'actual_class':y_train}\nfor k in range(0,model.layers[0].output_shape[1]):\n  activation_data[f\"activation_{k}\"] = hidden_layer_activation[:,k]\n\nactivation_df = pd.DataFrame(activation_data)\nactivation_df.T # Transpose the dataframe so rows become columns","metadata":{"id":"X55qwA0fiKEb","outputId":"61fa4a86-d048-4740-a0e8-86f48af45c2a"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"node = 10 # Random node from the hidden layer\n\n# Visualize the activation values from the selected node\nplt.figure(figsize=(16,8))\nsns.set(font_scale=1)\nsns.boxplot(y='activation_'+str(node), \n            x='actual_class', \n            data=activation_df, \n            width=0.5,\n            palette=\"colorblind\")\nplt.title('Activation Values Boxplots')\nplt.show();","metadata":{"id":"oM9e7hZ_i2Xj","outputId":"420479d2-1b7c-4260-fc18-e951b587bee9"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission =pd.DataFrame({\"ImageId\": list(range(1,len(pred)+1)),\n                         \"Label\": pred})\nsubmission.to_csv(\"mnistbp.csv\", index=False, header=True)\n","metadata":{"id":"Dcyh6MjFGSNY"},"execution_count":null,"outputs":[]}]}