{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport tensorflow as tf\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten, Lambda, Conv2D, MaxPool2D\nfrom sklearn.model_selection import train_test_split\nfrom tensorflow.keras.callbacks import EarlyStopping\nimport tensorflow.keras.layers.experimental.preprocessing as preprocessing\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Competition Description\nMNIST (\"Modified National Institute of Standards and Technology\") is the de facto “hello world” dataset of computer vision. Since its release in 1999, this classic dataset of handwritten images has served as the basis for benchmarking classification algorithms. As new machine learning techniques emerge, MNIST remains a reliable resource for researchers and learners alike.\n\nIn this competition, your goal is to correctly identify digits from a dataset of tens of thousands of handwritten images. We’ve curated a set of tutorial-style kernels which cover everything from regression to neural networks. We encourage you to experiment with different algorithms to learn first-hand what works well and how techniques compare."},{"metadata":{},"cell_type":"markdown","source":"# The Data\nThe data files train.csv and test.csv contain gray-scale images of hand-drawn digits, from zero through nine.\n\nEach image is 28 pixels in height and 28 pixels in width, for a total of 784 pixels in total. Each pixel has a single pixel-value associated with it, indicating the lightness or darkness of that pixel, with higher numbers meaning darker. This pixel-value is an integer between 0 and 255, inclusive.\n\nThe training data set, (train.csv), has 785 columns. The first column, called \"label\", is the digit that was drawn by the user. The rest of the columns contain the pixel-values of the associated image.\n\nEach pixel column in the training set has a name like pixelx, where x is an integer between 0 and 783, inclusive. To locate this pixel on the image, suppose that we have decomposed x as x = i * 28 + j, where i and j are integers between 0 and 27, inclusive. Then pixelx is located on row i and column j of a 28 x 28 matrix, (indexing by zero).\n\nFor example, pixel31 indicates the pixel that is in the fourth column from the left, and the second row from the top, as in the ascii-diagram below."},{"metadata":{},"cell_type":"markdown","source":"# Load the training and testing data"},{"metadata":{"trusted":true},"cell_type":"code","source":"train= pd.read_csv('../input/digit-recognizer/train.csv')\ntrain.info()\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test= pd.read_csv('../input/digit-recognizer/test.csv')\ntest.info()\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#lets assign the training data and test data and validation data. #the training data can be floats and y data as integers (to predict)\nX_train=train.iloc[:,1:].values.astype('float32')\ny_train=train.iloc[:,0].values.astype('int32')\npred_test=test.values.astype('float32')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_test","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data Vizualisation"},{"metadata":{"trusted":true},"cell_type":"code","source":"#We need to reshape the train, test and validation data (num of images, img_rows ,img_columns), it will be a matrix where each value in the matrix \n#represents a pixel.\nX_train=X_train.reshape(X_train.shape[0],28,28)\npres_test=pred_test.reshape(pred_test.shape[0],28,28)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#lets see some image 0-6\n#title shows true label \n\nfig, axes =plt.subplots(nrows=1,ncols=7,figsize=(12,6))\n\n\nfor i in range(7):\n    axes[i].imshow(X_train[i],cmap=plt.get_cmap('gray'))\n    axes[i].set_title(y_train[i])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Hot enconding validation data y_train\nwe currently have a multi class cathegorization problem where the last layer is going to have as many perceptrons(neurons) as there are cathegorys to predict, this is gona contain as many dimensions as there are cathegories to predict. Therefor our predicted value cant be compared to just an integer. We hot encode it to an a row of n_dim= catheogories "},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train[1]#what label is picture 2?","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.utils import to_categorical\ny_train=to_categorical(y_train,num_classes=10)\ny_train[0]#first\ny_train.shape[1]#shape of the array representing the label ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Standardize the data\nWe will standardize the data, this is common since many of the pixels will have high values. For the ai model, this will be problematic during backpropagation when trying to calibrate the weights. This will result in noise bieng amplified that wont work in favor of the model. To solve this problem we use a MinMax function that normalizes the values  between 0 and 1. \n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"#We create a function that will be used inside the Lambda of the model on the training data. This is the MinMaxScaler\n#and makes sure the feature (pixel data) is standardized to be between 0 and 1. Depending on min max range choosen.\nx_max=X_train.max().astype(np.float32)\nx_min=X_train.min().astype(np.float32)\nmin_range=np.float32(0)\nmax_range=np.float32(1)\ndef Min_Max_Scaler(x,min_range=min_range,max_range=max_range):\n    return ((x-x_min)/(x_max-x_min))*(max_range-min_range)+min_range","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Lets start building simple neural network, later we will try to use a convolutional neural network (cnn) which is suited for image recognition."},{"metadata":{},"cell_type":"markdown","source":"# Simple Neural Network"},{"metadata":{"trusted":true},"cell_type":"code","source":"model_1 = Sequential()\nmodel_1.add(Lambda(Min_Max_Scaler,input_shape=(28,28)))\nmodel_1.add(Flatten())#this will make the data 1d array which is gonna be possible to use in the model\nmodel_1.add(Dense(units=512,activation='relu'))#we have around 720 pixel types or features lets have the first layer close to that number\nmodel_1.add(Dropout(0.2))#turning of 20% of neurons to try to avoid overfitting\nmodel_1.add(Dense(units=10,activation='softmax'))#and then the last layer to classify, since this is a multi classification task activation function softmax is used.\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model evaluation, first we make a testsplit"},{"metadata":{},"cell_type":"markdown","source":"# Test_split\nThe test split is like the name implies a splitting of testing data and validation data, when training the model\nwe use the trained test split and trained validation split, of which both will be fractions of total training/validation data with regards to test_size. Once the model is trained, the second part (rest of the split) now named X_test and y_test is used to validate the model."},{"metadata":{"trusted":true},"cell_type":"code","source":"X=X_train\ny=y_train\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)\nX_train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Compile model\nWe will use Adam optimizer, and since its multiclass cathegorization task, we use crossentropy as loss function, with metrics of accuracy"},{"metadata":{"trusted":true},"cell_type":"code","source":"model_1.compile(\n    optimizer=tf.keras.optimizers.Adam(epsilon=0.01),\n    loss='categorical_crossentropy',\n    metrics=['accuracy']\n               )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Earlystopping"},{"metadata":{"trusted":true},"cell_type":"code","source":"early_stop=EarlyStopping(monitor='val_loss',mode='min',verbose=1,patience=5,restore_best_weights=True)\n#we want to monitor the minimizing of validation loss.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Now we fit the model to training data"},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model_1.fit(\n    X_train,\n    y_train,\n    epochs=200,\n    validation_data=(X_test,y_test),\n    callbacks=[early_stop],\n    batch_size=64\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"losses=pd.DataFrame(model_1.history.history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Comparing Loss to validation Loss"},{"metadata":{"trusted":true},"cell_type":"code","source":"losses_training=losses[['loss','val_loss']]\nlosses_training.plot()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#we can see some divergence from loss to validation loss already around 4 epochs however, the validation loss is not increasing","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"losses","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred=model_1.predict_classes(pred_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred=pd.DataFrame(pred,columns=['pred_number'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred.head(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Now we will try Convolutional Neural Network "},{"metadata":{},"cell_type":"markdown","source":"Convolution Neural Networks are optimal for image classification tasks.\nEssentially the network is built up from two parts:\nThe base and the Head.\n\nIn the base we extract information from the images, this part is formed from layers performing convolutional operations but can contain other layers aswell, usually pooling layers.\nThe head is used to determine the class of the image and is formed primarily of dense layers (like the simple neural network). \n\nWe will also include augmentation, which is a way of providing fake data by making shifts of the images, can be anything from contrast, rotations to flipping. This makes sure the model can indentify smaller changes in the classes (for example a if a number is rotated 30 degrees the model will train for this variations). In this task however we will be carefull with the flipping part and rotation part since a 9 flipped will not be distinguishable from a 6.\n\nThe convolutional part:\nThe convolution filters the images for particular features, than we specify activation functions that will detect features within the filtered image.\nThe pooling part:\nThe pooling part of the base is used for condensing the filtered images, simply enhancing the features and removing unecessary information.\n\n"},{"metadata":{},"cell_type":"markdown","source":"# CNN Model"},{"metadata":{},"cell_type":"markdown","source":"When using the convolutional network functions within keras we also need a 3th dimension, this dimension represents the color channels(RGB) for this task it will be one since we are using grey-scale"},{"metadata":{"trusted":true},"cell_type":"code","source":"#reassigning values\nX_train=train.iloc[:,1:].values.astype('float32')\ny_train=train.iloc[:,0].values.astype('int32')\npred_test=test.values.astype('float32')\n#Hot coding\ny_train=to_categorical(y_train,num_classes=10)\n#reshaping with the new dimension aswell\nX_train=X_train.reshape(X_train.shape[0],28,28,1)\npred_test=pred_test.reshape(pred_test.shape[0],28,28,1)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Lets start with a CNN containing three convolutional layers.\nThe kernel_size will be 3 with three of this layers the perceptive field for each neuron is 7x7 which  is could be fine enough, we will have to try it out. Padding is same meaning we dont lose size after each convolutional layer, pooling have padding same to in first pooling layer and valid in the next two layers some size will be lost, thats the reason we can have more filters."},{"metadata":{"trusted":true},"cell_type":"code","source":"model_2 = Sequential([\n    #Standardizing\n    Lambda(Min_Max_Scaler,input_shape=(28,28,1)),\n    #Augmentation\n    preprocessing.RandomContrast(0.2),\n    preprocessing.RandomRotation(0.1),\n\n    #The Base, first convolutional block\n    Conv2D(filters=64,kernel_size=3,activation='relu',padding='same',\n          input_shape=[28,28,1]),\n    #filters is how many filters to be created from each picture,kernel_size is the dimension of the sliding window\n    MaxPool2D(padding='same'),\n    #Second convolutional block\n    Conv2D(filters=64,kernel_size=3,activation='relu',padding='same',\n          input_shape=[28,28,1]),\n    MaxPool2D(),\n    #third convolutional block\n    Conv2D(filters=128,kernel_size=3,activation='relu',padding='same',\n          input_shape=[28,28,1]),\n    MaxPool2D(),\n    #Classified Head\n    Flatten(),\n    Dense(units=512,activation='relu'),\n    Dropout(0.2),\n    Dense(units=10,activation='softmax')\n])\nmodel_2.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Compile the model"},{"metadata":{"trusted":true},"cell_type":"code","source":"model_2.compile(\n    optimizer=tf.keras.optimizers.Adam(epsilon=0.01),\n    loss='categorical_crossentropy',\n    metrics=['accuracy']\n               )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Fit the model including early stopping"},{"metadata":{"trusted":true},"cell_type":"code","source":"history_2 = model_2.fit(\n    X_train,\n    y_train,\n    epochs=200,\n    validation_data=(X_test,y_test),\n    callbacks=[early_stop],\n    batch_size=64\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Comparing loss to validation loss"},{"metadata":{"trusted":true},"cell_type":"code","source":"losses_2=pd.DataFrame(model_2.history.history)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"losses_training_2=losses_2[['loss','val_loss']]\nlosses_training_2.plot()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Well seems like CNN did a good job, submitting predictions to kaggle"},{"metadata":{"trusted":true},"cell_type":"code","source":"pred=model_2.predict_classes(pred_test, verbose=0)\nsubmissions=pd.DataFrame({\"ImageId\": list(range(1,len(pred)+1)),\n                        \"Label\": pred})\nsubmissions.to_csv('Sub_dig_true.csv',index=False,header=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"True prediction score was 0.99371"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}