{"cells":[{"metadata":{},"cell_type":"markdown","source":"For this kernel, we attempt to create a model from scratch to recognise handwritten digits. We'll be using keras in this project. Then later visualise how the model is learning."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"from keras.models import Model, Sequential\nfrom keras.optimizers import Adam\nfrom keras.layers import Dense, Conv2D, MaxPool2D , Flatten\nfrom keras import metrics\nfrom keras.losses import categorical_crossentropy\nfrom keras.utils.np_utils import to_categorical \n\nimport matplotlib.pyplot as plt\n\nfrom sklearn.model_selection import train_test_split\n\nimport pandas as pd\nimport numpy as np","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"First of all always check where are our input data are and load it using to a pandas Dataframe."},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls ../input/digit-recognizer","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/digit-recognizer/train.csv')\ntest = pd.read_csv('../input/digit-recognizer/test.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Then check what our data looks like"},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next is double checking if we have missing values and try to address it."},{"metadata":{"trusted":true},"cell_type":"code","source":"train.isnull().sum().describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.isnull().sum().describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Fortunately for us, we don't have missing values. So then we can move on to our next task.\n\nFirst of all, we have to preprocess our data. Drop the label column to get the vanilla x train data. Then use the label column as our y train."},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train = train['label']\nx_train = train.drop(['label'], axis=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We then check the distribution if we have a fairly distributed data (balanced data). Otherwise we need to fix it."},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.hist(y_train, bins=10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Yes so we do have a fairly distributed data. We can now proceed.\n\nProcessing our x_train and test data to a 255x255 px to get our exact image(s)"},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train = x_train / 255.0\ntest = test / 255.0","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Then we reshape them to a 28x28x1 (1 is channel, meaning we have a grayscale image)"},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train = x_train.values.reshape(-1,28,28,1)\ntest = test.values.reshape(-1,28,28,1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Then our y_train is converted to categorical with 10 different classes because we have 10 different numbers (0-9)"},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train = to_categorical(y_train, num_classes = 10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And finally get our train and test split using x_train and y_train. We will set the test_size to 0.2 meaning we have a 80-20 split. Then set the random_state for reproducibility."},{"metadata":{"trusted":true},"cell_type":"code","source":"xx_train, x_test, yy_train, y_test = train_test_split(x_train, y_train, test_size = 0.2, random_state=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we create our model which is a sequential model. The first layer is basically our input layer. It is a convolutional2d layer with 28x28x1 input--remember we reshaped our images to the same size earlier?\n\nThen from taking the notable features using conv2d layers, we pool it using a maxpool2d layer, then flatten to be able to feed the processed data to a dense layer for categorising later on.\n\nThe output layer is a dense layer with 10 units. Meaning 10 different outputs (it's because we have 10 categories)."},{"metadata":{"trusted":true},"cell_type":"code","source":"model = Sequential()\nmodel.add(Conv2D(input_shape=(28,28,1),filters=32,kernel_size=(3,3),padding=\"same\", activation=\"relu\"))\nmodel.add(Conv2D(filters=64,kernel_size=(3,3),padding=\"same\", activation=\"relu\"))\nmodel.add(MaxPool2D(pool_size=(2,2),strides=(2,2)))\nmodel.add(Flatten())\nmodel.add(Dense(units=4096,activation=\"relu\"))\nmodel.add(Dense(units=10, activation=\"softmax\"))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We'll then compile our model with adam optimiser with a learning rate of 0.0001 and a loss of categorical crossentropy. We'll also use accuracy as our metric to see the effectivity of our model."},{"metadata":{"trusted":true},"cell_type":"code","source":"opt = Adam(lr=0.0001)\nmodel.compile(optimizer=opt, loss=categorical_crossentropy, metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Then finally, we'll start training. We'll train using 5 epochs."},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model.fit(xx_train, yy_train, epochs = 5, \n                    validation_data = (x_test, y_test), verbose = 1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Done. So we're getting fairly a 98% accuracy. Let's visualise this to understand better."},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.plot(history.history['loss'])\nplt.plot(history.history['val_loss'])\nplt.legend()\nplt.title('Loss Trend for training and validation data')\nplt.ylabel('Epochs')\nplt.xlabel('Loss')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.plot(history.history['accuracy'])\nplt.plot(history.history['val_accuracy'])\nplt.legend()\nplt.title('Accuracy Trend for training and validation data')\nplt.ylabel('Epochs')\nplt.xlabel('Accuracy')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We see that loss is steadily decreasing and accuracy is increasing. This means our model does not overfit--which is great!\n\nNow let's try to use our model against the test and create a dataframe with the expected output for the competition."},{"metadata":{"trusted":true},"cell_type":"code","source":"results = model.predict(test)\nresults = np.argmax(results,axis = 1)\nresults = pd.Series(results,name=\"label\")\nresults","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.concat([pd.Series(range(1,28001),name = \"ImageId\"),results],axis = 1)\nsubmission","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Then finally we create a submission file."},{"metadata":{"trusted":true},"cell_type":"code","source":"submission.to_csv(\"cnn_mnist_submission.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And that's it. If this notebook helped you in any way, please upvote! Thanks!"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}