{"cells":[{"metadata":{},"cell_type":"markdown","source":"# **Importing all the important libraries**\n\nStarting my Kernel for Digit Recognizer.\nBefore this, I have been working on the same problem with 20by20 Pixel MNSIT images in MATLAB.\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n%matplotlib inline\nfrom sklearn.metrics import confusion_matrix\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **Importing Keras**"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.utils.np_utils import to_categorical # convert to one-hot-encoding\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D, BatchNormalization\nfrom keras.optimizers import Adam\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.callbacks import LearningRateScheduler\nfrom keras.losses import categorical_crossentropy","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **Load the Data**\n\nTraning Data is saved in X and test data is X_test. These are saved as Dataframe object.\n\nMost of the pixels are empty. Having values in only few pixels and logically it should be this way. \n\nRemoving output 'label' from traning data and saving it as target variable.\n\nThe countplot tells us that training data is balanced or has almost equal number of samples for all the classes.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"X = pd.read_csv(\"/kaggle/input/digit-recognizer/train.csv\")\nX_test = pd.read_csv(\"/kaggle/input/digit-recognizer/test.csv\")\n# Drop 'label' column\ny = X.label\nX.drop(['label'], axis=1, inplace=True)\n\ng = sns.countplot(y)\ny.value_counts()\n\nprint(X.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Spliting data into train and cross-valdiaton."},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\nX_train, X_val, y_train, y_val = train_test_split(X, y,\n                                                      train_size=0.8, test_size=0.2,\n                                                      random_state=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Earlier operations were done on Dataframe, we are converting it to array for further processes."},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train = np.array(X_train)\nX_val = np.array(X_val)\ny_train = np.array(y_train)\ny_val = np.array(y_val)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **Visualization of Data**\n\nThis is to show what exactly do these numbers in 784 columns represent. In the second image assigned label (corresponding 'y') is in yellow box on top of the image."},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize= [3,3])\nplt.imshow(X_train[3].reshape(28,28), cmap='gray')\nax.set_title(y_train[3])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import random\nindices = range(len(y_train))\nbox = dict(facecolor='yellow', pad=5, alpha=1)\n\nfig, ax = plt.subplots(10, 10, squeeze=True, figsize=(24,12))\n\nfor n in range(10):\n    for m in range(10):\n        d=random.choice(indices)\n        ax[n][m].imshow(X_train[d].reshape(28,28), cmap='gray')\n        ax[n][m].set_title(y_train[d],y=0.9,bbox=box)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Reshaping the data for CNN"},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train = X_train.reshape(-1, 28, 28, 1)\nX_val = X_val.reshape(-1, 28, 28, 1)\nprint(X_train.shape, y_train.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The pixel fill strength is a vlaue between 0-255. Normalization gives better esults."},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train = X_train.astype(\"float32\")/255.\nX_val = X_val.astype(\"float32\")/255.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"One-Hot encoding the target variable"},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train = to_categorical(y_train)\ny_val = to_categorical(y_val)\n\nprint(y_train[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **Training the model**\n\nModel is sequential. I tried three layers of Conv2D. In second and third layer stride length is set to 2 and after first one there is 0.2 Dropout. Ih had added MaxPool2D layer but chose against it after completing the Deep Learning Course. It suggested the 'Stride Length' parameter in Conv2D covers this aspect more or less with similar effect. Before the final Dense Layer two more dense layers are added each followed by Dropout. Number of classes that we need finally is 10 (numbers 0-9) so last Dense layers has 10 nodes. \n\nAnother point to note here is that as we are using Dropout, validation accuracy here may be lower that we can achieve with given layers. Because we are not overfitting the data here, it should give as better result on private dataset compared to results with model had we not used dropout.\nMoreover, I'm experimenting with different configuration of layers and parameters so output keeps changing based on that."},{"metadata":{"trusted":true},"cell_type":"code","source":"model = Sequential()\nmodel.add(Conv2D(20, kernel_size=(3, 3),\n                 activation='relu',\n                 input_shape=(28, 28, 1)))\n# model.add(BatchNormalization())\nmodel.add(Conv2D(20, kernel_size=(3, 3), activation='relu', strides = 2))\nmodel.add(Dropout(0.2))\nmodel.add(Conv2D(20, kernel_size=(3, 3), activation='relu'))\n# model.add(BatchNormalization())\n# model.add(MaxPool2D(pool_size=(2,2)))\n\nmodel.add(Flatten())\nmodel.add(Dense(512, activation='relu'))\nmodel.add(Dropout(0.25))\nmodel.add(Dense(1024, activation='relu'))\nmodel.add(Dropout(0.2))\nmodel.add(Dense(10, activation='softmax'))\n\nmodel.compile(loss='categorical_crossentropy', optimizer = Adam(lr=1e-4), metrics=[\"accuracy\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"datagen = ImageDataGenerator(zoom_range = 0.1,\n                            height_shift_range = 0.1,\n                            width_shift_range = 0.1,\n                            rotation_range = 10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Learning Rate Scheduler lets you keep a high learning rate close to a crest in gradient descent and small when close to bottom of a 'valley'. This way code is optimized to take larger steps when it can. Model summary will list out all the layers we have in the model."},{"metadata":{"trusted":true},"cell_type":"code","source":"annealer = LearningRateScheduler(lambda x: 1e-3 * 0.9 ** x)\n\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"hist = model.fit_generator(datagen.flow(X_train, y_train, batch_size=16),\n                           steps_per_epoch=500,\n                           epochs=30, #Increase this when not commiting the Kernel or testing few changes\n                           verbose=2,  #1 for ETA, 0 for silent\n                           validation_data=(X_val, y_val),\n                           callbacks=[annealer])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"final_loss, final_acc = model.evaluate(X_val, y_val, verbose=0)\nprint(\"Final loss: {0:.4f}, final accuracy: {1:.4f}\".format(final_loss, final_acc))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.plot(hist.history['loss'], color='b')\nplt.plot(hist.history['val_loss'], color='r')\nplt.show()\nplt.plot(hist.history['acc'], color='b')\nplt.plot(hist.history['val_acc'], color='r')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Confusion Matrix helps you identify which type of data maybe causing the higher losses and lower accuracy. Y-axis is True Lable and X-axis is Predicted Label. In some cases I get  16 '9's being read as '5's. This makes sense as sometimes a nine can look like a five if upper segment and middle segment is not connected well. Such mistkaes in recognizing digits can be made by humans as well. iwht more epochs such false positives are reduced."},{"metadata":{"trusted":true},"cell_type":"code","source":"y_hat = model.predict(X_val)\ny_pred = np.argmax(y_hat, axis=1)\ny_true = np.argmax(y_val, axis=1)\nconfusion_matrix(y_true, y_pred)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Submit**"},{"metadata":{"trusted":true},"cell_type":"code","source":"X_test = np.array(X_test)\nX_test = X_test.reshape(-1, 28, 28, 1)/255.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_hat = model.predict(X_test, batch_size=64)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_pred = np.argmax(y_hat,axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"output_file = \"submission.csv\"\nwith open(output_file, 'w') as f :\n    f.write('ImageId,Label\\n')\n    for i in range(len(y_pred)) :\n        f.write(\"\".join([str(i+1),',',str(y_pred[i]),'\\n']))","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}