{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"df=pd.read_csv('/kaggle/input/Kannada-MNIST/train.csv')\ntest = pd.read_csv('/kaggle/input/Kannada-MNIST/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Label : TARGET FOR CLASSIFICATION\n    \nPixel0 to Pixel783: These are the grayscale pixel values of the image.That is each row contains 28 * 28 = 784 pixel (0-783) values here"},{"metadata":{"trusted":true},"cell_type":"code","source":"df.label.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that each digit has the same number of observations this tells us that the dataset is balanced"},{"metadata":{},"cell_type":"markdown","source":"# LABEL AND FEATURES \n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"X=df.drop('label',axis=1)\ny=df.label\n\ntest = test.drop('id',axis=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Normalize pixels\n\nThis can be achieved by dividing all pixels values by the largest pixel value; that is 255. This is performed across all channels, regardless of the actual range of pixel values that are present in the image.\n\nFor most image data, the pixel values are integers with values between 0 and 255.\n\nNeural networks process inputs using small weight values, and inputs with large integer values can disrupt or slow down the learning process. As such it is good practice to normalize the pixel values so that each pixel value has a value between 0 and 1.\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"X=X/255\n\ntest = test/255","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# reshape data to 28 28 2d matrix"},{"metadata":{"trusted":true},"cell_type":"code","source":"X=X.values.reshape(-1,28,28,1)\ntest=test.values.reshape(-1,28,28,1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Encoding target"},{"metadata":{"trusted":true},"cell_type":"code","source":"len(X[0]),len(X[0][0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.utils.np_utils import to_categorical\n\ny=to_categorical(y)\ny","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# TRAIN TEST SPLIT FOR VALIDATION \n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nX_train,X_test,y_train,y_test=train_test_split(X,y,random_state=42,test_size=0.15)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train[0][:,:,0] # pixel values of first entry ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\n\n\nplt.imshow(X_train[0][:,:,0]) #Visualizing this matrix as picture\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# DATA AUGMENTATION \n\nFor example, the number is not centered The scale is not the same (some who write with big/small numbers) The image is rotated...\n\nApproaches that alter the training data in ways that change the array representation while keeping the label the same are known as data augmentation techniques. Some popular augmentations people use are grayscales, horizontal flips, vertical flips, random crops, color jitters, translations, rotations, and much more.\n\nBy applying just a couple of these transformations to our training data, we can easily double or triple the number of training examples and create a very robust model.\n\n\n\nFor the data augmentation, i choosed to :\n\n    Randomly rotate some training images by 10 degrees\n    Randomly Zoom by 10% some training images\n    Randomly shift images horizontally by 10% of the width\n    Randomly shift images vertically by 10% of the height\n\nI did not apply a vertical_flip nor horizontal_flip since it could have lead to misclassify symetrical numbers such as 6 and 9.\n\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.preprocessing.image import ImageDataGenerator\n\ndatagen = ImageDataGenerator(\n        featurewise_center=False, \n        samplewise_center=False,  \n        featurewise_std_normalization=False, \n        samplewise_std_normalization=False,  \n        zca_whitening=False,  \n        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)\n        zoom_range = 0.1, # Randomly zoom image \n        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)\n        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)\n        horizontal_flip=False,  \n        vertical_flip=False) \n\n\ndatagen.fit(X_train) # only augment training dataset ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# BUILDING CNN"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.models import Sequential\n\nfrom keras.layers import Dense,Conv2D,Flatten,MaxPool2D,Dropout,BatchNormalization\n\n\nmodel = Sequential()\n\n# add conv layers \n\n\nmodel.add(Conv2D(filters = 32, kernel_size = (5,5),padding = 'Same', \n                 activation ='relu', input_shape = (28,28,1)))\nmodel.add(Conv2D(filters = 32, kernel_size = (5,5),padding = 'Same', \n                 activation ='relu'))\n\n\n# normalization latyer \n\nmodel.add(BatchNormalization(momentum=.15))\n\n#Pooling layer \n\nmodel.add(MaxPool2D(pool_size=(2,2)))\n\n#dropout \n\nmodel.add(Dropout(0.25))\n\n# same structure again with different parameters \n\nmodel.add(Conv2D(filters = 64, kernel_size = (3,3),padding = 'Same', \n                 activation ='relu'))\nmodel.add(Conv2D(filters = 64, kernel_size = (3,3),padding = 'Same', \n                 activation ='relu'))\n\nmodel.add(BatchNormalization(momentum=0.15))\n\nmodel.add(MaxPool2D(pool_size=(2,2), strides=(2,2)))\n\nmodel.add(Dropout(0.25))\n\n# again \n\nmodel.add(Conv2D(filters = 32, kernel_size = (5,5),padding = 'Same', \n                 activation ='relu', input_shape = (28,28,1)))\nmodel.add(Conv2D(filters = 32, kernel_size = (5,5),padding = 'Same', \n                 activation ='relu'))\nmodel.add(BatchNormalization(momentum=.15))\nmodel.add(MaxPool2D(pool_size=(2,2)))\nmodel.add(Dropout(0.25))\n\n#flatteining our output to feed into a ANN \nmodel.add(Flatten())\n\nmodel.add(Dense(256, activation = \"relu\"))\nmodel.add(Dropout(0.4))\n\n#output layer that will give us the label \nmodel.add(Dense(10, activation = \"softmax\"))\n\n\n\nmodel.summary()\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### structure of our model \n"},{"metadata":{},"cell_type":"markdown","source":"# OPTIMIZER\n\n## we need to choose a good optimizer so that our model can learn effectively \n\n# loss function - categorical_crossentropy"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.optimizers import Adam #,RMSprop\noptimizer=Adam(learning_rate=0.001,beta_1=0.9,beta_2=0.999)\n\n\nmodel.compile(optimizer=optimizer,loss=['categorical_crossentropy'],metrics=['accuracy'])\n\n\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The LR is the step by which the optimizer walks through the 'loss landscape'. The higher LR, the bigger are the steps and the quicker is the convergence. However the sampling is very poor with an high LR and the optimizer could probably fall into a local minima.\n\nIts better to have a decreasing learning rate during the training to reach efficiently the global minimum of the loss function.\n\nTo keep the advantage of the fast computation time with a high LR, i decreased the LR dynamically every X steps (epochs) depending if it is necessary (when accuracy is not improved).\n\nWith the ReduceLROnPlateau function from Keras.callbacks, i choose to reduce the LR by half if the accuracy is not improved after 3 epochs."},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.callbacks import ReduceLROnPlateau\nlearning_rate_reduction = ReduceLROnPlateau(monitor='val_acc', \n                                            patience=3, \n                                            verbose=1, \n                                            factor=0.5, \n                                            min_lr=0.00001)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# FITTING/TRAINING OUR MODEL "},{"metadata":{"trusted":true},"cell_type":"code","source":"epochs=5 #change this to 30 if you need to get better score\nbatch_size=64\n\n# Fit the model\nhistory = model.fit_generator(datagen.flow(X_train,y_train, batch_size=batch_size),\n                              epochs = epochs, \n                              validation_data = (X_test,y_test),\n                              verbose = 2, \n                              steps_per_epoch=X_train.shape[0] // batch_size, \n                              callbacks=[learning_rate_reduction])\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig,ax=plt.subplots(2,1)\nfig.set\nx=range(1,1+epochs)\nax[0].plot(x,history.history['loss'],color='red')\nax[0].plot(x,history.history['val_loss'],color='blue')\n\nax[1].plot(x,history.history['accuracy'],color='red')\nax[1].plot(x,history.history['val_accuracy'],color='blue')\nax[0].legend(['trainng loss','validation loss'])\nax[1].legend(['trainng acc','validation acc'])\nplt.xlabel('Number of epochs')\nplt.ylabel('accuracy')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history.history['accuracy'][-1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test=pd.read_csv('../input/Kannada-MNIST/test.csv')\n\n\n\n\ntest=test.drop('id',axis=1)\ntest=test/255\ntest=test.values.reshape(-1,28,28,1)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(test[0][:,:,0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test[[0]][:,:,:].shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# PREDICTIONS "},{"metadata":{"trusted":true},"cell_type":"code","source":"test[0].shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"preds = model.predict(test)\n\n\npreds[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\n\npreds=np.argmax(preds,axis=1)\npreds","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test[[55]][:,:,:].shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# PREDICTING WITH A SINGLE IMAGE "},{"metadata":{"trusted":true},"cell_type":"code","source":"preds = model.predict(test[[55]][:,:,:])\npreds=np.argmax(preds,axis=1)\npreds","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# SAVE MODEL AS H5\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"model.save('my_model.h5')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":""}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}