{"cells":[{"metadata":{},"cell_type":"markdown","source":"Forked from this nice kernel https://www.kaggle.com/shahules/indian-way-to-learn-cnn"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as  np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix\n\nfrom keras.utils.np_utils import to_categorical\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.models import Sequential\nfrom keras.layers import Dense,Conv2D,Flatten,MaxPool2D,Dropout,BatchNormalization\nfrom keras.optimizers import RMSprop,Adam\nfrom keras.callbacks import ReduceLROnPlateau","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Loading data"},{"metadata":{"trusted":true},"cell_type":"code","source":"train=pd.read_csv('../input/Kannada-MNIST/train.csv')\ntest=pd.read_csv('../input/Kannada-MNIST/test.csv')\nsample_sub=pd.read_csv('../input/Kannada-MNIST/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Before jumping to all complex stuff about Convolutions and all,we will simply understand our data.We will learn and gain basic understanding about this data."},{"metadata":{"trusted":true},"cell_type":"code","source":"print('The Train  dataset has {} rows and {} columns'.format(train.shape[0],train.shape[1]))\nprint('The Test  dataset has {} rows and {} columns'.format(test.shape[0],test.shape[1]))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head(3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.head(3)\ntest=test.drop('id',axis=1)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Checking Target class distribution..\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"y=train.label.value_counts()\nsns.barplot(y.index,y)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we can see that all of the classes has equal distribution.There are 6000 examples of each numbers in kannada in the the training dataset.Cool !"},{"metadata":{},"cell_type":"markdown","source":"## Data preparation <a id='2'></a>"},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train=train.drop('label',axis=1)\nY_train=train.label","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Normalize Pixel Values\n\nFor most image data, the pixel values are integers with values between 0 and 255.\n\nNeural networks process inputs using small weight values, and inputs with large integer values can disrupt or slow down the learning process. As such it is good practice to normalize the pixel values so that each pixel value has a value between 0 and 1.\n\nIt is valid for images to have pixel values in the range 0-1 and images can be viewed normally.\n\nThis can be achieved by dividing all pixels values by the largest pixel value; that is 255. This is performed across all channels, regardless of the actual range of pixel values that are present in the image."},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train=X_train/255\ntest=test/255","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Reshape"},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train=X_train.values.reshape(-1,28,28,1)\ntest=test.values.reshape(-1,28,28,1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('The shape of train set now is',X_train.shape)\nprint('The shape of test set now is',test.shape)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"All Set,We have our data reshape into 60000 examples of height 28 and width 28 and 1 channel."},{"metadata":{},"cell_type":"markdown","source":"### Splitting train and test"},{"metadata":{},"cell_type":"markdown","source":"Now we will split out training data into train and validation data.15percent of the training data will be used for validation purpose."},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train,X_test,y_train,y_test=train_test_split(X_train,Y_train,random_state=42,test_size=0.15)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.imshow(X_train[0][:,:,0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It's Nine in Kannada\n"},{"metadata":{},"cell_type":"markdown","source":"### More data !"},{"metadata":{},"cell_type":"markdown","source":"In order to avoid overfitting problem, we need to expand artificially our handwritten digit dataset. We can make your existing dataset even larger. The idea is to alter the training data with small transformations to reproduce the variations occuring when someone is writing a digit.\n\nFor example, the number is not centered The scale is not the same (some who write with big/small numbers) The image is rotated...\n\nApproaches that alter the training data in ways that change the array representation while keeping the label the same are known as data augmentation techniques. Some popular augmentations people use are grayscales, horizontal flips, vertical flips, random crops, color jitters, translations, rotations, and much more.\n\nBy applying just a couple of these transformations to our training data, we can easily double or triple the number of training examples and create a very robust model.\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"datagen = ImageDataGenerator(\n        featurewise_center=False,  # set input mean to 0 over the dataset\n        samplewise_center=False,  # set each sample mean to 0\n        featurewise_std_normalization=False,  # divide inputs by std of the dataset\n        samplewise_std_normalization=False,  # divide each input by its std\n        zca_whitening=False,  # apply ZCA whitening\n        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)\n        zoom_range = 0.1, # Randomly zoom image \n        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)\n        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)\n        horizontal_flip=False,  # randomly flip images\n        vertical_flip=False)  # randomly flip images\n\n\ndatagen.fit(X_train)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For the data augmentation, i choosed to :\n\n   - Randomly rotate some training images by 10 degrees\n   - Randomly Zoom by 10% some training images\n   - Randomly shift images horizontally by 10% of the width\n   - Randomly shift images vertically by 10% of the height\n\nI did not apply a vertical_flip nor horizontal_flip since it could have lead to misclassify symetrical numbers such as 6 and 9."},{"metadata":{},"cell_type":"markdown","source":"## Modelling <a id='4' ></a>"},{"metadata":{"trusted":true},"cell_type":"code","source":"model = Sequential()\n\nmodel.add(Conv2D(filters = 32, kernel_size = (5,5),padding = 'Same', \n                 activation ='relu', input_shape = (28,28,1)))\nmodel.add(Conv2D(filters = 32, kernel_size = (5,5),padding = 'Same', \n                 activation ='relu'))\nmodel.add(BatchNormalization(momentum=.15))\nmodel.add(MaxPool2D(pool_size=(2,2)))\nmodel.add(Dropout(0.25))\n\n\nmodel.add(Conv2D(filters = 64, kernel_size = (3,3),padding = 'Same', \n                 activation ='relu'))\nmodel.add(Conv2D(filters = 64, kernel_size = (3,3),padding = 'Same', \n                 activation ='relu'))\nmodel.add(BatchNormalization(momentum=0.15))\nmodel.add(MaxPool2D(pool_size=(2,2), strides=(2,2)))\nmodel.add(Dropout(0.25))\n\nmodel.add(Conv2D(filters = 32, kernel_size = (5,5),padding = 'Same', \n                 activation ='relu', input_shape = (28,28,1)))\nmodel.add(Conv2D(filters = 32, kernel_size = (5,5),padding = 'Same', \n                 activation ='relu'))\nmodel.add(BatchNormalization(momentum=.15))\nmodel.add(MaxPool2D(pool_size=(2,2)))\nmodel.add(Dropout(0.25))\n\n\nmodel.add(Flatten())\nmodel.add(Dense(256, activation = \"relu\"))\nmodel.add(Dropout(0.4))\nmodel.add(Dense(1))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"model.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"optimizer=Adam(learning_rate=0.001,beta_1=0.9,beta_2=0.999)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import keras.backend as K\ndef Acc(y_true, y_pred, from_logits=False, label_smoothing=0):\n    y_pred=K.round(y_pred)\n    return K.mean(K.cast(K.equal(y_true,y_pred),y_pred.dtype))\n\nmodel.compile(optimizer=optimizer,loss='mae',metrics=['accuracy',Acc])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Learning rate reduction"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Set a learning rate annealer\nlearning_rate_reduction = ReduceLROnPlateau(monitor='val_accuracy', \n                                            patience=3, \n                                            verbose=1, \n                                            factor=0.5, \n                                            min_lr=0.00001)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Fitting our model <a id='5'></a>"},{"metadata":{"trusted":true},"cell_type":"code","source":"epochs=40 \nbatch_size=64","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":false},"cell_type":"code","source":"# Fit the model\nhistory = model.fit_generator(datagen.flow(X_train,y_train, batch_size=batch_size),\n                              epochs = epochs, validation_data = (X_test,y_test),\n                              verbose = 1, steps_per_epoch=X_train.shape[0] // batch_size\n                              , callbacks=[learning_rate_reduction])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Evaluating our approach <a id='6'></a>"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig,ax=plt.subplots(2,1)\nfig.set\nx=range(1,1+epochs)\nax[0].plot(x,history.history['loss'],color='red')\nax[0].plot(x,history.history['val_loss'],color='blue')\n\nax[1].plot(x,history.history['accuracy'],color='red')\nax[1].plot(x,history.history['val_accuracy'],color='blue')\nax[0].legend(['trainng loss','validation loss'])\nax[1].legend(['trainng acc','validation acc'])\nplt.xlabel('Number of epochs')\nplt.ylabel('accuracy')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have plotted the performance of our model.We can see the number of epochs in the X axis and change in model performance in Y axis."},{"metadata":{"trusted":true},"cell_type":"code","source":"y_pre_test=model.predict(X_test)\ny_pre_test=np.round(y_pre_test)\ny_pre_test[y_pre_test>9]=9\ny_pre_test[y_pre_test<0]=0\ny_pre_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Correct predictions\",np.sum((y_pre_test.flat==y_test).astype(int)))\nprint(\"Incorrect predictions\",np.sum((y_pre_test.flat!=y_test).astype(int)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Making a Submission <a id='7'></a>\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"test=pd.read_csv('../input/Kannada-MNIST/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_id=test.id\ntest=test.drop('id',axis=1)\ntest=test/255\ntest=test.values.reshape(-1,28,28,1)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We will make our prediction using our CNN model."},{"metadata":{"trusted":true},"cell_type":"code","source":"y_pre=model.predict(test)     ##making prediction\ny_pre=np.round(y_pre).astype(int) ##changing the prediction intro labels\ny_pre[y_pre>9]=9\ny_pre[y_pre<0]=0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_sub['label']=y_pre.flat\nsample_sub.to_csv('submission.csv',index=False)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_sub.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Things to try:\n1. Different Thresholds\n2. Other loss fuctions"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}