{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"**CNN**\n\nIn neural networks, Convolutional neural network (ConvNets or CNNs) is one of the main categories to do images recognition, images classifications. Objects detections, recognition faces etc., are some of the areas where CNNs are widely used.where we will use a Sequential Convolutional Neural Network for digits recognition trained on MNIST dataset. we choosed to build it with keras API which is very powerfull"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":" Firstly, we will concentrate on the data then we will focus on the CNN modeling and evaluation.we have 3 main task they are data prepation and cnn modeling and The results prediction and submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\nimport seaborn as sns\n%matplotlib inline\nimport tensorflow.keras as keras\nfrom keras.utils.np_utils import to_categorical # convert to one-hot-encoding\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D\nfrom keras.optimizers import RMSprop\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.callbacks import ReduceLROnPlateau\nnp.random.seed(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv(\"../input/digit-recognizer/train.csv\")\ntest = pd.read_csv(\"../input/digit-recognizer/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#now as label is the target we will saparate for training \nY_train = train[\"label\"]\nX_train = train.drop(labels = [\"label\"],axis = 1) \nX_train.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"missing data handling is one another big task so lets see if any missing values "},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train.isnull().sum()\nY_train.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"so there is no missing values in the data we have so next is to we will see there count and normalize them"},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.countplot(Y_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"again we have similar counts for the 10 digits so now lets start playing with our data "},{"metadata":{},"cell_type":"markdown","source":"Before modifying our data lets see how CNN works..!\n\nCNN image classifications takes an input image, process it and classify it under certain categories (Eg., Dog, Cat, Tiger, Lion). Computers sees an input image as array of pixels and it depends on the image resolution. Based on the image resolution, it will see h x w x d( h = Height, w = Width, d = Dimension ). Eg., An image of 6 x 6 x 3 array of matrix of RGB (3 refers to RGB values) and an image of 4 x 4 x 1 array of matrix of grayscale image.\n\nTechnically, deep learning CNN models to train and test, each input image will pass it through a series of convolution layers with filters (Kernals), Pooling, fully connected layers (FC) and apply Softmax function to classify an object with probabilistic values between 0 and 1. The below figure is a complete flow of CNN to process an input image and classifies the objects based on values."},{"metadata":{},"cell_type":"markdown","source":"so now lets normalization our data . We perform a grayscale normalization to reduce the effect of illumination's differences.\n\nMoreover the CNN converg faster on [0..1] data than on [0..255]."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Normalize the data\nX_train = X_train / 255.0\ntest = test / 255.0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train = X_train.values.reshape(-1,28,28,1)\ntest = test.values.reshape(-1,28,28,1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":" so now we Reshape image in 3 dimensions (height = 28px, width = 28px , canal = 1). now lets see how it looks"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure( figsize = (10,10))\nfor i in range(1,10):\n    plt.subplot(3,4,i)\n    plt.imshow(X_train[i+1].reshape([28,28]),cmap=\"gray\")\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Wow ... it look good ok now lets  Encode labels to one hot vectors (ex : 2 -> [0,0,1,0,0,0,0,0,0,0])"},{"metadata":{"trusted":true},"cell_type":"code","source":"Y_train = to_categorical(Y_train, num_classes = 10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"ok now lets Split training and valdiation set "},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nX_train, X_val, Y_train, Y_val = train_test_split(X_train, Y_train, test_size = 0.1, random_state=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"we now split the train set in two parts a small fraction (10%) became the validation set which the model is evaluated and the rest (90%) is used to train the model."},{"metadata":{},"cell_type":"markdown","source":"**Build CNN **\n\nnow we try to build Sequential Convolutional Neural Network \n\nthe first convolutional (Conv2D) layer ,this layer creates a convolution kernel that is wind with layers input which helps produce a tensor of outputs.genneraly It is like a set of learnable filters . CNN cab able to do feature maps i.e The CNN can isolate features that are useful everywhere from these transformed images.\nhere in side CNN there are some works going on like Convolution,Strided Convolution,padding,pooling \nconvolution It is a process where we take a small matrix of numbers (called kernel or filter), we pass it over our image and transform it based on the values from filter.and calculate by following formula \n\n![image.png](attachment:image.png)\n\nThe second important layer in CNN is the pooling (MaxPool2D) layer. Max pooling is a sample-based discretization process. The objective is to down-sample an input representation by reducing its dimensionality\n\nDropout is a regularization method,i.e to reduce overfitting and improve generalization error in deep neural networks of all kinds.\n\n'relu' is the rectifier (activation function max(0,x). The rectifier activation function is used to add non linearity to the network.\n\nThe Flatten layer is use to convert the final feature maps into a one single 1D vector. It combines all the found local features of the previous convolutional layers.\n","attachments":{"image.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAdYAAAAqCAYAAAAOPXqMAAANT0lEQVR4Ae1d7WnkPBB2Xo77HVJCSAVhSwjHFRBSwpIKjpQQUsGxJYRUEBaugZAKwpYQ0kFeHjPPnlbWl2VJ9uZmYbGtj/l4ZjSSxl7vSacfRUARWBwCp6ennx8fH71cp6eno+RjP1enx8fH7ubm5sRVh7K5+Prk0XJFQBFQBBQBRaAIAr9///7suq7/Xl1d4Tj68/b29gk619fXe1qXl5dBWnPxHa2cdlAEFAFFQBFQBMYiIBNqPyne398HJ8QY7ff3989fv371tF5eXoK05uIb00HrFYF/AoHz8/N+oHJlHVsNLwkUCVQH8mOFP5eM3ClI8BuIAdmw80CqDnjn7mIGhK0CyvH8/BzE4vHxMSrHsfrHUnwDkyHtDZvHJkTLlM5L2G29XgdtOxffVN+DYi18iz4uRyeeKBzjL6Fx3EInrxJaMUAgxf7JtofhEdwxSXJQ44hrEMGgowRwBJQjCONbYuCTdu0j9KDcXMnPNbEKX6bsBqoz0AFv2ABfCUKDtlMLjLRhlBSdSo6D9sfqH0vyDRlT/SJGAu8A57EF8HdzHLv6z8F3jO+l+haDY2wx4cJA+kSxSvWX2DhO1ckl65LKpmC+JD1S7J9ke04wmEixI8I1giYYyG60H+AclHCEWjunlgBzpTzXxEpsiautOwOOr95uP+UaNsU3lUZI9q/gH3P7BuzAcYldq/hCqnkmtWvNd4zvpfoWgjxiVCwD4wIKvi2bC1e1syzkL7FxnKqTk/GCCqdgviA1urH2d9peJsh+4PoCOJzTnEi/iiM4AWlkYe4MfLs+EaNJQBW7R1OFJjSUX4KwWdWn60x/Oag8kos5fcOEiAsYTK6yIzCrq5234jvW9xrFntHjLuIvQXqNdKrmK1+QcNBetr6m7f9DJYLidrvt1ut19/T0dHJ2duZ8HP/Hjx8n2+3WWWczwTVTAmCILwcpjgwOCMxYyWFliO+UFbmPH+hGJi6X+E3K7u/vez53d3cDXGkoNHh6esKhzxhIeXH54AP4XF5e9rajvUL4rVarE7TfbDaj5fHZq4Z/+HiFdButUMUOsA1/dnNzc9O1yq604pvjez64OUlzvOTEFO5wr66uDtjQj+Cjvg3IQYeu60qNY/IGPXw5PmuMF1uH2HUJzMHDp2PrcTrZ/gQEgsfAs+tjKyymkkAbbZFSZjrESGv1u2DUiQzZk6DJD7RsfrK7stXYO35KsAJeqd8BI0cB5PTt6iAP8cJAgtPhKzZzUDsscu0iD1scXuXiJ4uWwT32JflHrm4MinP4xqF1uk4Ge7+4kqBqN6ly3YLvWPuEfAvjA+OENHMW1S6fJj3EFR/QLn9JHcchncCP/FvFU5+OrvISmNs6jonhLpmmlOXY/8D2vHA5HxzC/prCxhyBKyrhse8qjtlPGuYkwbRizgoTxH38qKMtBwVifUrwhM5cCYeOKYFP+AUXEkzRU9YxR5E1uUsufrSb7UNL8o9c3ebyDZ/ROHbge6EA7+ufW16b71j7xHwLerqCY6r+9rgzF7ghGj5/sem5aMR08mFE20DGkvHUJWOsbArmoO3TkbjKMSbG5HrbXin2p4yI69/MFIwpDYLlxcWFWdSfw3C+VLHd+PX1tUMq5fb2dpDmRFukD120zs/PbVJJ1zF+Z2dnSXRCjfDmmvf391CTvg68VqtVsN1ut+vrQ/q+vLz0GNJOQYJd18EBQO/h4aFP0YIHDI5UciyNn4sf0sEI9Cm4mPLH+JX0jxivpfmGiZN5vtlsThCAYVfgg90kbtGYbWqc1+Zbwz4cM+Kfo2BBX8Qu7HxxxPh5fn7ucrEeO45dwsYwKjleXPxTyqZgDvoxHUuM01Q9ptj/my+4IzhjEuEH93Vwj8c1EbKNecTEjIkFwhFs1sPJXLQoS2xCIh3zGOIHY+EDWaZ+cgapjydfPeebWLHyweIG9zBtDH00cY8cq0ZZYfUTLfjAliGnLIEf7eeTzSwP8SvtHyFeS/UNEyv7HAGei16MyzGLXZvWmOtafGvZB36EsUMbp+rKcYcYhViEBePb2xswz1rAkN6YcWzLGsKo9HixeY+5zsUcPEI60oYlYnhMH9priv2/+YKtOYEyvQCl5CGamGz7icAFBEC6vr4e0OLk4eoTYxjq63O8GE1XPbFw1dllJoZ2Xcq1qdPDw0NKl74NHoTC5IoVLHDGQ2kxWUxeNqOS+JF2iF9p/wjxKqlbK99AgEcW4vb2tsOiCZNri08tvjXsw+CIWMKgnIoR5UGsw6IXGI+lYfIiPcgyZhz7aJjlOJ8yXjCZ3d3d2SSd15Df9ZAlG0/BHDRMnEiTx9g4LakH5Zhkf94QD+WukQ5Bqs9uE7onYOeoCRAfhLBpoR708GXbMUcfP6Gx38G5aIos/f1kV71ZJvJF77PKvQKz6+CcWAi+g3reOxkTsKELZISjoT+OkEXsPODBgin4CY0BD8ghdMlmf/TxIyYl/cPHi3Jzd78XzjiZyzcMEbynvO/j8x9vx4kVpfnm2CfkW1CPMSsHG1s/jnlMHDHoXP6SOo5DOvkwqjFeYjr66qdgDpo+HYVfMIb7ZMopz7W/afv/uNrF6hdB2CUIVgv4jEnRog9SH/aHKz+bFiYPpBJzdqvg4eMnjpdN15YfKVWkxGJf7BZjH6aAfSlUrJzQJrbbNPmg/W63O0HKGpjg+Pr6ehLDdQp+DDi+7IcpH899/Gr4h4/Xkn2DOIWO8Bv8XCv0bzWh/rl1pfnWsA93HTG/d2EAH8Q4Iq68JcZY6eoTKssZxzY9H0Y1xovNO/V6Cubg4dOx9DiN6VPM/pxpsSvFqgG7G6QSsdLiU1qyajuQybfCkkA72MGgs29VwpWXa4WJSZcyHQggFyF+fEqNwd/Vn/qH2rj6lSjz/dxGdqmTVmmgnSJjKfzshdkS/KOUbnP4Rsh2WFXLyjrUrHhdab659vH5FhVGvc//Q/GE4052mSS3f8I4lvmxYwnppdjKp1MIo5x4uleq8Eku5hAjpGNKDC+lCu2VY3/b9r1MUIwTqTjk/rendsCkEj5HIAOuNNhejs70oDisMx1Lej7nZL2LHxzPN8AoF/uLcVnc5Cg6HTwmD8ZcaIhsWbKIk0T7Uv9c/HwYL8E/purG/nP4hs9wGCsptxp8/XPLa/AlvmN9z+dbhm7ehQd5uuIJ05nSxiD3d1PgkpUNSZv+MmYc+3QiTQ/f0fGUslY4ZmEOOUI6+uJLBfn3txBy7E8daPts+XyOkE3Q05GrMt8E7+mWXFwMkGSOfxty4MVWwn97LOtMbOLMTnwF/5jTN1yWhjzANXXR5KKRUzYXX5+sId/imBLbDUjUjCdT/CWk00CJhRXMiflSoJhi+wMdGjpC1RV6MUAO0Em/YLq9dbBMl9DfkkHKJftX8I+5fcNEHsEL2ZfJK2KTaML5XHxDooV8y5cFMuhViydT/CWkkyH7Ik/nxHwpgEyx/YEOcARMCti14FtjwHMlJMcD/lMuMBFQ7lAaegqP1L7c9ckkldpt9nZ0JLkHMpDnWP1jSb5BUDG2MKmWGAfQz7UQIi/zOBdfUwbXuelbf/78+fz582f/XAgDvM8na8STUv5i6lQrnrqwzCmDzsAaOM+BeY7MNfqUsv2BbHAEPPDEb437Ppj0hM8B76kXvCFO2XGssTBIlZOTlH3TPLV/63YMUKHFwLH6x9J8A4MXWIqPTDZ1qo/NxTdFQdu3vn//3sehGE414kkpf7F1qhFPU7BNaYOJn/LOgXmKjC3alLJ9C1mVhyKgCBgIIMCWugePyVl2GAYH9+lcfN3SaKkioAgoAoqAIlAAAUyCqRNhiB2yMUzZpaST5+Ib0kHrFAFFQBFQBBSBSQgwbYmUbCohtMUkilQdfjoCGpJS7FOlsZ+egc9cfFN11HaKgCKgCCgCisBoBPh7SvMZgBLnsZTyXHxHA6QdFIEFI5D1bw0L1kdFUwS+BAJ4KGLs3/ClKI4XrodekTkX3xTZtY0ioAgoAoqAIqAIKAKKgCKgCCgCioAioAgoAoqAIqAIKAKKgCKgCCgCioAioAgoAoqAIqAIKAKKgCKgCCwUAfyURp7qrS4h3t4Ue8K4uhDKQBFQBBQBRUARqImAvHO6Jos9bfw2ttUkvmeqJ4qAIqAIKAKKQAsE+A9D8iKIFizBI/lF/q0EUj6KgCKgCCgCikAxBFqmZrEzNt/ghPcP40Xsc/6JRTEglZAioAgoAoqAIgAEWqZm8RIJ/qsRX5k45rWLajFF4KsjoG9e+uoWVv3+FQSwY+wuLi6qj2m8oH+1WnW73a7/brfb6jz/FSOqnoqAIqAIKAILQMBOzdYWCWlgSQXrfdbaYCv9o0Tgv6OUWoVWBBSBPQIvLy8ddpAtPkj5fnx8dJvNpru+vu7W63ULtspDEVAEFAFFQBFohwBSs7jv2YIj/ssVDyqBFybZlr+dbaGf8lAEFAFFQBFQBDpMbil/Xl4CKjysJH+Y3pMTvvpEcAlwlYYioAgoAorAfAjwKVy8pEHud84njHJWBBQBRUARUASOHQFJ/fZp2ZZvXDp23FR+RaAFAv8DUNpjDlo4MeIAAAAASUVORK5CYII="}}},{"metadata":{"trusted":true},"cell_type":"code","source":"model = keras.models.Sequential([\n    keras.layers.Conv2D(128, (3,3), input_shape=(28,28,1), activation='relu'),\n    keras.layers.MaxPooling2D(2,2),\n    keras.layers.BatchNormalization(),\n    keras.layers.Conv2D(256, (3,3), activation='relu'),\n    keras.layers.MaxPooling2D(2,2),\n    keras.layers.BatchNormalization(),\n    keras.layers.Conv2D(512, (3,3), activation='relu'),\n    keras.layers.MaxPooling2D(2,2),\n    keras.layers.BatchNormalization(),\n    keras.layers.Flatten(),\n    keras.layers.Dense(128),\n    keras.layers.BatchNormalization(),\n    keras.layers.Activation('relu'), \n    keras.layers.Dropout(0.1),\n    keras.layers.Dense(64),\n    keras.layers.BatchNormalization(),\n    keras.layers.Activation('relu'),\n    keras.layers.Dense(10, activation='softmax')\n])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"once our layers are added to the model, we need to set up  \na loss function and an optimisation algorithm.\n\nThe most important function is the optimizer. This function will iteratively improve parameters (filters kernel values, weights and bias of neurons in order to minimise the loss.\n\nhere we will use adam . The adam update adjusts the Adagrad method in a very simple way "},{"metadata":{"trusted":true},"cell_type":"code","source":"model.compile(optimizer = 'adam',loss = \"categorical_crossentropy\", metrics=[\"accuracy\"])\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train ,X_test_test,Y_train,Y_test= train_test_split(X_train,Y_train,test_size=0.2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.keras.callbacks import EarlyStopping,ModelCheckpoint\nes = EarlyStopping(monitor='accuracy', mode='min', verbose=1, patience=5,baseline=0.99)\n\nmodel.fit(X_train,Y_train, epochs=20, callbacks=[es], validation_data = (X_test_test,Y_test))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"its time to evaluate our model"},{"metadata":{"trusted":true},"cell_type":"code","source":"model.evaluate(X_test_test,Y_test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"we are with 98% accuracy goood . but it is always better to avoid overfitting problem. The idea is to alter the training data with small transformations to reproduce the variations occuring when someone is writing a digit.this we generally call as Data augmentation . lets do ..."},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndatagen = ImageDataGenerator(\n        featurewise_center=False,  # set input mean to 0 over the dataset\n        samplewise_center=False,  # set each sample mean to 0\n        featurewise_std_normalization=False,  # divide inputs by std of the dataset\n        samplewise_std_normalization=False,  # divide each input by its std\n        zca_whitening=False,  # apply ZCA whitening\n        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)\n        zoom_range = 0.1, # Randomly zoom image \n        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)\n        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)\n        horizontal_flip=False,  # randomly flip images\n        vertical_flip=False)  # randomly flip images\n\n\ndatagen.fit(X_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"now lets do the fitting process again i \nthink we should end up with good accracy this time "},{"metadata":{"trusted":true},"cell_type":"code","source":"checkpoint = ModelCheckpoint('best_weigths.h5', monitor='val_loss', verbose=0, save_best_only=True, mode='auto', period=1)\n\nhistory = model.fit_generator(datagen.flow(X_train,Y_train),\n                              epochs = 30, validation_data = (X_test_test,Y_test),\n                              verbose = 1, callbacks=[es,checkpoint])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_test_np = np.array(test)\n\nprint(X_test_np.shape)\n\nplt.figure( figsize = (10,10))\nfor i in range(1,20):\n    toBePredicted = X_test_np[i].reshape(-1,28,28,1)\n    result2 = model.predict(toBePredicted)\n    print(np.argmax(result2,axis = 1))\n    \n    plt.subplot(10,4,i)\n    plt.imshow(X_test_np[i].reshape([28,28]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"after Data Augmentation its making good prediction ....now our model is well trained  lets see how accracy vary with no.of epochs"},{"metadata":{"trusted":true},"cell_type":"code","source":"# predict results\nresults = model.predict(X_test_np)\n# select the indix with the maximum probability\nresults = np.argmax(results,axis = 1)\nresults = pd.Series(results,name=\"Label\")\nsubmission = pd.concat([pd.Series(range(1,28001),name = \"ImageId\"),results],axis = 1)\nprint(results)\nsubmission.to_csv(\"cnn_mnist_datagen.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"acc = history.history['acc']\nepochs_s = range(0,30)\n\nplt.plot(epochs_s, acc , label='accuracy')\nplt.xlabel('epochs')\nplt.ylabel('accuracy')\n\nplt.title('accuracy vs epochs')\nplt.legend()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**you just liked it place upvotes that would be very much appreciated - That will keep me motivated ..**"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}