{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Digit Recognizer In Computer Vision\n\n\n*    ** 1. Data Preprocessing / Feature Engineering**\n    *             a. Load Dataset\n    *             b. Descriptive Statistics\n    *             c. Checking for null and missing values\n    *             d. outliers\n    *             e. Feature Split \n    *             f. Feature scale (Normalisation)\n    *             g. Reshape\n    *             h. Label Encoding/One Hot Encoder\n    *             i. Evaluation Resample Methods  \n*    ** 2. CNN Algorithm**\n    *             a. Build Model\n    *             b. Building CNN Layers\n    *             c. Data Augmentation\n    *             d. Evaluate Model Confusion Matrix\n*    ** 3. Evaluate Model**\n    *             a. Evaluate Model With Test Set\n*   ** 4. Submission**\n    *             a. Submit Result As .csv File\n    "},{"metadata":{},"cell_type":"markdown","source":"# 1. Data Preprocessing / Feature Engineering\n\n# **a. Load Dataset**"},{"metadata":{"_cell_guid":"","_uuid":"","trusted":true},"cell_type":"code","source":"# Import the datasets\nimport pandas as pd\nsample_submission = pd.read_csv(\"../input/digit-recognizer/sample_submission.csv\")\ntest_set = pd.read_csv(\"../input/digit-recognizer/test.csv\")\ntraining_set = pd.read_csv(\"../input/digit-recognizer/train.csv\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **b. Descriptive Statistics**"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Displaying the shape and datatype of each attribute\n\nprint(training_set.shape)\ntraining_set.dtypes","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We got almost 785 columns and 42000 rows in our dataset.\n\nAll columns having integer as a datatype."},{"metadata":{},"cell_type":"markdown","source":"# Descriptive Visualisation\n\nHistogram visualisation for output attribute to know what kind of distribution it is."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Histogram Visualisation for Output attribute\n\nimport seaborn as sb\nsb.distplot(training_set['label'])\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As per above Histogram plot we get to know output attribute is discrete outcome and it is not normal distribution aswell."},{"metadata":{},"cell_type":"markdown","source":"# **c. Checking for null and missing values**"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Displaying Null values info in each column\n\ntraining_set.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Displaying the sum count of null or empty values in each count..Due too many columns unable to view\ntraining_set.isna().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Now we going to display only those column which have null value and remaining columns wont display\ntraining_set.isnull().any().describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_set.isnull().any().describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There is no missing or empty values in both training and test set, So now we can safely go ahead."},{"metadata":{},"cell_type":"markdown","source":"# **d. outliers**\n\nAs we know we dont get any missing value and we converted image pixel values into array if image so most probabliy it wont have any outliers in each attribute.\n\nIn each row it will contains 0's only some where in that row it have high value due to high pattern detection happen there.."},{"metadata":{},"cell_type":"markdown","source":"# e. Feature Split\n\nSplit the input and output attributes."},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train=training_set['label'].values\nx_train=training_set.drop(['label'],axis=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# e. Feature scale (Normalisation)\n\nWe are perform a grayscale normalization to reduce the effect of illumination's differences.\n\nMoreover CNN coverage faster if values lies in between 0 to 1.So by using Normalisation scale we are rescaling all values from [0...255] to [0..1]"},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train=x_train/255.0\ntest_set=test_set/255.0","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# f. Reshape\n\nReshape image in 3 dimensions\n\nTrain and test images (28px x 28px) has been stock into pandas.Dataframe as 1D vectors of 784 values. We reshape all data to 28x28x1 3D matrices.\n\nKeras requires an extra dimension in the end which correspond to channels. MNIST images are gray scaled so it use only one channel. For RGB images, there is 3 channels, we would have reshaped 784px vectors to 28x28x3 3D matrices\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train=x_train.values.reshape(-1,28,28,1)\nx_test=test_set.values.reshape(-1,28,28,1)\ndel test_set\ndel training_set","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# h. Label Encoding/One Hot Encoder\n\nLabels are 10 digits numbers from 0 to 9. We need to encode these lables to one hot vectors"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Encoding numerics in one hot encoder vector\n'''from sklearn.preprocessing import OneHotEncoder\nonehotencoder=OneHotEncoder(Categorical_features=[0])\ny_train=onehotencoder.fit_transform(y_train)\n\nTypeError: __init__() got an unexpected keyword argument 'Categorical_features' some api issue there\n'''\nfrom keras.utils.np_utils import to_categorical # convert to one-hot-encoding\n\ny_train=to_categorical(y_train,num_classes=10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# i. Evaluation Resample Methods\n\nNow we are going to split again x_train and y_train into training and test set to evaluate performance with training set data.\n\nLater we will predict label values using test set..."},{"metadata":{"trusted":true},"cell_type":"code","source":"# split the training set into train and test set\n\nseed=5\ntrain_size=0.80\ntest_size=0.20\n\nfrom sklearn.model_selection import train_test_split\nx1_train,x1_test,y1_train,y1_test=train_test_split(x_train,y_train,train_size=train_size,test_size=test_size,random_state=seed)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2. CNN Algorithm\n\n# a. Build Model\n\nI used the Keras Sequential API, where you have just to add one layer at a time, starting from the input."},{"metadata":{},"cell_type":"markdown","source":"# b. Building CNN Layers"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Building CNN layers\n\n# intialisaing the sequence of layers\nfrom keras.models import Sequential\ncnn=Sequential()\n\n# Building fist Convolutional Layer\nfrom keras.layers import Convolution2D\nfrom keras.layers import Dropout\ncnn.add(Convolution2D(input_shape=(28,28,1),activation='relu',filters=32,kernel_size=(5,5)))\ncnn.add(Dropout(0.2))\n\n# Building first pooling Layer\nfrom keras.layers import MaxPooling2D\ncnn. add(MaxPooling2D(pool_size=(2,2)))\ncnn.add(Dropout(0.2))\n\n# Building Second Colvolution and pooling layers\ncnn.add(Convolution2D(kernel_size=(5,5),filters=32,activation='relu'))\ncnn.add(Dropout(0.2))\ncnn.add(MaxPooling2D(pool_size=(2,2)))\ncnn.add(Dropout(0.2))\n\n# Building Third Convolution and pooling layer\ncnn.add(Convolution2D(kernel_size=(3,3),filters=32,activation='relu'))\ncnn.add(Dropout(0.2))\ncnn.add(MaxPooling2D(pool_size=(2,2)))\ncnn.add(Dropout(0.2))\n\n# Building Flatten layer\nfrom keras.layers import Flatten\ncnn.add(Flatten())\n\n# Building Fully Connected Layers\nfrom keras.layers import Dense\n# First fully connected hidden layer\ncnn.add(Dense(256,activation='relu'))\ncnn.add(Dropout(0.2))\n# Second fully connected hidden layer\ncnn.add(Dense(256,activation='relu'))\ncnn.add(Dropout(0.2))\n# Third Fully connected hidden layer\ncnn.add(Dense(128,activation='relu'))\ncnn.add(Dropout(0.2))\n\n# Output layer with 10 neurons\ncnn.add(Dense(10,activation='softmax'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Try changing this line:\n\nmodel.add(Dense(output_dim=NUM_CLASSES, activation='softmax'))\n\nto\n\nmodel.add(Dense(NUM_CLASSES, activation='softmax'))\n\nI could not find a parameter called output_dim on the documentation page for Dense. I think you meant to provide units but labelled it as output_dim"},{"metadata":{"trusted":true},"cell_type":"code","source":"# compile the CNN Model\ncnn.compile(optimizer='adam',loss='categorical_crossentropy',metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# c. Without Data Augmentation"},{"metadata":{"trusted":true},"cell_type":"code","source":"result=cnn.fit(x1_train, y1_train, batch_size = 32, epochs = 10,validation_data = (x1_test,y1_test), verbose = 2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For 10 epochs without data augmentation we got...\n\ntraining accuracy: 97.54% and loss:0.0892\ntest accuracy: 98.79% and loss:0.0525\n\n\nNow we are excepting more accuracy in both training and test set by using data augmentation"},{"metadata":{},"cell_type":"markdown","source":"# With Data Augmentation"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create Data Augmentation Generator\nfrom keras.preprocessing.image import ImageDataGenerator\ndatagen=ImageDataGenerator(width_shift_range=0.1,height_shift_range=0.1,# randomly shift images\n                          horizontal_flip=False,vertical_flip=False,# randomly flip images\n                          rotation_range=10,# randomly rotate images in the range (degrees, 0 to 180)\n                          #brightness_range=[0.1,1.0],# randomly brightning images\n                          zoom_range=0.1,# Randomly zoom image\n                          zca_whitening=False,# apply ZCA whitening\n                          featurewise_center=False,  # set input mean to 0 over the dataset\n                          samplewise_center=False,  # set each sample mean to 0\n                          featurewise_std_normalization=False,  # divide inputs by std of the dataset\n                          samplewise_std_normalization=False,  # divide each input by its std\n                          )\n\ndatagen.fit(x1_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_size=86\n# makeing iteration flow\nsample=datagen.flow(x1_train,y1_train,batch_size=batch_size)\n# fit and generate the outcome\ntrain_predictions=cnn.fit_generator(sample,epochs=10,validation_data=(x1_test,y1_test))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"After Applying image data augmentation to our dataset we got good accuracy\n\ntraining set : 97.57% and loss:0.0848\ntest set: 99.13% and loss:0.0392"},{"metadata":{},"cell_type":"markdown","source":"# d. Evaluate Model Confusion Matrix"},{"metadata":{"trusted":true},"cell_type":"code","source":"# predicting the training set test accuracy\nimport numpy as np\ny_trainpred=cnn.predict(x1_test)\n# Convert predictions classes to one hot vectors \ny_pred_one=np.argmax(y_trainpred,axis=1)\n# Convert validation observations to one hot vectors\ny1_test_one=np.argmax(y1_test,axis=1)\nfrom sklearn.metrics import confusion_matrix\naccuracy=confusion_matrix(y1_test_one,y_pred_one)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(accuracy)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Evaluate Model\n\n# a. Evaluate Model With Test Set"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Predict the result for test set\ny_pred=cnn.predict(x_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# argmax() is used to decode the onehotencoder value to numerical value\nresult=np.argmax(y_pred,axis=1)\n# storing those value as a column name label\nresult=pd.Series(result,name='label')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. Submission\n\n# a. Submit Result As .csv File"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Submission \nsubmission=pd.concat([pd.Series(range(1,28001),name = \"ImageId\"),result],axis = 1)\nsubmission.to_csv(\"My_Submission.csv\",index=False)\nprint(\"Submission Successfully\")","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}