{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Handwritten digits classification using neural network\n\n# Introduction\n\nThe data files contain gray-scale images of hand-drawn digits, from zero through nine.\n\nEach image is 28 pixels in height and 28 pixels in width, for a total of 784 pixels in total. Each pixel has a single pixel-value associated with it, indicating the lightness or darkness of that pixel. This pixel-value is an integer between 0 and 255, inclusive.\n\n# Problem-Statement\nIn this notebook we will classify handwritten digits using a simple neural network which has only input and output layers. We will than add a hidden layer and see how the performance of the model improves","metadata":{}},{"cell_type":"markdown","source":"## ANN - NN","metadata":{}},{"cell_type":"code","source":"from IPython.display import Image\nImage(\"../input/annarchitecture/Architecture-ANN.jpg\")","metadata":{"execution":{"iopub.status.busy":"2022-08-18T03:25:48.178655Z","iopub.execute_input":"2022-08-18T03:25:48.179772Z","iopub.status.idle":"2022-08-18T03:25:48.192864Z","shell.execute_reply.started":"2022-08-18T03:25:48.179732Z","shell.execute_reply":"2022-08-18T03:25:48.191867Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Activation functions","metadata":{}},{"cell_type":"code","source":"from IPython.display import Image\nImage(\"../input/activationfunctions/Commonly-used-activation-functions.png\")","metadata":{"execution":{"iopub.status.busy":"2022-08-18T03:25:48.195035Z","iopub.execute_input":"2022-08-18T03:25:48.196041Z","iopub.status.idle":"2022-08-18T03:25:48.210394Z","shell.execute_reply.started":"2022-08-18T03:25:48.196005Z","shell.execute_reply":"2022-08-18T03:25:48.209541Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Loss/Cost functions","metadata":{}},{"cell_type":"markdown","source":"The purpose of loss functions is to compute the quantity that a model should seek to minimize during training.\n\n 1. binary_crossentropy function\n 2. categorical_crossentropy function\n 3. sparse_categorical_crossentropy function\n 4. mean_squared_error function\n 5. mean_absolute_error function\n 6. mean_absolute_percentage_error function\n 7. mean_squared_logarithmic_error function","metadata":{}},{"cell_type":"markdown","source":"## Optimizers\n\n1. SGD\n2. RMSprop\n3. Adam\n","metadata":{}},{"cell_type":"markdown","source":"# Sigmoid function - Derivative","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport numpy as np\n\ndef sigmoid(x):\n    s=1/(1+np.exp(-x))\n    ds=s*(1-s)  \n    return s,ds\nx=np.arange(-6,6,0.1)\n#print(x)\n#print(len(x))\n#sigmoid(x)\n# Create and show plot\nfig, ax = plt.subplots(figsize=(6, 4))\nplt.grid()\nax.plot(x,sigmoid(x)[0], color=\"blue\", linewidth=3, label=\"sigmoid\")\nax.plot(x,sigmoid(x)[1], color=\"red\", linewidth=3, label=\"derivative\")\nax.legend(loc=\"upper left\")\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-18T03:25:48.212814Z","iopub.execute_input":"2022-08-18T03:25:48.213277Z","iopub.status.idle":"2022-08-18T03:25:48.414412Z","shell.execute_reply.started":"2022-08-18T03:25:48.213232Z","shell.execute_reply":"2022-08-18T03:25:48.413360Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Observations\n1. Derivate values are in between 0 and 0.3\n2. Derivate values become non zero , if sigmoid values are in range -4 to 4 , otherwise it is zero","metadata":{}},{"cell_type":"markdown","source":"# tanh or Hyperbolic function - Derivative","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport numpy as np\n\ndef tanh(x):\n    t=(np.exp(x)-np.exp(-x))/(np.exp(x)+np.exp(-x))\n    dt=1-t**2\n    return t,dt\n\nz=np.arange(-4,4,0.01)\n# Create and show plot\nfig, ax = plt.subplots(figsize=(6, 4))\nplt.grid()\nax.plot(z,tanh(z)[0], color=\"blue\", linewidth=3, label=\"tanh\")\nax.plot(z,tanh(z)[1], color=\"red\", linewidth=3, label=\"derivative\")\nax.legend(loc=\"upper left\")\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-18T03:25:48.417531Z","iopub.execute_input":"2022-08-18T03:25:48.417810Z","iopub.status.idle":"2022-08-18T03:25:48.639720Z","shell.execute_reply.started":"2022-08-18T03:25:48.417782Z","shell.execute_reply":"2022-08-18T03:25:48.638804Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Observations\n1. Derivate values are in between 0 and 1\n2. Derivate values become non zero , if tanh values are in range -3 to 3 , otherwise it is zero","metadata":{}},{"cell_type":"markdown","source":"# Vanishing gradient\nAs more layers using certain activation functions are added to neural networks, the gradients of the loss function approaches zero, making the network hard to train.","metadata":{}},{"cell_type":"markdown","source":"# 1.Import Python modules","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\n\nimport tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Dense\nfrom keras.preprocessing.image import load_img\nfrom keras.preprocessing.image import img_to_array","metadata":{"execution":{"iopub.status.busy":"2022-08-18T03:25:48.643593Z","iopub.execute_input":"2022-08-18T03:25:48.643867Z","iopub.status.idle":"2022-08-18T03:25:48.652002Z","shell.execute_reply.started":"2022-08-18T03:25:48.643839Z","shell.execute_reply":"2022-08-18T03:25:48.651090Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. Load Dataset - MNIST","metadata":{}},{"cell_type":"code","source":"(X_train, Y_train) , (X_test, Y_test) = keras.datasets.mnist.load_data()\nprint(\"train data length:\",len(X_train))\nprint(\"test data length:\",len(X_test))","metadata":{"execution":{"iopub.status.busy":"2022-08-18T03:25:48.655408Z","iopub.execute_input":"2022-08-18T03:25:48.657658Z","iopub.status.idle":"2022-08-18T03:25:48.921577Z","shell.execute_reply.started":"2022-08-18T03:25:48.657620Z","shell.execute_reply":"2022-08-18T03:25:48.920504Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. Scaling","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.utils import to_categorical\nY_train_saved = Y_train\nY_test_saved = Y_test\nY_train = to_categorical(Y_train, 10)\nY_test = to_categorical(Y_test, 10)\nprint(Y_train.shape)\nprint(Y_test.shape)\n# Normalize the features\nX_train = X_train / 255\nX_test = X_test / 255\n\n# reashaping the 784 pixels into 28 x 28 matrix which represents the pixels of the actual image\n# as input we will have an array of these matrices\nX_train_scaled = X_train.reshape(X_train.shape[0],28*28)\nX_test_scaled = X_test.reshape(X_test.shape[0],28*28)","metadata":{"execution":{"iopub.status.busy":"2022-08-18T03:25:48.922915Z","iopub.execute_input":"2022-08-18T03:25:48.923758Z","iopub.status.idle":"2022-08-18T03:25:49.074491Z","shell.execute_reply.started":"2022-08-18T03:25:48.923718Z","shell.execute_reply":"2022-08-18T03:25:49.073488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. Model","metadata":{}},{"cell_type":"markdown","source":"## 4.1 Using hidden layer - Using Dropout on Hidden Layers\nUsing hidden layer - Adam,softmax","metadata":{}},{"cell_type":"code","source":"model_1 = keras.Sequential([\n    keras.layers.Dense(100,input_shape=(784,),activation='relu'),\n    keras.layers.Dropout(0.2),\n    keras.layers.Dense(50, activation='relu'),\n    keras.layers.Dropout(0.2),\n    keras.layers.Dense(30, activation='relu'),\n    keras.layers.Dropout(0.2),\n    keras.layers.Dense(10, activation='softmax')\n])\n\nmodel_1.compile(optimizer='adam',\n              loss='categorical_crossentropy',\n              metrics=['accuracy'])\nmodel_1.summary()\n#param_number = output_channel_number * (input_channel_number + 1)\n\nhistory=model_1.fit(X_train_scaled, Y_train, validation_data=(X_test_scaled, Y_test),epochs=30)","metadata":{"execution":{"iopub.status.busy":"2022-08-18T03:25:49.075894Z","iopub.execute_input":"2022-08-18T03:25:49.077141Z","iopub.status.idle":"2022-08-18T03:27:11.850263Z","shell.execute_reply.started":"2022-08-18T03:25:49.077102Z","shell.execute_reply":"2022-08-18T03:27:11.849267Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_1.evaluate(X_test_scaled, Y_test)","metadata":{"execution":{"iopub.status.busy":"2022-08-18T03:27:11.852080Z","iopub.execute_input":"2022-08-18T03:27:11.852494Z","iopub.status.idle":"2022-08-18T03:27:12.572750Z","shell.execute_reply.started":"2022-08-18T03:27:11.852449Z","shell.execute_reply":"2022-08-18T03:27:12.571838Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.grid()\nplt.plot(history.history['accuracy'], label='train_accuracy')\nplt.plot(history.history['loss'], label = 'train_loss')\nplt.plot(history.history['val_accuracy'], label='val_accuracy')\nplt.plot(history.history['val_loss'], label = 'val_loss')\nplt.xlabel('Epoch')\nplt.ylabel('Accuracy')\nplt.ylim([-0.5, 1.2])\nplt.legend(loc='lower right')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-18T03:27:12.574095Z","iopub.execute_input":"2022-08-18T03:27:12.574572Z","iopub.status.idle":"2022-08-18T03:27:12.795413Z","shell.execute_reply.started":"2022-08-18T03:27:12.574533Z","shell.execute_reply":"2022-08-18T03:27:12.794464Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4.2 Using hidden layer - Using Dropout on Hidden Layers\nUsing hidden layer - SGD,Softmax ","metadata":{}},{"cell_type":"code","source":"model_2 = keras.Sequential([\n    keras.layers.Dense(100,input_shape=(784,),activation='relu'),\n    keras.layers.Dropout(0.2),\n    keras.layers.Dense(50, activation='relu'),\n    keras.layers.Dropout(0.2),\n    keras.layers.Dense(30, activation='relu'),\n    keras.layers.Dropout(0.2),\n    keras.layers.Dense(10, activation='softmax')\n])\n\nmodel_2.compile(optimizer='SGD',\n              loss='categorical_crossentropy',\n              metrics=['accuracy'])\nmodel_2.summary()\n#param_number = output_channel_number * (input_channel_number + 1)\n\nhistory=model_2.fit(X_train_scaled, Y_train, validation_data=(X_test_scaled, Y_test),epochs=30)","metadata":{"execution":{"iopub.status.busy":"2022-08-18T03:27:12.799122Z","iopub.execute_input":"2022-08-18T03:27:12.800057Z","iopub.status.idle":"2022-08-18T03:28:35.895199Z","shell.execute_reply.started":"2022-08-18T03:27:12.800018Z","shell.execute_reply":"2022-08-18T03:28:35.893955Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_2.evaluate(X_test_scaled, Y_test)","metadata":{"execution":{"iopub.status.busy":"2022-08-18T03:28:35.896997Z","iopub.execute_input":"2022-08-18T03:28:35.897665Z","iopub.status.idle":"2022-08-18T03:28:36.585866Z","shell.execute_reply.started":"2022-08-18T03:28:35.897620Z","shell.execute_reply":"2022-08-18T03:28:36.584965Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.grid()\nplt.plot(history.history['accuracy'], label='train_accuracy')\nplt.plot(history.history['loss'], label = 'train_loss')\nplt.plot(history.history['val_accuracy'], label='val_accuracy')\nplt.plot(history.history['val_loss'], label = 'val_loss')\nplt.xlabel('Epoch')\nplt.ylabel('Accuracy')\nplt.ylim([-0.5, 1.2])\nplt.legend(loc='lower right')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-18T03:28:36.587332Z","iopub.execute_input":"2022-08-18T03:28:36.587703Z","iopub.status.idle":"2022-08-18T03:28:36.811188Z","shell.execute_reply.started":"2022-08-18T03:28:36.587668Z","shell.execute_reply":"2022-08-18T03:28:36.810225Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5.Performance Matrix","metadata":{}},{"cell_type":"code","source":"y_predicted = model_1.predict(X_test_scaled)\ny_predicted_labels = [np.argmax(i) for i in y_predicted]\ncm = tf.math.confusion_matrix(labels=Y_test_saved,predictions=y_predicted_labels)\ncm","metadata":{"execution":{"iopub.status.busy":"2022-08-18T03:28:36.813075Z","iopub.execute_input":"2022-08-18T03:28:36.813475Z","iopub.status.idle":"2022-08-18T03:28:37.272687Z","shell.execute_reply.started":"2022-08-18T03:28:36.813412Z","shell.execute_reply":"2022-08-18T03:28:37.271494Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import seaborn as sn\nplt.figure(figsize = (10,7))\nsn.heatmap(cm, annot=True, fmt='d')\nplt.xlabel('Predicted')\nplt.ylabel('Truth')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-18T03:28:37.274335Z","iopub.execute_input":"2022-08-18T03:28:37.274712Z","iopub.status.idle":"2022-08-18T03:28:37.806117Z","shell.execute_reply.started":"2022-08-18T03:28:37.274675Z","shell.execute_reply":"2022-08-18T03:28:37.805036Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 6. Predict\n1. Take one row of new set of values/image[not from exisitng] and Predict\n2. Show as new feature prediction.","metadata":{}},{"cell_type":"code","source":"def load_image(filename):\n    # load the image\n    img = load_img(filename, grayscale=True, target_size=(28, 28))\n    # convert to array\n    img = img_to_array(img)\n    # reshape into a single sample with 1 channel\n    img = img.reshape(1, 28*28)\n    # prepare pixel data\n    img = img.astype('float32')\n    img = img / 255.0\n    return img","metadata":{"execution":{"iopub.status.busy":"2022-08-18T03:28:37.807746Z","iopub.execute_input":"2022-08-18T03:28:37.808355Z","iopub.status.idle":"2022-08-18T03:28:37.814965Z","shell.execute_reply.started":"2022-08-18T03:28:37.808317Z","shell.execute_reply":"2022-08-18T03:28:37.814004Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.image as mpimg\nimg = mpimg.imread(\"../input/imghbbuu/sample_image.png\")\nimgplot = plt.imshow(img)\nplt.show()\n\nimg = load_image('../input/imghbbuu/sample_image.png')\npredict_value = model_1.predict(img)\ndigit = np.argmax(predict_value)\nprint(\"predict_value/digit:\",digit) ","metadata":{"execution":{"iopub.status.busy":"2022-08-18T03:28:37.816842Z","iopub.execute_input":"2022-08-18T03:28:37.817308Z","iopub.status.idle":"2022-08-18T03:28:38.397335Z","shell.execute_reply.started":"2022-08-18T03:28:37.817273Z","shell.execute_reply":"2022-08-18T03:28:38.395724Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 7. Summary\n1. Very simple neural network with hidden layer(100-50-30-10) with dropout Regularization - accuray=97%(test=96% and train=97%),Trainable params: 85,390","metadata":{}},{"cell_type":"markdown","source":"# 8. Test & Submit","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nnew_test=pd.read_csv(\"../input/digit-recognizer/test.csv\")\nprint(new_test.shape)\nnew_test = new_test / 255\ny_predicted = model_2.predict(new_test)\ny_predicted_labels = [np.argmax(i) for i in y_predicted]\n\nresult=pd.DataFrame()\nresult[\"ImageId\"]=[i for i in range(1,28001)]\nresult[\"Label\"]=y_predicted_labels\nresult.to_csv('./submission.csv',index=False)","metadata":{"execution":{"iopub.status.busy":"2022-08-18T03:28:38.398849Z","iopub.execute_input":"2022-08-18T03:28:38.399198Z","iopub.status.idle":"2022-08-18T03:28:41.948467Z","shell.execute_reply.started":"2022-08-18T03:28:38.399162Z","shell.execute_reply":"2022-08-18T03:28:41.947353Z"},"trusted":true},"execution_count":null,"outputs":[]}]}