{"cells":[{"metadata":{},"cell_type":"markdown","source":"This webpage is known as a Notebook as it includes documentation along with program code.\n\n**Note :** The documentation in the notebook depicts my knowledge or understanding, may not be accurate as I'm a learning. I wrote in such a way that, anybody could be able to understand intuitively the abstracted processes including some BTS. If you find any errors or suggestions, please comment at the end. Thanks!\n\nKeywords/Variable names are Italicized along with single quotes, e.g. *'train_data'*\n\nLet's get started!"},{"metadata":{},"cell_type":"markdown","source":"# 1. Let's First Plan\nThis program identifies digits from a dataset of tens of thousands of handwritten images, from 0 to 9 and outputs corresponding integer value.\nMNIST (\"Modified National Institute of Standards and Technology\") is the de facto “hello world” dataset of computer vision. Since its release in 1999, this classic dataset of handwritten images has served as the basis for benchmarking classification algorithms. As new machine learning techniques emerge, MNIST remains a reliable resource for researchers and learners alike.\n\nThe algorithm used in the program is Convolutional Neural Networks. The algorithm is built with a Sequential model which is a stack of layers that take an input and yield only one output. In the program, the model contains 3 Convolutional layers + 2 Dense layers, this forms a interconnected network of operations or nodes thus called as Convolutional Neural Network. \n\nCNN is considered as an efficient technique to classify/recognize images and is nowadays very popular. With Keras' high level and performace API's , it is easy to use and gives accurate results. "},{"metadata":{},"cell_type":"markdown","source":"# 2. Get Resources and Data\n1. Import Modules - Let's gather essential resources for programming."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# Input data files are available in the read-only \"../input/\" directory\n# Above are the instructions by Kaggle\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom tensorflow import keras\nfrom tensorflow.keras.utils import to_categorical # to One-Hot-Code values\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Activation, Dropout, Dense, Flatten\nfrom tensorflow.keras.layers import Conv2D, MaxPooling2D # create convolutional layer\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nimport matplotlib.pyplot as plt # plotting graphs for visual analysis\nfrom sklearn.model_selection import train_test_split #splitting dataset\nimport os","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"2. Load Image Data and Analyse - Let's find the path of image data files."},{"metadata":{"trusted":true},"cell_type":"code","source":"for dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"3. Get *'DataFrames'* (which is a data structure analogous to an Excel sheet or dictionary of dictionaries) from the corresponding files and store them."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data=pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\ntest_data=pd.read_csv('/kaggle/input/digit-recognizer/test.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"4. Let's preview the data for analysis. This codeline describes about the datat contained by *train_data*. Thus, we get metadata as no. of entries, datatype and format."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's preview a few rows (default 5)."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's confirm the number of rows(no. of images) and columns(no. of *pixels +/-label*) in the data."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.shape, test_data.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Preprocess the Image Data\n1. Store sufficient data to input into our algorithm.\nSince 'DataFrame' works similar to dictionary of dictionaries, analogous to Excel sheet, thus we need to delete/drop the column 'label' for the image dataset while extract the single column for the label dataset. However, the 'test.csv' has only image dataset thus we don't need to preprocess it."},{"metadata":{"trusted":true},"cell_type":"code","source":"label_train=train_data['label']\nimage_train=train_data.drop(labels=['label'], axis=1)\nimage_test=test_data","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"2. Normalize the Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"image_train=image_train/255.\nimage_test=image_test/255.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"3. Represent both Image Data into 2D Matrices\nWe need to convert 1D array of pixels values into (obviously) 2D array."},{"metadata":{"trusted":true},"cell_type":"code","source":"image_train=image_train.values.reshape(-1, 28, 28, 1)\nimage_test=image_test.values.reshape(-1, 28, 28, 1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"4. One-Hot-Ecnode the Labels\n* This is just another binary representation of a number that sets value '1' where it matches with index in an array of class_size/category_size/length=n.\n* i.e. if input is 'i' then set a[i]=1, for other values of i, set a[i]=0.\n(Only one 1 will be present thus said as one-hot).\n* We're considereing index values as our input values and when an index place holds '1' indicates that the corresponding value is present or represented."},{"metadata":{"trusted":true},"cell_type":"code","source":"label_train=to_categorical(label_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"label_train[0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. Create validation set of data\nLet's save some dataset for validation to improve performance of our trained model before we test it. This code splits the input dataset into train and validation datasets(0.2%)."},{"metadata":{"trusted":true},"cell_type":"code","source":"image_train, image_val, label_train, label_val = train_test_split(image_train, label_train,\n                                        test_size=0.1, random_state=144)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"image_train.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 5. Build Model\n**1.  Build a model with Convolutional Neural Network**"},{"metadata":{},"cell_type":"markdown","source":"* The algorithm used in the program is Convolutional Neural Networks. The algorithm is built with a **Sequential model** which is a stack of layers that take an input and yield only one output. And a model is just a systematic encapsulation of our algorithms in a format that will be later used. In the program, the model contains 3 Convolutional layers + 2 Dense layers.\n\n*  The convolutional layers are used for feature extraction that is extracting elements that will help to identify the image. Covolution is a process/function that relates two or more entities. Here, the machine will evaluate value of each pixel also considering neighbouring pixels by applying filter(multiplying matrix called as weights and adding values, called dot product). Then finally at the end of this layer maxpooling is applied which compresses the whole image. \n\n*  Next we need Two Dense layers, which perform dot product on input and weight matrices, apply activation function at the end and give results to next layer. Here, the 1st Dense layer forms patterns from the ouput of convolutinal layers, identify and the 2nd layer calculates predictions in 10 categories for 0 to 9 digits."},{"metadata":{"trusted":true},"cell_type":"code","source":"#1. Create 3 Convolutional layers\nmodel=Sequential()\n\nmodel.add(Conv2D(64, (3,3), \n                 activation='relu',\n                 input_shape=(28, 28, 1)))\nmodel.add(MaxPooling2D(pool_size=(2,2)))\n#model.add(Dropout(0.25))\n\nmodel.add(Conv2D(64, (3,3),\n                activation='relu',\n                input_shape=(28, 28, 1)))\nmodel.add(MaxPooling2D(pool_size=(2,2)))\n#model.add(Dropout(0.25))\n\nmodel.add(Conv2D(64, (3,3),\n                activation='relu',\n                input_shape=(28, 28, 1)))\nmodel.add(MaxPooling2D(pool_size=(2,2)))\n\nmodel.add(Dropout(0.25))\n\n#2. Create 2 Dense layers, one is hidden and other is for prediction(output)\n#Flatten(convert 2D to 1D array) the results for Dense Neural Network\nmodel.add(Flatten())\n#model.add(Dropout(0.25))\n\n#Add a dense layer to recognize and define pixel patterns\nmodel.add(Dense(256, activation='relu'))\n\n#Add another dense layer to identify and find probability of class 0-9\nmodel.add(Dense(10, activation='softmax'))\n\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"How does this model work?\n\n* When the program reads array of image pixels from the input file, it first converts that into a 2D matrix/tensor and then is used for training our machine. THis initial step is Data preprocessing. During training, machine starts execution of our constructed Sequential model. The image data first enters into the convolutional layers which are used for feature extraction that is extracting elements that characterize and will help identifying the image e.g. straight lines,  cruves, etc.\n* The output of this layer is then passed through the 1st Dense layer, to form patterns from the ouput of convolutinal layers and this output is passed through 2nd Dense layer to identify and calculate predictions for 0 to 9 digits. And then finally it gives probabilities of the 10 numbers to be the correct answer for the input image. The highest value indicates the highest probability."},{"metadata":{},"cell_type":"markdown","source":"**2. Compile model**\n\nTo Optimize and Error control by finding better nodes and get metrics in Convolutional layers and 1st Dense layer. This to specify the training **configuration** as (optimizer, loss, metrics)."},{"metadata":{"trusted":true},"cell_type":"code","source":"model.compile(loss='categorical_crossentropy',\n                optimizer='Adam',    #rmsprop\n                metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"epochs=5\nbatch_size=64","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**3. Data Augmentation to extend Dataset** \n\nThis is a technique to introduce our model to new dataset by applying some modifications (e.g. resize, rescale, rotation, object position, etc.) and generating from the existing dataset. This allows us to train the model even more broadly and diversely  with the available dataset.\n\nI'll use this later."},{"metadata":{"trusted":true},"cell_type":"code","source":"#datagen=ImageDataGenerator( rotation_range=5,\n#                            horizontal_flip=False,\n#                            vertical_flip=False)\n#datagen.fit(image_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 5. Model Training \nFit the Model and Get Performance. I'm working on this to imrpove accuracy.\n\nDuring training, machine maps its predictions with the provided correct answers and makes changes in its parameters(internal values/weight matrices) which are used for calculating output inside every layer. These parameters are machine's skills that are learned from the data used for training. During training machine makes changes in its values or algorithms by correcting mistakes and learning good skills. \n\nHere, More Data => More Learning => More Intelligent."},{"metadata":{"trusted":true},"cell_type":"code","source":"history=model.fit(image_train,label_train,\n                  batch_size=batch_size, epochs=epochs,\n                  validation_data = (image_val, label_val),\n                  verbose=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We don't need this, since we've evaluated the validation data above.\n\nWhile training, we gave validation data as input which will not be used for training but for testing the performance of the model we built. We manually need to monitor the performance with visual or numeral analysis and make chnages in the model for further imporvements.\nThen finally the model becomes ready to use!"},{"metadata":{"trusted":true},"cell_type":"code","source":"#valscore = model.evaluate(image_val,label_val)\n#print('ValTest Loss:', history.history['val_loss'])\n#print('ValTest accuracy:', history.history['val_accuracy'])\n#print(valscore)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 6. Plot the Performance\nPlot visually to understand better. Oops! the graph is not visible, i shall correct it shortly."},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(2,1, figsize=(15, 10))\nax[0].plot(history.history['loss'], color='b', label=\"Training loss\")\nax[0].plot(history.history['val_loss'], color='r', label=\"validation loss\")\nax[0].legend()\n\nax[1].plot(history.history['accuracy'], color='b', label=\"Training accuracy\")\nax[1].plot(history.history['val_accuracy'], color='r',label=\"Validation accuracy\")\nax[1].legend()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 7. Now, Let's test!\nWe are now testing our model on new data to Classify the New Images. This is the actual test of this project."},{"metadata":{"trusted":true},"cell_type":"code","source":"test_result = model.predict(image_test, verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"results = np.argmax(test_result,axis = 1)\nresults = pd.Series(results,name=\"Label\")\nimage_ids=pd.Series(range(1,28001),name = \"ImageId\")\nmySubmission = pd.concat([image_ids,results],axis = 1)\nmySubmission.to_csv(\"submission.csv\", index=False)\nmySubmission.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's see!"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax=plt.subplots(1, 5, figsize=(10,10))\nfor i in range(0,5):\n    ax[i].imshow(image_test[i], cmap='binary')\n    ax[i].set(title='ImageId %d'%(i+1))\n              ","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}