{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### Dataset Information\n\nThis MNIST dataset consists of 70000 images of handwritten digits where each digit of size 28 x 28 is flattened into a vector of size 784. These 70000 images are split into training set and testing set where the training set consists of 42000 images and the test set consists of 28000 images.\n\n### Problem Statement\nCorrectly classify the digits from the dataset\n\n### Overview\nIn this kernel, I first use 2D-DCT to reduce the dimension of the dataset. Two approaches are used to classify the dimension reduced datapoints - PCA and ANN. The classification is relatively faster as compared to the CNN approaches as now we are dealing with a vector for each data sample instead of an image.\n\n### References\nD. Ismailova and W. Lu, \"Fast classification of handwritten digits using 2D-DCT based sparse PCA,\" 2015 IEEE Pacific Rim Conference on Communications, Computers and Signal Processing (PACRIM), 2015, pp. 131-135, doi: 10.1109/PACRIM.2015.7334822.\n\nPlease upvote if you like my work. Thank you!","metadata":{}},{"cell_type":"markdown","source":"# **1. Importing Libraries**","metadata":{}},{"cell_type":"code","source":"import os\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport cv2\nfrom sklearn.preprocessing import LabelEncoder, OneHotEncoder\nimport sklearn\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.decomposition import PCA\nfrom scipy.sparse.linalg.eigen.arpack import eigsh as largest_eigsh\nfrom sklearn.metrics import confusion_matrix,classification_report,accuracy_score\nfrom tensorflow.keras.layers import Dense, Input, Conv1D,MaxPooling1D, GlobalMaxPooling1D, GaussianNoise, BatchNormalization,Flatten,Dropout\nfrom tensorflow.keras.models import Model,Sequential\nfrom tensorflow.keras.layers.experimental.preprocessing import Normalization\nfrom tensorflow.keras.callbacks import EarlyStopping\nfrom keras.utils.np_utils import to_categorical\nfrom tensorflow.keras.optimizers import RMSprop, Adam\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-07-25T19:32:16.49326Z","iopub.execute_input":"2021-07-25T19:32:16.493699Z","iopub.status.idle":"2021-07-25T19:32:23.445604Z","shell.execute_reply.started":"2021-07-25T19:32:16.493601Z","shell.execute_reply":"2021-07-25T19:32:23.444591Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **2. Data Handling and Exploration**","metadata":{"execution":{"iopub.status.busy":"2021-07-24T01:16:31.845496Z","iopub.execute_input":"2021-07-24T01:16:31.84587Z","iopub.status.idle":"2021-07-24T01:16:31.850091Z","shell.execute_reply.started":"2021-07-24T01:16:31.845837Z","shell.execute_reply":"2021-07-24T01:16:31.849137Z"}}},{"cell_type":"code","source":"# load data set\ntrain_data = pd.read_csv('../input/digit-recognizer/train.csv')\ntest_data = pd.read_csv('../input/digit-recognizer/test.csv')","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:32:23.44713Z","iopub.execute_input":"2021-07-25T19:32:23.447412Z","iopub.status.idle":"2021-07-25T19:32:29.363643Z","shell.execute_reply.started":"2021-07-25T19:32:23.447384Z","shell.execute_reply":"2021-07-25T19:32:29.362761Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:32:29.367781Z","iopub.execute_input":"2021-07-25T19:32:29.368416Z","iopub.status.idle":"2021-07-25T19:32:29.405007Z","shell.execute_reply.started":"2021-07-25T19:32:29.368367Z","shell.execute_reply":"2021-07-25T19:32:29.404133Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_data.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:32:29.406402Z","iopub.execute_input":"2021-07-25T19:32:29.406715Z","iopub.status.idle":"2021-07-25T19:32:29.425579Z","shell.execute_reply.started":"2021-07-25T19:32:29.406686Z","shell.execute_reply":"2021-07-25T19:32:29.424678Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For training set, the first column is the label. So we need to split the labels from the actual image data to perform classification. ","metadata":{}},{"cell_type":"code","source":"#Splitting labels from actual data\nXtrain = train_data.iloc[:,1:]\nytrain = train_data.iloc[:,0]\nXtest = test_data","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:32:29.42681Z","iopub.execute_input":"2021-07-25T19:32:29.427128Z","iopub.status.idle":"2021-07-25T19:32:29.438491Z","shell.execute_reply.started":"2021-07-25T19:32:29.427092Z","shell.execute_reply":"2021-07-25T19:32:29.437606Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Overview of the dataset\nnum_pixels = len(Xtrain.loc[0])\nimg_dim = np.sqrt(num_pixels).astype(int)\nnum_classes = len(set(ytrain))\nnum_samples = len(Xtrain)\nnum_samples_te = len(Xtest)\nprint(10*\"-\",\"Overview of the Dataset\",10*\"-\")\nprint(\"Size of each image: {} x {}\".format(img_dim,img_dim))\nprint(\"Number of pixels in each image:\",num_pixels)\nprint(\"Number of classes:\",num_classes)\nprint(\"Number of images in the training set:\",num_samples)\nprint(\"Number of images in the testing set:\",num_samples_te)","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:32:29.439941Z","iopub.execute_input":"2021-07-25T19:32:29.44058Z","iopub.status.idle":"2021-07-25T19:32:29.460504Z","shell.execute_reply.started":"2021-07-25T19:32:29.440549Z","shell.execute_reply":"2021-07-25T19:32:29.459431Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#samples per class\nsample_per_class = np.unique(ytrain,return_counts = True)\nsns.barplot( x = sample_per_class[0], y = sample_per_class[1])\nplt.title('Samples per class')\nplt.xlabel('Label')\nplt.ylabel('Count')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:32:29.462039Z","iopub.execute_input":"2021-07-25T19:32:29.46245Z","iopub.status.idle":"2021-07-25T19:32:29.841651Z","shell.execute_reply.started":"2021-07-25T19:32:29.462419Z","shell.execute_reply":"2021-07-25T19:32:29.840731Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for m,n in zip(sample_per_class[0],sample_per_class[1]):\n    print(\"Number of samples for {} : {} \".format(m,n))","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:32:29.844193Z","iopub.execute_input":"2021-07-25T19:32:29.844507Z","iopub.status.idle":"2021-07-25T19:32:29.851406Z","shell.execute_reply.started":"2021-07-25T19:32:29.844476Z","shell.execute_reply":"2021-07-25T19:32:29.850415Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The dataset appears to be fairly balanced","metadata":{}},{"cell_type":"code","source":"# Normalization\nXtrain = Xtrain / 255.0\nXtest = Xtest / 255.0","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:32:29.853212Z","iopub.execute_input":"2021-07-25T19:32:29.853527Z","iopub.status.idle":"2021-07-25T19:32:29.990799Z","shell.execute_reply.started":"2021-07-25T19:32:29.85349Z","shell.execute_reply":"2021-07-25T19:32:29.990046Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Normalization is performed to reduce illumination differences","metadata":{}},{"cell_type":"code","source":"#Splitting the training data into training and validation set\nXtrain,Xval,ytrain,yval = train_test_split(Xtrain,ytrain,test_size = 0.2,random_state = 2)","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:32:29.991949Z","iopub.execute_input":"2021-07-25T19:32:29.992344Z","iopub.status.idle":"2021-07-25T19:32:30.276901Z","shell.execute_reply.started":"2021-07-25T19:32:29.992307Z","shell.execute_reply":"2021-07-25T19:32:30.27597Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Reshaping Xtrain and Xtest\nXtrain  = Xtrain.to_numpy().reshape(-1,28,28)\nXval = Xval.to_numpy().reshape(-1,28,28)\nytrain  = ytrain.to_numpy()\nXtest = Xtest.to_numpy().reshape(-1,28,28)","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:32:30.27834Z","iopub.execute_input":"2021-07-25T19:32:30.278743Z","iopub.status.idle":"2021-07-25T19:32:30.284887Z","shell.execute_reply.started":"2021-07-25T19:32:30.278703Z","shell.execute_reply":"2021-07-25T19:32:30.283803Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Plotting images for each sign\nplt.figure(figsize=(20,6))\n\nfor i,j in enumerate([4,7,0,1,14,9,32,5,8,3]):\n    plt.subplot(2,5,i+1)\n    plt.subplots_adjust(top = 2, bottom = 1)\n    img = Xtrain[j]\n    img = img.reshape((28,28))\n    plt.imshow(img,cmap = 'gray')\n    plt.title(ytrain[j])\n    plt.axis('off')\nplt.suptitle('Images from the training set',fontsize = 20)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:32:30.286551Z","iopub.execute_input":"2021-07-25T19:32:30.28724Z","iopub.status.idle":"2021-07-25T19:32:30.866515Z","shell.execute_reply.started":"2021-07-25T19:32:30.287195Z","shell.execute_reply":"2021-07-25T19:32:30.865475Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **3. Dimension reduction using DCT**","metadata":{}},{"cell_type":"markdown","source":"The 2D Discrete Cosine Transform (DCT) can be effectively used to compress digital images. The 2D DCT removes redundancies between neighbouring pixels and gives us uncorrelated transform coefficients which can be independently encoded. The DCT has a variety of useful properties such as energy preservability, compaction, seperability and decorrelation ability that makes it suitable for compression. \n\nFor dimensionality reduction, the energy compaction and signal decorrelation property of 2D DCT is very useful which allows representation of digital image by a relatively small number of DCT coefficients. To achieve this, zig zag scanning is used to scan each digit image of the data base of size 28 x 28 from top-left to bottom-right converting it into a vector of length 784. The DCT coefficients with larger magnitude tend to occur in the top-left corner. Hence, in a vector obtained by zig-zag scanning from top-left, larger DCT coefficients will be concentrated in the top part of the vector. Therefore, it is possible to get a fairly accurate estimation of the raw input image using a low dimensional vector obtained by retaining only the first several DCT coefficients.","metadata":{}},{"cell_type":"code","source":"#Lets plot the images compressed using DCT\nn_coeff = 14 #retaining only 14 x 14 DCT coefficients of a 28 x 28 image\ncompress = np.round((1-(n_coeff**2/num_pixels))*100,2)\nplt.figure(figsize=(20,6))\nfor i,j in enumerate([4,7,0,1,14,9,32,5,8,3]):\n    plt.subplot(2,5,i+1)\n    plt.subplots_adjust(top = 2, bottom = 1)\n    C = cv2.dct(Xtrain[j]) #dct of image\n    C_ = C[0:n_coeff,0:n_coeff] #taking only 30 x 30 i.e. 900 coefficients\n    C_n = np.zeros([28,28]) #original image shape\n    C_n[0:n_coeff,0:n_coeff] = C_\n    X_n = cv2.idct(C_n)\n    plt.imshow(X_n,cmap = 'gray')\n    plt.title(ytrain[j])\n    plt.axis('off')\nplt.suptitle('Images after compression using DCT by {} %'.format(compress),fontsize = 20)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:32:30.875646Z","iopub.execute_input":"2021-07-25T19:32:30.875955Z","iopub.status.idle":"2021-07-25T19:32:31.481977Z","shell.execute_reply.started":"2021-07-25T19:32:30.875928Z","shell.execute_reply":"2021-07-25T19:32:31.480906Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#function for zigzag scanning\ndef zigzag(a):\n    b =  np.concatenate([np.diagonal(a[::-1,:], i)[::(2*(i % 2)-1)] for i in range(1-a.shape[0], a.shape[0])])\n    return b","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:32:31.483524Z","iopub.execute_input":"2021-07-25T19:32:31.483937Z","iopub.status.idle":"2021-07-25T19:32:31.489641Z","shell.execute_reply.started":"2021-07-25T19:32:31.483895Z","shell.execute_reply":"2021-07-25T19:32:31.488675Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def dct_reduce(img,n):\n    #n is determine using how much energy we wanna retain\n    C = cv2.dct(img) #Perform discrete cosine transfor\n    c = zigzag(C) #scans image from top left to bottom right\n    s = c[:n] #retains first n entries\n    return s","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:32:31.491199Z","iopub.execute_input":"2021-07-25T19:32:31.491855Z","iopub.status.idle":"2021-07-25T19:32:31.502568Z","shell.execute_reply.started":"2021-07-25T19:32:31.491801Z","shell.execute_reply":"2021-07-25T19:32:31.501827Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"r = np.random.choice(range(len(Xtrain)))\nI = Xtrain[r,:,:] #select a random image\nC = cv2.dct(I) #calculate dct\nc = zigzag(C) #perform zigzag scanning\nfig,ax = plt.subplots(1,3,figsize=(20,5))\nax[0].imshow(I,cmap = 'gray') #original image\nax[0].set_title('Original Image')\nax[0].axis('off')\nax[1].imshow(C,cmap = 'gray')\nax[1].set_title('2D-DCT')\nax[1].axis('off')\nax[2].plot(c)\nax[2].grid(True)\nax[2].set_title('DCT coefficients arranged by zigzag scanning')\nplt.suptitle('Illustration of energy compaction property of 2D-DCT')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:32:31.504047Z","iopub.execute_input":"2021-07-25T19:32:31.504671Z","iopub.status.idle":"2021-07-25T19:32:31.82762Z","shell.execute_reply.started":"2021-07-25T19:32:31.50463Z","shell.execute_reply":"2021-07-25T19:32:31.826667Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"n =   196 #retaining only first 196 coefficients\nE_i = np.round(np.linalg.norm(I),2)  #frobenius norm of matrix after flattening\nE_c = np.round(np.linalg.norm(C),2)\nE_cr = np.round(np.linalg.norm(zigzag(C)[:n]),2)\ne_ret = E_cr/E_c*100\nprint(\"Frobenius norm of Image and its DCT is {} and {} respectively\".format(E_i,E_c))\nprint(\"Frobenius norm with only {} coefficients: {}\".format(n,E_cr))\nprint('Percentage of energy retained: {} %'.format(np.round(e_ret,2)))","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:32:31.828807Z","iopub.execute_input":"2021-07-25T19:32:31.829124Z","iopub.status.idle":"2021-07-25T19:32:31.837707Z","shell.execute_reply.started":"2021-07-25T19:32:31.829094Z","shell.execute_reply":"2021-07-25T19:32:31.836694Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Perform DCT based dimension reduction on train set\nxtrain = np.zeros([len(Xtrain),n])\nfor i in range(len(Xtrain)):\n    Itr = Xtrain[i]\n    s_tr = dct_reduce(Itr,n)\n    xtrain[i] = s_tr #truncated \n    ","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:32:31.839051Z","iopub.execute_input":"2021-07-25T19:32:31.83941Z","iopub.status.idle":"2021-07-25T19:32:41.109006Z","shell.execute_reply.started":"2021-07-25T19:32:31.839379Z","shell.execute_reply":"2021-07-25T19:32:41.107973Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Perform DCT based dimension reduction on validation set\nxval = np.zeros([len(Xval),n])\nfor i in range(len(Xval)):\n    Ival = Xval[i]\n    s_val = dct_reduce(Ival,n)\n    xval[i] = s_val","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:32:41.110293Z","iopub.execute_input":"2021-07-25T19:32:41.110605Z","iopub.status.idle":"2021-07-25T19:32:43.41806Z","shell.execute_reply.started":"2021-07-25T19:32:41.11057Z","shell.execute_reply":"2021-07-25T19:32:43.417111Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Perform DCT based dimension reduction on test set\nxtest = np.zeros([len(Xtest),n])\nfor i in range(len(Xtest)):\n    Itest = Xtest[i]\n    s_test = dct_reduce(Itest,n)\n    xtest[i] = s_test","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:32:43.419285Z","iopub.execute_input":"2021-07-25T19:32:43.419579Z","iopub.status.idle":"2021-07-25T19:32:51.200611Z","shell.execute_reply.started":"2021-07-25T19:32:43.419551Z","shell.execute_reply":"2021-07-25T19:32:51.199505Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **4. Performing Classification**","metadata":{}},{"cell_type":"markdown","source":"### Using PCA for Classification","metadata":{}},{"cell_type":"code","source":"def PCA_extract(xtrain,ytrain,q):\n    # q  - number of principal component components to be retained .\n    U = []\n    M = []\n    for i in range(ytrain.max()+1):\n        D = xtrain[np.where(ytrain == i)]  #get data for each digit\n        m = np.mean(D, axis=0) #calculate the mean of each column\n        C = D - m #centralize each data point\n        V = np.cov(C.T) #calculate covariance\n        # eigendecomposition of covariance matrix\n        values, vectors = largest_eigsh(V,q,which = 'LM') # we want q eigen vectors corresponding to q largest magnitude eigen values\n        U.append(vectors)\n        M.append(m)  \n    return U,M","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:32:51.202032Z","iopub.execute_input":"2021-07-25T19:32:51.202385Z","iopub.status.idle":"2021-07-25T19:32:51.209131Z","shell.execute_reply.started":"2021-07-25T19:32:51.202354Z","shell.execute_reply":"2021-07-25T19:32:51.208238Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q = 14\nU,M = PCA_extract(xtrain,ytrain,q)","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:32:51.212621Z","iopub.execute_input":"2021-07-25T19:32:51.212922Z","iopub.status.idle":"2021-07-25T19:32:51.572891Z","shell.execute_reply.started":"2021-07-25T19:32:51.212883Z","shell.execute_reply":"2021-07-25T19:32:51.571741Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Perform classification using PCA\ndef PCA_classify(test,U,M):\n    e = np.zeros([len(test),10])\n    for i in range(len(test)):\n        for j in range(10):\n            Uq = U[j] #get the eigen vector corresponding to the class (n x q)\n            te = test[i] - M[j] # (n,)\n            f =  np.dot(Uq.T,te) #lower dimension of the point (q x n)*(n x 1) = q x 1\n            z = np.dot(f,Uq.T) # projecting low dimensional data point onto subspace of each class\n            d = np.linalg.norm(z - te) #distance between testing datapoint and its projection\n            e[i,j]= d\n    # identify the target class for data points \n    ypred = np.argmin(e,axis =1 ) \n    return ypred","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:32:51.575208Z","iopub.execute_input":"2021-07-25T19:32:51.575916Z","iopub.status.idle":"2021-07-25T19:32:51.585904Z","shell.execute_reply.started":"2021-07-25T19:32:51.575868Z","shell.execute_reply":"2021-07-25T19:32:51.584581Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#classification on the validation set\nypred = PCA_classify(xval,U,M)","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:32:51.587743Z","iopub.execute_input":"2021-07-25T19:32:51.588446Z","iopub.status.idle":"2021-07-25T19:32:53.319619Z","shell.execute_reply.started":"2021-07-25T19:32:51.588397Z","shell.execute_reply":"2021-07-25T19:32:53.318534Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#plot confusion matrix\ncm = confusion_matrix(yval,ypred)\nplt.figure(figsize=(10,10))\nsns.heatmap(cm,cmap = 'Blues',annot = True,fmt = 'g')\nplt.xlabel('Predicted value')\nplt.ylabel('True')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:32:53.320855Z","iopub.execute_input":"2021-07-25T19:32:53.321139Z","iopub.status.idle":"2021-07-25T19:32:53.91404Z","shell.execute_reply.started":"2021-07-25T19:32:53.321112Z","shell.execute_reply":"2021-07-25T19:32:53.913061Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"accuracy = np.round(accuracy_score(yval,ypred),4)\nprint(\"The accuracy of PCA based classifier on validation set is {} %\".format(accuracy*100))","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:32:53.915124Z","iopub.execute_input":"2021-07-25T19:32:53.915385Z","iopub.status.idle":"2021-07-25T19:32:53.92166Z","shell.execute_reply.started":"2021-07-25T19:32:53.91536Z","shell.execute_reply":"2021-07-25T19:32:53.920634Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#classification on the test set\nypred_t = PCA_classify(xtest,U,M)","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:32:53.923001Z","iopub.execute_input":"2021-07-25T19:32:53.923469Z","iopub.status.idle":"2021-07-25T19:32:59.520044Z","shell.execute_reply.started":"2021-07-25T19:32:53.923438Z","shell.execute_reply":"2021-07-25T19:32:59.519171Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"results = pd.Series(ypred_t,name=\"Label\")\nsubmission = pd.concat([pd.Series(range(1,28001),name = \"ImageId\"),results],axis = 1)\nsubmission.to_csv(\"dct_pca_mnist.csv\",index=False)","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:32:59.521417Z","iopub.execute_input":"2021-07-25T19:32:59.521701Z","iopub.status.idle":"2021-07-25T19:32:59.583471Z","shell.execute_reply.started":"2021-07-25T19:32:59.521674Z","shell.execute_reply":"2021-07-25T19:32:59.582385Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Using ANN for classification","metadata":{}},{"cell_type":"code","source":"def plotLearningCurve(history,epochs):\n  epochRange = range(1,epochs+1)\n  fig , ax = plt.subplots(1,2,figsize = (10,5))\n  \n  ax[0].plot(epochRange,history.history['accuracy'],label = 'Training Accuracy')\n  ax[0].plot(epochRange,history.history['val_accuracy'],label = 'Validation Accuracy')\n  ax[0].set_title('Training and Validation accuracy')\n  ax[0].set_xlabel('Epoch')\n  ax[0].set_ylabel('Accuracy')\n  ax[0].legend()\n  ax[1].plot(epochRange,history.history['loss'],label = 'Training Loss')\n  ax[1].plot(epochRange,history.history['val_loss'],label = 'Validation Loss')\n  ax[1].set_title('Training and Validation loss')\n  ax[1].set_xlabel('Epoch')\n  ax[1].set_ylabel('Loss')\n  ax[1].legend()\n  fig.tight_layout()\n  plt.show()","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:32:59.584814Z","iopub.execute_input":"2021-07-25T19:32:59.58513Z","iopub.status.idle":"2021-07-25T19:32:59.594631Z","shell.execute_reply.started":"2021-07-25T19:32:59.585099Z","shell.execute_reply":"2021-07-25T19:32:59.593347Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.random.seed(12)\ni = Input(shape=len(xtrain.T))\nx = Dense(256, activation='relu')(i)\nx = Dropout(0.4)(x)\nx = Dense(128, activation='relu')(x)\nx = Dropout(0.3)(x)\nx = Dense(64, activation='relu')(x)\nx = Dropout(0.2)(x)\nx = Dense(32,activation = 'relu')(x)\nx = Dropout(0.1)(x)\nx = Dense(10, activation='softmax')(x)\nmodel = Model(i, x)\nmodel.summary()\n","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:32:59.596401Z","iopub.execute_input":"2021-07-25T19:32:59.596816Z","iopub.status.idle":"2021-07-25T19:32:59.727671Z","shell.execute_reply.started":"2021-07-25T19:32:59.59677Z","shell.execute_reply":"2021-07-25T19:32:59.726756Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.compile(loss='sparse_categorical_crossentropy', optimizer=Adam(0.001),metrics=['accuracy']) # use categorical_crossentropy when one hot encoding is used","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:32:59.728899Z","iopub.execute_input":"2021-07-25T19:32:59.729199Z","iopub.status.idle":"2021-07-25T19:32:59.744505Z","shell.execute_reply.started":"2021-07-25T19:32:59.72917Z","shell.execute_reply":"2021-07-25T19:32:59.743474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":" r  = model.fit(x=xtrain, \n          y=ytrain, \n          epochs=10,\n          batch_size = 16,\n          validation_data=(xval, yval),\n          )","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:32:59.745975Z","iopub.execute_input":"2021-07-25T19:32:59.746311Z","iopub.status.idle":"2021-07-25T19:33:39.380556Z","shell.execute_reply.started":"2021-07-25T19:32:59.746281Z","shell.execute_reply":"2021-07-25T19:33:39.379678Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plotLearningCurve(r,10)","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:33:39.381804Z","iopub.execute_input":"2021-07-25T19:33:39.382113Z","iopub.status.idle":"2021-07-25T19:33:39.757399Z","shell.execute_reply.started":"2021-07-25T19:33:39.382053Z","shell.execute_reply":"2021-07-25T19:33:39.756668Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ypred = model.predict(xval)\nypred = np.argmax(ypred,axis = 1) ","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:33:39.75922Z","iopub.execute_input":"2021-07-25T19:33:39.759604Z","iopub.status.idle":"2021-07-25T19:33:40.093227Z","shell.execute_reply.started":"2021-07-25T19:33:39.759561Z","shell.execute_reply":"2021-07-25T19:33:40.092237Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#plot confusion matrix\ncm = confusion_matrix(yval,ypred)\nplt.figure(figsize=(10,10))\nsns.heatmap(cm,cmap = 'Blues',annot = True,fmt = 'g')\nplt.xlabel('Predicted value')\nplt.ylabel('True')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:33:40.094542Z","iopub.execute_input":"2021-07-25T19:33:40.094864Z","iopub.status.idle":"2021-07-25T19:33:40.679654Z","shell.execute_reply.started":"2021-07-25T19:33:40.094835Z","shell.execute_reply":"2021-07-25T19:33:40.67871Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Train score:\", model.evaluate(xtrain,ytrain))\nprint(\"Validation score:\", model.evaluate(xval,yval))","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:33:40.680941Z","iopub.execute_input":"2021-07-25T19:33:40.681252Z","iopub.status.idle":"2021-07-25T19:33:42.900183Z","shell.execute_reply.started":"2021-07-25T19:33:40.681221Z","shell.execute_reply":"2021-07-25T19:33:42.899152Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"accuracy = np.round(accuracy_score(yval,ypred),4)\nprint(\"The accuracy of ANN on validation set is {} %\".format(accuracy*100))","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:33:42.901482Z","iopub.execute_input":"2021-07-25T19:33:42.901795Z","iopub.status.idle":"2021-07-25T19:33:42.908216Z","shell.execute_reply.started":"2021-07-25T19:33:42.901764Z","shell.execute_reply":"2021-07-25T19:33:42.907088Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ypred_t = model.predict(xtest)","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:33:42.9093Z","iopub.execute_input":"2021-07-25T19:33:42.909583Z","iopub.status.idle":"2021-07-25T19:33:43.940771Z","shell.execute_reply.started":"2021-07-25T19:33:42.909555Z","shell.execute_reply":"2021-07-25T19:33:43.939737Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# select the indix with the maximum probability\nresults = np.argmax(ypred_t,axis = 1)\n\nresults = pd.Series(results,name=\"Label\")","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:33:43.942053Z","iopub.execute_input":"2021-07-25T19:33:43.942374Z","iopub.status.idle":"2021-07-25T19:33:43.948223Z","shell.execute_reply.started":"2021-07-25T19:33:43.942344Z","shell.execute_reply":"2021-07-25T19:33:43.947267Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = pd.concat([pd.Series(range(1,28001),name = \"ImageId\"),results],axis = 1)\n\nsubmission.to_csv(\"dct_ann_mnist.csv\",index=False)","metadata":{"execution":{"iopub.status.busy":"2021-07-25T19:33:43.949611Z","iopub.execute_input":"2021-07-25T19:33:43.950027Z","iopub.status.idle":"2021-07-25T19:33:44.01138Z","shell.execute_reply.started":"2021-07-25T19:33:43.949987Z","shell.execute_reply":"2021-07-25T19:33:44.010365Z"},"trusted":true},"execution_count":null,"outputs":[]}]}