{"cells":[{"metadata":{},"cell_type":"markdown","source":"# MNIST Data\n**The MNIST database (Modified National Institute of Standards and Technology database)** is a large database of handwritten digits that is commonly used for training various image processing systems. The database is also widely used for training and testing in the field of machine learning. IT was created by \"re-mixing\" the samples from NIST's original datasets. The creators felt that since NIST's training dataset was taken from American Census Bureau employees, while the testing dataset was taken from American high school students, it was not well-suited for machine learning experiments.Furthermore, the black and white images from NIST were normalised to fit into a 28x28 pixel bounding box and anti-aliased, which introduced grayscale levels."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.utils.data import DataLoader\nimport torchvision.transforms as transforms\nimport torchvision\nfrom torch.utils.data import TensorDataset\nfrom torch.optim import Adam, SGD\n\nfrom sklearn.decomposition import PCA\nimport pylab\n\n# Basic Numeric Computation\nimport numpy as np\nimport pandas as pd\n\n# Look at data\nimport seaborn as sns\nfrom matplotlib import pyplot\nimport matplotlib.pyplot as plt\nimport pandas as pd\n#from math import pi\n#from collections import Counter\nimport seaborn as sns\nfrom sklearn.decomposition import PCA\nimport pylab\nimport time\nfrom sklearn.manifold import TSNE\nfrom sklearn import manifold\nimport matplotlib.pyplot as plt\nfrom mpl_toolkits.mplot3d import Axes3D\nfrom sklearn.datasets import make_swiss_roll","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import warnings\nwarnings.filterwarnings('ignore')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"device = torch.device(\"cpu\")\nepochs=10\n\ntrain = pd.read_csv('../input/train.csv')\ntest = pd.read_csv('../input/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_viz_train = train['label']\nX_viz_train = train.drop('label', axis=1)\nX_viz_test = test\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.preprocessing import StandardScaler\nscaler = StandardScaler()\nX_viz_train = scaler.fit_transform(X_viz_train)\nX_viz_test = scaler.fit_transform(X_viz_test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# PCA- Principal Component Analysis\nPrincipal Component Analysis, or PCA, is a dimensionality-reduction method that is often used to reduce the dimensionality of large data sets, by transforming a large set of variables into a smaller one that still contains most of the information in the large set. Reducing the number of variables of a data set naturally comes at the expense of accuracy, but the trick in dimensionality reduction is to trade a little accuracy for simplicity. Because smaller data sets are easier to explore and visualize and make analyzing data much easier and faster for machine learning algorithms without extraneous variables to process. So to sum up, the idea of PCA is simple — reduce the number of variables of a data set, while preserving as much information as possible."},{"metadata":{"trusted":true},"cell_type":"code","source":"pca = PCA(n_components=2)\nprincipalComponents = pca.fit_transform(X_viz_train)\nprincipalDf =pd.DataFrame(data = principalComponents, columns = ['principalcomponent1',  'principalcomponent2'])\n\nlabel = pd.DataFrame(list(train['label']))\nprincipalDf = pd.concat([principalDf,label],axis = 1,join='inner', ignore_index=True)\nprincipalDf = principalDf.loc[:,~principalDf.columns.duplicated()]\nprincipalDf.columns = [\"principalcomponent1\", \"principalcomponent2\", \"label\"] ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"principalDf.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"flatui = [\"#9b59b6\", \"#3498db\", \"orange\"]\nsns.set_palette(flatui)\nsns.lmplot( x=\"principalcomponent1\", y=\"principalcomponent2\", data=principalDf, fit_reg=False,\n           hue='label', legend=False)\n\nplt.figure(figsize=(13,10))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"N = 10000\ndf_subset = X_viz_train[:N,:].copy()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# t-SNE\n\n**t-Distributed Stochastic Neighbor Embedding (t-SNE)** is a (prize-winning) technique for dimensionality reduction that is particularly well suited for the visualization of high-dimensional datasets. The technique can be implemented via Barnes-Hut approximations, allowing it to be applied on large real-world datasets.\nt-Distributed Stochastic Neighbor Embedding (t-SNE) is an unsupervised, non-linear technique primarily used for data exploration and visualizing high-dimensional data. In simpler terms, t-SNE gives you a feel or intuition of how the data is arranged in a high-dimensional space. It was developed by Laurens van der Maatens and Geoffrey Hinton in 2008."},{"metadata":{"trusted":true},"cell_type":"code","source":"time_start = time.time()\ntsne = TSNE(n_components=3, verbose=1, perplexity=40, n_iter=300)\ntsne_results = tsne.fit_transform(df_subset)\nprint('t-SNE done in {} seconds'.format(time.time()-time_start))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tsne3_subset = pd.DataFrame(columns=['tsne-3d-one', 'tsne-3d-two', 'tsne-3d-three'])\n\ntsne3_subset['tsne-3d-one'] = tsne_results[:,0]\ntsne3_subset['tsne-3d-two'] = tsne_results[:,1]\ntsne3_subset['tsne-3d-three'] = tsne_results[:,2]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## t-SNE vs PCA\nYou’re probably wondering the difference between PCA and t-SNE. The first thing to note is that PCA was developed in 1933 while t-SNE was developed in 2008. A lot has changed in the world of data science since 1933 mainly in the realm of compute and size of data. Second, PCA is a linear dimension reduction technique that seeks to maximize variance and preserves large pairwise distances. In other words, things that are different end up far apart. This can lead to poor visualization especially when dealing with non-linear manifold structures. Think of a manifold structure as any geometric shape like: cylinder, ball, curve, etc.\n\nt-SNE differs from PCA by preserving only small pairwise distances or local similarities whereas PCA is concerned with preserving large pairwise distances to maximize variance. You can see that due to the non-linearity of this toy dataset (manifold) and preserving large distances that PCA would incorrectly preserve the structure of the data."},{"metadata":{"trusted":true},"cell_type":"code","source":"n_samples = 10000\nX, color = make_swiss_roll(n_samples)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"time_start = time.time()\ntsne2d = TSNE(n_components=2, verbose=1, perplexity=40, n_iter=300)\ntsne2d_results = tsne.fit_transform(df_subset)\nprint('t-SNE done in {} seconds'.format(time.time()-time_start))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tsne2_subset = pd.DataFrame(columns=['tsne-2d-one', 'tsne-2d-two'])\n\ntsne2_subset['tsne-2d-one'] = tsne2d_results[:,0]\ntsne2_subset['tsne-2d-two'] = tsne2d_results[:,1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(20,10))\nax = fig.add_subplot(2, 1, 1,projection='3d')\nax.set_title('TSNE 3-d', fontsize=10)\nax.scatter(tsne_results[:,0],tsne_results[:,1],tsne_results[:,2],c = color,cmap=\"Accent\",s=60)# we are picking up the x,y,z co-ordinate values from dataset\nax = fig.add_subplot(2, 1, 2) \nax.set_title('TSNE - 2d', fontsize=10)\nax.scatter(tsne2d_results[:,0],tsne2d_results[:,1],c = color,cmap=\"Accent\",s=60)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# ISOMAP\n\nIsomap stands for isometric mapping. Isomap is a non-linear dimensionality reduction method based on the spectral theory which tries to preserve the geodesic distances in the lower dimension. Isomap starts by creating a neighborhood network. After that, it uses graph distance to the approximate geodesic distance between all pairs of points. And then, through eigenvalue decomposition of the geodesic distance matrix, it finds the low dimensional embedding of the dataset. In non-linear manifolds, the Euclidean metric for distance holds good if and only if neighborhood structure can be approximated as linear. If neighborhood contains holes, then Euclidean distances can be highly misleading. In contrast to this, if we measure the distance between two points by following the manifold, we will have a better approximation of how far or near two points are. Let's understand this with an extremely simple 2-D example. "},{"metadata":{"trusted":true},"cell_type":"code","source":"##############\n######\n#ISOMAP\n######\n##############\n\niso = manifold.Isomap(n_neighbors=6, n_components=2)\niso.fit(X)\nmanifold_iso_data = iso.transform(X)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Suppose our data lies on a circular manifold in a 2-D structure like in the image below.\n\n> Why geodesic distances are better than the Euclidean distances in nonlinear manifolds?\n\n We will reduce the data to 1-D using the 'Euclidean' distances and approximate geodesic distances. Now, if we look at the 1-D mapping based on the Euclidean metric, we see that for points which are far apart(a & b) have been mapped poorly. Only the points which can be approximated to lie on a linear manifold(c & d) give satisfactory results. On the other hand, see the mapping with geodesic distances, it nicely approximates the close points as neighbors and far away points as distant.\nThe geodesic distances between two points in the image are approximated by graph distance between the two points. Thus, Euclidean distances should not be used for approximating the distance between two points in non-linear manifolds while geodesic distances can be used.\n\n![ISOMAP GEO](https://i.imgur.com/9KqpHx3.png)"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(20,10))\nax = fig.add_subplot(2, 1, 1,projection='3d')\nax.set_title('Here is the swiss roll maniflod', fontsize=10)\nax.scatter(X[:,0],X[:,1],X[:,2],c = color,cmap=\"Accent\",s=60)\nx = X[:,0][2:10000] \ny = X[:,1][2:10000] # Just as abovve, this time for column 1\nax.scatter(x,y,c = \"black\") #Now we randomly plot this in both 3D manifold (this may not be clearly visible as the  existing\nax.plot(x[2:4],y[2:4],c = \"red\")\nax = fig.add_subplot(2, 1, 2) \n#Now we plot 2D after ISOMAP...\nax.set_title('When compressing with ISOMAP', fontsize=10)\nax.scatter(manifold_iso_data[:,0],manifold_iso_data[:,1],c = color,cmap=\"Accent\",s=60)\nx = X[:,0][2:10000]#Now we plot the same 'black' samples, after ISOMAP in 2D and observe the distance in 2D.\ny = X[:,1][2:10000]\nax.scatter(x,y,c = \"black\")\nax.plot(x[2:4],y[2:4],c = \"red\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n> Isomap uses the above principle to create a similarity matrix for eigenvalue decomposition.\n\nUnlike other non-linear dimensionality reduction like LLE & LPP which only use local information, isomap uses the local information to create a global similarity matrix. The isomap algorithm uses euclidean metrics to prepare the neighborhood graph. Then, it approximates the geodesic distance between two points by measuring shortest path between these points using graph distance. Thus, it approximates both global as well as the local structure of the dataset in the low dimensional embedding."},{"metadata":{"trusted":true},"cell_type":"code","source":"train['label'].head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Convolutional Neural Network\n\nA Convolutional Neural Network (ConvNet/CNN) is a Deep Learning algorithm which can take in an input image, assign importance (learnable weights and biases) to various aspects/objects in the image and be able to differentiate one from the other. The pre-processing required in a ConvNet is much lower as compared to other classification algorithms. While in primitive methods filters are hand-engineered, with enough training, ConvNets have the ability to learn these filters/characteristics.\nThe architecture of a ConvNet is analogous to that of the connectivity pattern of Neurons in the Human Brain and was inspired by the organization of the Visual Cortex. Individual neurons respond to stimuli only in a restricted region of the visual field known as the Receptive Field. A collection of such fields overlap to cover the entire visual area."},{"metadata":{"trusted":true},"cell_type":"code","source":"def Image_Data(raw: pd.DataFrame):\n    y = raw['label'].values\n    y.resize(y.shape[0],1)\n    x = raw[[i for i in raw.columns if i != 'label']].values\n    x = x.reshape([-1,1, 28, 28])\n    y = y.astype(int).reshape(-1)\n    x = x.astype(float)\n    return x, y\n\n## Convert to One Hot Embedding\ndef one_hot_embedding(labels, num_classes=10):\n    y = torch.eye(num_classes) \n    return y[labels] \n\nx_train, y_train = Image_Data(train)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"![CNN](https://i.imgur.com/eBQL6AL.jpg)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Normalization\nmean = x_train.mean()\nstd = x_train.std()\nx_train = (x_train-mean)/std\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Input Image:\nWe have an RGB image which has been separated by its three color planes — Red, Green, and Blue. \nThere are a number of such color spaces in which images exist — Grayscale, RGB, HSV, CMYK, etc.\nYou can imagine how computationally intensive things would get once the images reach dimensions, say 8K (7680×4320). The role of the ConvNet is to reduce the images into a form which is easier to process, without losing features which are critical for getting a good prediction. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# Numpy to Torch Tensor\nx_train = torch.from_numpy(np.float32(x_train)).to(device)\ny_train = torch.from_numpy(y_train.astype(np.long)).to(device)\ny_train = one_hot_embedding(y_train)\n#x_val = torch.from_numpy(np.float32(x_val))\n#y_val = torch.from_numpy(y_val.astype(np.long))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Convolution Layer\nThe objective of the Convolution Operation is to extract the high-level features such as edges, from the input image. ConvNets need not be limited to only one Convolutional Layer. Conventionally, the first ConvLayer is responsible for capturing the Low-Level features such as edges, color, gradient orientation, etc. With added layers, the architecture adapts to the High-Level features as well, giving us a network which has the wholesome understanding of images in the dataset, similar to how we would.There are two types of results to the operation — one in which the convolved feature is reduced in dimensionality as compared to the input, and the other in which the dimensionality is either increased or remains the same. This is done by applying Valid Padding in case of the former, or Same Padding in the case of the latter."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Convert into Torch Dataset\ntrain_ds = TensorDataset(x_train, y_train)\ntrain_dl = DataLoader(train_ds, batch_size=64)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def init_weights(m):\n    if type(m) == nn.Linear:\n        torch.nn.init.xavier_uniform(m.weight)\n        m.bias.data.fill_(0.01)\n\n## Flatten Later\nclass Flatten(nn.Module):\n    def forward(self, input):\n        return input.view(input.size(0), -1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Pooling Layer\n\nSimilar to the Convolutional Layer, the Pooling layer is responsible for reducing the spatial size of the Convolved Feature. This is to decrease the computational power required to process the data through dimensionality reduction. Furthermore, it is useful for extracting dominant features which are rotational and positional invariant, thus maintaining the process of effectively training of the model.\nThere are two types of Pooling:** Max Pooling and Average Pooling**. Max Pooling returns the maximum value from the portion of the image covered by the Kernel. On the other hand, Average Pooling returns the average of all the values from the portion of the image covered by the Kernel."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Train the network and print accuracy and loss overtime\ndef fit(train_dl, model, loss, optim, epochs=10):\n    model = model.to(device)\n    print('Epoch\\tAccuracy\\tLoss')\n    accuracy_overtime = []\n    loss_overtime = []\n    for epoch in range(epochs):\n        avg_loss = 0\n        correct = 0\n        total=0\n        for x, y in train_dl: # Iterate over Data Loder\n    \n            # Forward pass\n            yhat = model(x) \n            l = loss(y, yhat)\n            \n            #Metrics\n            avg_loss+=l.item()\n            \n            # Backward pass\n            optim.zero_grad()\n            l.backward()\n            optim.step()\n            \n            # Metrics\n            _, original =  torch.max(y, 1)\n            _, predicted = torch.max(yhat.data, 1)\n            total += y.size(0)\n            correct = correct + (original == predicted).sum().item()\n            \n        accuracy_overtime.append(correct/total)\n        loss_overtime.append(avg_loss/len(train_dl))\n        print(epoch,accuracy_overtime[-1], loss_overtime[-1], sep='\\t')\n    return accuracy_overtime, loss_overtime","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_accuracy_loss(accuracy, loss):\n    f = pyplot.figure(figsize=(15,5))\n    ax1 = f.add_subplot(121)\n    ax2 = f.add_subplot(122)\n    ax1.title.set_text(\"Accuracy over epochs\")\n    ax2.title.set_text(\"Loss over epochs\")\n    ax1.plot(accuracy)\n    ax2.plot(loss, 'r:')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ff_model = nn.Sequential(\n    Flatten(),\n    nn.Linear(28*28, 100),\n    nn.ReLU(),\n    nn.Linear(100, 10),\n    nn.Softmax(1),\n).to(device)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ff_model.apply(init_weights)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\noptim = Adam(ff_model.parameters())\nloss = nn.MSELoss()\noutput = fit(train_dl, ff_model, loss, optim, epochs)\nplot_accuracy_loss(*output)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"index = 6\npyplot.imshow(x_train.cpu()[index].reshape((28, 28)), cmap=\"gray\")\nprint(y_train[index])\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class ConvNet(nn.Module):\n    def __init__(self, num_classes=10):\n        super(ConvNet, self).__init__()\n        self.layer1 = nn.Sequential(\n            nn.Conv2d(1, 16, kernel_size=5, stride=1, padding=2),\n            nn.BatchNorm2d(16),\n            nn.ReLU(),\n            nn.MaxPool2d(kernel_size=2, stride=2))\n        self.layer2 = nn.Sequential(\n            nn.Conv2d(16, 32, kernel_size=5, stride=1, padding=2),\n            nn.BatchNorm2d(32),\n            nn.ReLU(),\n            nn.MaxPool2d(kernel_size=2, stride=2))\n        self.fc = nn.Linear(7*7*32, num_classes)\n        \n    def forward(self, x):\n        out = self.layer1(x)\n        out = self.layer2(out)\n        out = out.reshape(out.size(0), -1)\n        out = self.fc(out)\n        return out","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Classification — Fully Connected Layer (FC Layer)\nAdding a Fully-Connected layer is a (usually) cheap way of learning non-linear combinations of the high-level features as represented by the output of the convolutional layer. The Fully-Connected layer is learning a possibly non-linear function in that space.\nNow that we have converted our input image into a suitable form for our Multi-Level Perceptron, we shall flatten the image into a column vector. The flattened output is fed to a feed-forward neural network and backpropagation applied to every iteration of training. Over a series of epochs, the model is able to distinguish between dominating and certain low-level features in images and classify them using the Softmax Classification technique."},{"metadata":{"trusted":true},"cell_type":"code","source":"\nnum_classes = 10\nmodel = ConvNet(num_classes).to(device)\nmodel.apply(init_weights)\nloss = nn.MSELoss()\noptim = SGD(model.parameters(), lr=0.003, momentum=0.9)\nplot_accuracy_loss(*fit(train_dl, model,loss,optim,epochs))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x_test = test.values\nx_test = x_test.reshape([-1, 28, 28]).astype(float)\nx_test = (x_test-mean)/std\nx_test = torch.from_numpy(np.float32(x_test))\nx_test.shape\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def export_csv(model_name, predictions):\n    df = pd.DataFrame(prediction.tolist(), columns=['Label'])\n    df['ImageId'] = df.index + 1\n    file_name = f'submission_{model_name}.csv'\n    print('Saving ',file_name)\n    df[['ImageId','Label']].to_csv(file_name, index = False)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ff_test = ff_model(x_test.float())\nprediction = torch.argmax(ff_test,1)\nprint('Prediction',prediction)\nexport_csv('ff_model',prediction)\ntorch.save(model.state_dict(), 'model_ff.ckpt')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}