{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## [Link to original Notebook by Data Storm](https://www.kaggle.com/paulrohan2020/tutorial-dimensionality-reduction-pca-maths)","metadata":{}},{"cell_type":"markdown","source":"## Dimensionality Reduction\n\nIn many practical applications, although the data reside in a high-dimensional space, the true dimensionality, known as intrinsic dimensionality, can be of a much lower value.\n\n For example, in a three-dimensional space, the data may cluster around a straight line, or around the circumference of a circle or the graph of a parabola, arbitrarily placed in R^3. In all previous cases, the intrinsic dimensionality of the data is equal to one, as any of these curves can equivalently\nbe described in terms of a single parameter.\n\nBelow figure illustrates the three cases. Learning the lower-dimensional structure associated with a given set of data is gaining in importance in the context of big data processing and analysis. Some typical examples are the disciplines of computer vision, robotics, medical imaging, and computational neuroscience.\n\n![img](https://i.imgur.com/IkFGd65.png)\n\nThe data reside close to (A) a straight line, (B) the circumference of a circle, and (C) the graph of a parabola in the three-dimensional space. In all three cases, the intrinsic dimensionality of the data is equal to one. In (A) the data are clustered around a (translated/affine) linear subspace and in (B) and (C) around one-dimensional manifolds.\n\n#### Some notes on Terminologies used here - When I say x_i belongs to R^d that means x_i is a d-dimensional column vector\n\n![img](https://i.imgur.com/ppkf04S.png)\n\nWhich is a d * 1 Matrix\n\nSo in the case of this famous [iris dataset](https://www.kaggle.com/arshid/iris-flower-dataset?select=IRIS.csv) x_i belongs to R^4 as it has 4-features and is a 4-dimensional dataset.\n\n![img](https://i.imgur.com/GifMvTp.png)\n\n#### The various methods used for dimensionality reduction include:\n\n- Principal Component Analysis (PCA)\n\n- Linear Discriminant Analysis (LDA)\n\n- Generalized Discriminant Analysis (GDA)\n\nDimensionality reduction may be both linear or non-linear, depending upon the method used. The prime linear method, called Principal Component Analysis, or PCA, is discussed below.\n\n\n\n## When should I use Dimensionality reduction\n\nThink of PCA as an exploratory technique to investigate and study your system before doing other things. Dimensionality reduction, by nature, loses some information, just like image compression for instance. Consequently, it will often reduce the prediction quality of your model, but depending on the data it could also leave it unchanged or even improve it in some cases (very noisy data). In general, its main benefit will be to speed up training.\n\nBesides speed of training, feature reduction also help with multicollinearity in some cases. Which is mainly an issue if you are interested in parameters estimation, like in causal analysis. For instance, if you use a multiple linear regression model to estimate the effect of some regressors on a dependent variable, multicollinearity will prevent proper parameters estimation because you won't be able to identify the effect of each regressor.\n\nHowever, in many other ML applications, we don't really care about parameters identification, we care about how a set of variables can be used to predict another variable. If some variables are highly correlated, then there will be some redundant information in your data, but this shouldn't be problematic in terms of prediction quality.\n\nIn my experience, in couple of Kaggle Challenges, after doing some PCA, the accuracy has actually went down. After googling found out that some infer that PCA assumed that the variance could translate to high differentiable information which classifier could use. But at the same time, it threw some information away(not just noise), and these information could play a huge role in classification, although the variance is low.\n\n---\n\n## Data preparation for dimensionality reduction\n\nThere is no best technique for dimensionality reduction and no mapping of techniques to problems.\n\nInstead, the best approach is to use systematic controlled experiments to discover what dimensionality reduction techniques, when paired with your model of choice, result in the best performance on your dataset.\n\nTypically, linear algebra and manifold learning methods assume that all input features have the same scale or distribution. This suggests that it is good practice to either normalize or standardize data prior to using these methods if the input variables have differing scales or units.\n\n## Comparison of feature selection, PCA, and clustering as dimension reduction schemes\n\n![img](https://i.imgur.com/1KUD3gN.png)\n\nThe first 2 methods, Feature Selection and PCA reduce the dimension of the feature space, or in other words the number of rows in a data matrix. However, the two methods work differently: while feature selection literally selects rows from the original matrix to keep, PCA uses the geometry of the feature space to produce a new data matrix based on a lower feature dimensional version of the data. K-means, on the other hand, reduces the dimension of the data/number of data points, or equivalently the number of columns in the input data matrix. It does so by finding a small number of new averaged representatives or “centroids” of the input data, forming a new data matrix whose fewer columns (which are not\npresent in the original data matrix) are precisely these centroids.\n\n## Mean Vector\n\nThe first step in analyzing multivariate data is computing the mean vector and the variance-covariance matrix.\n\nThe sample mean is a vector each of whose elements is the sample mean of one of the random variables – that is, each of whose elements is the arithmetic average of the observed values of one of the variables.\n\n![img](https://i.imgur.com/XG2Yag7.png)\n\nFor example, take the following two vectors\n\nx1 = [2.2, 4.2, …]\n\nx2 = [1.2, 3.2, …]\n\nSo the formulae for calculating `x_mean = 1/2(x1+x2)`\n\n= 0.5* [3.4, 7.4, ..]\n\n= [1.7,3.7, .. ]\n\n#### So we just summed up elements at ith index of the first array and the corresponding index of the second array.\n\n#### That also means, that we can say every array can be considered as a vector with each of its indices as one of the dimensions.\n\n## Covariance Matrix\n\nCovariances Matrix consists of covariances between each pair of variables in the other matrix positions.\n\nFirst a quick look at the difference between covariance and variance. Variance measures the variation of a single random variable (like the height of a person in a population), whereas covariance is a measure of how much two random variables vary together (like the height of a person and the weight of a person in a population). The formula for variance is given by\n\n![img](https://i.imgur.com/CnnSCxv.png)\n\n### Definition of Co-Variance for 2 scaler values from [Wikipedia](https://en.wikipedia.org/wiki/Covariance)\n\nFor two jointly distributed real-valued random variables X and Y with finite second moments, the covariance is defined as the expected value (or mean) of the product of their deviations from their individual expected values\n\n\n![img](https://i.imgur.com/jE6qkN7.png)\n\nwhere E[X] is the expected value of X, also known as the mean of X.\n\n\n### Definition of Co-Variance for Matrix form of data - i.e. where the X (as defined in the above formulae is not a Scaler but is a Vector )\n\n![img](https://i.imgur.com/eS35C7X.png)\n\n#### and  very importantly the transpose appears in this case (for the Matrix form ) because we are multiplying two vectors.\n\n#### In another form the formulae for the Covariance Matrix is\n\n![img](https://i.imgur.com/AZ10i59.png)\n\nStressing the last point from above if the columns of the original Matrix are standardized, meaning making the column vectors such that the mean is zero and Standard Deviation is 1 the above formulate for Covariance of a Matrix becomes simply\n\n#### XX^T/(n-1)\n\n## Matrix Multiplication Basics - Explanation on why we are using the transpose of the second element in calculating Covariance Matrix\n\n### Rules of matrix multiplication\n\n#### a⋅b = a^Tb\n\nOn the other hand, **b⋅a≠baT**, which is an n×n matrix. The correct expression is\n\n#### b⋅a = b^Ta\n\n![img](https://i.imgur.com/ZrW2xUy.png)\n\nWe are indeed taking dot products here. Note, though, that a is a column vector, but a^T is a row vector.\n\nBy the rules of matrix multiplication, a^Ta and a^Tb result in a 1×1 matrix, which is equivalent to a scalar, while aa^T produces an n×n matrix:\n\n![img](https://i.imgur.com/6xjyH0y.png)\n\n[Source](https://math.stackexchange.com/a/1853890/517433)\n\nLets see a simple example of a 2*1 Matrix to understand the above vector multiplications and using the Transformed vector for the second element in the above formulae.\n\n![img](https://i.imgur.com/KVAIo6b.png)\n\n#### [To Read further on Dot product in matrix notation](https://mathinsight.org/dot_product_matrix_notation)\n\n![img](https://i.imgur.com/sIBKvlv.png)\n\n\n### Note on using the unbiased estimator (1/n-1) instead of (1/n)\n\nThis is termed as Bessel's correction. Think of it this way. When we calculate the sample standard deviation from a sample of n values, we are using the sample mean already calculated from that same sample of n values. The calculated sample mean has already \"used up\" one of the \"degrees of freedom of variability\"(which is the mean itself) that is available in the sample. Only n-1 degrees of freedom of variability are left for the calculation of the sample standard deviation.\n\nFor further details on Bessel's Correction read the following\n\n- http://vortex.ihrc.fiu.edu/MET4570/members/Lectures/Lect05/m10divideby_nminus1.pdf\n- https://math.stackexchange.com/questions/61251/intuitive-explanation-of-bessels-correction\n\n---\n\nLet’s start with the below matrix.\n\n```python\nA = [[1, 3, 5],\n       [5, 4, 1],\n       [3, 8, 6]]\n```\n\nThat is our original dataframe is of the below form.\n\n![img](https://i.imgur.com/UMS11Dc.png)\n\n### We will calculate the covariance between the first and the third column vectors (x1 and x3 features):\n\n![img](https://i.imgur.com/J2A8cos.png)\n\n![img](https://i.imgur.com/BZbzA1m.png[/img])\n\nSo, that’s the value (-2.67 ) of the covariance matrix.\n\nNow use numpy to do build the full Covariance Matrix. Importantly, here I want NumPy to use the columns as vectors, so the parameter `rowvar=False` has to be used. Also, `bias=True` divides by n and not by n-1.","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\n\nA = np.array([[1, 3, 5], [5, 4, 1], [3, 8, 6]])\n# A = np.array([[ 3, 5], [4, 1], [8, 6]])\nprint(A)\n\n# array([[1, 3, 5],\n#        [5, 4, 1],\n#        [3, 8, 6]])\n\ncov_matrix = np.cov(A, rowvar=False, bias=True )\ncov_matrix\n\n# array([[ 2.66666667, 0.66666667, -2.66666667],\n#        [ 0.66666667, 4.66666667, 2.33333333],\n#        [-2.66666667, 2.33333333, 4.66666667]])","metadata":{"jupyter":{"outputs_hidden":false},"pycharm":{"name":"#%%\n"},"collapsed":false,"execution":{"iopub.status.busy":"2021-06-17T16:26:37.034755Z","iopub.execute_input":"2021-06-17T16:26:37.035087Z","iopub.status.idle":"2021-06-17T16:26:37.075024Z","shell.execute_reply.started":"2021-06-17T16:26:37.035053Z","shell.execute_reply":"2021-06-17T16:26:37.074121Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### The diagonal elements of this matrix are the variances of the variables, and the off-diagonal elements are the covariances between the variables.\n\nIn below image all green highlighted are the variances and the rest (red highlighted) are the co-variances\n\n![img](https://i.imgur.com/8PJ5lMV.png)\n\n### Note, the Dimension of the Co-Variance Matrix will always be a Square Matrix -\n\nA Square Matrix is one with same number of rows and columns). Because they capture the covariance between any Variable and any other Variable. So we need all the Variables (Features in a dataset) on the columns and also all the variables on the rows in the resultant Co-Variance Matrix. Thats why in the above image I have put all of x1, x2, x3 in both the columns and rows.\n\n#### So, m-dimensional data will result in mxm covariance matrix\n\nBasically for three variables (or 3 dimensions of a dataset or 3 features of a dataset) X, Y and Z we will get the Covariance Matrix as following\n\n![img](https://i.imgur.com/AAq7ZC1.png)\n\n![img](https://i.imgur.com/j0vnWlR.png)\n\nThus,\n\n- 2.66666667 is the variance of the x1 variable,\n\n- 0.66666667 is the covariance between x1 and x2 variables,\n\n- -2.66666667 is the covariance between x1 and x3 variables,\n\n- 4.66666667 is the variance of x2,\n\n- 2.33333333 is the covariance between x2 and x3 and\n\n- 4.66666667 is the variance of x3.\n\n#### The mean vector is often referred to as the centroid and the variance-covariance matrix as the dispersion or dispersion matrix. Also, the terms variance-covariance matrix and covariance matrix are used interchangeably.\n\n## Properties of Co-Variance Matrix\n\n\n\n---\n\n## Principal Component Analysis\n\nPCA stands for Principal Component Analysis. Here is what Wikipedia says about PCA.\n\n\"Given a collection of points in two, three, or higher dimensional space, a “best fitting” line can be defined as one that minimizes the average squared distance from a point to the line. The next best-fitting line can be similarly chosen from directions perpendicular to the first. Repeating this process yields an orthogonal basis in which different individual dimensions of the data are uncorrelated. These basis vectors are called principal components, and several related procedures principal component analysis (PCA).\"\n\nSo, PCA means extracting the components (features) which are most effective in forecasting the target variable in a dataset, and discarding the redundant features. Thus, we calculate the Principal Components to achieve Dimensionality reduction.\n\nPCA ( aka Karhunen–Loève transform in Mathematics ) is among the oldest and most widely used methods for dimensionality reduction [105]. The assumption underlying PCA, as well as any dimensionality reduction technique, is that the observed data are generated by a system or process that is driven by a (relatively) small number of latent (not directly observed) variables. The goal is to learn this latent structure.\n\nHere's a visual representation of PCA being applied to a simple 2-D data set (left, in blue)\n\n![img](https://i.imgur.com/lKK6Gts.png)\n\nAs we can see above dimension reduction via PCA retains much of the structure of the original data. The ideal subspace is shown in black in the left panel, and the data projected onto this subspace is shown in blue on the right. (bottom panels)\n\nBefore using PCA, we must standardize our dataset. The method won’t work if we have entirely different scales for our data. Standardization helps to avoid biased outcomes.\n\nFor example, suppose there are three variables – v1 in a 1-10 range, v2 in a 10-100 range, and v3 in a 100,000 to 1,000,000 range. If we just go ahead and compute an output using these predictors, we’ll get a heavily biased result because the third variable will have a disproportionately large impact on the output value.\n\n#### Step-1 So, Before applying PCA, we must ensure that all our attributes (dimensions) are centered around zero and have a standard deviation of 1. So, the first thing we do is calculate the means for all of them. Then, we center the values in each column (to do so we subtract the mean column value) and calculate the covariance matrix for the resulting centered matrix.\n\nNow, we can calculate our first principal component.\n\n#### Step-2 - Next, we utilize one of the methods for breaking up matrices (eigendecomposition or singular value decomposition) to turn our matrix into a list of eigenvectors that will define directions of axes in the new subspace and eigenvalues that will show the magnitude of those directions.\n\nWe sort the vectors by their eigenvalues in decreasing order and thus get a ranking of the principal components.\n\n#### Step-3 - Now finally, we can ditch some of the dimensions and project our dataset into a reduced subspace without losing important information about the original dataset. And since new axes capture the maximum variance in the data, any clustering algorithm we’re going to apply next will have an easier time grouping the data instances.\n\nLets look at it visually,\n\nImagine this is our dataset that we’re trying to cluster; we only have two dimensions.\n\n![img](https://i.imgur.com/CUWkBzi.png)\n\nIf we project the data onto the horizontal axis (our attribute 1) we won’t see much spread; it will show a nearly equal distribution of the observations.\n\n![img](https://i.imgur.com/s5O8hYn.png)\n\nAttribute 2, obviously, isn’t hugely helpful either.\n\n![img](https://i.imgur.com/6VnyFk7.png)\n\nAs the data points in this case are spreading diagonally so we need a new line that would better capture this.\n\n![img](https://i.imgur.com/B9MtmIg.png)\n\n#### This axis is our first Principle Component (PC) – a new direction (we can imagine it to be an attribute) that maximizes the variance of the data (and thus the clusters become much more obvious.) Besides maximizing the spread, this first PC sits through the direction in the data that minimizes the distances between all the points and the new axis.\n\n#### The second PC must represent the second maximum amount of variance; it’s going to be a line that’s orthogonal to our first axis.\n\n![img](https://i.imgur.com/ocDf36C.png)\n\nBecause of the underlying PCA-Math being based on eigenvectors and eigenvalues, new principal components will always come out orthogonal to the ones before them. **Eigenvectors**, simply, are vectors that aren’t knocked off of their span by a linear transformation; they can hold on to their original direction while being stretched, shrunk or reversed. Eigenvalues are factors by which these special vectors are scaled.\n\n\n\n## Key concept of Preserving the Variance by Projection of Data\n\n#### Before we can project the training set onto a lower-dimensional hyperplane, we first need to choose the right hyperplane. For example, in the below figure, a simple 2D dataset is represented on the left, along with three different axes (i.e., 1D hyperplanes). On the right is the result of the projection of the dataset onto each of these axes. As we can see, the projection onto the solid line preserves the maximum variance, while the projection onto the dotted line preserves very little variance and the projection onto the dashed line preserves an intermediate amount of variance.\n\n![img](https://i.imgur.com/Fp8kqgJ.png)\n\n#### It seems reasonable to select the axis that preserves the maximum amount of variance, as it will most likely lose less information than the other projections. Another way to justify this choice is that it is the axis that minimizes the mean squared distance between the original dataset and its projection onto that axis. This is the basic idea behind PCA.\n\nPCA identifies the axis that accounts for the largest amount of variance in the train-ing set. In above figure, it is the solid line. It also finds a second axis, orthogonal to the first one, that accounts for the largest amount of remaining variance. In this 2D example there is no choice: it is the dotted line (i.e. the orthogonal one). If it were a higher-dimensional data‐set, PCA would also find a third axis, orthogonal to both previous axes, and a fourth, a fifth, and so on—as many axes as the number of dimensions in the dataset. The ith axis is called the ith principal component (PC) of the data. In above figure, the first PC is the axis on which vector c1 lies, and the second PC is the axis on which vector c2 lies. In case of 3-D dataset the first two PCs are the orthogonal axes on which the two arrows lie, on the plane, and the third PC is the axis orthogonal to that plane, like in the below figure.\n\n![img](https://i.imgur.com/wEHwUMl.png)\n\n\nFor each principal component, PCA finds a zero-centered unit vector pointing in the direction of the PC. Since two opposing unit vectors lie on the same axis, the direction of the unit vectors returned by PCA is not stable: if you perturb the training set slightly and run PCA again, the unit vectors may point in the oppo‐\nsite direction as the original vectors. However, they will generally still lie on the same axes. In some cases, a pair of unit vectors may even rotate or swap (if the variances along these two axes are close), but the plane they define will generally remain the same.\n\n\n## Eigenvalues and Eigenvectors\nMany problems present themselves in terms of an eigenvalue problem:\n\n###  A·v=λ·v\n\nIn this equation A is an n-by-n matrix, v is a non-zero n-by-1 vector and λ is a scalar (which may be either real or complex).  Any value of λ for which this equation has a solution is known as an eigenvalue of the matrix A.\n\nTo begin, let $v$ be a vector (shown as a point) and $A$ be a matrix with columns $a_1$ and $a_2$ (shown as arrows). If we multiply $v$ by $A$, then $A$ sends $v$ to a new vector $Av$.\n\n![img](https://i.imgur.com/xBy93Vd.png)\n\n\nIf you can draw a line through the three points $(0,0)$, $v$ and $Av$, then $Av$ is just $v$ multiplied by a number $\\lambda$; that is, $Av = \\lambda v$. In this case, we call $\\lambda$ an eigenvalue and $v$ an eigenvector.\n\nThis means that the linear transformation A on vector **v** is completely defined by λ.\n\nIn the linear algebra literature and software, it is often a convention that eigenvalues are sorted in descending order, so that the largest\neigenvalue and associated eigenvector are called the first eigenvalue and its associated eigenvector, and the second largest called the second eigenvalue and its associated eigenvector, and so on.\n\nAn eigenvector is a vector whose direction remains unchanged when a linear transformation is applied to it.\n\n\n### Usefullness of Eigenvalues and Eigenvectors\n\nIf you keep multiplying $v$ by $A$, you get a sequence ${ v, Av, A^2v,}$ etc. Eigenspaces attract that sequence and eigenvalues tell you whether it ends up at $(0,0)$ or far away. Therefore, eigenvectors/values tell us about systems that evolve step-by-step.\n\n![img]https://i.imgur.com/1hXen7o.png[/img]\n\n### Properties of eigenvalues and eigenvectors\n\n- Let A be a $K\times K$ square matrix. A scalar $lambda $ is an eigenvalue of A if and only if it is an eigenvalue of $A^{intercal }$.\n\n-  For symmetric matrix, the eigenvectors are always orthogonal\n\nIn general, for any matrix, the eigenvectors are NOT always orthogonal. But for a special type of matrix, symmetric matrix, the eigenvalues are always real and the corresponding eigenvectors are always orthogonal.\n\nFor any matrix M with n rows and m columns, M multiplies with its transpose, either M*M' or M'M, results in a symmetric matrix, so for this symmetric matrix, the eigenvectors are always orthogonal.\n\nIn the application of PCA, a dataset of n samples with m features is usually represented in a n* m matrix D. The variance and covariance among those m features can be represented by a m*m matrix D'*D, which is symmetric (numbers on the diagonal represent the variance of each single feature, and the number on row i column j represents the covariance between feature i and j). The PCA is applied on this symmetric matrix, so the eigenvectors are guaranteed to be orthogonal.\n\n\n#### During Steps PCA calculation steps we need the Eigenvectors as below\n\n- Step-A => Standardize the Columns\n\n- Step-B => Build the Covariance-Matrix S which is $(X^T * X)$\n\n- Step-C => Get the EigneVectors of this Covariance-Matrix S. These will be sorted from largest λ and downwards.\n\n- Step-D => And my **u1** is the Eigenvector **v1** which is the largest Eigenvector corresponding to the largest Eigenvalue\n\n![img](https://i.imgur.com/grOzxqv.png)\n\n### Significance of Eigenvectors for a 10-Dimensional dataset - Below is a Geometric interpretation\n\n#### The highest Eigenvector v1 (corresponding to the max lambda i.e. max Eigenvalue ) - is the vector representing the direction with the highest Variance (Principal Component). Then v2 is the direction with the second highest variance\n\n![img](https://i.imgur.com/Y4NLf1I.png)\n\n[Source-AppliedAI Course Youtube Video](https://www.youtube.com/watch?v=b0DHvcdIStE&feature=emb_logo&ab_channel=AppliedAICourse)\n\n\n### Significance of Eigenvalues (Lambdas) for a 2-Dimensional dataset - a Geometric interpretation\n\n#### A particular Lambda in a particular direction / Summation of All Lambdas - tells me the % of the Variances retained in that direction.\n\n![img](https://i.imgur.com/WCX268w.png)\n\nSo in the above picture, for the middle graph (representing a 2-d dataset) - lambda-1 retains (3/4)th or 75% of the variances.\n\n[Source-AppliedAI Course Youtube Video](https://www.youtube.com/watch?v=b0DHvcdIStE&feature=emb_logo&ab_channel=AppliedAICourse)\n\n\n#### That means the dot product of two eigenvectors of a Symmetric Matrix is always zero\n\n![img](https://i.imgur.com/lzlNX4Y.png)\n\n\n---\n\n## The Mathematics of PCA\n\n### Under the Maximum Variance Formulae\n\n### First here is the what the Projection Vector looks like geometrically, for point x_i and also the mathematical formula of point x_i on the Principle Component Vector u_1\n\n![img](https://i.imgur.com/WirAvCo.png)\n\nIn English the here the Math Objective of the Problem - assuming **u1** represents the Principle component Vector\n\n### Find u1, such that the the Variance of points x_i ( i going from 1 to n) projected onto u1 is maximized\n\nMathematically\n\n![img](https://i.imgur.com/E5mMcaD.png)\n\nA slightly more formally\n\n![img](https://i.imgur.com/teprtEQ.png)\n\n[Source of above Image](https://www.cs.toronto.edu/~urtasun/courses/CSC411/tutorial8.pdf)\n\n#### But note our original dataset Matrix has been centered or column-standardized, hence\n\n![img](https://i.imgur.com/yAj6dE6.png)\n\n### So the final Objective function becomes\n\n![img](https://i.imgur.com/ByyY9Wr.png)\n\n### So extending the above, our absolute final target is to find u_1 for which the above variance will be maximized.\n\n---\n\nDon’t confuse PCA with linear regression. PCA is to minimize the distance of the orthogonal projection, while linear regression is to minimize the distance on the y-axis.\n\n![img](https://i.imgur.com/x3cFb0o.png)\n\n[Source of above image](http://alexhwilliams.info/itsneuronalblog/2016/03/27/pca/)\n\n## Why is minimizing squared residuals equivalent to maximizing variance?\n\nTo ans this question I could not find a better explanations than has been described **[here in this blog](http://alexhwilliams.info/itsneuronalblog/2016/03/27/pca/)**\n\n![img](https://i.imgur.com/w7gM6lu.png)\n\n Where **w** is the projection of each datapoint onto the top principal component and the vector **c**  is the top principal component\n\n## Important Note - PCA assumes that the dataset is centered around the origin.\n\nAs we will see, Scikit-Learn’s PCA classes take care of centering the data for you. If you implement PCA yourself, or if you use other libraries, don’t forget to center the data first\n\n## Now I will Calculate the PCA for MNIST data quite manually using the steps described earlier, which are\n\n- Step-A => Standardize the Columns\n\n- Step-B => Build the Covariance-Matrix S which is $(X^T * X)$\n\n- Step-C => Get the EigneVectors of this Covariance-Matrix S. These will be sorted from largest λ and downwards. And my **u1** is the Eigenvector **v1** which is the largest Eigenvector corresponding to the largest Eigenvalue\n\n- Step-D => Now I need to project the original data sample on the plane formed by two principle eigenvectors by vector-vector multiplication.\n\n### The training dataset is 42,000 small square 28×28 pixel grayscale images of handwritten single digits between 0 and 9. The task is to classify a given image of a handwritten digit into one of 10 classes representing integer values from 0 to 9, inclusively.","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nsns.set_style(\"dark\")\nimport matplotlib.pyplot as plt\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn import decomposition\nfrom sklearn.preprocessing import StandardScaler","metadata":{"jupyter":{"outputs_hidden":false},"pycharm":{"name":"#%%\n"},"collapsed":false,"execution":{"iopub.status.busy":"2021-06-17T16:26:37.077485Z","iopub.execute_input":"2021-06-17T16:26:37.077998Z","iopub.status.idle":"2021-06-17T16:26:38.190139Z","shell.execute_reply.started":"2021-06-17T16:26:37.077953Z","shell.execute_reply":"2021-06-17T16:26:38.189331Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# In my local Machine, I ah only taking fewer rows for faster local training and development\n# In Kaggle comment-out below line\n# train_df_org = pd.read_csv('kaggle/input/digit-recognizer/train.csv',  nrows=1000)\n# test_df_org = pd.read_csv('kaggle/input/digit-recognizer/test.csv',  nrows=1000)\n\n# Instead, uncomment the below one, where the whole dataset is taken for traiing\ntrain_df_org = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\ntest_df_org = pd.read_csv('/kaggle/input/digit-recognizer/test.csv')\n\nprint(\"train_df_org shape \", train_df_org.shape) #  (48000, 785)\nprint(\"test_df_org shape \", test_df_org.shape)\n\ntrain_df_org.head()","metadata":{"jupyter":{"outputs_hidden":false},"pycharm":{"name":"#%%\n"},"collapsed":false,"execution":{"iopub.status.busy":"2021-06-17T16:26:38.19174Z","iopub.execute_input":"2021-06-17T16:26:38.192276Z","iopub.status.idle":"2021-06-17T16:26:43.265345Z","shell.execute_reply.started":"2021-06-17T16:26:38.192235Z","shell.execute_reply":"2021-06-17T16:26:43.264599Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# In test df there should not be any label column.\n# lets see it.\ntest_df_org.head()","metadata":{"jupyter":{"outputs_hidden":false},"pycharm":{"name":"#%%\n"},"collapsed":false,"execution":{"iopub.status.busy":"2021-06-17T16:26:43.268362Z","iopub.execute_input":"2021-06-17T16:26:43.268654Z","iopub.status.idle":"2021-06-17T16:26:43.28936Z","shell.execute_reply.started":"2021-06-17T16:26:43.268625Z","shell.execute_reply":"2021-06-17T16:26:43.288041Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In below the labels variable contains the labels and then from the 'train_df_org' (which contains all the actual image pixel) I will drop 'label' column","metadata":{}},{"cell_type":"code","source":"labels = train_df_org['label']\nlabels.shape\n","metadata":{"jupyter":{"outputs_hidden":false},"pycharm":{"name":"#%%\n"},"collapsed":false,"execution":{"iopub.status.busy":"2021-06-17T16:26:43.292599Z","iopub.execute_input":"2021-06-17T16:26:43.293013Z","iopub.status.idle":"2021-06-17T16:26:43.299601Z","shell.execute_reply.started":"2021-06-17T16:26:43.29297Z","shell.execute_reply":"2021-06-17T16:26:43.298589Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df_org.drop(['label'], axis=1, inplace=True)\ntrain_df_org.shape","metadata":{"jupyter":{"outputs_hidden":false},"pycharm":{"name":"#%%\n"},"collapsed":false,"execution":{"iopub.status.busy":"2021-06-17T16:26:43.30165Z","iopub.execute_input":"2021-06-17T16:26:43.302132Z","iopub.status.idle":"2021-06-17T16:26:43.390329Z","shell.execute_reply.started":"2021-06-17T16:26:43.302091Z","shell.execute_reply":"2021-06-17T16:26:43.389392Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Step-A: Data-preprocessing: Standardizing the train dataframe\n#### `(x - mean) / s.d`","metadata":{"jupyter":{"outputs_hidden":false},"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"standardized_data = StandardScaler().fit_transform(train_df_org)\nprint(standardized_data.shape)","metadata":{"jupyter":{"outputs_hidden":false},"pycharm":{"name":"#%%\n"},"collapsed":false,"execution":{"iopub.status.busy":"2021-06-17T16:26:43.392041Z","iopub.execute_input":"2021-06-17T16:26:43.392456Z","iopub.status.idle":"2021-06-17T16:26:44.285651Z","shell.execute_reply.started":"2021-06-17T16:26:43.392415Z","shell.execute_reply":"2021-06-17T16:26:44.284194Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Step-B: Now build the c-variance matrix which is :\n\n### $A^T * A$","metadata":{"jupyter":{"outputs_hidden":false},"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"sample_data = standardized_data\n\n# Matrix multiplication with numpy\ncovariance_matrix = np.matmul(sample_data.T, sample_data)\n\n# As the sample_data has 784 columns, so the co-variance matrix shape\n# should be 784 * 784\nprint('Shape of Co-variance matrix = ', covariance_matrix.shape)\n","metadata":{"jupyter":{"outputs_hidden":false},"pycharm":{"name":"#%%\n"},"collapsed":false,"execution":{"iopub.status.busy":"2021-06-17T16:26:44.286962Z","iopub.execute_input":"2021-06-17T16:26:44.287315Z","iopub.status.idle":"2021-06-17T16:26:44.734571Z","shell.execute_reply.started":"2021-06-17T16:26:44.287284Z","shell.execute_reply":"2021-06-17T16:26:44.73362Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Step-C => Get the EigneVectors of this Covariance-Matrix S for projecting onto a 2-D space. These Eigenvectors will be sorted by the value of λ.","metadata":{}},{"cell_type":"code","source":"from scipy.linalg import eigh\n\n# eigh function will give me the eigenvalues in ascending order with zero-index\n# subset_by_index is an optional and iterable parameter\n# If provided, this two-element iterable defines the start and the end indices\n# of the desired eigenvalues (ascending order and 0-indexed).\n# To return only the second smallest to fifth smallest eigenvalues, [1, 4] is used.\n# [n-3, n-1] returns the largest three.\n# Only available with “evr”, “evx”, and “gvx” drivers. The entries are directly converted to integers via int().\n\n# Needed to comment out below line containing 'subset_by_index' parameter, as it would fail\n# in Kaggle, probably because of the driver requirement.\n# eigenvalues, eigenvectors = eigh(covariance_matrix, subset_by_index=(782, 783) )\n\n# instead using 'eigvals' which is deprecated per doc\neigenvalues, eigenvectors = eigh(covariance_matrix, eigvals=(782, 783) )\nprint('Shape of Eigenvectors ', eigenvectors.shape )\n\n# In above I am only passing 782 and 783 as eigvalues becuase\n# Our target is to reduce the 784-dimension data to 2-dimension data and plot it.\n# So we need the top 2 eigvalues.","metadata":{"jupyter":{"outputs_hidden":false},"pycharm":{"name":"#%%\n"},"collapsed":false,"execution":{"iopub.status.busy":"2021-06-17T16:26:44.736302Z","iopub.execute_input":"2021-06-17T16:26:44.736703Z","iopub.status.idle":"2021-06-17T16:26:44.833268Z","shell.execute_reply.started":"2021-06-17T16:26:44.736664Z","shell.execute_reply":"2021-06-17T16:26:44.832317Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So you have the principal components. They are eigenvectors of the _covariance_ matrix $X^T X$.\n\nA way to retrieve the eigenvalues from there is to apply this matrix to each principal components and project the results onto the component.\nLet v_1  be the first principal component and lambda_1 the associated eigenvalue. We have:\n[![eq][1]][1] and thus:\n[![eq2][2]][2] since [![eq3][3]][3]. (x, y)  the scalar product of vectors x and y.\n\n[1]: http://i.stack.imgur.com/6OApA.gif\n[2]: http://i.stack.imgur.com/iCZcI.gif\n[3]: http://i.stack.imgur.com/8pGd1.gif","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"# now converting the eigenvectors into (2, d) shape for ease of computation further ahead\neigenvectors = eigenvectors.T\n","metadata":{"jupyter":{"outputs_hidden":false},"pycharm":{"name":"#%%\n"},"collapsed":false,"execution":{"iopub.status.busy":"2021-06-17T16:26:44.834556Z","iopub.execute_input":"2021-06-17T16:26:44.83507Z","iopub.status.idle":"2021-06-17T16:26:44.839278Z","shell.execute_reply.started":"2021-06-17T16:26:44.835029Z","shell.execute_reply":"2021-06-17T16:26:44.838374Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" #### eigenvectors[0] represents the eigenvector corresponding to 1st Principal Component\n #### eigenvectors[1] represents the eigenvector corresponding to 2nd Principal Component\n\nNow finally our target which was to reduce the 784-dimension data to 2-dimension data and plot it.\n\n#### Step*D => Now I need to project the original data sample on the plane formed by two principle eigenvectors by vector-vector multiplication.\n\n#### Simplest Formulate to Refer for Vector Projection\n\n![img](https://i.imgur.com/enenSdD.png)\n\n[Source](https://users.math.msu.edu/users/gnagy/teaching/11-fall/mth234/L03-234.pdf)","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"projected_vec = np.matmul(eigenvectors, sample_data.T)\nprojected_vec.shape\n# Now we will see this is a 2-dimensional data","metadata":{"jupyter":{"outputs_hidden":false},"pycharm":{"name":"#%%\n"},"collapsed":false,"execution":{"iopub.status.busy":"2021-06-17T16:26:44.840679Z","iopub.execute_input":"2021-06-17T16:26:44.841258Z","iopub.status.idle":"2021-06-17T16:26:44.878021Z","shell.execute_reply.started":"2021-06-17T16:26:44.841207Z","shell.execute_reply":"2021-06-17T16:26:44.877064Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now I will use vstack() function to Stack arrays in sequence vertically (row wise). This function makes most sense for arrays with up to 3 dimensions. For instance, for pixel-data with a height (first axis), width (second axis), and r/g/b channels (third axis).\n\n```python\na = np.array([1, 2, 3])\nb = np.array([2, 3, 4])\nnp.vstack((a,b))\narray([[1, 2, 3],\n       [2, 3, 4]])\n```\n","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"# appending label to the 2-d projected data\nprojected_vec = np.vstack((projected_vec, labels)).T\nprojected_vec.shape","metadata":{"jupyter":{"outputs_hidden":false},"pycharm":{"name":"#%%\n"},"collapsed":false,"execution":{"iopub.status.busy":"2021-06-17T16:26:44.879738Z","iopub.execute_input":"2021-06-17T16:26:44.880124Z","iopub.status.idle":"2021-06-17T16:26:44.887675Z","shell.execute_reply.started":"2021-06-17T16:26:44.880082Z","shell.execute_reply":"2021-06-17T16:26:44.886514Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The projected_vec at this point is still in array form and so we can not apply\n# head() function to it directly and also for plotting purpose, I need to make a dataframe from it.\n# creating a new dataframe from plotting the labeled points\npca_dataframe = pd.DataFrame(data=projected_vec, columns=('1st_principal_comp', \"2nd_principal_comp\", 'labels') )\npca_dataframe.head()\n","metadata":{"jupyter":{"outputs_hidden":false},"pycharm":{"name":"#%%\n"},"collapsed":false,"execution":{"iopub.status.busy":"2021-06-17T16:26:44.889472Z","iopub.execute_input":"2021-06-17T16:26:44.88989Z","iopub.status.idle":"2021-06-17T16:26:44.905877Z","shell.execute_reply.started":"2021-06-17T16:26:44.889851Z","shell.execute_reply":"2021-06-17T16:26:44.904536Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So in above each row is x_i which was originally 784 dimensions and now they are in 2-dimensions by applying PCA.","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"sns.FacetGrid(pca_dataframe, hue='labels', size=6).map(plt.scatter, '1st_principal_comp', \"2nd_principal_comp\").add_legend()\nplt.show()","metadata":{"jupyter":{"outputs_hidden":false},"pycharm":{"name":"#%%\n"},"collapsed":false,"execution":{"iopub.status.busy":"2021-06-17T16:26:44.908107Z","iopub.execute_input":"2021-06-17T16:26:44.909019Z","iopub.status.idle":"2021-06-17T16:26:45.724561Z","shell.execute_reply.started":"2021-06-17T16:26:44.908969Z","shell.execute_reply":"2021-06-17T16:26:45.723538Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And we can see some degree of classifications among the digits. For example, all zeros are on the top left corner. All the sixes (oranges) are towards the bottom between 0 and 4 on the x-axis.\n\n## Now deriving PCA with scikit-learn's PCA function","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"pca = decomposition.PCA()\npca.n_components = 2\npca_data_with_scikit = pca.fit_transform(standardized_data)\npca_data_with_scikit.shape\n","metadata":{"jupyter":{"outputs_hidden":false},"pycharm":{"name":"#%%\n"},"collapsed":false,"execution":{"iopub.status.busy":"2021-06-17T16:26:45.725768Z","iopub.execute_input":"2021-06-17T16:26:45.726106Z","iopub.status.idle":"2021-06-17T16:26:47.158107Z","shell.execute_reply.started":"2021-06-17T16:26:45.72607Z","shell.execute_reply":"2021-06-17T16:26:47.157377Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pca_data_with_scikit = np.vstack((pca_data_with_scikit.T, labels)).T\npca_data_with_scikit.shape","metadata":{"jupyter":{"outputs_hidden":false},"pycharm":{"name":"#%%\n"},"collapsed":false,"execution":{"iopub.status.busy":"2021-06-17T16:26:47.159357Z","iopub.execute_input":"2021-06-17T16:26:47.159716Z","iopub.status.idle":"2021-06-17T16:26:47.168044Z","shell.execute_reply.started":"2021-06-17T16:26:47.159675Z","shell.execute_reply":"2021-06-17T16:26:47.166806Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_PCA_scikit = pd.DataFrame(data=pca_data_with_scikit, columns=('f1_PC', 'f2_PC', 'labels'))\ndf_PCA_scikit.head()","metadata":{"jupyter":{"outputs_hidden":false},"pycharm":{"name":"#%%\n"},"collapsed":false,"execution":{"iopub.status.busy":"2021-06-17T16:26:47.170166Z","iopub.execute_input":"2021-06-17T16:26:47.170676Z","iopub.status.idle":"2021-06-17T16:26:47.186611Z","shell.execute_reply.started":"2021-06-17T16:26:47.170629Z","shell.execute_reply":"2021-06-17T16:26:47.185558Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.FacetGrid(df_PCA_scikit, hue='labels', size=6).map(plt.scatter, 'f1_PC', 'f2_PC').add_legend()\nplt.show()","metadata":{"jupyter":{"outputs_hidden":false},"pycharm":{"name":"#%%\n"},"collapsed":false,"execution":{"iopub.status.busy":"2021-06-17T16:26:47.188429Z","iopub.execute_input":"2021-06-17T16:26:47.189083Z","iopub.status.idle":"2021-06-17T16:26:48.055379Z","shell.execute_reply.started":"2021-06-17T16:26:47.189036Z","shell.execute_reply":"2021-06-17T16:26:48.054251Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And I am getting almost similar plotting that we got with the manual steps to calculate Principle Components. Main difference with sickit-learn the axes are reversed.\n\n\n## Plot to show how Variance Retained increases as we increase the Number of Principle components","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"print('Shape of standardized_data ', standardized_data.shape)\n\npca.n_components = 784\npca_data_scikit_2 = pca.fit_transform(standardized_data)\npercent_variance_retained = pca.explained_variance_ / np.sum(pca.explained_variance_)\ncumulative_variance_retained = np.cumsum(percent_variance_retained)","metadata":{"jupyter":{"outputs_hidden":false},"pycharm":{"name":"#%%\n"},"collapsed":false,"execution":{"iopub.status.busy":"2021-06-17T16:26:48.059338Z","iopub.execute_input":"2021-06-17T16:26:48.05975Z","iopub.status.idle":"2021-06-17T16:26:54.783191Z","shell.execute_reply.started":"2021-06-17T16:26:48.059709Z","shell.execute_reply":"2021-06-17T16:26:54.78204Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(1, figsize=(10, 6))\nplt.clf()\nplt.plot(cumulative_variance_retained, linewidth=2)\nplt.axis('tight')\nplt.grid()\nplt.xlabel('Number of Components')\nplt.ylabel('Cumulative variance Retained')\nplt.show()","metadata":{"jupyter":{"outputs_hidden":false},"pycharm":{"name":"#%%\n"},"collapsed":false,"execution":{"iopub.status.busy":"2021-06-17T16:26:54.784668Z","iopub.execute_input":"2021-06-17T16:26:54.785192Z","iopub.status.idle":"2021-06-17T16:26:55.190003Z","shell.execute_reply.started":"2021-06-17T16:26:54.785151Z","shell.execute_reply":"2021-06-17T16:26:55.189242Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So above was all about Dimensionality Reduction and PCA\n\n## Now for the Competition I will run a CNN Model on the MNIST dataset\n","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"import tensorflow as tf\n\nfrom tensorflow.keras import layers\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras import metrics\nfrom tensorflow.keras import backend as K\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Dense, Dropout, Lambda, Flatten, BatchNormalization\nfrom tensorflow.keras.layers import Conv2D, MaxPool2D, AvgPool2D\nfrom tensorflow.keras.optimizers import Adadelta\nfrom keras.utils.np_utils import to_categorical\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.callbacks import ReduceLROnPlateau\nfrom tensorflow.keras.callbacks import LearningRateScheduler","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2021-06-17T16:26:55.191479Z","iopub.execute_input":"2021-06-17T16:26:55.19182Z","iopub.status.idle":"2021-06-17T16:27:00.142723Z","shell.execute_reply.started":"2021-06-17T16:26:55.191782Z","shell.execute_reply":"2021-06-17T16:27:00.141903Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df_org = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\n\n# Now dataframe with only 'label' column\nmnist_train_label = train_df_org.loc[:, \"label\"]\n\n\nmnist_train_df = train_df_org.loc[:, \"pixel0\":]\n\n","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2021-06-17T16:27:00.14398Z","iopub.execute_input":"2021-06-17T16:27:00.144321Z","iopub.status.idle":"2021-06-17T16:27:02.615123Z","shell.execute_reply.started":"2021-06-17T16:27:00.144283Z","shell.execute_reply":"2021-06-17T16:27:02.614135Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Now convert the dataframes into numpy arrays\nmnist_train_array = np.array(mnist_train_df)\nmnist_train_label_array = np.array(mnist_train_label)\n\n# Now need to do a little reshaping to get the structure which the model will expect to receive going forward\nmnist_train_array = mnist_train_array.reshape(mnist_train_array.shape[0], 28, 28, 1)\nprint(mnist_train_array.shape, mnist_train_label_array.shape)\nprint(mnist_train_label_array)","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2021-06-17T16:27:02.617306Z","iopub.execute_input":"2021-06-17T16:27:02.617938Z","iopub.status.idle":"2021-06-17T16:27:02.727556Z","shell.execute_reply.started":"2021-06-17T16:27:02.617885Z","shell.execute_reply":"2021-06-17T16:27:02.726572Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## CNN Model Building\nTraining a neural network with one input layer, one hidden layer and one output layer for learning the digits in images.\n\n#### Encoding the labels and making them as the class value and finally converting them as categorical values.\n\nUsing the method **to_categorical()**, a numpy array (or) a vector which has integers that represent different categories, can be converted into a numpy array (or) a matrix which has binary values and has columns equal to the number of categories in the data. For example, label ‘3’ should be converted to a vector [0, 0, 0, 1, 0, 0, 0, 0, 0, 0] for model building. This function returns a matrix of binary values (either ‘1’ or ‘0’). It has number of rows equal to the length of the input vector and number of columns equal to the number of classes.\n\nSo the whole mnist_train_label would look like below\n\n```python\n[[0. 1. 0. ... 0. 0. 0.]\n [1. 0. 0. ... 0. 0. 0.]\n [0. 1. 0. ... 0. 0. 0.]\n ...\n [0. 0. 0. ... 0. 0. 1.]\n [0. 0. 0. ... 0. 0. 0.]\n [0. 0. 0. ... 0. 0. 0.]]\n```","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"num_of_digit_classes = mnist_train_label_array.max() - mnist_train_label_array.min() + 1\nmnist_train_label_array = to_categorical(mnist_train_label_array, num_classes=num_of_digit_classes)\nprint('Shape of ytrain after encoding and converting to categorical values ', mnist_train_label_array.shape)\nprint(mnist_train_label_array)\n","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2021-06-17T16:27:02.729445Z","iopub.execute_input":"2021-06-17T16:27:02.729828Z","iopub.status.idle":"2021-06-17T16:27:02.739932Z","shell.execute_reply.started":"2021-06-17T16:27:02.729785Z","shell.execute_reply":"2021-06-17T16:27:02.738824Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def run_model(input_shape=(28, 28, 1)):\n    model = Sequential()\n    model.add(Conv2D(32, kernel_size = 3, activation='relu', input_shape = input_shape))\n    model.add(BatchNormalization())\n    model.add(Conv2D(32, kernel_size = 3, activation='relu'))\n    model.add(BatchNormalization())\n    model.add(Conv2D(32, kernel_size = 5, strides=2, padding='same', activation='relu'))\n    model.add(BatchNormalization())\n    model.add(Dropout(0.4))\n\n    model.add(Conv2D(64, kernel_size = 3, activation='relu'))\n    model.add(BatchNormalization())\n    model.add(Conv2D(64, kernel_size = 3, activation='relu'))\n    model.add(BatchNormalization())\n    model.add(Conv2D(64, kernel_size = 5, strides=2, padding='same', activation='relu'))\n    model.add(BatchNormalization())\n    model.add(Dropout(0.4))\n\n    model.add(Conv2D(128, kernel_size = 4, activation='relu'))\n    model.add(BatchNormalization())\n    model.add(Flatten())\n    model.add(Dropout(0.4))\n    model.add(Dense(10, activation='softmax'))\n    return model\n\ndef run_model_compilation(model, optimizer='adam', loss='categorical_crossentropy'):\n    model.compile(optimizer=optimizer, loss=loss, metrics=['accuracy'])\n\ndef run_model_training(model, train, test, epochs, split):\n    history=model.fit(train, test, shuffle=True, epochs=epochs, validation_split=split )\n    return history","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2021-06-17T16:27:02.741611Z","iopub.execute_input":"2021-06-17T16:27:02.742302Z","iopub.status.idle":"2021-06-17T16:27:02.760311Z","shell.execute_reply.started":"2021-06-17T16:27:02.74225Z","shell.execute_reply":"2021-06-17T16:27:02.759155Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Neural Network Model's parameters explained\n\n### Sequential() -\n\nA sequential model, which is a linear stack of layers as the name suggests, allows you to create models layer-by-layer in a step-by-step fashion. Keras Sequential API is by far the easiest way to get up and running with Keras, but it is s also the most limited — you cannot create models that:\n\nShare layers\nHave branches (at least not easily)\nHave multiple inputs\nHave multiple outputs\n\nThe model needs to know what input shape it should expect. For this reason, the first layer in a sequential model (and only the first, because following layers can do automatic shape inference) needs to receive information about its input shape.\n\nAs illustrated in the example above, this is done by passing an input_shape argument to the first layer. This is a list of integers or NULL entries, where NULL indicates that any positive integer may be expected. In input_shape, the batch dimension is not included.\n\nIf you ever need to specify a fixed batch size for your inputs (this is useful for stateful recurrent networks), you can pass a batch_size argument to a layer. If you pass both batch_size=32 and input_shape=c(6, 8) to a layer, it will then expect every batch of inputs to have the batch shape (32, 6, 8).\n\n### The activation parameter\n\nThe activation parameter to the Conv2D class is simply a convenience parameter, allowing you to supply a string specifying the name of the activation function you want to apply after performing the convolution.\n\n### ReLU (Rectified Linear Unit) Activation Function\n\n- The ReLU is the most used activation function in the world right now.Since, it is used in almost all the convolutional neural networks or deep learning.\n\n### The ReLU function is f(x)=max(0,x).\n\n![img](https://i.imgur.com/2ezpnE3.png)\n\nAs you can see, the ReLU is half rectified (from bottom). f(z) is zero when z is less than zero and f(z) is equal to z when z is above or equal to zero.\nRange: [ 0 to infinity)\nThe function and its derivative both are monotonic.\nBut the issue is that all the negative values become zero immediately which decreases the ability of the model to fit or train from the data properly. That means any negative input given to the ReLU activation function turns the value into zero immediately in the graph, which in turns affects the resulting graph by not mapping the negative values appropriately.\n\nOne way ReLUs improve neural networks is by speeding up training. The gradient computation is very simple (either 0 or 1 depending on the sign of x). Also, the computational step of a ReLU is easy: any negative elements are set to 0.0 -- no exponentials, no multiplication or division operations.\n\nGradients of logistic and hyperbolic tangent networks are smaller than the positive portion of the ReLU. This means that the positive portion is updated more rapidly as training progresses. However, this comes at a cost. The 0 gradient on the left-hand side is has its own problem, called \"dead neurons,\" in which a gradient update sets the incoming values to a ReLU such that the output is always zero; modified ReLU units such as ELU (or Leaky ReLU, or PReLU, etc.) can ameliorate this.\n\n### BatchNormalization in Keras -\n\nThe layer will transform inputs so that they are standardized, meaning that they will have a mean of zero and a standard deviation of one.\n\nDuring training, the layer will keep track of statistics for each input variable and use them to standardize the data.\n\nFurther, the standardized output can be scaled using the learned parameters of Beta and Gamma that define the new mean and standard deviation for the output of the transform. The layer can be configured to control whether these additional parameters will be used or not via the “center” and “scale” attributes respectively. By default, they are enabled.\n\nThe statistics used to perform the standardization, e.g. the mean and standard deviation of each variable, are updated for each mini batch and a running average is maintained.\n\nA “momentum” argument allows you to control how much of the statistics from the previous mini batch to include when the update is calculated. By default, this is kept high with a value of 0.99. This can be set to 0.0 to only use statistics from the current mini-batch, as described in the original paper.\n\n### Keras Conv2D is a 2D Convolution Layer\n\nThis layer creates a convolution kernel that is wind with layers input which helps produce a tensor of outputs.\n\nKernel: In image processing kernel is a convolution matrix or masks which can be used for blurring, sharpening, embossing, edge detection, and more by doing a convolution between a kernel and an image.\n\n#### Filters\n\nThe first required Conv2D parameter is the number of filters that the convolutional layer will learn.\n\n![img](https://i.imgur.com/j38MUoW.png)\n\nThe Keras Conv2D parameter, filters determines the number of kernels to convolve with the input volume. Each of these operations produces a 2D activation map.\n\nLayers early in the network architecture (i.e., closer to the actual input image) learn fewer convolutional filters while layers deeper in the network (i.e., closer to the output predictions) will learn more filters.\n\nConv2D layers in between will learn more filters than the early Conv2D layers but fewer filters than the layers closer to the output. Let’s go ahead and take a look at an example:\n\nSo filter is a mandatory Conv2D parameter. It is an integer value and also determines the number of output filters in the convolution.\n\n`model.add(Conv2D(32, kernel_size = 3, activation='relu', input_shape = input_shape))`\n\nIn above line we are learning a total of 32 filters.\nAs far as choosing the appropriate value for no. of filters, it is always recommended to use powers of 2 as the values.\n\nRead more on Keras Conv2D [here](https://missinglink.ai/guides/keras/keras-conv2d-working-cnn-2d-convolutions-keras/) - Some excerpts from this\n\n#### What is a 2D Convolution Layer, the Convolution Kernel and its Role in CNN Image Classification\n\nBriefly, some background. A convolution layer “scans” A source image with a filter of, for example, 5×5 pixels, to extract features which may be important for classification. This filter is also called the convolution kernel. The kernel also contains weights, which are tuned in the training of the model to achieve the most accurate predictions.\n\nIn a 5×5 kernel, for each 5×5 pixel region, the model computes the dot products between the image pixel values and the weights defined in the filter.\n\nA 2D convolution layer means that the input of the convolution operation is three-dimensional, for example, a color image which has a value for each pixel across three layers: red, blue and green. However, it is called a “2D convolution” because the movement of the filter across the image happens in two dimensions. The filter is run across the image three times, once for each of the three layers.\n\nAfter the convolution ends, the features are downsampled, and then the same convolutional structure repeats again. At first, the convolution identifies features in the original image (for example in a cat, the body, legs, tail, head), then it identifies sub-features within smaller parts of the image (for example, within the head, the ears, whiskers, eyes). Eventually, this process is meant to identify the essential features that can help classify the image.\n\n### The kernel_size.\n\nThe kernel_size must be an odd integer.\n\nTypical values for kernel_size include: (1, 1) , (3, 3) , (5, 5) , (7, 7) . It’s rare to see kernel sizes larger than 7×7.\n\nSo, when do you use each?\n\nIf your input images are greater than 128×128 you may choose to use a kernel size > 3 to help (1) learn larger spatial filters and (2) to help reduce volume size.\n\n### strides\n\nThe strides parameter is a 2-tuple of integers, specifying the “step” of the convolution along the x and y axis of the input volume.\n\nThe strides value defaults to (1, 1) , implying that:\n\nA given convolutional filter is applied to the current location of the input volume\nThe filter takes a 1-pixel step to the right and again the filter is applied to the input volume\nThis process is performed until we reach the far-right border of the volume in which we move our filter one pixel down and then start again from the far left\nTypically you’ll leave the strides parameter with the default (1, 1) value; however, you may occasionally increase it to (2, 2) to help reduce the size of the output volume (since the step size of the filter is larger).\n\nTypically you’ll see strides of 2×2 as a replacement to max pooling\n\n### padding\n\nThe padding parameter to the Keras Conv2D class can take on one of two values: valid or same .\n\nWith the valid parameter the input volume is not zero-padded and the spatial dimensions are allowed to reduce via the natural application of convolution.\n\nIf you instead want to preserve the spatial dimensions of the volume such that the output volume size matches the input volume size, then you would want to supply a value of same for the padding :\n\n### Dropout\n\nFrom Tensorflow [Docs](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dropout)\n\nThe parameter `rate` - Float between 0 and 1. Fraction of the input units to drop.\n\nThe Dropout layer randomly sets input units to 0 with a frequency of rate at each step during training time, which helps prevent overfitting. Inputs not set to 0 are scaled up by 1/(1 - rate) such that the sum over all inputs is unchanged.\n\nNote that the Dropout layer only applies when training is set to True such that no values are dropped during inference. When using model.fit, training will be appropriately set to True automatically, and in other contexts, you can set the kwarg explicitly to True when calling the layer.\n\nThe following taken from this very informative [blog](https://towardsdatascience.com/understanding-and-implementing-dropout-in-tensorflow-and-keras-a8a3a02c1bfa)\n\nNeural networks have hidden layers in between their input and output layers, these hidden layers have neurons embedded within them, and it’s the weights within the neurons along with the interconnection between neurons is what enables the neural network system to learn.\nThe general idea in Neural networks is that the more neurons and layers within a neural network architecture, the greater the representational power it has. This increase in representational power means that the neural network can fit more complex functions and generalize well to training data.\nSimply kept, there are more configurations for the interconnections between the neurons within the neural network layers.\n\nThe disadvantage of utilizing deeper neural networks is that they are highly prone to overfitting.\nOverfitting is a common problem that is defined as the inability for a trained machine learning model to generalized well to unseen data, but the same model performs well on the data it was trained on.\nThe primary purpose of dropout is to minimize the effect of overfitting within a trained network.\nDropout technique works by randomly reducing the number of interconnecting neurons within a neural network. At every training step, each neuron has a chance of being left out, or rather, dropped out of the collated contribution from connected neurons.\n\nThis technique minimizes overfitting because each neuron becomes independently sufficient, in the sense that the neurons within the layers learn weight values that are not based on the cooperation of its neighbouring neurons. Supposedly you trained 7,000 different neural network architecture, to select the best one you simply take the average of all 7,000 trained neural network.\n\nWell, the dropout technique actually mimics this scenario.\nIf the probability of a neuron getting dropped out in a training step is set to 0.5; we are actually training a variety of different network at each training step as it’s highly impossible that the same neurons are excluded at any two training steps. Therefore a neural network that has been trained utilizing the dropout technique is an average of all the different neurons connection combinations that have occurred at each training step.","metadata":{}},{"cell_type":"markdown","source":"## Training and evaluating the model\n\nAfter the model architecture is defined and compiled, the model needs to be trained with training data to be able to recognize the handwritten digits.\nHere, one epoch means one forward and one backward pass of all the training samples. Batch size implies number of training samples in one forward/backward pass.","metadata":{}},{"cell_type":"code","source":"# Now actually train the model implementing all the functions above\ncnn_model = run_model((28,28, 1))\nrun_model_compilation(cnn_model, 'adam', 'categorical_crossentropy')\n\n# Taining the model specifying epocs\nmodel_history = run_model_training(cnn_model, mnist_train_array, mnist_train_label_array, 100, 0.2)","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2021-06-17T16:27:02.762169Z","iopub.execute_input":"2021-06-17T16:27:02.762635Z","iopub.status.idle":"2021-06-17T16:37:23.378725Z","shell.execute_reply.started":"2021-06-17T16:27:02.762587Z","shell.execute_reply":"2021-06-17T16:37:23.377837Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Now like the training data transformation to array I do the same with test data\n# Convert the test dataframes into numpy arrays\nmnist_test_array = np.array(test_df_org)\nprint(\"mnist_test_array.shape after converting to array \", mnist_test_array.shape)\n\n# And also like the training data, need to do reshaping to get the structure which the model will expect to receive going forward\nmnist_test_array = mnist_test_array.reshape(mnist_test_array.shape[0], 28, 28, 1)\nprint(\"mnist_test_array.shape after reshaping \", mnist_test_array.shape)","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2021-06-17T16:37:23.381527Z","iopub.execute_input":"2021-06-17T16:37:23.381899Z","iopub.status.idle":"2021-06-17T16:37:23.454947Z","shell.execute_reply.started":"2021-06-17T16:37:23.381856Z","shell.execute_reply":"2021-06-17T16:37:23.45415Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Final Model Predictions and creating output file for Submission","metadata":{}},{"cell_type":"code","source":"final_predictions = cnn_model.predict(mnist_test_array)\n\nprediction_test_array = []\n\nfor i in final_predictions:\n    prediction_test_array.append(np.argmax(i))","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2021-06-17T16:37:23.457205Z","iopub.execute_input":"2021-06-17T16:37:23.457598Z","iopub.status.idle":"2021-06-17T16:37:24.966371Z","shell.execute_reply.started":"2021-06-17T16:37:23.457556Z","shell.execute_reply":"2021-06-17T16:37:24.965482Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = pd.DataFrame({\n    'ImageId': test_df_org.index+1,\n    'Label': prediction_test_array\n})\n\nsubmission.to_csv('final_submission.csv', index=False)","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2021-06-17T16:37:24.967728Z","iopub.execute_input":"2021-06-17T16:37:24.968093Z","iopub.status.idle":"2021-06-17T16:37:25.388219Z","shell.execute_reply.started":"2021-06-17T16:37:24.968054Z","shell.execute_reply":"2021-06-17T16:37:25.387449Z"},"trusted":true},"execution_count":null,"outputs":[]}]}