{"cells":[{"metadata":{},"cell_type":"markdown","source":"# MNIST Data\nThe **MNIST database *(Modified National Institute of Standards and Technology database)*** is a large database of handwritten digits that is commonly used for training various image processing systems. The database is also widely used for training and testing in the field of machine learning. IT was created by \"re-mixing\" the samples from NIST's original datasets. The creators felt that since NIST's training dataset was taken from American Census Bureau employees, while the testing dataset was taken from American high school students, it was not well-suited for machine learning experiments.Furthermore, the black and white images from NIST were normalised to fit into a 28x28 pixel bounding box and anti-aliased, which introduced grayscale levels."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\nimport warnings\nwarnings.filterwarnings('ignore') \n# Any results you write to the current directory are saved as output.\n\nimport matplotlib.pyplot as plt\nimport pandas as pd\n#from math import pi\nfrom collections import Counter\nimport numpy as np\nimport seaborn as sns\nfrom sklearn.decomposition import PCA\nimport pylab","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"![MNIST](https://i.imgur.com/5P8Zd9n.png)"},{"metadata":{"trusted":true},"cell_type":"code","source":"pylab.rcParams['figure.figsize'] = (9.0, 9.0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# PCA- Principal Component Analysis\nPrincipal Component Analysis, or PCA, is a dimensionality-reduction method that is often used to reduce the dimensionality of large data sets, by transforming a large set of variables into a smaller one that still contains most of the information in the large set.\nReducing the number of variables of a data set naturally comes at the expense of accuracy, but the trick in dimensionality reduction is to trade a little accuracy for simplicity. Because smaller data sets are easier to explore and visualize and make analyzing data much easier and faster for machine learning algorithms without extraneous variables to process.\nSo to sum up, the idea of PCA is simple — reduce the number of variables of a data set, while preserving as much information as possible."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"\n\n\n\ntrain = pd.read_csv('../input/train.csv')#extract values\ntrain_x = train.drop('label',axis = 1)\ntest = pd.read_csv('../input/test.csv',index_col = None)\n\n#train_x = train.iloc[:,1:].values\n#test_x = test.iloc[:,:].values\n\nfrom sklearn.preprocessing import StandardScaler\nscaler = StandardScaler()\ntrain_x = scaler.fit_transform(train_x)\ntest_x = scaler.fit_transform(test)\n\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n# Standardization\nThe aim of this step is to standardize the range of the continuous initial variables so that each one of them contributes equally to the analysis.\nMore specifically, the reason why it is critical to perform standardization prior to **PCA**, is that the latter is quite sensitive regarding the variances of the initial variables. That is, if there are large differences between the ranges of initial variables, those variables with larger ranges will dominate over those with small ranges (For example, a variable that ranges between 0 and 100 will dominate over a variable that ranges between 0 and 1), which will lead to biased results. So, transforming the data to comparable scales can prevent this problem.\nMathematically, this can be done by subtracting the mean and dividing by the standard deviation for each value of each variable."},{"metadata":{},"cell_type":"markdown","source":"# Covariance Matrix computation\n\nThe aim of this step is to understand how the variables of the input data set are varying from the mean with respect to each other, or in other words, to see if there is any relationship between them. Because sometimes, variables are highly correlated in such a way that they contain redundant information. So, in order to identify these correlations, we compute the covariance matrix."},{"metadata":{},"cell_type":"markdown","source":"# Eigenvectors and Eigenvalues\nEigenvectors and eigenvalues are the linear algebra concepts that we need to compute from the covariance matrix in order to determine the principal components of the data. Before getting to the explanation of these concepts, let’s first understand what do we mean by principal components.\nPrincipal components are new variables that are constructed as linear combinations or mixtures of the initial variables. These combinations are done in such a way that the new variables (i.e., principal components) are uncorrelated and most of the information within the initial variables is squeezed or compressed into the first components."},{"metadata":{"trusted":true},"cell_type":"code","source":"\npca = PCA(n_components=2)\nprincipalComponents = pca.fit_transform(train_x)\nprincipalDf = pd.DataFrame(data = principalComponents, columns = ['principalcomponent1',\n                                                                  'principalcomponent2'])\nlabel = pd.DataFrame(data = train['label'])\nprincipalDf = pd.concat([principalDf,label],axis = 1,ignore_index=True)\n\nprincipalDf.columns = [\"principalcomponent1\", \"principalcomponent2\", \"label\"] \n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Dimensionality reduction using PCA\nPCA is a technique for reducing the number of dimensions in a dataset whilst retaining most information. It is using the correlation between some dimensions and tries to provide a minimum number of variables that keeps the maximum amount of variation or information about how the original data is distributed. It does not do this using guesswork but using hard mathematics and it uses something known as the eigenvalues and eigenvectors of the data-matrix. These eigenvectors of the covariance matrix have the property that they point along the major directions of variation in the data. These are the directions of maximum variation in a dataset.\n\nThe above code reduces the **No of Components** to 2 and converts that to a dataframe of two columns. Interpretation becomes easier to plot when it has been reduced to 2 or 3 directions"},{"metadata":{"trusted":true},"cell_type":"code","source":"# give a list to the marker argument\nsns.regplot(x=principalDf[\"principalcomponent1\"], y=principalDf[\"principalcomponent2\"],\n            fit_reg=False, scatter_kws={\"color\":\"darkred\",\"alpha\":0.3,\"s\":200} )\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.lmplot( x='principalcomponent1', y='principalcomponent2', data=principalDf, fit_reg=False, \n           hue='label', legend=False, palette=\"Blues\")\nplt.figure(figsize=(13,10))\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"flatui = [\"#9b59b6\", \"#3498db\", \"orange\"]\nsns.set_palette(flatui)\nsns.lmplot( x=\"principalcomponent1\", y=\"principalcomponent2\", data=principalDf, fit_reg=False,\n           hue='label', legend=False)\n\nplt.figure(figsize=(13,10))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nfrom sklearn.decomposition import PCA\n# Make an instance of the Model\npca2 = PCA(.95)\n\npca2.fit(train_x)\n\ntrain_2 = pca.transform(train_x)\ntest_2 = pca.transform(test_x)\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Logistic Regression\nLogistic Regression is a Machine Learning classification algorithm that is used to predict the probability of a categorical dependent variable. In logistic regression, the dependent variable is a binary variable that contains data coded as 1 (yes, success, etc.) or 0 (no, failure, etc.). In other words, the logistic regression model predicts P(Y=1) as a function of X.\nLogistic Regression is one of the most popular ways to fit models for categorical data, especially for binary response data in Data Modeling. It is the most important (and probably most used) member of a class of models called generalized linear models. Unlike linear regression, logistic regression can directly predict probabilities (values that are restricted to the (0,1) interval); furthermore, those probabilities are well-calibrated when compared to the probabilities predicted by some other classifiers, such as Naive Bayes. Logistic regression preserves the marginal probabilities of the training data. The coefficients of the model also provide some hint of the relative importance of each input variable.\nLogistic Regression is used when the dependent variable (target) is categorical."},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.linear_model import LogisticRegression\nlogisticRegr = LogisticRegression(solver = 'lbfgs')\n\nlogisticRegr.fit(train_2, train['label'])\ntest_pred=logisticRegr.predict(test_2)\n\n\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.DataFrame({\n        \"ImageId\": np.arange(1,28001),\n        \"Label\": test_pred\n    })\nsubmission.to_csv('submission.csv', index=False)\n\n#a=np.arange(1,28001)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}