{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-10-01T18:29:32.944266Z","iopub.execute_input":"2022-10-01T18:29:32.944907Z","iopub.status.idle":"2022-10-01T18:29:34.145828Z","shell.execute_reply.started":"2022-10-01T18:29:32.944788Z","shell.execute_reply":"2022-10-01T18:29:34.144541Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Introduction**\n\nLet me start by saying, this is not the best way to get a high score in this competation! This notebook is to prove that sometimes using a simple supervised machine learning algorithms can be a good start for getting acceptable scores (of course not in case of big-data).\nAny suggestions for improvement or comments on poor coding practices are appreciated.","metadata":{}},{"cell_type":"markdown","source":"**Reading Datasets to Kernel**","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\ntest  = pd.read_csv('/kaggle/input/digit-recognizer/test.csv')\nsample_submission = pd.read_csv('/kaggle/input/digit-recognizer/sample_submission.csv')\nlabel=train.iloc[:,0].values ## isolate the target variable.","metadata":{"execution":{"iopub.status.busy":"2022-10-01T18:29:34.148370Z","iopub.execute_input":"2022-10-01T18:29:34.149167Z","iopub.status.idle":"2022-10-01T18:29:40.431344Z","shell.execute_reply.started":"2022-10-01T18:29:34.149119Z","shell.execute_reply":"2022-10-01T18:29:40.430096Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Knowing your Data**\n\nFirst of all after reading datasets to the kernel, we should meet with our data, and knowing some information from it by few lines of code.\n\n1. Looking at the shape of the data.\n\n2. Looking at random sample from the data.\n\n3. Looking at null values if exist in order to deal with them after.","metadata":{}},{"cell_type":"code","source":"print('Training Dataset:\\n')\nprint('The Shape is: ',train.shape)\nprint('\\nSome Samples:\\n',train.sample(5))\nprint('\\nThere is ',str((train.isna().sum()>0).sum()),' null values in the training dataset')\nprint('-'*80)\nprint('Testing Dataset:\\n')\nprint('The Shape is: ',test.shape)\nprint('\\nSome Samples:\\n',test.sample(5))\nprint('\\nThere is ',str((test.isna().sum()>0).sum()),' null values in the testing dataset')\n","metadata":{"execution":{"iopub.status.busy":"2022-10-01T18:29:40.432820Z","iopub.execute_input":"2022-10-01T18:29:40.433185Z","iopub.status.idle":"2022-10-01T18:29:40.538956Z","shell.execute_reply.started":"2022-10-01T18:29:40.433151Z","shell.execute_reply":"2022-10-01T18:29:40.537692Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Great!! No NaNs at both of train / test datasets.\n\nNow we need to know more details about the data like:\n\n1- Pixel value range for photos.\n\n2- The frequency for each label ","metadata":{}},{"cell_type":"code","source":"plt.hist(train['label'],bins=10)\nplt.title('The labels Frequencies')\nplt.xlabel('Label')\nplt.ylabel('Frequency')\nplt.show()\n\ni=np.random.randint(0,train.shape[0],1)\nplt.hist(train.iloc[i,1:])\nplt.title('Pixel range')\nplt.xlabel('Pixel Value')\nplt.ylabel('Frequency')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-10-01T18:29:40.542293Z","iopub.execute_input":"2022-10-01T18:29:40.542826Z","iopub.status.idle":"2022-10-01T18:29:54.429445Z","shell.execute_reply.started":"2022-10-01T18:29:40.542773Z","shell.execute_reply":"2022-10-01T18:29:54.428224Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From above figure we conclude the most values are 0s \"The most dark pixels\" and some numbers above 0 to 254 distributed with very small frequencies. \nSo we can apply a trick by converting each number above \"0\" to \"1\" and thus will facilitate and accelerate the model fitting and learning.\n\nBut first I will try to reduce the data dimenssions using PCA.","metadata":{}},{"cell_type":"markdown","source":"**Dimenssionality Reduction using PCA**","metadata":{}},{"cell_type":"markdown","source":"The datasets contain high number of features, which will negatively impact our model accuracy.\n\nSo I will try to reduce the data dimenssions using PCA without losing informative features and see if we can success.\n\nfirst of all, I will determine the best number of components for PCA by plotting n_components vs. explained variance ratio, and from diagram I can determine the best n_components.","metadata":{}},{"cell_type":"code","source":"train_without_label=train.drop('label',axis=1)\n\nfrom sklearn.decomposition import PCA\n\npca=PCA()\npca.fit(train_without_label)\nexplained_variance=pca.explained_variance_\nplt.plot(range(100),explained_variance[:100])\nplt.title('Number of PCA components vs. Variance Ratio')\nplt.xlabel('n_components')\nplt.ylabel('Explained Variance Ratio')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-10-01T18:29:54.431055Z","iopub.execute_input":"2022-10-01T18:29:54.431891Z","iopub.status.idle":"2022-10-01T18:30:00.430527Z","shell.execute_reply.started":"2022-10-01T18:29:54.431851Z","shell.execute_reply":"2022-10-01T18:30:00.429687Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From above figure, we can see a high varince is correlated with number of components smaller than \"20\", while the flateness of the plot occur at 60 components and bigger..\n\nI will chose n_components= 60","metadata":{}},{"cell_type":"code","source":"# Plotting a photo before PCA:\n\ni=6\nx=train_without_label.iloc[i,:].values.reshape(28,28)\nplt.imshow(x,cmap='gray')\nplt.title(label[i])\nplt.show()\n\n# Performing PCA with n_components=60\n\npca=PCA(n_components=60)\ntrain_pca = pca.fit_transform(train_without_label)\ntest_pca  = pca.transform(test)\n\n# Plotting a photo after PCA:\n\nx=pca.inverse_transform(train_pca[i,:]).reshape(28,28)\nplt.imshow(x,cmap='gray')\nplt.title(label[i])\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-10-01T18:30:00.431831Z","iopub.execute_input":"2022-10-01T18:30:00.432838Z","iopub.status.idle":"2022-10-01T18:30:05.643802Z","shell.execute_reply.started":"2022-10-01T18:30:00.432800Z","shell.execute_reply":"2022-10-01T18:30:05.642445Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## The shape of train / test datasets after PCA\nprint('The new shape for train dataset after PCA: ',train_pca.shape)\nprint('The new shape for test dataset after PCA: ',test_pca.shape)","metadata":{"execution":{"iopub.status.busy":"2022-10-01T18:30:05.645466Z","iopub.execute_input":"2022-10-01T18:30:05.645963Z","iopub.status.idle":"2022-10-01T18:30:05.652131Z","shell.execute_reply.started":"2022-10-01T18:30:05.645917Z","shell.execute_reply":"2022-10-01T18:30:05.651087Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Great!!! we succeed to reduce the dimenssions for datasets to 60 features only without losing informative features...","metadata":{}},{"cell_type":"markdown","source":"**Model Fitting: KNeighborsClassifier**\n\nNow we reached the most sexy part after we prepared the data for model fitting and predicting..\n\nI will choose KNeighborsClassifier algorithm with its default parameters, check the accuracy after fitting on training data, then try to improve the accuracy by tunning hyperparameters.","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nfrom sklearn.neighbors import KNeighborsClassifier\n\n# Splitting the original dataset (after PCA) to training and validation datasets.\n\nX_train,X_val,y_train,y_val=train_test_split(train_pca,label,stratify=label,test_size=0.2,random_state=42)\n\n# Model fitting and scoring:\n\nknn=KNeighborsClassifier()\nknn.fit(X_train,y_train)\nprint('KNighborsClassifier accuracy by default parameters: ',np.round(knn.score(X_val,y_val),4))","metadata":{"execution":{"iopub.status.busy":"2022-10-01T18:30:05.653925Z","iopub.execute_input":"2022-10-01T18:30:05.654685Z","iopub.status.idle":"2022-10-01T18:30:12.173421Z","shell.execute_reply.started":"2022-10-01T18:30:05.654575Z","shell.execute_reply":"2022-10-01T18:30:12.172162Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"WoW, I got a relatively high accuracy by just first model fitting with default hyperparameters. \n\nNow, let's tune the hyperparameters for our data using GridSearchCV.","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import GridSearchCV\n\nparams={'n_neighbors':[3,5,7],'weights':['uniform','distance']}\ngridcv=GridSearchCV(knn,params,cv=5,n_jobs=-1)\ngridcv.fit(train_pca,label)\nprint('The best parameters are: ',gridcv.best_params_)\nprint('The best accuracy is: ',np.round(gridcv.best_score_,4))","metadata":{"execution":{"iopub.status.busy":"2022-10-01T18:30:12.175008Z","iopub.execute_input":"2022-10-01T18:30:12.175338Z","iopub.status.idle":"2022-10-01T18:31:37.054538Z","shell.execute_reply.started":"2022-10-01T18:30:12.175308Z","shell.execute_reply":"2022-10-01T18:31:37.052895Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Labelling the test data**\n\nNow for competition submission, we can predict the unlabeled data from test.csv. We then output this data to a results.csv for competition submission.","metadata":{}},{"cell_type":"code","source":"best_estimator=gridcv.best_estimator_\nresults=best_estimator.predict(test_pca)\nsample_submission['Label']=results\nsample_submission.to_csv('/kaggle/working/results.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-10-01T18:31:37.059259Z","iopub.execute_input":"2022-10-01T18:31:37.059633Z","iopub.status.idle":"2022-10-01T18:32:02.946670Z","shell.execute_reply.started":"2022-10-01T18:31:37.059596Z","shell.execute_reply":"2022-10-01T18:32:02.945464Z"},"trusted":true},"execution_count":null,"outputs":[]}]}