{"cells":[{"metadata":{},"cell_type":"markdown","source":"# MNIST Dataset with PCA+RandomForest\n- Loading Data.\n- Visulization of Images.\n- Preprocessing.\n    - Scaling the data.\n    - Principal Component Analysis.\n- Chosing Number Of Components.\n- RandomForest classifier\n    - Tuning Model\n- Submission"},{"metadata":{},"cell_type":"markdown","source":"# Loading Data"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing,\nimport warnings    # warning to show or hide\nimport random     # For random pocess\nimport matplotlib.pyplot as plt     # Plotting Images, Graph\nfrom IPython.display import display, Markdown\n\n# Ignoring Wornings\nwarnings.filterwarnings(\"ignore\")\n\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"# Reading Training Data\ntrain_data = pd.read_csv('../input/train.csv')\n# Reading Testing Data\ntest_data = pd.read_csv('../input/test.csv')\nprint(\"Training data informataion\")\nprint(train_data.info())\nprint(\"Training Data Shape: {0}\\nTesting Data Shape: {1}\".format(train_data.shape, test_data.shape))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Training Data Columns\ntrain_data.columns","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Separating features and labels from traning data."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Get Featrues and Labels\nfeatrue = train_data.drop(['label'], axis=1)\nlabel = np.asarray(train_data[['label']])\nprint(\"Unique Digit Labels: {}\".format(np.unique(label)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visulization of Images\nLets have visulization of MNIST handwritten digit dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Creating custom subplot function to visualize the image\ndef plotImage(image_data, label, number_of_plots=3):\n    \n    # Generate Random number for position\n    pos_of_image = [random.randint(0, image_data.shape[0]) for p in range(0, number_of_plots*number_of_plots)]\n    \n    # Defining figure\n    fig = plt.figure(figsize=(10, 10))\n    \n    # plotting subplots\n    for pos, plotnumber in zip(pos_of_image, range(1, (number_of_plots*number_of_plots)+1)):\n        ax = fig.add_subplot(number_of_plots, number_of_plots, plotnumber)\n        \n        # pLotting image\n        ax.imshow(image_data[pos], cmap='gray')\n        ax.set_title(\"Digit-{}\".format(label[pos][0]), fontsize=14)\n        fig.tight_layout()\n\n# Converting the 1-D digit array to 2-D array.\n# Using reshape function 784 >> 28,28\nimage_array = np.reshape(np.asarray(featrue), (len(featrue), 28, 28))\n\n# Calling custom plot funcation\ndisplay(Markdown(\"### MNIST Handwritten Digits\"))\nplotImage(image_array, label, number_of_plots=5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Preprocessing\n## Scaling the data\n- Scaling is most important when it comes to data preprocssing. \n- The independent feature which ranges in 0-1000 values makes the optimization slower i.e. training gets slower. \n- In our case the digit pixel values ranges from 0-255. Which will make our classifier system slower. \n- So to imporve the training we normalize the data using Normalizer which will convert the data in range of -1 to 1 or 0 to 1."},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.preprocessing import Normalizer\n\n# Scaling Data with Normalizer\ndef norm(input_data):\n    nm = Normalizer()\n    nm.fit(input_data)\n    input_data_scale = nm.transform(featrue)\n    return input_data_scale, nm\n\nfeatrue_scale, nm = norm(featrue)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Dimensionality Reduction: Principal Component Analysis\n- The dimensionality reduction is another most important part of data preprocessing.\n- Lets say we have data of vehicles. In which type of vehicle (2-wheeler,4-wheeler), number of tyres (2 tyre,4 tyre) which are co-related to a high degree. Hence, we can reduce the number of features in such problems.\n- In this case we have 784 features. Whcih represnts the pixels values of digit. These feature are correalted so we can readuce the number of features.\n- Approch is done using Principal Component Analysis. \n- Lets calculate optimum number of feature is which covers 90% of PCA variance."},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.decomposition import PCA\n\ncomponent_with_var = {}\n# We will use variacne range instead  of number features. Looping 784 features is slower than 10 varince range.\nvariance_list = np.arange(0.1, 1.1, 0.1)\n\nfor var in variance_list:\n    if var < 1.0:\n        # Selecting 2nd shape value which is nothing but number of components.\n        component_with_var[str(var*100)+' %'] = PCA(var).fit_transform(featrue_scale).shape[1]\n    else:\n        component_with_var[str(var*100)+' %'] = featrue_scale.shape[1]\ncomponent_with_var","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"variance_cover = 0.9    # 90% Of varince cover\n# Get component list\ncomponent_list = list(component_with_var.values())\n# using numpy interp function we will get number of features(components) for the 90% varince coverage.\ncomponents = int(np.interp(variance_cover, variance_list, component_list))\nprint(\"Optimum number of feature or component is {0}\".format(components))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.plot(variance_list, component_list)\nplt.scatter(variance_cover, components, color='red')\nplt.annotate('{}% variance covered by \\n{} features'.format(variance_cover*100,components), \n             xy=(variance_cover, components), xytext=(0.3,400),\n             arrowprops=dict(facecolor='black', shrink=0.05),\n             fontsize=14\n            )\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From above calculation we found that up to 229 we can reduce the features. Which covers 90% of variance."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Lets reduce the dimension of given dataset\ndef pca_transform(input_data, components):\n    pca = PCA(n_components=components)\n    pca.fit(input_data)\n    input_data_pca = pca.transform(input_data)\n    return input_data_pca, pca\nfeature_pca, pca_model = pca_transform(featrue_scale, components)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train-Test-Split\nLets split the training data into train-test data for cross validations"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n# Train data 80%, Test data 20%\nX_train, X_test, Y_train, Y_test = train_test_split(feature_pca, label, test_size=0.2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# RandomForsest Classifier"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Calling Model\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.model_selection import GridSearchCV","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Taking effective params \nrfr_grid = {\n    \"n_estimators\":[100, 200],\n    \"max_depth\":[2, 5, 10, None]\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n# Fitting model\nclf = GridSearchCV(RandomForestClassifier(),param_grid=rfr_grid,verbose=5)\nclf.fit(X_train, Y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Best params are\nclf.best_params_","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n# Fitting with best params\nrfr = RandomForestClassifier(\n    n_estimators=200\n)\nrfr.fit(X_train,Y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Train score\nrfr.score(X_train,Y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# ValidationTest Score\nrfr.score(X_test, Y_test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Validation Testing scores 95% "},{"metadata":{},"cell_type":"markdown","source":"# Submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Test data\ntest_data.head(3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Preprocessing Test data\ntest_scale = nm.transform(test_data)\ntest_pca = pca_model.transform(test_scale)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Predicted Digit\npredicted_digit = rfr.predict(test_pca)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Submission\nsub =  pd.DataFrame(range(1,len(test_data)+1),columns=['ImageId'])\nsub['Label'] = predicted_digit\nsub.to_csv('pca_rfr_digit.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}