{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Digit Recognizer","metadata":{}},{"cell_type":"markdown","source":"### Data Description","metadata":{}},{"cell_type":"markdown","source":"* The data files train.csv file contain gray-scale images of hand-drawn digits, from zero through nine.\n* Each image is 28 pixels in height and 28 pixels in width, for a total of 784 pixels in total. \n* Each pixel has a single pixel-value associated with it, indicating the lightness or darkness of that pixel, with higher numbers meaning darker. \n* This pixel-value is an integer between 0 and 255, inclusive.","metadata":{}},{"cell_type":"markdown","source":"* The training data set, (train.csv), has 785 columns. \n* The first column, called \"label\", is the digit that was drawn by the user. \n* The rest of the columns contain the pixel-values of the associated image.","metadata":{}},{"cell_type":"markdown","source":"* Each pixel column in the training set has a name like pixelx, where x is an integer between 0 and 783, inclusive. \n* To locate this pixel on the image, suppose that we have decomposed x as x = i * 28 + j, where i and j are integers between 0 and 27, inclusive. \n* Then pixelx is located on row i and column j of a 28 x 28 matrix, (indexing by zero).","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"execution":{"iopub.status.busy":"2022-03-13T08:29:34.756951Z","iopub.execute_input":"2022-03-13T08:29:34.758249Z","iopub.status.idle":"2022-03-13T08:29:34.767212Z","shell.execute_reply.started":"2022-03-13T08:29:34.758189Z","shell.execute_reply":"2022-03-13T08:29:34.766029Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nfrom sklearn.tree import DecisionTreeClassifier\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.model_selection import train_test_split,GridSearchCV\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.metrics import accuracy_score, confusion_matrix\nimport numpy as np \nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nsns.set()","metadata":{"execution":{"iopub.status.busy":"2022-03-13T08:29:34.816457Z","iopub.execute_input":"2022-03-13T08:29:34.816990Z","iopub.status.idle":"2022-03-13T08:29:34.825571Z","shell.execute_reply.started":"2022-03-13T08:29:34.816932Z","shell.execute_reply":"2022-03-13T08:29:34.824335Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the above code of line , we have imported all the necessary librabies","metadata":{}},{"cell_type":"code","source":"data = pd.read_csv(\"/kaggle/input/digit-recognizer/train.csv\")\ndata.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-13T08:29:34.827885Z","iopub.execute_input":"2022-03-13T08:29:34.828330Z","iopub.status.idle":"2022-03-13T08:29:38.908537Z","shell.execute_reply.started":"2022-03-13T08:29:34.828291Z","shell.execute_reply":"2022-03-13T08:29:38.907363Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the above line of code, we have Imported our Dataset and stored in a variable called 'data'","metadata":{}},{"cell_type":"code","source":"data.shape","metadata":{"execution":{"iopub.status.busy":"2022-03-13T08:29:38.910497Z","iopub.execute_input":"2022-03-13T08:29:38.910770Z","iopub.status.idle":"2022-03-13T08:29:38.918920Z","shell.execute_reply.started":"2022-03-13T08:29:38.910738Z","shell.execute_reply":"2022-03-13T08:29:38.917447Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Got the sahpe of our Dataset.\n* There are 42000 Rows & 785 Columns","metadata":{}},{"cell_type":"code","source":"col_names = data.columns\n\ncol_names","metadata":{"execution":{"iopub.status.busy":"2022-03-13T08:29:38.920687Z","iopub.execute_input":"2022-03-13T08:29:38.921889Z","iopub.status.idle":"2022-03-13T08:29:38.932641Z","shell.execute_reply.started":"2022-03-13T08:29:38.921837Z","shell.execute_reply":"2022-03-13T08:29:38.931481Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data.info()","metadata":{"execution":{"iopub.status.busy":"2022-03-13T08:29:38.935065Z","iopub.execute_input":"2022-03-13T08:29:38.936199Z","iopub.status.idle":"2022-03-13T08:29:39.001742Z","shell.execute_reply.started":"2022-03-13T08:29:38.936146Z","shell.execute_reply":"2022-03-13T08:29:39.001119Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data.isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2022-03-13T08:29:39.003226Z","iopub.execute_input":"2022-03-13T08:29:39.003553Z","iopub.status.idle":"2022-03-13T08:29:39.057400Z","shell.execute_reply.started":"2022-03-13T08:29:39.003512Z","shell.execute_reply":"2022-03-13T08:29:39.056354Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Lets find the Unique values in our Target variable","metadata":{}},{"cell_type":"code","source":"print(data.label.unique())","metadata":{"execution":{"iopub.status.busy":"2022-03-13T08:29:39.059026Z","iopub.execute_input":"2022-03-13T08:29:39.059353Z","iopub.status.idle":"2022-03-13T08:29:39.066475Z","shell.execute_reply.started":"2022-03-13T08:29:39.059300Z","shell.execute_reply":"2022-03-13T08:29:39.065116Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.countplot(x='label', data=data)\nplt.title('Count of label', fontsize=30)","metadata":{"execution":{"iopub.status.busy":"2022-03-13T08:29:39.067980Z","iopub.execute_input":"2022-03-13T08:29:39.068853Z","iopub.status.idle":"2022-03-13T08:29:39.394173Z","shell.execute_reply.started":"2022-03-13T08:29:39.068811Z","shell.execute_reply":"2022-03-13T08:29:39.393095Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* CountPlot of Target Variable i.e Label","metadata":{}},{"cell_type":"code","source":"data.label.value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-03-13T08:29:39.395554Z","iopub.execute_input":"2022-03-13T08:29:39.395783Z","iopub.status.idle":"2022-03-13T08:29:39.403530Z","shell.execute_reply.started":"2022-03-13T08:29:39.395755Z","shell.execute_reply":"2022-03-13T08:29:39.402861Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Now Lets do Training & Testing of our Dataset","metadata":{}},{"cell_type":"code","source":"X = data.drop('label', axis=1)   \n\ny = data['label']        # Target Variable","metadata":{"execution":{"iopub.status.busy":"2022-03-13T08:29:39.404617Z","iopub.execute_input":"2022-03-13T08:29:39.405398Z","iopub.status.idle":"2022-03-13T08:29:39.524439Z","shell.execute_reply.started":"2022-03-13T08:29:39.405358Z","shell.execute_reply":"2022-03-13T08:29:39.523350Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x_train,x_test,y_train,y_test = train_test_split(X,y,test_size=0.3, random_state= 355)","metadata":{"execution":{"iopub.status.busy":"2022-03-13T08:29:39.527672Z","iopub.execute_input":"2022-03-13T08:29:39.527919Z","iopub.status.idle":"2022-03-13T08:29:39.831852Z","shell.execute_reply.started":"2022-03-13T08:29:39.527893Z","shell.execute_reply":"2022-03-13T08:29:39.831137Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Now Lets first try with Decision Tree classifier","metadata":{}},{"cell_type":"code","source":"attr_forest = DecisionTreeClassifier( min_samples_split= 2)\n\nattr_forest.fit(x_train,y_train)","metadata":{"execution":{"iopub.status.busy":"2022-03-13T08:29:39.833004Z","iopub.execute_input":"2022-03-13T08:29:39.833464Z","iopub.status.idle":"2022-03-13T08:29:48.347211Z","shell.execute_reply.started":"2022-03-13T08:29:39.833423Z","shell.execute_reply":"2022-03-13T08:29:48.346449Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print the scores on training and test set\n\nprint('Training set score: {:.4f}'.format(attr_forest.score(x_train, y_train)))\n\nprint('Test set score: {:.4f}'.format(attr_forest.score(x_test, y_test)))","metadata":{"execution":{"iopub.status.busy":"2022-03-13T08:29:48.348379Z","iopub.execute_input":"2022-03-13T08:29:48.349256Z","iopub.status.idle":"2022-03-13T08:29:48.496599Z","shell.execute_reply.started":"2022-03-13T08:29:48.349208Z","shell.execute_reply":"2022-03-13T08:29:48.495390Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Using Decision Tree classifier we got 100 % Training Accuracy & 84 % Testing Accuracy .\n* But from this info. we can say that our Model with Decision Tree classifier is OverFitted.","metadata":{}},{"cell_type":"markdown","source":"### Random Forest without HyperParameter Tunning","metadata":{}},{"cell_type":"code","source":"classifier = RandomForestClassifier(random_state=6)","metadata":{"execution":{"iopub.status.busy":"2022-03-13T08:29:48.498033Z","iopub.execute_input":"2022-03-13T08:29:48.498408Z","iopub.status.idle":"2022-03-13T08:29:48.503967Z","shell.execute_reply.started":"2022-03-13T08:29:48.498375Z","shell.execute_reply":"2022-03-13T08:29:48.502769Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"classifier.fit(x_train,y_train)","metadata":{"execution":{"iopub.status.busy":"2022-03-13T08:29:48.505609Z","iopub.execute_input":"2022-03-13T08:29:48.506237Z","iopub.status.idle":"2022-03-13T08:30:09.422421Z","shell.execute_reply.started":"2022-03-13T08:29:48.506189Z","shell.execute_reply":"2022-03-13T08:30:09.421696Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print the scores on training and test set\n\nprint('Training set score: {:.4f}'.format(classifier.score(x_train, y_train)))\n\nprint('Test set score: {:.4f}'.format(classifier.score(x_test, y_test)))","metadata":{"execution":{"iopub.status.busy":"2022-03-13T08:30:09.423491Z","iopub.execute_input":"2022-03-13T08:30:09.424220Z","iopub.status.idle":"2022-03-13T08:30:11.178461Z","shell.execute_reply.started":"2022-03-13T08:30:09.424184Z","shell.execute_reply":"2022-03-13T08:30:11.177218Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Using Random Forest classifier we got 100 % Training Accuracy & 96 % Testing Accuracy .\n* Now we can see by changing an Algorithm we have improved a Testing Accuracy.","metadata":{}},{"cell_type":"markdown","source":"### Random Forest with HyperParameter Tunning","metadata":{}},{"cell_type":"code","source":"n_estimators = [5,20,50,100,90,115,130] # number of trees in the random forest\nmax_features = ['auto', 'sqrt'] # number of features in consideration at every split\nmax_depth = range(2,20,1) # maximum number of levels allowed in each decision tree\nmin_samples_split = range(2,10,1)  # minimum sample number to split a node\nmin_samples_leaf = range(1,10,1) # minimum sample number that can be stored in a leaf node\nbootstrap = [True, False] # method used to sample data points\n\nrandom_grid = {'n_estimators': n_estimators,\n\n'max_features': max_features,\n\n'max_depth': max_depth,\n\n'min_samples_split': min_samples_split,\n\n'min_samples_leaf': min_samples_leaf,\n\n'bootstrap': bootstrap}","metadata":{"execution":{"iopub.status.busy":"2022-03-13T08:30:11.180281Z","iopub.execute_input":"2022-03-13T08:30:11.180616Z","iopub.status.idle":"2022-03-13T08:30:11.188061Z","shell.execute_reply.started":"2022-03-13T08:30:11.180574Z","shell.execute_reply":"2022-03-13T08:30:11.186969Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.ensemble import RandomForestClassifier\n\nrf = RandomForestClassifier()","metadata":{"execution":{"iopub.status.busy":"2022-03-13T08:30:11.189593Z","iopub.execute_input":"2022-03-13T08:30:11.189917Z","iopub.status.idle":"2022-03-13T08:30:11.202700Z","shell.execute_reply.started":"2022-03-13T08:30:11.189876Z","shell.execute_reply":"2022-03-13T08:30:11.201842Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import RandomizedSearchCV\nrf_random = RandomizedSearchCV(estimator = rf,param_distributions = random_grid,\n               n_iter = 100, cv = 5, verbose=2, random_state=35, n_jobs = -1)","metadata":{"execution":{"iopub.status.busy":"2022-03-13T08:30:11.204239Z","iopub.execute_input":"2022-03-13T08:30:11.204489Z","iopub.status.idle":"2022-03-13T08:30:11.218566Z","shell.execute_reply.started":"2022-03-13T08:30:11.204460Z","shell.execute_reply":"2022-03-13T08:30:11.217451Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rf_random.fit(x_train, y_train)","metadata":{"execution":{"iopub.status.busy":"2022-03-13T08:30:11.220524Z","iopub.execute_input":"2022-03-13T08:30:11.220864Z","iopub.status.idle":"2022-03-13T08:56:52.653956Z","shell.execute_reply.started":"2022-03-13T08:30:11.220818Z","shell.execute_reply":"2022-03-13T08:56:52.653216Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print ('Random grid: ', random_grid, '\\n')\n# print the best parameters\nprint ('Best Parameters: ', rf_random.best_params_, ' \\n')","metadata":{"execution":{"iopub.status.busy":"2022-03-13T08:56:52.655800Z","iopub.execute_input":"2022-03-13T08:56:52.656214Z","iopub.status.idle":"2022-03-13T08:56:52.663770Z","shell.execute_reply.started":"2022-03-13T08:56:52.656181Z","shell.execute_reply":"2022-03-13T08:56:52.662506Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rand_clf = RandomForestClassifier(criterion= 'gini',\n max_depth = 16,\n max_features = 'auto',\n min_samples_leaf = 4,\n min_samples_split= 6,\n n_estimators = 115,random_state=6)","metadata":{"execution":{"iopub.status.busy":"2022-03-13T09:00:57.351465Z","iopub.execute_input":"2022-03-13T09:00:57.351897Z","iopub.status.idle":"2022-03-13T09:00:57.356606Z","shell.execute_reply.started":"2022-03-13T09:00:57.351865Z","shell.execute_reply":"2022-03-13T09:00:57.355420Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rand_clf.fit(x_train,y_train)","metadata":{"execution":{"iopub.status.busy":"2022-03-13T09:00:57.357822Z","iopub.execute_input":"2022-03-13T09:00:57.358454Z","iopub.status.idle":"2022-03-13T09:01:16.822349Z","shell.execute_reply.started":"2022-03-13T09:00:57.358417Z","shell.execute_reply":"2022-03-13T09:01:16.821489Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print the scores on training and test set\n\nprint('Training set score: {:.4f}'.format(rand_clf.score(x_train, y_train)))\n\nprint('Test set score: {:.4f}'.format(rand_clf.score(x_test, y_test)))","metadata":{"execution":{"iopub.status.busy":"2022-03-13T09:02:54.685817Z","iopub.execute_input":"2022-03-13T09:02:54.686144Z","iopub.status.idle":"2022-03-13T09:02:56.965791Z","shell.execute_reply.started":"2022-03-13T09:02:54.686109Z","shell.execute_reply":"2022-03-13T09:02:56.964783Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Now from This Score we got 99 % Training Accuracy & 95 % Testing Accuracy with HyperParameter Tunning.","metadata":{}},{"cell_type":"code","source":"y_pred = rand_clf.predict(x_test)","metadata":{"execution":{"iopub.status.busy":"2022-03-13T09:03:09.036550Z","iopub.execute_input":"2022-03-13T09:03:09.037509Z","iopub.status.idle":"2022-03-13T09:03:09.610048Z","shell.execute_reply.started":"2022-03-13T09:03:09.037464Z","shell.execute_reply":"2022-03-13T09:03:09.609023Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Confusion Matrix\nconf_mat = confusion_matrix(y_test,y_pred)\nconf_mat","metadata":{"execution":{"iopub.status.busy":"2022-03-13T09:03:12.690879Z","iopub.execute_input":"2022-03-13T09:03:12.691875Z","iopub.status.idle":"2022-03-13T09:03:12.703354Z","shell.execute_reply.started":"2022-03-13T09:03:12.691826Z","shell.execute_reply":"2022-03-13T09:03:12.702424Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import classification_report \n","metadata":{"execution":{"iopub.status.busy":"2022-03-13T09:03:15.265429Z","iopub.execute_input":"2022-03-13T09:03:15.266486Z","iopub.status.idle":"2022-03-13T09:03:15.271279Z","shell.execute_reply.started":"2022-03-13T09:03:15.266411Z","shell.execute_reply":"2022-03-13T09:03:15.270457Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nfrom sklearn.model_selection import cross_val_predict, cross_val_score \nfrom sklearn.metrics import accuracy_score, classification_report \nfrom sklearn.metrics import confusion_matrix \nfrom sklearn.neighbors import KNeighborsClassifier \n\n  \ndef print_score(rand_clf, x_train, y_train, x_test, y_test, train = True): \n    if train: \n        print(\"Train Result:\") \n        print(\"------------\") \n        print(\"Classification Report: \\n {}\\n\".format(classification_report( \n                y_train, rand_clf.predict(x_train)))) \n        print(\"Confusion Matrix: \\n {}\\n\".format(confusion_matrix( \n                y_train, rand_clf.predict(x_train)))) \n  \n        res = cross_val_score(rand_clf, x_train, y_train,  \n                              cv = 10, scoring ='accuracy') \n        print(\"Average Accuracy: \\t {0:.4f}\".format(np.mean(res))) \n        print(\"Accuracy SD: \\t\\t {0:.4f}\".format(np.std(res))) \n        print(\"accuracy score: {0:.4f}\\n\".format(accuracy_score( \n                y_train, rand_clf.predict(x_train)))) \n        print(\"----------------------------------------------------------\") \n                 \n    elif train == False: \n        print(\"Test Result:\") \n        print(\"-----------\") \n        print(\"Classification Report: \\n {}\\n\".format( \n                classification_report(y_test, rand_clf.predict(x_test)))) \n        print(\"Confusion Matrix: \\n {}\\n\".format( \n                confusion_matrix(y_test, rand_clf.predict(x_test))))  \n        print(\"accuracy score: {0:.4f}\\n\".format( \n                accuracy_score(y_test, rand_clf.predict(x_test)))) \n        print(\"-----------------------------------------------------------\") \n          \nknn = KNeighborsClassifier(n_neighbors = 7) \nknn.fit(x_train, y_train) \nprint_score(knn, x_train, y_train, x_test, y_test, train = True) \nprint_score(knn, x_train, y_train, x_test, y_test, train = False) \n","metadata":{"execution":{"iopub.status.busy":"2022-03-13T09:03:18.415804Z","iopub.execute_input":"2022-03-13T09:03:18.416146Z","iopub.status.idle":"2022-03-13T09:07:02.055302Z","shell.execute_reply.started":"2022-03-13T09:03:18.416111Z","shell.execute_reply":"2022-03-13T09:07:02.054450Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(classification_report(y_test, y_pred))","metadata":{"execution":{"iopub.status.busy":"2022-03-13T09:07:10.133854Z","iopub.execute_input":"2022-03-13T09:07:10.134685Z","iopub.status.idle":"2022-03-13T09:07:10.171788Z","shell.execute_reply.started":"2022-03-13T09:07:10.134639Z","shell.execute_reply":"2022-03-13T09:07:10.170840Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}