{"cells":[{"metadata":{},"cell_type":"markdown","source":"## <div style=\"text-align: center\">Machine Learning on Kannada MNIST  </div>\n\n<img src=\"https://storage.googleapis.com/kaggle-media/competitions/Kannada-MNIST/kannada.png\">\nKannada is a language spoken predominantly by people of Karnataka in southwestern India. The language has roughly 45 million native speakers and is written using the Kannada script. \n\n</div>\n\n\n-------------------------------------------------------------\n\n **I hope this kernel helpful and some <font color=\"red\"><b>UPVOTES</b></font> would be very much appreciated**\n "},{"metadata":{},"cell_type":"markdown","source":"<a id=\"top\"></a> <br>\n## Notebook  Content\n\n1. [Scikit-learn and Keras](#1)\n1. [Import](#2)\n1. [Estimator](#3)\n1. [Load Data](#4)\n1. [Prepare Train and Test](#5)\n1. [Visualization](#6)\n1. [Machine Learning Algorithms](#7)\n    1. [Logistic Regression](#10)\n    1. [Decision Tree](#11)\n    1. [PCA ams SVM](#12)\n    1. [XGBOOST](#13)\n    1. [AdaBoost classifier](#14)\n1. [Submit](#15)"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"1\"></a> <br>\n## 1-Scikit-learn\n\n- Simple and efficient tools for data mining and data analysis\n- Accessible to everybody, and reusable in various contexts\n- Built on NumPy, SciPy, and matplotlib\n- Open source, commercially usable - BSD license\n\n<div style=\"text-align:center\">Website: http://scikit-learn.org</div>\n\n"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"2\"></a> <br>\n## 2- Import"},{"metadata":{"trusted":true},"cell_type":"code","source":"%matplotlib inline\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import classification_report\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.metrics import confusion_matrix\nimport pylab as pl\nimport os","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"3\"></a> <br>\n## 3- Estimator for ML\n\nGiven a scikit-learn estimator object named **model**, the following methods are available:\n\n#### Available in all Estimators\n\n**model.fit()** : fit training data. For supervised learning applications, this accepts two arguments: the data X and the labels y (e.g. model.fit(X, y)). For unsupervised learning applications, this accepts only a single argument, the data X (e.g. model.fit(X)).\n\n---------------------------------------------------------\n\n#### Available in supervised estimators\n\n**model.predict()** : given a trained model, predict the label of a new set of data. This method accepts one argument, the new data X_new (e.g. model.predict(X_new)), and returns the learned label for each object in the array.\n\n**model.predict_proba()** : For classification problems, some estimators also provide this method, which returns the probability that a new observation has each categorical label. In this case, the label with the highest probability is returned by model.predict().\n**model.score()** : for classification or regression problems, most (all?) estimators implement a score method. Scores are between 0 and 1, with a larger score indicating a better fit.\n\n---------------------------------------------------------\n#### Available in unsupervised estimators\n\n**model.predict()** : predict labels in clustering algorithms.\n**model.transform()** : given an unsupervised model, transform new data into the new basis. This also accepts one argument X_new, and returns the new representation of the data based on the unsupervised model.\n**model.fit_transform()** : some estimators implement this method, which more efficiently performs a fit and a transform on the same input data."},{"metadata":{},"cell_type":"markdown","source":"<a id=\"4\"></a> <br>\n## 4- Load Data"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"for dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Total File sizes')\nprint('-'*10)\nfor f in os.listdir('../input/Kannada-MNIST'):\n    if 'zip' not in f:\n        print(f.ljust(30) + str(round(os.path.getsize('../input/Kannada-MNIST/' + f) / 1000000, 2)) + 'MB')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/Kannada-MNIST/train.csv')\ntest = pd.read_csv('../input/Kannada-MNIST/test.csv')\nsubmission = pd.read_csv('../input/Kannada-MNIST/sample_submission.csv')\nval= pd.read_csv('../input/Kannada-MNIST/Dig-MNIST.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.rename(columns={'id':'label'}, inplace=True)\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Train Shape: ', train.shape)\nprint('Test Shape:',test.shape)\nprint('Submission Shape: ',submission.shape)\nprint('Validation Shape: ',val.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.groupby(by='label').size()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"5\"></a> <br>\n## 5- Prepare Train and Test\n\nscikit-learn provides a helpful function for partitioning data, train_test_split, which splits out your data into a training set and a test set.\n\n- Training set for fitting the model\n- Test set for evaluation only"},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train, X_test, y_train, y_test = train_test_split(train.iloc[:, 1:], train.iloc[:, 0], test_size=0.2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_test.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"6\"></a> <br>\n## 6- Visualization\n some graphical representation of information and data."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Visualization Reference Kernel https://www.kaggle.com/josephvm/kannada-with-pytorch\n# Some quick data visualization \n# First 10 images of each class in the training set\n\nfig, ax = plt.subplots(nrows=10, ncols=10, figsize=(10,10))\n\n# I know these for loops look weird, but this way num_i is only computed once for each class\nfor i in range(10): # Column by column\n    num_i = X_train[y_train == i]\n    ax[0][i].set_title(i)\n    for j in range(10): # Row by row\n        ax[j][i].axis('off')\n        ax[j][i].imshow(num_i.iloc[j, :].to_numpy().astype(np.uint8).reshape(28, 28), cmap='gray')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"7\"></a> <br>\n## 7- Machine Learning Algorithm\n"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"10\"></a> <br>\n## 7.1 Logistic Regression\n\nDon’t get confused by its name! It is a classification not a regression algorithm. It is used to estimate discrete values ( Binary values like 0/1, yes/no, true/false ) based on given set of independent variable(s). In simple words, it predicts the probability of occurrence of an event by fitting data to a logit function. Hence, it is also known as logit regression. Since, it predicts the probability, its output values lies between 0 and 1 (as expected)."},{"metadata":{"trusted":true},"cell_type":"code","source":"# LogisticRegression\nfrom sklearn.linear_model import LogisticRegression\nModelLR = LogisticRegression(C=5, solver='lbfgs', multi_class='multinomial')\nModelLR.fit(X_train, y_train)\n\ny_predLR = ModelLR.predict(X_test)\n\n# Accuracy score\nprint('accuracy is',accuracy_score(y_predLR,y_test))\n\nscore = accuracy_score(y_predLR,y_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cm = confusion_matrix(y_test, y_predLR)\nprint(cm)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(9,9))\nsns.heatmap(cm, annot=True, fmt=\".3f\", linewidths=.5, square = True, cmap = 'Blues_r');\nplt.ylabel('Actual label');\nplt.xlabel('Predicted label');\nall_sample_title = 'Accuracy Score: {0}'.format(score)\nplt.title(all_sample_title, size = 15);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"11\"></a> <br>\n## 7.2 Decision Tree \n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Seed for reproducability\nseed = 1234\nnp.random.seed(seed)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.tree import DecisionTreeClassifier, export_graphviz\n\nDT = DecisionTreeClassifier(max_depth=10, random_state=seed)\nDT.fit(X_train, y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_predDT = DT.predict(X_test)\n\n# Accuracy score\nprint('accuracy DT',accuracy_score(y_predDT,y_test))\n\nscoreDT= accuracy_score(y_predDT,y_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"DTm =confusion_matrix(y_test, y_predDT)\nprint(DTm)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(9,9))\nsns.heatmap(DTm, annot=True, fmt=\".3f\", linewidths=.5, square = True, cmap = 'Blues_r');\nplt.ylabel('Actual label');\nplt.xlabel('Predicted label');\nall_sample_title = 'Accuracy Score Desicion Tree: {0}'.format(scoreDT)\nplt.title(all_sample_title, size = 15);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"12\"></a> <br>\n## 7.3 PCA svm"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn import svm\nfrom sklearn.decomposition import PCA\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pca = PCA(n_components=0.7,whiten=True)\nX_train_PCA = pca.fit_transform(X_train)\nX_test_PCA = pca.transform(X_test)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sv = svm.SVC(kernel='rbf',C=9)\nsv.fit(X_train_PCA , y_train)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_predsv = sv.predict(X_test_PCA)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('accuracy is',accuracy_score(y_predsv,y_test))\n\nscoreclf= accuracy_score(y_predsv,y_test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"13\"></a> <br>\n## 7.4 XGBOOST"},{"metadata":{"trusted":true},"cell_type":"code","source":"from xgboost import XGBClassifier\n# fit model no training data\nmodel = XGBClassifier()\neval_set = [(X_test,y_test)]\nmodel.fit(X_train, y_train, early_stopping_rounds= 5, eval_set=eval_set, verbose=True)\n# make predictions for test data\ny_pred = model.predict(X_test)\npredictions = [round(value) for value in y_pred]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import accuracy_score\n# evaluate predictions\naccuracy = accuracy_score(y_test, predictions)\nprint(\"Accuracy XGBOOST: %.2f%%\" % (accuracy * 100.0))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"14\"></a> <br>\n## 7.5 AdaBoost"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.ensemble import AdaBoostClassifier\nModel=AdaBoostClassifier()\nModel.fit(X_train, y_train)\ny_predAda=Model.predict(X_test)\n\n# Summary of the predictions made by the classifier\nprint(classification_report(y_test,y_predAda))\nprint(confusion_matrix(y_pred,y_test))\n#Accuracy Score\nprint('accuracy is ',accuracy_score(y_predAda,y_test))\n\nAdaB = accuracy_score(y_predAda,y_test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## SCORES "},{"metadata":{"trusted":true},"cell_type":"code","source":"models = pd.DataFrame({\n    'Model': ['LogisticRegression','Decision Tree', 'PCA', 'XGBOOST', \"AdaBoost classifier\"\n              ],\n    'Score': [score,scoreDT,scoreclf,accuracy,AdaB]})\nmodels.sort_values(by='Score', ascending=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.subplots(figsize =(10, 5))\n\nsns.barplot(x='Score', y = 'Model', data = models, palette=\"Set3\")\n\n#prettify using pyplot: https://matplotlib.org/api/pyplot_api.html\nplt.title('Machine Learning Algorithm Accuracy Score \\n')\nplt.xlabel('Accuracy Score (%)')\nplt.ylabel('Algorithm')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"15\"></a> <br>\n## 15- Submit Prediction"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_x = test.values[:,1:]\ntest_x = pca.transform(test_x)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"preds = sv.predict(test_x)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission['label'] = preds\nsubmission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"[Go to top](#top)"},{"metadata":{},"cell_type":"markdown","source":"\n **I hope this kernel helpful and some <font color=\"red\"><b>UPVOTES</b></font> would be very much appreciated**\n "}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}