{"cells":[{"metadata":{},"cell_type":"markdown","source":"MNIST is a classic dataset with handwritten digits. The challenge is classify digits from 0 to 9 without using neural networks.  \nFirst of all, let's read train and test datasets from CSV files using pandas lib."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Loading train and test data from csv files\ntrain_full = pd.read_csv('../input/digit-recognizer/train.csv')\n\nX_train_full = train_full.drop(['label'], axis=1)\ny_train_full = train_full['label']\nX_test = pd.read_csv('../input/digit-recognizer/test.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Check if data is loaded correctly and preview some images from input data source"},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\n\ndef show_images(images, titles, cols = 1):\n    assert((titles is None) or (len(images) == len(titles)))\n    \n    n_images = len(images)\n    fig = plt.figure()\n    images = np.reshape(images.values, (-1, 28, 28))\n\n    for n, (image, title) in enumerate(zip(images, titles)):\n        a = fig.add_subplot(cols, np.ceil(n_images/float(cols)), n + 1)\n        a.set_axis_off()\n        a.set_title(title, size=32)\n        plt.imshow(image, interpolation='nearest', cmap='gray', vmin=0, vmax=255)\n\n    fig.set_size_inches(np.array(fig.get_size_inches()) * n_images)\n    plt.show()\n    \nshow_images(X_train_full[:8], y_train_full[:8].values)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Split all data to train and validation datasets. Limits the size of dataset to speed up hyperparameter tunning."},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\nn_samples = 5000\nX_train, X_val, y_train, y_val = train_test_split(X_train_full[:n_samples], y_train_full[:n_samples], test_size=0.1, shuffle=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Train model with default parameters and get base score.  \nUsed SVM (support vector machines) classifier as the most accurete for such image classification tasks.  \nAlso I tried SGDClassifier and didn't get better results."},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.svm import SVC\n\nclf = SVC().fit(X_train, y_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn import metrics\n\npred_val = clf.predict(X_val)\n\nprint(\"Classification report for classifier %s:\\n%s\\n\"\n      % (clf, metrics.classification_report(y_val, pred_val)))\ndisp = metrics.plot_confusion_matrix(clf, X_val, y_val)\ndisp.figure_.suptitle(\"Confusion Matrix\")\n\nplt.show() ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Accuracy 0.96 is good enough. But let's improve this.  \nTrying to fing best hyperparameters for SVC classifier using GridSearchCV."},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import GridSearchCV\n\nparams = {\n    \"kernel\" : [\"rbf\", 'linear'],\n    \"C\" : [1, 10, 100]\n}\n\ngs_clf = GridSearchCV(SVC(), params, verbose=2)\ngs_clf.fit(X_train, y_train)\nprint(\"Best params for SVC: \", gs_clf.best_params_)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Train model using better parameters and dataset with same size."},{"metadata":{"trusted":true},"cell_type":"code","source":"clf = SVC(**gs_clf.best_params_).fit(X_train, y_train)\n\npred_val = clf.predict(X_val)\n\nprint(\"Classification report for classifier %s:\\n%s\\n\"\n      % (clf, metrics.classification_report(y_val, pred_val)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The accuracy has improved.  \nTrain model using better parameters and a large dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"n_samples = 20000\nX_train, X_val, y_train, y_val = train_test_split(X_train_full[:n_samples], y_train_full[:n_samples], test_size=0.1, shuffle=False)\n\nclf = SVC(**gs_clf.best_params_).fit(X_train, y_train)\n\npred_val = clf.predict(X_val)\n\nprint(\"Classification report for classifier %s:\\n%s\\n\"\n      % (clf, metrics.classification_report(y_val, pred_val)))\ndisp = metrics.plot_confusion_matrix(clf, X_val, y_val)\ndisp.figure_.suptitle(\"Confusion Matrix\")\n\nplt.show() ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Display some false predictions to show where our model is weak."},{"metadata":{"trusted":true},"cell_type":"code","source":"val_df = pd.DataFrame(y_val)\nval_df['preds'] = pred_val\nfalse_pred = val_df[val_df['label'] != val_df['preds']][:8]\n\nshow_images(X_val.loc[false_pred.index], \"Pred: \" + false_pred['preds'].astype(str) + \", GT:\" + false_pred['label'].astype(str))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Predict labels for test dataset and save results for Kaggle leaderboard."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Getting final results\")\n\ntest_preds = clf.predict(X_test)\nshow_images(X_test[:8], test_preds[:8])\n\n# The lines below shows how to save predictions in format used for competition scoring\noutput = pd.DataFrame({'ImageId': X_test.index + 1,\n                       'Label': test_preds})\noutput.to_csv('submission.csv', index=False)\n\nprint(\"Results saved\")","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}