{"cells":[{"metadata":{},"cell_type":"markdown","source":"I read a detailed [discussion](https://www.kaggle.com/c/digit-recognizer/discussion/61480) on how to score top scores in the mnist dataset. It had multiple approches. I was curious to check if it's actually possible to get those score.\n\nI'm going to try the first three options on the list\n* 92%: Multinomial logistic regression aka softmax regression is a simple first attempt scoring 92%.\n* 97%: Non-linear methods will score 97%. These include kNN, random forest, etc\n* 98%: SVM with a nonlinear kernel, or a fully connected neural network is the next thing that people try. If tuned well, these methods can climb to 99%. Add preprocessing and they can go a little higher."},{"metadata":{},"cell_type":"markdown","source":"I have taken bits and pieces from the following kernals:\n* https://www.kaggle.com/kmader/capsulenet-on-mnist\n* https://www.kaggle.com/archaeocharlie/a-beginner-s-approach-to-classification"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.model_selection import train_test_split\nfrom skimage.feature import hog\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn import svm\nfrom sklearn.metrics import accuracy_score\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"train = pd.read_csv(\"../input/train.csv\")\ntest = pd.read_csv(\"../input/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_full = train.iloc[:,1:]\ny_full = train.iloc[:,:1]\nx_train, x_test, y_train, y_test = train_test_split(X_full, y_full, test_size = 0.15)\nxt_non, xte_non, yt_non, yte_non = train_test_split(X_full, y_full, test_size = 0.15)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#reshape and normalise the dataset\nx_train = x_train.values.reshape(-1, 28, 28).astype('float32') / 255.\nx_test = x_test.values.reshape(-1, 28, 28).astype('float32') / 255.\ntest_set = test.values.reshape(-1, 28, 28).astype('float32') / 255.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For those whose wondering what's hog(Histogram of Oriented Gradients), find more details [here](https://scikit-image.org/docs/dev/auto_examples/features_detection/plot_hog.html)"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_features(data):\n    features = []\n    for image in data:\n        feat, hog_image = hog(image, orientations=8, pixels_per_cell=(2, 2), cells_per_block=(1, 1), block_norm='L2-Hys',\\\n                              visualize=True, multichannel=False)\n        features.append(feat)\n    return features","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tr_feat = get_features(x_train)\nts_feat = get_features(x_test)\ntest_set = get_features(test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"knn_classifier = KNeighborsClassifier(n_neighbors=10, weights='uniform', n_jobs=4)\nknn_classifier.fit(tr_feat, y_train['label'].values)\n#y_pred = knn_classifier.predict(ts_feat)\n#accuracy_score(y_test, y_pred)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"log_clf = LogisticRegression()\nlog_clf.fit(xt_non, yt_non)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"After submission of the result I got 90%, pretty close to what he had mentioned in the post. Maybe if I tune the hyper-parameters then there that could improve the result."},{"metadata":{"trusted":true},"cell_type":"code","source":"svc_clf = svm.SVC()\nsvc_clf.fit(xt_non, yt_non)\n#svc_clf.score(xte_non, yte_non)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For some reason the SVC gave 11% accuracy. I'm pretty sure that I screwed something up. Will need look into it a little more."},{"metadata":{"trusted":true},"cell_type":"code","source":"results = knn_classifier.predict(test_set)\n\nresults = pd.Series(results, name=\"Label\")\nsubmission = pd.concat([pd.Series(range(1,28001),name = \"ImageId\"),results],axis = 1)\n\nsubmission.to_csv(\"cnn_digit_knn.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"results = svc_clf.predict(test)\n\nresults = pd.Series(results, name=\"Label\")\nsubmission = pd.concat([pd.Series(range(1,28001),name = \"ImageId\"),results],axis = 1)\n\nsubmission.to_csv(\"cnn_digit_svm.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"results = log_clf.predict(test)\n\nresults = pd.Series(results, name=\"Label\")\nsubmission = pd.concat([pd.Series(range(1,28001),name = \"ImageId\"),results],axis = 1)\n\nsubmission.to_csv(\"cnn_digit_log.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As of now the discussion is yet to disappoint. All the information provided were precise and useful. Will try out the second part to see if we are able to replicate the results."},{"metadata":{},"cell_type":"markdown","source":"Please let me know if there are a few other things that I could try or improve. If anyone could suggest other [features](https://scikit-image.org/docs/dev/api/skimage.feature.html) from skimage which coult lead to a better result."},{"metadata":{},"cell_type":"markdown","source":"If you had liked it then vote, if you had loved it then fork."},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}