{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction\n\nFor a change of pace I'd like to try an unusual approach to this common challenge. Looking at how methods normally reserved for tabular data problems perform on the dataset.\n\nIt should be noted that all of these methods are likely to face a similar issue, that all columns are equally important. In image recognition it is often very helpful that pixels nearer to a selected pixel are assigned a higher importance in regard to that selected pixel than pixels far away from it. This is not desirable in tabular data and hence there is an issue, still it is interesting to see how far we might get.  ","metadata":{}},{"cell_type":"code","source":"# Import Libraries\n\nimport numpy as np\nimport pandas as pd \nfrom PIL import Image\nfrom matplotlib import pyplot as plt\nimport seaborn as sns\nimport os ","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-04T03:08:45.836809Z","iopub.execute_input":"2022-02-04T03:08:45.837122Z","iopub.status.idle":"2022-02-04T03:08:45.84251Z","shell.execute_reply.started":"2022-02-04T03:08:45.837083Z","shell.execute_reply":"2022-02-04T03:08:45.841753Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's look at the images","metadata":{}},{"cell_type":"code","source":"# load data\ndf = pd.read_csv(r\"../input/digit-recognizer/train.csv\",dtype = np.float32)\n\n# Split data into features X and labels y\nX = df.iloc[:,1:].values / 255 \ny = df.iloc[:,0].values\n\nfig,ax = plt.subplots(2,5)\nfor i in range(10):\n    nparray = X[i].reshape(28,28)\n    image = Image.fromarray(nparray * 255)\n    ax[i%2][i//2].imshow(image)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-04T03:08:45.844051Z","iopub.execute_input":"2022-02-04T03:08:45.844462Z","iopub.status.idle":"2022-02-04T03:08:49.115125Z","shell.execute_reply.started":"2022-02-04T03:08:45.844432Z","shell.execute_reply":"2022-02-04T03:08:49.113768Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's look at the distribution of classes in the dataset","metadata":{}},{"cell_type":"code","source":"sns.histplot(y)","metadata":{"execution":{"iopub.status.busy":"2022-02-04T03:08:49.116903Z","iopub.execute_input":"2022-02-04T03:08:49.117133Z","iopub.status.idle":"2022-02-04T03:08:49.397578Z","shell.execute_reply.started":"2022-02-04T03:08:49.117102Z","shell.execute_reply":"2022-02-04T03:08:49.396699Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Roughly equal which is good. Interestingly I always figured MNIST would be completely balanced.","metadata":{}},{"cell_type":"markdown","source":"# Create Decision Tree and Visualize Result","metadata":{}},{"cell_type":"code","source":"from sklearn.tree import DecisionTreeClassifier\nfrom sklearn import tree\n\ntree_set = df.copy()\ntarget = tree_set.iloc[:,0]\ntree_set_X = tree_set.iloc[:,1:] \n\nclf = DecisionTreeClassifier(max_depth=4)\nclf.fit(tree_set_X, target)\nclf.score(tree_set_X, target)","metadata":{"execution":{"iopub.status.busy":"2022-02-04T03:08:49.399327Z","iopub.execute_input":"2022-02-04T03:08:49.399651Z","iopub.status.idle":"2022-02-04T03:08:52.284049Z","shell.execute_reply.started":"2022-02-04T03:08:49.39961Z","shell.execute_reply":"2022-02-04T03:08:52.283519Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"With a max depth of 4 I find even this accuracy kind of crazy. Let's see it:","metadata":{}},{"cell_type":"code","source":"lst = tree_set_X.columns.tolist()\ntext_representation = tree.export_text(clf, feature_names=lst)\nprint(text_representation)","metadata":{"execution":{"iopub.status.busy":"2022-02-04T03:08:52.285129Z","iopub.execute_input":"2022-02-04T03:08:52.285434Z","iopub.status.idle":"2022-02-04T03:08:52.290818Z","shell.execute_reply.started":"2022-02-04T03:08:52.285394Z","shell.execute_reply":"2022-02-04T03:08:52.290172Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I'd say this is indicative that a few cental pixels can be definitive in this dataset in regards to which number they are.","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nfrom sklearn import preprocessing\n\ntrain_ratio = 0.90\n\nX_train, X_val, y_train, y_val = train_test_split(X, y, test_size=1 - train_ratio, stratify = y, random_state = 0)\n\nscaler = preprocessing.StandardScaler().fit(X_train)\nX_train = scaler.transform(X_train)\nX_val = scaler.transform(X_val)","metadata":{"execution":{"iopub.status.busy":"2022-02-04T03:08:52.292058Z","iopub.execute_input":"2022-02-04T03:08:52.29229Z","iopub.status.idle":"2022-02-04T03:08:52.976958Z","shell.execute_reply.started":"2022-02-04T03:08:52.29226Z","shell.execute_reply":"2022-02-04T03:08:52.976044Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.ensemble import ExtraTreesClassifier\n\nclf = ExtraTreesClassifier(random_state=1)\n\nclf.fit(X_train, y_train)\nprint(clf.score(X_val, y_val))","metadata":{"execution":{"iopub.status.busy":"2022-02-04T03:08:52.978232Z","iopub.execute_input":"2022-02-04T03:08:52.978488Z","iopub.status.idle":"2022-02-04T03:09:15.933272Z","shell.execute_reply.started":"2022-02-04T03:08:52.978453Z","shell.execute_reply":"2022-02-04T03:09:15.932598Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let's try it in binary","metadata":{}},{"cell_type":"code","source":"from sklearn.preprocessing import Binarizer\n\nX_train_bin = Binarizer().fit_transform(X_train)\nX_val_bin = Binarizer().fit_transform(X_val)\n\nfig,ax = plt.subplots(2,5)\nfor i in range(10):\n    nparray = X_train_bin[i].reshape(28,28)\n    image = Image.fromarray(nparray * 255)\n    ax[i%2][i//2].imshow(image)\n    ax[i%2][i//2].title.set_text(round(y_train[i]))\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-04T03:09:15.934431Z","iopub.execute_input":"2022-02-04T03:09:15.935576Z","iopub.status.idle":"2022-02-04T03:09:17.096106Z","shell.execute_reply.started":"2022-02-04T03:09:15.93547Z","shell.execute_reply":"2022-02-04T03:09:17.094731Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"et_clf = ExtraTreesClassifier(random_state=1)\n\net_clf.fit(X_train_bin, y_train)\nprint(et_clf.score(X_val_bin, y_val))","metadata":{"execution":{"iopub.status.busy":"2022-02-04T03:09:17.097436Z","iopub.execute_input":"2022-02-04T03:09:17.097665Z","iopub.status.idle":"2022-02-04T03:09:39.391758Z","shell.execute_reply.started":"2022-02-04T03:09:17.097632Z","shell.execute_reply":"2022-02-04T03:09:39.390877Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"That's getting better in terms of accuracy. Let's try and see if we can find the bad images.","metadata":{}},{"cell_type":"code","source":"bad_image_list = []\nbad_image_list_y = []\nbad_predict = []\nfor _X, _y in zip(X_val_bin, y_val):\n    if et_clf.predict(_X.reshape(1, -1)) != _y:\n        bad_image_list.append(_X)\n        bad_image_list_y.append(_y)\n        bad_predict.append(et_clf.predict(_X.reshape(1, -1)).item())\n        if len(bad_image_list) > 9:\n            break\n    \nfig,ax = plt.subplots(2,5)\nfor i in range(10):\n    nparray = bad_image_list[i].reshape(28,28)\n    image = Image.fromarray(nparray * 255)\n    ax[i%2][i//2].imshow(image)\n    ax[i%2][i//2].title.set_text(\"Real: \" + str(round(bad_image_list_y[i])))\nfig.show()\n\nprint(\"Bad predictions:\")\nprint(bad_predict)","metadata":{"execution":{"iopub.status.busy":"2022-02-04T03:09:39.395146Z","iopub.execute_input":"2022-02-04T03:09:39.395479Z","iopub.status.idle":"2022-02-04T03:09:43.603285Z","shell.execute_reply.started":"2022-02-04T03:09:39.395431Z","shell.execute_reply":"2022-02-04T03:09:43.601316Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's see a confusion matrix of this","metadata":{}},{"cell_type":"code","source":"from sklearn.metrics import confusion_matrix\n\nconf_mat = confusion_matrix(y_val, et_clf.predict(X_val_bin))\nsns.heatmap(conf_mat, vmax=10, cmap=\"Blues\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-04T03:09:43.604908Z","iopub.execute_input":"2022-02-04T03:09:43.605198Z","iopub.status.idle":"2022-02-04T03:09:44.134863Z","shell.execute_reply.started":"2022-02-04T03:09:43.605158Z","shell.execute_reply":"2022-02-04T03:09:44.133293Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Stacking Models\n\nWe can try stacking models for better results.","metadata":{}},{"cell_type":"code","source":"from sklearn.ensemble import RandomForestClassifier\n\nrf_clf = RandomForestClassifier(random_state=1)\n\nrf_clf.fit(X_train_bin, y_train)\nprint(rf_clf.score(X_val_bin, y_val))","metadata":{"execution":{"iopub.status.busy":"2022-02-04T03:09:44.136387Z","iopub.execute_input":"2022-02-04T03:09:44.136613Z","iopub.status.idle":"2022-02-04T03:09:57.194392Z","shell.execute_reply.started":"2022-02-04T03:09:44.136586Z","shell.execute_reply":"2022-02-04T03:09:57.193099Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.tree import DecisionTreeClassifier\n\ndt_clf = DecisionTreeClassifier(random_state=1)\n\ndt_clf.fit(X_train_bin, y_train)\nprint(dt_clf.score(X_val_bin, y_val))","metadata":{"execution":{"iopub.status.busy":"2022-02-04T03:09:57.195659Z","iopub.execute_input":"2022-02-04T03:09:57.19587Z","iopub.status.idle":"2022-02-04T03:10:01.844652Z","shell.execute_reply.started":"2022-02-04T03:09:57.195844Z","shell.execute_reply":"2022-02-04T03:10:01.843432Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.ensemble import StackingClassifier\nfrom sklearn.linear_model import LogisticRegression\n\nestimators = [\n     ('rf', RandomForestClassifier(random_state=1)),\n     ('et', ExtraTreesClassifier(random_state=1))\n]\n\nstack_clf = StackingClassifier(\n    estimators=estimators, final_estimator=LogisticRegression()\n)\n\nstack_clf.fit(X_train_bin, y_train)\nprint(stack_clf.score(X_val_bin, y_val))","metadata":{"execution":{"iopub.status.busy":"2022-02-04T03:10:01.845946Z","iopub.execute_input":"2022-02-04T03:10:01.846134Z","iopub.status.idle":"2022-02-04T03:13:12.27465Z","shell.execute_reply.started":"2022-02-04T03:10:01.846109Z","shell.execute_reply":"2022-02-04T03:13:12.273977Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Create Submission","metadata":{}},{"cell_type":"code","source":"test_df = pd.read_csv(\"../input/digit-recognizer/test.csv\")\ntest_X = scaler.transform(test_df.values)\ntest_X = Binarizer().fit_transform(test_X)\npredicted = stack_clf.predict(test_X)","metadata":{"execution":{"iopub.status.busy":"2022-02-04T03:13:12.275865Z","iopub.execute_input":"2022-02-04T03:13:12.27652Z","iopub.status.idle":"2022-02-04T03:13:14.615057Z","shell.execute_reply.started":"2022-02-04T03:13:12.276483Z","shell.execute_reply":"2022-02-04T03:13:14.612906Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Check output visually","metadata":{}},{"cell_type":"code","source":"fig,ax = plt.subplots(2,5)\nfor i in range(10):\n    nparray = test_X[i].reshape(28,28)\n    image = Image.fromarray(nparray * 255)\n    ax[i%2][i//2].imshow(image)\n    ax[i%2][i//2].title.set_text(round(predicted[i]))\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-04T03:19:23.972134Z","iopub.execute_input":"2022-02-04T03:19:23.972419Z","iopub.status.idle":"2022-02-04T03:19:24.952218Z","shell.execute_reply.started":"2022-02-04T03:19:23.972393Z","shell.execute_reply":"2022-02-04T03:19:24.950756Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = pd.read_csv(\"../input/digit-recognizer/sample_submission.csv\")\nprint(submission.head())\nsubmission[\"Label\"] = np.rint(predicted)\nsubmission = submission.astype(int)\nsubmission.to_csv(\"submission.csv\", index=False)\nprint(submission.head())","metadata":{"execution":{"iopub.status.busy":"2022-02-04T03:20:33.636134Z","iopub.execute_input":"2022-02-04T03:20:33.636382Z","iopub.status.idle":"2022-02-04T03:20:33.692155Z","shell.execute_reply.started":"2022-02-04T03:20:33.636356Z","shell.execute_reply":"2022-02-04T03:20:33.691071Z"},"trusted":true},"execution_count":null,"outputs":[]}]}