{"cells":[{"metadata":{"_uuid":"41b603957b11981792e4c4374b6230b368e529fb"},"cell_type":"markdown","source":"### Ari Purnama Aji | 1313617008 | Ilmu Komputer 2017 | Universitas Negeri Jakarta\n\n# Digit Recognition Experiment"},{"metadata":{"_uuid":"54168039de1351bbe513132a12f198766db8746f"},"cell_type":"markdown","source":"## Library Python\nImport numpy, pandas, dan os\n### 1. Numpy \nNumpy memiliki kegunaan untuk operasi vektor dan matriks. Fiturnya hampir sama dengan MATLAB dalam mengelola array dan array multidimensi. Numpy merupakan salah satu library yang digunakan oleh library lain seperti Scikit-Learn untuk keperluan analisis data.\n\n### 2. Pandas\nDengan menggunakan sistem dataframe, kamu dapat memuat sebuah file ke dalam tabel virtual ala spreadsheet dengan menggunakan Pandas. Dengan menggunakan Pandas, kamu dapat mengolah suatu data dan mengolahnya seperti join, distinct, group by, agregasi, dan teknik seperti pada SQL. Hanya saja dilakukan pada tabel yang dimuat dari file ke RAM.\nPandas juga dapat membaca file dari berbagai format seperti .txt, .csv, .tsv, dan lainnya."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"scrolled":false},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"504da5a5138c0729e8d510c86e4195288554fb80"},"cell_type":"markdown","source":"# Library Import\n#### 1. At least a classification algorithm (SVM or Decision Tree is a Good Choice)\nSupport vector machines (SVMs) adalah satu set metode pembelajaran yang digunakan untuk deteksi klasifikasi, regresi dan deteksi outliers. Sedangkan,\nDecision Trees (DTs) adalah metode pembelajaran  yang digunakan untuk klasifikasi dan regresi . Tujuannya adalah untuk membuat model yang memprediksi nilai variabel target dengan mempelajari aturan keputusan sederhana yang disimpulkan dari fitur data.\n#### 2. Matplotlib\nMatplotlib adalah library yang digunakan oleh data science untuk menyajikan datanya ke dalam visual yang lebih baik.\n#### 3. Preprocessing tools\n#### 4. Train test split\nPembagian dataset menjadi data train dan data test."},{"metadata":{"trusted":true,"_uuid":"19f6558573e6282e4a6c72cb79566e039e76356a"},"cell_type":"code","source":"import matplotlib.pyplot as plt, matplotlib.image as mpimg\nfrom sklearn.model_selection import train_test_split\nfrom sklearn import svm, tree\n#%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dfcf44e971185a0567031b3a0e799e2689485e99"},"cell_type":"markdown","source":"# Load Data"},{"metadata":{"trusted":true,"_uuid":"6c37f07666e5f34cd7910e89572f6a89f74d032a"},"cell_type":"code","source":"# load the data\nlabeled_images = pd.read_csv('../input/train.csv')\nimages = labeled_images.iloc[0:5000,1:]\nlabels = labeled_images.iloc[0:5000,:1]\ntrain_images, test_images,train_labels, test_labels = train_test_split(images, labels, test_size=0.2, random_state=0)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c7246a86ac0b5bb5e5c6b35f2624db4e4b293705"},"cell_type":"markdown","source":"### Q1\nNotice in the above we used labeled_images.iloc?, can you confirm on the documentation? what is the role?"},{"metadata":{"_uuid":"7165a6ee3c8d409a4cdd5562e255361a00788b4d"},"cell_type":"markdown","source":"### Answer Q1 :\nPeran \"Iloc\" di panda digunakan untuk memilih baris dan kolom dengan nomor, dalam urutan yang muncul pada data frame. Pada deklarasi variabel images terdapat perintah labeled_images.iloc[0:5000, 1:] berarti, variabel images mengambil data dari labeled_images sebanyak 5000 baris pertama dan kolom kedua sampai kolom terakhir. \nDan pada deklarasi variabel labels terdapat perintah labeled_images.iloc[0:5000, :1] berarti, variabel labels mengambil data dari labeled_images sebanyak 5000 baris pertama dan kolom pertama. \nKemudian kita melakukan split data pada variabel images dan labels menggunakan fungsi train_test_split dengan rasio data test_size = 0.2 dari total jumlah data images dan labels."},{"metadata":{"trusted":true,"_uuid":"7885bcb7bcf5836e0ac89a47544e5dbac097a2d3"},"cell_type":"code","source":"# now we gonna load the second image, reshape it as matrix than display it\ni=1\nimg=train_images.iloc[i].values\nimg=img.reshape((28,28))\nplt.imshow(img,cmap='gray')\nplt.title(train_labels.iloc[i,0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ee3129ba06d0fe49fcec360f90d75b200641a95d"},"cell_type":"markdown","source":"### Q2\nNow plot an image for each image class"},{"metadata":{"_uuid":"fce3d38114e75375ba76f3a3d68a52cbc964a740"},"cell_type":"markdown","source":"### Answer Q2:"},{"metadata":{"trusted":true,"_uuid":"b300ee3c5e9988a45be772e3bb405c797877c3f6"},"cell_type":"code","source":"# Todo: Put your code here\narr = [[],[],[],[],[],[],[],[],[],[]]\nfor i in range(len(train_labels)):\n    a = train_labels.iloc[i, 0]\n    arr[a] += [i]\n    \nfor i in range(10):\n    j = arr[i][0]\n    img = train_images.iloc[j].values\n    img = img.reshape((28,28))\n    plt.figure()\n    plt.imshow(img,cmap = 'gray')\n    judul = 'Kelas ' + str(train_labels.iloc[j,0])\n    plt.title(judul)\n    plt.show()\n    plt.close()\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ca68bd4299688c943856c0ae9116038affd6fe30"},"cell_type":"markdown","source":"Now plot the histogram within img"},{"metadata":{"trusted":true,"_uuid":"05525f9ae745028873350cebd8ee654dcba6f547"},"cell_type":"code","source":"train_images.iloc[i].describe()\nprint(type(train_images.iloc[i]))\nplt.hist(train_images.iloc[i])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7f61ae0147949e346814cdf731a6eb919748e999"},"cell_type":"markdown","source":"### Q3\nCan you check in what class does this histogram represent?. How many class are there in total for this digit data?. How about the histogram for other classes"},{"metadata":{"trusted":true,"_uuid":"5d248b02e7a0b68050ef715c465dad08570ccbf3"},"cell_type":"code","source":"# create histogram for each class (data merged per class)\n# Todo\n#print(train_labels.iloc[:5])\n#data1 = train_images.iloc[1]\n#data2 = train_images.iloc[3]\n#data1 = np.array(data1)\n#data2 = np.array(data2)\n#data3 = np.append(data1,data2)\n#print(len(data3))\n#plt.hist(data3)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cbdc60c1a5418edff2005a8ae374caa656d148db"},"cell_type":"markdown","source":"### Answer Q3:"},{"metadata":{"trusted":true,"_uuid":"6bd9ec85a37abea864118644bb47f2a6326883c3"},"cell_type":"code","source":"# create histogram for each class (data merged per class)\n# Todo\n# This is the progam to answer Q3\n\n# Menampilkan histogram untuk perkelas\nfor i in range(10):\n    j = arr[i][0]\n    data = train_images.iloc[j]\n    data = np.array(data)\n    plt.figure()\n    plt.hist(data)\n    plt.title('Histogram Kelas ' + str(train_labels.iloc[j,0]))\n    plt.xlabel('Nilai Pixel')\n    plt.ylabel('Jumlah Kolom')\n    plt.show()\n    plt.close()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3b11a4d20ae8c883569a67c9a518cdcede89ba34"},"cell_type":"code","source":"# Menampilkan histogram untuk kelas 0 sampai 9   \nsemua_data = np.append(train_images.iloc[data[0]], (train_images.iloc[data[1:]]))\nplt.figure()\nplt.hist(semua_data)\nplt.title(\"Histogram Kelas 0 sampai 9\")\nplt.xlabel('Nilai Pixel')\nplt.ylabel('Jumlah Kolom')\nplt.show()\nplt.close()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"94db2108a6439f83ca1d4cbd6601e8c7c352a39d"},"cell_type":"markdown","source":"### Train the model\nNow we are ready to train the model, for starter let's use SVM. For the learning most model in SKLearn adopt the usual fit() and predict()."},{"metadata":{"trusted":true,"_uuid":"32c759ad74a1bd70cab3f5c9bc22e3f59eb0cdc7"},"cell_type":"code","source":"clf = svm.SVC()\nclf.fit(train_images, train_labels.values.ravel())\nclf.score(test_images,test_labels)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8401cd8917001e560125977d5d2a3793788d51c3"},"cell_type":"markdown","source":"### Q4\nIn above, did you see score() function?, open SVM.score() dokumentation at SKLearn, what is its role?. Does it the same as MAE discussed in class previously?. Ascertain it through running the MAE. Now does score() and mae() prooduce the same results?."},{"metadata":{"trusted":true,"_uuid":"66fee3a2d12be44d6642316aa00e6055c0f40540"},"cell_type":"code","source":"# Put your verification code here\n# Todo\n# print(train_labels.values.ravel())\n# print(np.unique(test_labels)) # to see class number\n\nfrom sklearn.tree import DecisionTreeRegressor\nfrom sklearn.metrics import mean_absolute_error\n\nclf_split_model = DecisionTreeRegressor()\nclf_split_model.fit(train_images, train_labels)\ntest_predictions = clf_split_model.predict(test_images)\nprint(mean_absolute_error(test_labels, test_predictions))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"072e1e5b211d2f2859d382e10401d95bc7e3c280"},"cell_type":"markdown","source":"### Answer Q4:\nsvm.SVC.Score() berperan untuk mencari akurasi pada data atau label uji yang diberikan.\n\nMAE berperan untuk mencari rata-rata error pada data atau label uji yang diberikan.\n\nKesimpulannya, baik svm.SVC.Score() dan MAE memiliki kesamaan peran yaitu untuk menguji kualitas model yang dibuat.\nNamun result yang diberikan antara keduanya berbeda, kalau svm.SVC.Score() memiliki result berupa rasio dari 0 sampai 1, yang artinya jika result yang dihasilkan semakin mendekati 1 berarti kualitas model yang dibuat semakin baik. \nSedangkan result dari MAE tidak menggunkan rasio tapi berupa besaran galat, yang  artinya jika result yang dihasilkan semakin mendekati bilangan 0 berarti kualitas model yang dibuat semakin baik."},{"metadata":{"_uuid":"2996ff6b239199721e07b60bc002f7105ea86dad"},"cell_type":"markdown","source":"### Improving Performance\nDid you noticed, that the performance is so miniscule in range of ~0.1. Before doing any improvement, we need to analyze what are causes of the problem?. But allow me to reveal one such factor. It was due to pixel length in [0, 255]. Let's see if we capped it into [0,1] how the performance are going to improved."},{"metadata":{"trusted":true,"_uuid":"dc4f2476b3832dd2016671b2dbd068c0b1e4fc67"},"cell_type":"code","source":"test_images[test_images>0]=1\ntrain_images[train_images>0]=1\n\nimg=train_images.iloc[i].values.reshape((28,28))\nplt.imshow(img,cmap='binary')\nplt.title(train_labels.iloc[i])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"28a2ec11ff79e4cf03584ddc672c94f1a748bcb1"},"cell_type":"code","source":"# now plot again the histogram\nplt.hist(train_images.iloc[i])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7e00a9bf4a5209427b12cc4bcd54f6bb4bdff297"},"cell_type":"markdown","source":"# Retrain the model\nUsing the now adjusted data, let's retrain our model to see the improvement"},{"metadata":{"trusted":true,"_uuid":"0a192210f1068c1eca82da16e4d32e25a4f83b4d"},"cell_type":"code","source":"clf = svm.SVC()\nclf.fit(train_images, train_labels.values.ravel())\nclf.score(test_images,test_labels)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f97d9efe56e2c874d51a554aa9f77f2be25f750c"},"cell_type":"markdown","source":"### Q5\nBased on this finding, Can you explain why if the value is capped into [0,1] it improved the performance significantly?. Perhaps you need to do several self designed test to see why."},{"metadata":{"_uuid":"d9862c7d8397eb9390a2212805747cbd7244a6b8"},"cell_type":"markdown","source":"### Answer Q5:\nPada data pixel yang pertama masih menggunakan data yang belum dinormalisasi yaitu dengan jumlah kelas pixel [0, 255] dan setelah dinormalisasi jumlah kelas pixel menjadi [0, 1]. Jadi kesimpulannya jumlah kelas pixel berpengaruh pada kompleksitas waktu karena jumlah kelas pixel merupakan parameter pada formula matematika milik SVC. Berikut dibawah ini pembuktian untuk melihat perbedaan waktu yang dihasilkan dari data pixel setelah dinormalisasi dan sebelum dinormalisasi."},{"metadata":{"trusted":true,"_uuid":"a4da8b676855327e65a1cd93bd156a24534b446b"},"cell_type":"code","source":"from time import time\nfrom time import sleep\n\n# Waktu yang dibutuhkan untuk mencari skor pada kasus data pixel setelah dinormalisasi\nclf.fit(train_images, train_labels.values.ravel())\na = float(time())\nclf.score(test_images,test_labels)\nb = float(time())\nelapsed = b - a\nprint(\"Waktu yang dibutuhkan: \", elapsed)\n\n# Waktu yang dibutuhkan untuk mencari skor pada kasus data pixel sebelum dinormalisasi\ntrain_img, test_img,train_lbl, test_lbl = train_test_split(images, labels, test_size=0.2, random_state=0)\nclf.fit(train_img, train_lbl.values.ravel())\na = float(time())\nclf.score(test_img,test_lbl)\nb = float(time())\nelapsed = b - a\nprint(\"Waktu yang dibutuhkan: \", elapsed)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"918841bffc6e25a062c6e6639f43939be68d8a98"},"cell_type":"markdown","source":"### Prediction labelling\nIn Kaggle competition, we don't usually submit the end test data performance on Kaggle. But what to be submitted is CSV of the prediction label stored in a file."},{"metadata":{"trusted":true,"_uuid":"3ad197cb7a950620e9184a830e5a60802595a633"},"cell_type":"code","source":"# Test again to data test\nclf.fit(train_images, train_labels.values.ravel())\ntest_data=pd.read_csv('../input/test.csv')\ntest_data[test_data>0]=1\nresults=clf.predict(test_data[0:5000])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d91cd05a9a0132a066886537d1f330490f16a5ca"},"cell_type":"code","source":"# separate code section to view the results\nprint(results)\nprint(len(results))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9ec60d0a9b596002c4e5d88edf789b839943f004"},"cell_type":"code","source":"# dump the results to 'results.csv'\ndf = pd.DataFrame(results)\ndf.index.name='ImageId'\ndf.index+=1\ndf.columns=['Label']\ndf.to_csv('results.csv', header=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3f0885705e4e39b86d2afcbb57bae87d013a1e14"},"cell_type":"code","source":"#check if the file created successfully\nprint(os.listdir(\".\"))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c978ad0179c48706ea47c3d1198fc29b811a9a39"},"cell_type":"markdown","source":"# Data Download\nWe have the file, can listed it but how we are take it from sever. Thus we also need to code the download link."},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"c3b565b37dbcd635e451a5da1df626c17de1e13d"},"cell_type":"code","source":"# from https://www.kaggle.com/rtatman/download-a-csv-file-from-a-kernel\n\n# import the modules we'll need\nfrom IPython.display import HTML\nimport pandas as pd\nimport numpy as np\nimport base64\n\n# function that takes in a dataframe and creates a text link to  \n# download it (will only work for files < 2MB or so)\ndef create_download_link(df, title = \"Download CSV file\", filename = \"data.csv\"):  \n    csv = df.to_csv()\n    b64 = base64.b64encode(csv.encode())\n    payload = b64.decode()\n    html = '<a download=\"{filename}\" href=\"data:text/csv;base64,{payload}\" target=\"_blank\">{title}</a>'\n    html = html.format(payload=payload,title=title,filename=filename)\n    return HTML(html)\n\ncreate_download_link(df)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"482098780b6edc6617b1bfcdfe690d77553bbdfd"},"cell_type":"markdown","source":"# Q6\nAlhamdulillah, we have completed our experiment. Here's things to do for your next task:\n* What are the overfitting factor of SVM algorithm?. Previously on decision tree regression, the factor was max_leaf nodes. Do similar expriment using SVM!\n* Apply Decision Tree Classifier on this dataset, seek the best overfitting factor, then compare it with results of SVM. \n* Apply Decision Tree Regressor on this dataset, seek the best overfitting factor, then compare it with results of SVM & Decision Tree Classifier. Provides the results in table/chart. I suspect they are basically the same thing.\n* Apply Decision Tree Classifier on the same dataset, use the best overfitting factor & value.  But do not use unnormalized dataset, before the value normalized to [0,1]"},{"metadata":{"_uuid":"bebda60aa2ecde132bd3af467bd65a1d0d9d01ad"},"cell_type":"markdown","source":"## Answer Q6:\n"},{"metadata":{"_uuid":"c67d5776fd9ac66bc624632cc8a72ab5ac4a38d1"},"cell_type":"markdown","source":"## SVM"},{"metadata":{"trusted":true,"_uuid":"2a75118d664f77d17d915e36e0c2270650828108"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"41ec5a0291b1b28151d8c007dcef8c6d8be52bb1"},"cell_type":"markdown","source":"## Decision Tree Classifier"},{"metadata":{"trusted":true,"_uuid":"65e1227989dca120e3ef463107c567c08f21750f"},"cell_type":"code","source":"from sklearn.tree import DecisionTreeClassifier\n\ndef getTrainMAE(max_leaf_nodes, train_images, test_images, train_labels, test_labels):\n    modelTrain = DecisionTreeClassifier(max_leaf_nodes = max_leaf_nodes, random_state = 0)\n    modelTrain.fit(train_images, train_labels)\n    predictTrain = modelTrain.predict(train_images)\n    mae = mean_absolute_error(train_labels, predictTrain)\n    return (mae)\n\ndef getTestMAE(max_leaf_nodes, train_images, test_images, train_labels, test_labels):\n    modelTest = DecisionTreeClassifier(max_leaf_nodes = max_leaf_nodes, random_state = 0)\n    modelTest.fit(train_images, train_labels)\n    predictTest = modelTest.predict(test_images)\n    mae = mean_absolute_error(test_labels, predictTest)\n    return (mae)\n\ntrainMAE = []\ntestMAE = []\n\nbestTrain = {}\nbestTest = {}\n\ncandidate_max_leaf_nodes = [5, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000]\nprint(\"Train MAE for each nodes:\")\nfor max_leaf_nodes in candidate_max_leaf_nodes:\n    myTrainMAE = getTrainMAE(max_leaf_nodes, train_images, test_images, train_labels, test_labels)\n    trainMAE.append(myTrainMAE)\n    bestTrain.update({max_leaf_nodes : myTrainMAE})\n    print(\"Max leaf nodes: %d  \\t\\t Train MAE:  %d\" %(max_leaf_nodes, myTrainMAE))\n\nprint(\"\\nTest MAE for each nodes:\")\nfor max_leaf_nodes in candidate_max_leaf_nodes:\n    myTestMAE = getTestMAE(max_leaf_nodes, train_images, test_images, train_labels, test_labels)\n    testMAE.append(myTestMAE)\n    bestTest.update({max_leaf_nodes : myTestMAE})\n    print(\"Max leaf nodes: %d  \\t\\t Test MAE:  %d\" %(max_leaf_nodes, myTestMAE))\n\nbest_tree_size_train  = min(bestTrain, key=bestTrain.get)\nbest_tree_size_test = min(bestTest, key=bestTest.get)\nprint(\"\\nTrain MAE for each nodes: \", trainMAE)\nprint(\"Test MAE for each nodes: \", testMAE)\nprint(\"\\nBest classifier tree size for train = \", best_tree_size_train)\nprint(\"Best classifier tree size for test = \", best_tree_size_test)\n\n\nplt.figure()\nplt.plot(candidate_max_leaf_nodes, trainMAE, color=\"red\")\nplt.plot(candidate_max_leaf_nodes, testMAE, color=\"green\")\nplt.xlabel(\"Tree Depth\")\nplt.ylabel(\"MAE\")\nplt.title(\"Decision Tree Classifier\")\nplt.show()\n\nCLS = DecisionTreeClassifier(max_leaf_nodes=best_tree_size_train, random_state=0)\nCLS.fit(train_images,train_labels)\nscoreCLS = CLS.score(test_images,test_labels)\nprint(\"Score Decision Tree Classifier = \",scoreCLS)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"055767a4e02f0246b17f3e732591778e7ae759ec"},"cell_type":"markdown","source":"## Decision Tree Regressor"},{"metadata":{"trusted":true,"_uuid":"26390b508f99c64cda1d8cb59c3ef153c7eaa48d"},"cell_type":"code","source":"def getTrainMAE(max_leaf_nodes, train_images, test_images, train_labels, test_labels):\n    modelTrain = DecisionTreeRegressor(max_leaf_nodes = max_leaf_nodes, random_state = 0)\n    modelTrain.fit(train_images, train_labels)\n    predictTrain = modelTrain.predict(train_images)\n    mae = mean_absolute_error(train_labels, predictTrain)\n    return (mae)\n\ndef getTestMAE(max_leaf_nodes, train_images, test_images, train_labels, test_labels):\n    modelTest = DecisionTreeRegressor(max_leaf_nodes = max_leaf_nodes, random_state = 0)\n    modelTest.fit(train_images, train_labels)\n    predictTest = modelTest.predict(test_images)\n    mae = mean_absolute_error(test_labels, predictTest)\n    return (mae)\n\ntrainMAE = []\ntestMAE = []\n\nbestTrain = {}\nbestTest = {}\n\ncandidate_max_leaf_nodes = [5, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000]\nprint(\"Train MAE for each nodes:\")\nfor max_leaf_nodes in candidate_max_leaf_nodes:\n    myTrainMAE = getTrainMAE(max_leaf_nodes, train_images, test_images, train_labels, test_labels)\n    trainMAE.append(myTrainMAE)\n    bestTrain.update({max_leaf_nodes : myTrainMAE})\n    print(\"Max leaf nodes: %d  \\t\\t Train MAE:  %d\" %(max_leaf_nodes, myTrainMAE))\n\nprint(\"\\nTest MAE for each nodes:\")    \nfor max_leaf_nodes in candidate_max_leaf_nodes:\n    myTestMAE = getTestMAE(max_leaf_nodes, train_images, test_images, train_labels, test_labels)\n    testMAE.append(myTestMAE)\n    bestTest.update({max_leaf_nodes : myTestMAE})\n    print(\"Max leaf nodes: %d  \\t\\t Test MAE:  %d\" %(max_leaf_nodes, myTestMAE))\n    \nbest_tree_size_train  = min(bestTrain, key=bestTrain.get)\nbest_tree_size_test = min(bestTest, key=bestTest.get)\nprint(\"\\nTrain MAE for each nodes:\", trainMAE)\nprint(\"\\nTest MAE for each nodes:\", testMAE)\nprint(\"\\nBest regression tree size for train = \", best_tree_size_train)\nprint(\"Best regression tree size for test = \", best_tree_size_test)\n\nplt.figure()\nplt.plot(candidate_max_leaf_nodes, trainMAE, color=\"red\")\nplt.plot(candidate_max_leaf_nodes, testMAE, color=\"green\")\nplt.xlabel(\"Tree Depth\")\nplt.ylabel(\"MAE\")\nplt.title(\"Decision Tree Regressor\")\nplt.show()\n\nRGS = DecisionTreeRegressor(max_leaf_nodes=best_tree_size_train, random_state=0)\nRGS.fit(train_images,train_labels)\nscoreRGS = RGS.score(test_images,test_labels)\nprint(\"Score Dicession Tree Regressor = \",scoreRGS)\nprint(\"Score Dicession Tree Classifier = \",scoreCLS)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b82cfa7cb0f2ce6818e7fbe79fc73e0a9872f104"},"cell_type":"markdown","source":"## Decsion Tree Classifier with Unnormalized Dataset"},{"metadata":{"trusted":true,"_uuid":"e70705f165055987a26a617be180711290a08ce5"},"cell_type":"code","source":"# Disini saya menggunkan data train dan data test yang data pixelnya belum dinormalkan\n# Sebelumnya data train dan data test disini telah saya buat terlebih dahulu pada Answer Q5\ndef getTrainMAE(max_leaf_nodes, train_img, test_img, train_lbl, test_lbl):\n    modelTrain = DecisionTreeClassifier(max_leaf_nodes = max_leaf_nodes, random_state = 0)\n    modelTrain.fit(train_img, train_lbl)\n    predictTrain = modelTrain.predict(train_img)\n    mae = mean_absolute_error(train_lbl, predictTrain)\n    return (mae)\n\ndef getTestMAE(max_leaf_nodes, train_img, test_img, train_lbl, test_lbl):\n    modelTest = DecisionTreeClassifier(max_leaf_nodes = max_leaf_nodes, random_state = 0)\n    modelTest.fit(train_img, train_lbl)\n    predictTest = modelTest.predict(test_img)\n    mae = mean_absolute_error(test_lbl, predictTest)\n    return (mae)\n\ntrainMAE = []\ntestMAE = []\n\nbestTrain = {}\nbestTest = {}\n\ncandidate_max_leaf_nodes = [5, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000]\nprint(\"Train MAE for each nodes:\")\nfor max_leaf_nodes in candidate_max_leaf_nodes:\n    myTrainMAE = getTrainMAE(max_leaf_nodes, train_img, test_img, train_lbl, test_lbl)\n    trainMAE.append(myTrainMAE)\n    bestTrain.update({max_leaf_nodes : myTrainMAE})\n    print(\"Max leaf nodes: %d  \\t\\t Train MAE:  %d\" %(max_leaf_nodes, myTrainMAE))\n    \nprint(\"\\nTest MAE for each nodes:\")\nfor max_leaf_nodes in candidate_max_leaf_nodes:\n    myTestMAE = getTestMAE(max_leaf_nodes, train_img, test_img, train_lbl, test_lbl)\n    testMAE.append(myTestMAE)\n    bestTest.update({max_leaf_nodes : myTestMAE})\n    print(\"Max leaf nodes: %d  \\t\\t Test MAE:  %d\" %(max_leaf_nodes, myTestMAE))\n\nbest_tree_size_train  = min(bestTrain, key=bestTrain.get)\nbest_tree_size_test = min(bestTest, key=bestTest.get)\nprint(\"\\nTrain MAE for each nodes:\", trainMAE)\nprint(\"Test MAE for each nodes:\", testMAE)\nprint(\"\\nBest tree size for train = \", best_tree_size_train)\nprint(\"Best tree size for test = \", best_tree_size_test)\n\nplt.figure()\nplt.plot(candidate_max_leaf_nodes, trainMAE, color=\"red\")\nplt.plot(candidate_max_leaf_nodes, testMAE, color=\"green\")\nplt.xlabel(\"Tree Depth\")\nplt.ylabel(\"MAE\")\nplt.title(\"Decision Tree Classifier with Unnormalized Dataset\")\nplt.show()\n\nCLSuN = DecisionTreeClassifier(max_leaf_nodes=best_tree_size_train, random_state=0)\nCLSuN.fit(train_img,train_lbl)\nscoreCLSuN = CLSuN.score(test_img,test_lbl)\nprint(\"Score Decision Tree Classifier (Unnormalized Dataset) = \", scoreCLSuN)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}