{"cells":[{"metadata":{"_uuid":"4a586d84a300626561e20b0ec8e7aad1ee412277"},"cell_type":"markdown","source":"# Disclaimer\nThe data in this notebook is mostly copied from [https://www.kaggle.com/archaeocharlie/a-beginner-s-approach-to-classification ](http://https://www.kaggle.com/archaeocharlie/a-beginner-s-approach-to-classification). I intended to do modification later to the tutorial, so please permit me for using it. "},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"# Library Import\nFor starter import any machine libary we wanted to use. SKLearn is good choice for beginner, the question is what the algorithm we interested to test. \nHere's what we are going to need:\n1. At least a classification algorithm (SVM or Decision Tree is a Good Choice)\n2. Matplotlib\n3. Preprocessing tools\n4. Train test split\nAnd since we have been import numpy and panda no need to import them. "},{"metadata":{"trusted":true,"_uuid":"80d2b5abc402e8023ce027e4fa9bdd71ee357f4e"},"cell_type":"code","source":"import pandas as pd\nimport matplotlib.pyplot as plt, matplotlib.image as mpimg\nfrom sklearn.model_selection import train_test_split\nfrom sklearn import svm, tree\nprint(\"sukses import package\")\n#%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d8534e45d98cf6fd686581e031e59283b7318491"},"cell_type":"markdown","source":"# Load Data\nIn case you haven't imported Digit Recognizer dataset from the competition, please do so. Then load the data with pandas. \nFor simplicity we'll only load first 5000 train images then split them again into our personal train & test set for testing."},{"metadata":{"trusted":true,"_uuid":"99cbaaf91a49d8e0fe5af00fd0559ab25ac74d37"},"cell_type":"code","source":"# load the data\nlabeled_images = pd.read_csv('../input/train.csv')\nimages = labeled_images.iloc[0:5000,1:]\nlabels = labeled_images.iloc[0:5000,:1]\ntrain_images, test_images,train_labels, test_labels = train_test_split(images, labels, test_size=0.2, random_state=0)\nprint(\"berhasil load data\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c99381ac3a70d7d5189e2d591eea32760f44af12"},"cell_type":"markdown","source":"### Q1\nNotice in the above we used _images.iloc?, can you confirm on the documentation? what is the role?\n\nAnswer :\nMenurut saya _images.iloc berfungsi untuk indexing dan selecting data\n\nAllowed inputs are:\n\nAn integer, e.g. 5.\nA list or array of integers, e.g. [4, 3, 0].\nA slice object with ints, e.g. 1:7.\nA boolean array.\nA callable function with one argument (the calling Series, DataFrame or Panel) and that returns valid output for indexing (one of the above)\n"},{"metadata":{"trusted":true,"_uuid":"4e1a0173e06bc25af7f08ed8d04a76c5ebda1676","scrolled":false},"cell_type":"code","source":"# now we gonna load the second image, reshape it as matrix than display it\ni=1\nimg=train_images.iloc[10].values\nimg=img.reshape((28,28))\nplt.imshow(img,cmap='gray')\nplt.title(train_labels.iloc[i,0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"be8a51d4c8285194cecffafc4c495a0b6d221ae7"},"cell_type":"markdown","source":"### Q2\nNow plot an image for each image class"},{"metadata":{"trusted":true,"_uuid":"bc94d6e2b8fd66f8bb2646455a8ca9b3c2042305"},"cell_type":"code","source":"# Todo: Put your code here\nplt.plot(img)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4ab61996db6534fa5839c2e444d1567008966b45"},"cell_type":"markdown","source":"Now plot the histogram within img"},{"metadata":{"trusted":true,"_uuid":"0bf1b1956917f593ab351c7c93710235826dbc00"},"cell_type":"code","source":"#train_images.iloc[i].describe()\n#print(type(train_images.iloc[i]))\nplt.hist(train_images.iloc[i])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c643622e8113a7b0e8be8de62117a0980a88e467"},"cell_type":"markdown","source":"### Q3\nCan you check in what class does this histogram represent?. How many class are there in total for this digit data?. How about the histogram for other classes"},{"metadata":{"trusted":true,"_uuid":"ff31f946f9b567ad336d8038afaecb91d9ce6b79"},"cell_type":"code","source":"# create histogram for each class (data merged per class)\n# Todo\n#print(train_labels.iloc[:5])\ndata1 = train_images.iloc[1]\ndata2 = train_images.iloc[3]\ndata1 = np.array(data1)\ndata2 = np.array(data2)\ndata3 = np.append(data1,data2)\nprint(len(data3))\nplt.hist(data3)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e3310272ec292e85b6e4823d310104259432b6e9"},"cell_type":"markdown","source":"### Train the model\nNow we are ready to train the model, for starter let's use SVM. For the learning most model in SKLearn adopt the usual fit() and predict(). "},{"metadata":{"trusted":true,"_uuid":"36e546a7f1e86a4b642e23cabb739cd91bb88a2f"},"cell_type":"code","source":"clf = svm.SVC()\nclf.fit(train_images, train_labels.values.ravel())\nclf.score(test_images,test_labels)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e10d830216bd49586b553c839ed180ba9ce23c4b"},"cell_type":"markdown","source":"### Q4\nIn above, did you see score() function?, open SVM.score() dokumentation at SKLearn, what does it's role?. Does it the same as MAE discussed in class previously?.\nAscertain it through running the MAE. Now does score() and mae() prooduce the same results?. "},{"metadata":{"trusted":true,"_uuid":"bd6b60c6859f9509acc95f14671f659a64509290"},"cell_type":"code","source":"# Put your verification code here\n# Todo\nprint(train_labels.values.ravel())\nprint(np.unique(test_labels)) # to see class number","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cf582d5bbe55a9b9b5a6d73a117811ca9e4acbb1"},"cell_type":"markdown","source":"### Improving Performance\nDid you noticed, that the performance is so miniscule in range of ~0.1. Before doing any improvement, we need to analyze what are causes of the problem?. \nBut allow me to reveal one such factor. It was due to pixel length in [0, 255]. Let's see if we capped it into [0,1] how the performance are going to improved."},{"metadata":{"trusted":true,"_uuid":"b1554e450ab974e907c77dd218ec0687560ba5a5"},"cell_type":"code","source":"test_images[test_images>0]=1\ntrain_images[train_images>0]=1\n\nimg=train_images.iloc[i].values.reshape((28,28))\nplt.imshow(img,cmap='binary')\nplt.title(train_labels.iloc[i])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7e5885f747c8e88622c4f190d1dffad23e354cfc"},"cell_type":"code","source":"# now plot again the histogram\nplt.hist(train_images.iloc[i])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"38e1b9509379ae403c4fbc006e0f3973f1c7864b"},"cell_type":"markdown","source":"# Retrain the model\nUsing the now adjusted data, let's retrain our model to see the improvement"},{"metadata":{"trusted":true,"_uuid":"d0d429f3564709aaa7b0e700a40b17ba7e2f8a3f"},"cell_type":"code","source":"clf = svm.SVC()\nclf.fit(train_images, train_labels.values.ravel())\nclf.score(test_images,test_labels)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"04e838aaba5e88326e24be1318d867be392f6eff"},"cell_type":"markdown","source":"### Q5\nBased on this finding, Can you explain why if the value is capped into [0,1] it improved the performance significantly?. \nPerharps you need to do several self designed test to see why. "},{"metadata":{"_uuid":"f5bc953bc28b13c479a7f8dee87589d31a3506f5"},"cell_type":"markdown","source":"### Prediction labelling\nIn Kaggle competition, we don't usually submit the end test data performance on Kaggle. But what to be submitted is CSV of the prediction label stored in a file. "},{"metadata":{"trusted":true,"_uuid":"c6818a3743735706cd0c20400dfc23bba38158c1"},"cell_type":"code","source":"# Test again to data test\ntest_data=pd.read_csv('../input/test.csv')\ntest_data[test_data>0]=1\nresults=clf.predict(test_data[0:5000])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"65475e13189c162ba105ab9b051b358842c83668"},"cell_type":"code","source":"# separate code section to view the results\nprint(results)\nprint(len(results))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"68c89fa23076b3b1b917b9d36035c993e9847ae5"},"cell_type":"code","source":"# dump the results to 'results.csv'\ndf = pd.DataFrame(results)\ndf.index.name='ImageId'\ndf.index+=1\ndf.columns=['Label']\ndf.to_csv('results.csv', header=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"610ccc2311762fa90a5c7a9d6bf720edb7c95a01"},"cell_type":"code","source":"#check if the file created successfully\nprint(os.listdir(\".\"))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6f99466305b337cb04343b29e743a8775187ad1f"},"cell_type":"markdown","source":"# Data Download\nWe have the file, can listed it but how we are take it from sever. Thus we also need to code the download link. "},{"metadata":{"trusted":true,"_uuid":"033461288f8a2756a55f7eca38fdeeef44bcacaf"},"cell_type":"code","source":"# from https://www.kaggle.com/rtatman/download-a-csv-file-from-a-kernel\n\n# import the modules we'll need\nfrom IPython.display import HTML\nimport pandas as pd\nimport numpy as np\nimport base64\n\n# function that takes in a dataframe and creates a text link to  \n# download it (will only work for files < 2MB or so)\ndef create_download_link(df, title = \"Download CSV file\", filename = \"data.csv\"):  \n    csv = df.to_csv()\n    b64 = base64.b64encode(csv.encode())\n    payload = b64.decode()\n    html = '<a download=\"{filename}\" href=\"data:text/csv;base64,{payload}\" target=\"_blank\">{title}</a>'\n    html = html.format(payload=payload,title=title,filename=filename)\n    return HTML(html)\n\ncreate_download_link(df)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5a32d17515a6005fd466ff0ab809d47c225458b1"},"cell_type":"markdown","source":"# Q6\nAlhamdulillah, we have completed our experiment. Here's things to do for your next task:\n* What is the overfitting factor of SVM algorithm?. Previously on decision tree regression, the factor was max_leaf nodes. Do similar expriment using SVM by seeking SVM documentation!\n* Apply Decision Tree Classifier on this dataset, seek the best overfitting factor, then compare it with results of SVM. \n* Apply Decision Tree Regressor on this dataset, seek the best overfitting factor, then compare it with results of SVM & Decision Tree Classifier. Provides the results in table/chart. I suspect they are basically the same thing. \n* Apply Decision Tree Classifier on the same dataset, use the best overfitting factor & value.  But use the unnormalized dataset, before the value normalized to [0,1]\n\n"},{"metadata":{"_uuid":"a60a62269f4716b8fdcbef9697f544df1f99ceda"},"cell_type":"markdown","source":""}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}